TypeSafe’s Jev is an example of a generative (in spirit) decision model as it generates probabilities over decision outcomes to help downstream systems take the correct decision. They also have the natural language understanding capabilities of a true generative model. These are often referred to as System One (S1) models on the basis of their speed of operation. The frontier LLMs like Gemini that offer reasoning modes are considered System Two (S2) models that take time to reason/reflect before answering and are therefore slower.
In reality S1/S2 labelling is more marketing than anything else because apart from purely instinctive ‘fast thinking’ (the kind that helps us survive) high value (expert level) S1 thinking comes from extreme task specialisation after years of training (S2 learning transferring to S1).
As an example champion sportspersons have practised particular movements so often that it becomes ‘muscle memory’. A chess player is able to play out entire strategies without thinking in blitz style quick games.
Jev is fast for a different reason: it isn’t drawing on deep expertise, it’s choosing from a constrained set of options someone else has defined. The speedup comes from narrowing the problem, not from mastering the problem domain.
Setting the Scene
So what is a Generative Decision Model? And how is it different from other types of models?
A Generative Decision Model generates decisions when the inputs (state in Jev), decision required (instructions in Jev), possible decision options (criteria key in Jev), and factors affecting the selection (criteria value in Jev) are given. The decision generation takes the form of a probability distribution across the available decision options when using a ‘Choice‘ form, a probabilistic true/false when using a ‘Noul‘ form, and a score on a scale when using a ‘Score‘ form.
These are highly structured input-output models and cannot be used to generate open-ended responses (such as generating summaries).
Common use-cases for Jev-like models include routing (e.g., ticket routing, model routing), business-process decision making, classification over unstructured data, and guardrails.
Decisioning Framework
Let us establish a decision framework so that we can in the following posts evaluate such models.
D = f (I, Q, O, C)
- Input provides the required information to take the decision.
- For each decision we have a question that defines the decision required.
- Attached to that decision we have possible outcomes as a defined finite set.
- Each outcome must be attached to a condition that needs to be met if that outcome is to be selected.
We also need a reasonable number of outcomes for human cognition as we have human experts providing the insights into outcomes and conditions. What reasonable means is up to interpretation.
These conditions make Jev-like models poorly suited to numeric forecasting decisions because the output loses precision and ordering if broken into buckets to make a defined finite set (consider forecasting the temperature tomorrow given the conditions today).
The Decision Model needs to understand the condition and the question to map to the appropriate outcome.
Compared to LLMs
LLMs can behave like a Decision Model (and provide explanations for decision probabilities) but the costs and latency would be quite high compared to models like Jev. So you trade off speed and cost against expertise and reflection.
The confidence scores of Decision models gives an (albeit model generated) metric to operate with.
Compared to ML Classifiers
Jev-like models are generic when compared to task specific ML models, only requiring the decision and options to be articulated in a structured manner instead of retraining/new model creation based on labelled datasets. Therefore, you avoid the traditional MLOps stages in building and testing an ML model.
Such decision models appear to combine the ‘understanding’ and ‘decide’ layers of an LLM with a custom trained decision layer that is fine-tuned for selecting the best decision from a set of options.
With enough labelled data (a tricky challenge in itself) an ML-based classifier may still be cheaper, easier to understand and analyse, and faster than Jev-like models.
Compared to Code
If the conditions are objective, write code. Use a decision model only where the conditions need interpretation. No model will match the transparency, speed, and cost of code.
Reflections
Generative Decision Models are interesting because they occupy that grey area between full-fledged LLMs and bespoke machine learning models. But they are not a silver bullet that removes the need for using LLMs for complex decisioning (e.g., where reflection or multi-step decisioning is required) and ML for task-specific decisioning (e.g., where labelled data captures the decision space accurately).
In future posts I will analyse the Jev model as we would a classifier and show how sensitive the decisioning can be.
I will also explain why writing outcomes and conditions that are resilient over different natural language inputs is not a trivial task. We will also examine some real-world use-cases with an example.
