Intellect, Senses, and the Language Model

Intellect: By convention there is sweetness, by convention bitter-ness, by convention color, in reality only atoms and the void.

Senses: Foolish intellect! Do you seek to overthrow us, while it is from us that you take your evidence?

– Democritus

Consider the humble language model with its attention heads, feed-forward layers and complete dependence on the prompt.

The prompt is the information coming in. To be ‘read’ by the senses (attention-heads) to be evaluated by the intellect (feed forward layers) distributed across stacked layers.

The quote above is highly relevant for how such models operate and how the combination of prompts and attention mechanisms impact the results. 

For the model the only place to sense is the context window and the only mechanism to action what changes there is via dependency on external tools. The intellect has limited functions to influence what it attempts to sense next both due to lack of sophisticated actuators (tools) and limited forms of sensing.

The so-what?

The implication is that prompts are becoming complex artefacts that evolve outside direct human control. The so called thought traces of models are a bit of smoke and mirrors in the process as supported by research into CoT faithfulness.

Cost estimates, token budgets, and tool integrations are art forms and not the hard science that a business case requires. The interdependence here creates the perfect fog of war uncertainty. Context enrichment via tool use will compound costs as each model run will require re-evaluation of all the context (assuming caching will be of limited use in this case).

When you have a human to validate outputs and continually tweak the sources of change the risk is less but not absent as humans can quickly lose situational awareness of what is changing.

Getting into the details…

You could have the sharpest intellect (generating layers) but they will generate rubbish if the senses are not aligned.

The senses are frozen in time – they don’t change with time as human senses do. Consider our senses that sharpen with experience (e.g., whilst driving or playing sports). Our sensory apparatus is constantly changing (and degrading as we age). No such advantage for AI models.

Therefore, same inputs broadly give similar outputs (perhaps with different language structures depending on the output randomisation). 

This makes prompting an optimisation problem where we are trying to find the right combination of words to ‘tickle’ the feed forward layers into providing the required response across each model layer.

There is no ‘taking a second look’ with these models by default unless we are operating in an ‘agentic’ manner. 

But then how do these models perform so well with such static senses? The secret lies in multiple attention heads. That is the model taking ‘n’ different looks at the same input and making the output align with the input. Furthermore, thanks to the layering the model is also taking ‘m’ different looks at different levels (e.g., text structures or syntax and semantics).

Now that said, these ‘n’ different views are also static and far less powerful than a single adaptive sensor. I am sure there is some level of sensor selection going on in state-of-the-art proprietary models (beyond plain MoE) where certain types of heads are preferred for specific input content. 

I also wonder if there is any kind of sensor augmentation going on when it comes to specific attention heads per user based on their content and vocabulary. That could enhance the responses. Where out of say 20 heads 5 might be user oriented and 15 generic. 

LoRA attempts to do the above but not at the per-head level. Where LoRA changes are clustered over specific heads in a layer similar outcomes may be achieved but at a macro level. The resolution of sensing will need to massively increase in the next generation of models for a material improvement in quality – specifically concepts like ‘targeted’ LoRA attaching itself to specific heads.

What about Agentic mode you ask?

Think of agentic mode like ‘continuous optimisation’ using external and internal sources of information.

The sensor heads are the same (and frozen) but we take an initial prompt (system + user) and the model is trained to keep tweaking and assessing the output. This tweaking is done using tools that:

• Allow reflection (feed output back into the model with additional prompting for checking/validating/fixing).
• Search the web for evidence/grounding.
• Use other (private) data sources for customisation.
• Navigate folder structures for grounding content such as skills and knowledge bundles.

This type of tweaking is quite ‘dynamic’ by nature and leads to complex interactions between static internal data (system and user prompt) and dynamic data (both internal and external). The complexity of the interaction grows with the length of the tweaking. 

In tools like Claude this is governed by the ‘Effort’ setting which is not a token budget but a behavioural signal, influencing how many tool calls the model makes and therefore how much external information enters the context.

All of this tweaking can be wasted effort if the senses and the intellect become misaligned due to the information flowing through. Imagine the model senses being hammered by all these bits of information. Each interaction nudging the internal layers and ultimately the next input (via the output) in a particular direction.

What is happening to the input in each loop?

Remember the senses are not changing – the input is. The intellect has the hard work of keeping the output pointing in the right direction as the input evolves. 

The lower layers (closer to the raw input) can influence both the senses and the intellect in the upper layers but not the other way around. Layer ‘m’ has no way to influence the input received from ‘m-1’ as there are no loops in an AI model. 

Loops make it harder for model training and inference to be parallelised like with Recurrent Neural Networks.

It is the richness of the senses working together with a sharp intellect that will help clear the fog of war. This augmentation of the senses and feedback (if it can be achieved at scale) from the intellect will enable longer unsupervised runs on complex tasks with minimum initial ‘prompt engineering’ allowing the agent to self-adapt the input to produce the required output.

Till then we will live with augmentation rather than true autonomy.