From Agent-to-Agent to Semantic Networks

There is a revolution underway in how we think about multi-agent systems. It is moving from an orchestra with a conductor to a bunch of one-person bands.

Modelling a journey as a set of interacting agents each with specific knowledge, tools, and skills is not trivial to scale. Each hop between agent represents cost, latency, and risk.

This burden is worth taking when operating across system boundaries to ensure high cohesion and low coupling between systems. But within system boundaries this burden can be reduced by using a semantic network instead of a network of agents.

A semantic network is a network of interconnected knowledge, skills and tools. So instead of moving data between agents, the agent moves across the network of knowledge assets, tools, and skills as required with state changes being captured.

Check this out for more info regarding state management.

Agents as Independent Entities

Consider the multi-agent system below. The Customer interacts with the Conversational Agent which in turn uses the Expert Problem Solver to define the solution and the Fulfilment Agent to execute the solution. Control passes along the green arrows.

The green arrows represent some form of agent to agent communication. This can be remote (e.g., via SSE) or local (e.g., defined by frameworks like Google’s ADK).

The above approach leads to the following:

• Each agent has (potentially) a separate set of instructions, tools, skills, and access to knowledge.
• Each agent can evolve independently (e.g., instructions can change).
• Each agent may have independent existence (failure of one may not mean the failure of others).
• Communication between agents is via state exchange (messaging/shared state).
• Non-linear scaling of operational effort (not per agent but also to manage the interaction between agents).

Agents as Knowledge-seeking Navigators

In the figure below the coloured ovals are subsets of knowledge, skills, and tools relevant to a specific task within a journey. Each subset can refer to common knowledge, skills and tools. For example, tone guidance common to all responses.

Conversational skills for example would explain how to hold a conversation across different channels (e.g., email, web-chat) whereas Problem Solving tools would include data discovery capabilities around process, and Fulfilment Knowledge would allow an agent to execute specific fulfilment workflows in a repeatable manner.

The purple arrows represent semantic links between the sub-sets. The agent here is a single entity that attaches itself to different knowledge sub-sets as it traverses through the enterprise’s knowledge base.

The green arrows map this view to the agent as an independent entity view where each agent would be tightly coupled to its knowledge subset and control would pass along the green paths.

This is the default architecture of Claude Cowork and you do not build individual agents and then connect them in a workflow. It can spawn specific agents for certain tasks but that is outside your direct control.

Now imagine all the knowledge, tools and skills for a given business domain is represented as a network of related items. These relationships can be of the following types:

1. Semantic – to maintain semantic links between different items to also be used for navigation.
2. Content – to maintain link between different content items (e.g., references and their use, tools and skills).

Semantic and Content relationships are navigable. This navigation is what I call a flexi-reasoning task which requires flexibility between reasoning about the problem, selecting an action, and thinking about how to think.

What is Changing?

In the second case above the only thing changing between each stage is the knowledge available to the agent to operate. There is nothing specific about the agent itself – it is merely an intelligent semantic processing function. It is able to ‘make sense’ of the knowledge and skills provided and then use available tools as required.

Session state is what helps keep the conversational context as the agent acts using different knowledge subsets.

Operational simplification is an added advantage of this approach. You are not monitoring multiple agents and comms between them. Instead you are monitoring a finite set of semantic network crawlers.

Issues with the Approach

There are a few issues with this approach:

1. heavy reliance on the model and agent architecture to deal with different knowledge subsets encountered whereas in the individual entity model each agent can be crafted to handle the knowledge subset it needs.
2. dynamic linking between knowledge can lead to long/deep thinking sessions which may not produce the required output and this would be wasted effort.
3. question of when we need actual independent agents communicating with each other versus agents navigating a semantic network (system boundary rules).
4. building semantic networks at scale is still an art-form, leveraging AI will help but only part way until it becomes ingrained in day-to-day ways of working.
5. retrieval latency can have high variability depending on the hop (e.g., when accessing content from local folders vs external stores).
6. each hop carries its own noise based on the specific task and there are no guarantees of information gain corresponding to the navigation effort.

How to Build This?

The navigation richness described above depend on three capabilities:

1. the AI model
2. the harness that orchestrates the model outputs (between tool calls, reflection loops, user I/O)
3. the tools available to the harness to access content in different locations

The AI Model

The AI Model will require decent context window size enough to hold a knowledge subset plus session state; in practice hundreds of thousands of tokens. Reasoning mode is also required for such flexi-reasoning tasks. Here newer models have a clear edge on older ones with respect to depth and width of tasks that can be carried out. For example I have seen a massive difference between different versions of Gemini Flash (2.5 vs 3.1) in such a situation.

The AI Model is what directs the switching between action, problem solving, and reasoning modes, makes sense of inputs, and maintains context. Therefore, it is a critical component.

The Harness

The harness includes system prompt templates, input and output parsers, routers, and callback functions. This is the hidden deterministic processing that sits around the model and is the real value-add beyond the ‘intelligence’ in the model.

One of the key tasks for the harness is to ensure output type is correctly routed to the appropriate handler. For example, if the model indicates execution of a tool then it should be routed to the appropriate tool handler.

The harness needs to be generic enough to deal with wide variety of tasks.

Callbacks

Functions that are called at specific points during the operation of the Agent and used to manage the system in a deterministic manner. Use-cases include injecting guardrails, critical knowledge, and rules using design-time integration (e.g., coded API call).

The Tools

The navigation of the semantic network is heavily reliant on the tools available. Semantic links are about accessing resources and can be of different types:

1. Local folders: a local folder structure with parent-child and peer-to-peer relationships that is used to organise content.
2. Internal reference via a URL: an internal link that can be followed and control remains within the org. For example: following a shared folder link.
3. External reference via a URL: a link pointing to an external resource. For example: following a web-link to a pdf.
4. Discovered links: links that are made available through internal or external search, these links can point to a variety of resources (e.g., pdf, website, documents). For example: internal SharePoint search which retrieves word documents, pages, and pdfs.
5. Embedded content: where content is embedded in a page structure as an attachment. This requires the tool to parse and extract the embedded content.

For the above we need tools that can access content of different types, extract content for the AI model to consume and if required to crawl the semantic network.

Each of the specific link types comes with its own challenges:

1. Control: local folders are under direct control of the agent whereas discovered links are completely outside its control.
2. Curation: external resources are outside org curation.
3. Permission: some sources provide detailed authorisation capabilities (e.g., a content management system like Confluence) but others may not (e.g., local folders).