This post attempts to bring together different low level components from the current agentic AI ecosystem to explore how things work under the hood in a multi-agent system. The components include:
As ever I want to show how these things can be used to build something real. This post is the first part where we treat the UI (a command-line one) as an agent that interacts with a bunch of other agents using A2A. Code at the end of the post. TLDR here.
We do not need to know anything about how the ‘serving’ agent is developed (e.g., if it uses LangGraph or ADK). It is about building a standard interface and a whole bit of data conversion and mapping to enable the bi-directional communication.
The selection step requires an Agent Registry. This means the Client in the image above which represent the Human in the ‘agentic landscape’ needs to be aware of the agents available to communicate with at the other end and their associated skills.
In this first part the human controls the agent selection via the UI.
There is a further step which I shall implement in the second part of this post where LLM-based agents discover and talk to other LLM-based agents without direct human intervention.
Key Insights
A2A is a heavy protocol – that is why it is restricted to the edge of the system boundary.
Production architectures depend on which framework is selected and that brings its own complexities which services like GCP Agent Engine aim to solve for.
Data injection works differently between LangGraph and ADK as these frameworks work at different levels of abstraction
LangGraph allows you full control on how you build the handling logic (e.g., is it even an agent) and what is the input and output schema for the same. There are pre=made graph constructs available (e.g., ReACT agent) in case you did not want to start from scratch.
ADK uses agents as the top level abstraction and everything happens through a templatised prompt. There is a lower level API available to build out workflows and custom agents.
Attempting to develop using the agentic application paradigm requires a lot of thinking and lot of hard work – if you are building a customer facing app you will not be able to ignore the details.
Tooling and platforms like AI Foundry, Agent Engine, and Copilot Studio are attempting to reduce the barriers to entry but that doesn’t help with customer facing applications where the control and customisation is required.
The missing elephant in the room – there are no controls or responsible AI checks. That is a whole layer of complexity missing. Maybe I will cover it in another part.
Setup
There are two agents deployed in the Agent Runner Server. One uses a simple LangGraph graph (‘lg_greeter’) with a Microsoft Phi-4 mini instruct running locally. The other agent uses ADK agent (‘adk_greeter’) using Gemini Flash 2.5. The API between the Client and the Agent Runner Server is A2A (supporting the Message structure).
Currently, the agent registry in the Agent Runner Server is simply a dict keyed against the string label which holds the agent artefact and appropriate agent runner.
It is relatively easy to add new agents using the agent registry data structure.
Memory is implemented at the Agent Runner Server and takes into account the user input and the agent response. It is persisted in Redis and is shared by all agents. This shared memory is a step towards agents with individual memories.
There is no remote agent to agent communication happening as yet.
Output
The command line tool first asks the user which agent they want to interact with. The labels presented are string labels so that we can identify which framework was used and run tests. The selected label is passed as metadata to the Agent Runner Server.
The labels just need to correspond to real agent labels loaded on the Agent Runner Server where they are used to route the request to the appropriate agent running function. It also ensures that correct agent artefact is loaded.
The code is in test/harness_client_test.py
In the test above you can see how when we ask a question to the lg_greeter agent it then remembers what was asked. Since the memory is handled at the Agent Runner level and is keyed by the user id and the session id it is retained across agent interactions. Therefore, the other agent (adk_greeter) has access to the same set of memories.
Adding Tools
I next added a stock info tool to the ADK agent (because LangGraph agent running on Phi-4 is less performant). The image below shows the output where I ask ADK (Gemini) for info on IBM and it uses the tool to fetch it from yfinance. This is shown by the yellow arrow towards the top.
Then I asked Phi-4 about my previous interaction which answered correctly (shown by the yellow arrow towards the bottom.
Adding More Agents
Let us now add a new agent using LangGraph that responds to the user queries but is a Dungeons&Dragons fan therefore rolls 1d6 as well and gives us the result! I call this agent dice (code in dice_roller.py)
You can see now we have three agents to choose from. The yellow arrows indicate agent choice. Once again we can see how we address the original question to dice agent and subsequent one to the lg_greeter and then the last two to the adk_greeter.
A thing to note is the performance of Gemini Flash 2.5 on the memory recall questions.
You will need a Gemini API key and access to my genai_web_server and locally deployed Phi-4. Otherwise you will need to change the lg_greeter.py to use your preferred model.
Check out the commands.txt for how to run the server and the test client.
In this post I will cover points 1, 2 and 4. Point 3 I feel is needed only in specific use-cases and current tooling for Long Term Memory is evolving rapidly.
Interaction between the Agentic Application, Agent Runtime and API Server.
The diagram above shows the major components of an Agentic Application. The API Server is responsible for providing an endpoint (e.g., https REST, message queue) that external applications (e.g., chat client) can use to access the Agent App 1.
The API Server then invokes the appropriate handler method when the API is invoked. The handler method is responsible for triggering the Agent Runner in the Agent Runtime that deals with the execution of the Agent App 1.
The Agent Runtime is the component that:
1. Sets up the correct session context (e.g., current session history if the conversation thread is resuming) using the Session Manager
2. Manages the Agent Runner which executes the Agent App 1 by triggering its Root Agent.
Remember as per ADK one Agentic Application can only have one Root Agent per deployment.
The Agent Runner is then responsible for finishing the agentic app execution (including handling any errors). Agent Runtime then cleans up after the Agent Runner and returns any response generated by the Agent App 1 back to the handler where it can be returned to the caller using API constructs.
Key Concept: If the Agentic App does not have a sequential workflow and instead depends on LLMs or contains loops then the app keeps going till it emits a result (or an error).
This makes it difficult to set meaningful time-outs for request-response style and we should look at async APIs (e.g., message based) instead.
API Server
REST API Handler function example using Flask.
The code above shows the API handler function using Flask server.
Lines 77 – 80 are all about extracting data from the incoming request to deal with User, Session management and the Query integration (e.g., incoming text from the user for a chat app). Here we assume the requesting application manages the User Id (e.g., a chat app that handles the user authentication and authorisation) and Session Id.
Lines 82-84 are all about setting up the session store if no existing session context is found. This will usually trigger when the user say first engages with the agent at the start of a new conversation. It is indexed by User Id and Session Id.
Key Concept: The session boundary from an agents perspective is something that needs to be decided based on the use-case and experience desired.
Line 88 is where the Agent Runtime is triggered in an async manner with the Application Name, User Id, Session Id, and the Query. The Application Name is important in case we have multiple Agentic Applications being hosted by the same Agent Runner. We would then have to change the session store to also be indexed by the App Name.
Line 90 extracts the final response of the Agent from the session state and is executed once the Agent Runtime has finished executing the Agentic Application (Line 88) which as per our Key Concept earlier is when the final result or an error is produced by the Root Agent.
Beyond Line 90 the method simply extracts the results and returns them.
This code is to show the API Server interacting with the Agent Runtime and must not be used in production. In production use async API style that decouples API Server from the Agent Runner.
Running Agents and Session State Management
ADK defines a session as a conversation thread. From an Agentic App perspective we have three things to think about when it comes to sessions:
Application Name
User ID
Session ID
These three items when put together uniquely identify a particular application handling requests from a given user within a specific conversation thread (session).
Typically, session management requires managing state and lots of record keeping. Nothing very interesting therefore ADK provides a few different types of Session Managers:
InMemorySessionManager – the most basic Session Manager that is only suitable for demos and learning more about session management.
DatabaseSessionManager – persisted version of the Session Manager.
VertexAISessionManager – the pro version which utilizes the VertexAI platform to manage the Session State. Best thing to use with Agent Engine for production workloads.
In this post I use the InMemorySessionManager to show how session management works and how we execute an agentic application.
Setting up the Agent Runner and session in ADK.
The main method (Line 62 onwards in the above – invoked on Line 88 in previous listing) represents the Agent Runtime (e.g., Agent Engine in GCP) triggering the agents it is hosting (Agent App 1). It is taking the App Name, Session Id, User Id, and the incoming Query as described previously.
The Agent Runner is setup on Line 64.
On Line 66 the Agent Runtime initialises the current session (an instance of InMemorySessionManager) in ADK and provides the starting session state from the session state store. This will either be a freshly initialised session (blank state) or an existing session as per the logic shown previously.
Finally, on Line 69 we use the ‘call_agent’ method to configure and execute the Agent Runner. As you can see we are passing the ‘root_agent’, current session, and other details like session Id and query to this method.
Running the Agentic App.
This is the fun bit now…
Lines 31 and 32 are all about extracting what we received from the external application (in this case what the user typed in the chat box) and preparing a ‘default’ response (in case of issues with agent execution).
Lines 35-42 is where the core execution happens for Agent App 1. Since the app is running in async mode it will go through a set of steps where the root agent is triggered and it in turn triggers sub-agents and tools as needed. The async for goes through the responses till the root agent provides the final response signalling the end of the execution of the app. The final response is extracted and stored for eventual return back to the API Server.
Lines 46-51 simply extract the final session state and log it. Nothing interesting there unless you are after an audit trail.
Lines 55-58 is where we build up the session which allows the agents to remember the context and previous inputs/outputs in the conversation session. We extract the current state from the state store, add to it the user’s request and the agent’s response (think of it like adding a request – response pair). Finally the state store is updated (using the ‘history’ key) so when user responds to the agent’s current output the session history is available to guide the agent on what to do next.
The session history is also called Short Term Memory. When you use VertexAISessionManager with Agent Engine or the ‘adk web’ testing utility you get all of this for free. But now you know how it works!
Logging and Monitoring
Line 35 is where we enter the mysterious async-probabilistic realm of Agent Execution and we need logging and monitoring to help us comprehend the flow of the agent execution as the control passes between Agents and from Agents to tools.
Utilities like ‘adk web’ show the flow of control within the Agentic Application through a connected graph. But how does this work? What mechanisms are available for developers to get telemetry information? By default Agent Runtimes like Google’s Agent Engine provide built-in capability to generate telemetry using OpenTelemetry standard that can then be consumed by likes of Cloud Trace or AgentOps.
In this section we look at the internals of the root agent and see how we collect information as it executes. I also show my own (vibe-coded no less) version of the ‘adk web’ flow visualisation.
Callbacks for the Root Agent.
The root agent is defined in Line 97 as per the standard constructor for LLMAgent till Line 103.
We see the usual parameters for the name, model, description, instruction (prompt), tools (using Agent as Tool paradigm) and the output key for the root agent.
Then come the callbacks (Lines 103-108) that allow us to track the flow of the Agent application. There are six types of callbacks and between them they tap the strategic points in the Agent-Agent and Agent-Tool flows.
The six callbacks supported by ADK and their touchpoints in the Agentic Application.
Before and After Tool: wraps a tool call. This allows us to tap into all the tool calls the agent makes and any responses returned by the tool. This is also the place to execute critical guardrails around tool calling and responses.
Before and After Model: wraps the call to the Large Language Model. This allows us to tap into all the prompts going into the LLM and any responses returned by the Model. This is also the place to execute critical guardrails around input prompts and responses – especially to ensure LLMs are not called with unsafe prompts and any harmful responses blocked.
Before and After Agent: wraps the call to the agent which allows us to tap into all the inputs going into the agent (including user inputs and agent requests) and any outputs.
These callbacks are defined at the level of the agent therefore, it can be used to track the flow through the Agent App 1 going from one agent to another.
Callbacks for the Sub-Agent.
The above shows callbacks registered for the Sub-Agent (named info_gather_agent).
Callbacks
Two examples of callbacks: After Tool and Before Agent.
The above examples show two callback examples out of the 6 possible for the root agent. The standard pattern I have used is:
Log the input provided using python logging for audit purposes.
Record the complete trace of the short term memory as a memory record (in MongoDb) for us to visualise the flow.
One example flow of Human to Root Agent to Sub-Agent to Tool and so on.
In the flow diagram above we see:
Time flowing from top of the image to the bottom.
Human input as the blue dots.
Purple dots are the Sub-Agent (called info_gather_agent).
Green dots are the Root Agent’s response.
Yellow dots are tool calls.
Given we use Agent as tool for communication we see Yellow -> Purple -> Yellow signifying the Root Agent invoking the Sub-Agent as a tool.
Yellow -> Green -> Yellow is the Sub-Agent responding to the Root Agent and the Root Agent processing that input.
Green -> Blue -> Yellow is the Root Agent responding to the Human and the Human responding with a follow up question.
This visualisation was completely vibe-coded based on the document structure of the memory record in MongoDb.
Key Concept: Note the short return trace after ‘were we talking about aapl’ the last question from the Human. The Root Agent does not need to engage with any other agent or tool. It can simply examine the history we have been collecting to answer the question.
Chat App output can be seen below (entry in blue is the human asking questions and grey is the Root Agent responding):
The Chat web-app was also vibe coded in 10 mins just to have something visual to show instead of Postman requests.
It also shows how the web-app could collect User Id and Session Id defaulting to the ‘application-led’ ID and Authorisation model.
We can play around with the session Ids to give parallel chat experience (same user with two open sessions) – see above.
session_1759699185719 – talking about IBM stock (yellow boxes)
session_1759699185720 – talking about AAPL stock (red box)
Code
By now I hope you are ready to start playing around under the hood to improve your understanding of the tech. Remember concepts never die – they just become abstract implementations.
How does one identify a ‘real’ AI Agent developer? They don’t use ‘adk web’ to demo their agent.
Silly jokes aside, in this post I want to talk about building an app with AI Agents. The focus is on GCP stack. Google’s Agent Engine – ADK combination looks like a really good option for pro-code development that abstracts out many common tasks.
I do not like magical abstraction as it blocks my understand of ‘how things work’. Therefore, I will use Google’s ADK to build our agents and instead of hosting it on Agent Engine I will write the application that provides a snug and safe environment for the agents.
This will be a text heavy post where I introduce key aspects. I will follow this up with a code heavy set of posts looking at Sessions, Memory, and Telemetry.
Let us first look at what Agent Engine helps us out with as an Agent developer:
Session State: the internal state that the agent uses to carry out the task – includes knowledge and facts gathered
Long Term Memory: the long term memory created by compressing and updating based on the internal state of the agent
Logging/Telemetry: the data stream coming from the agent that helps us monitor, troubleshoot and audit.
Agent Runner and API: the infrastructure that runs the agents and creates the API for our agent to be published for use.
All of the things in the list above are done for us when we use ‘adk web’ or similar ‘agent testing’ tool. This container for the agents is the ‘Agent Application’.
Agent Engine and adk are two examples of ‘Agent Applications’ that we can use to deploy Agents written in ADK. Agent Engine also supports LangGraph and other frameworks.
Writing an Agent Application
Writing an Agent Application requires a deep dive into concepts that are critical to agent operations: session, memory, and telemetry. Fortunately, ADK gives us all the core capabilities to build these out. For example, when using adk web our agents remember what the user said before, this is because the adk web Agent Application is taking care of session management for us.
One also needs to understand the lifecycle of an interaction so let us look at this first.
Interaction Lifecycle
In any agentic interaction there is usually a user or external system (or agent), there is also an agentic application they interact with, and the interaction is usually bound to an intent called a session (e.g., I want to open a new account). The interaction continues till the intent is satisfied or rejected.
The interaction lifecycle gives us scope for any state and memory we use.
User: Scope is the user. This typically will be information associated with the user like their username, preferences etc. This will span application and session scope. Identified by ‘user_id’.
Application: Scope is the application. This typically will be information required to operate the app. This allows the same agent to have different application state when used in different applications. For example, credentials for agents, application level guardrails (e.g., do not give advice) and global facts. Identified by ‘app_name’.
Session: Scope is the current session. This will contain the short-term (or scratchpad) state associated with the ongoing interaction for the agent to use while the session is active. This includes the users inputs like the reason to contact and any responses by the agent. The client in this case will need to ensure state management by reusing the same user, session and app ID when it wishes to maintain the session continuity (e.g., in case of a disconnect).
Now let us look at the Session, Memory and Telemetry.
Session
Session is all about the local state of the Agent. ADK has the Session object that provides the data structure to store session info and a SessionService object that manages the session data structure.
The SessionService is responsible for the entire session state management and Google do not want us to manually write to the session object. The only two special cases being when the interaction starts (to preload state) and when it finishes (to save state). Even for this we are expected to use the appropriate lifecycle callbacks. More on callbacks later.
SessionService comes in three flavours:
InMemorySessionService – for testing locally (outside Agent Engine).
VertexAISessionService – for production deployment using Agent Engine (abstracts away).
DatabaseSessionService – developer managed generic persistent session service for production environment that can use any SQL db.
The Session object is identified by the combination of session_id, user_id, and app_name.
Session object has the following properties:
History – an automatically updated list of significant events (managed by ADK).
Session State – the temporary state of the agent to be managed by the application for use by the agent as a scratchpad.
Activity Tracking – this is a timestamp of the last event for tracking (managed by ADK).
A key question for me is how does one manage shared user sessions (e.g., in case of a joint bank account). Some creative coding and user management is needed.
Memory
This represents the session state processed and stored as a ‘memory’ for use in later interactions. The retrieval happens based on the user_id and app_name to ensure compartmentalisation.
The MemoryService manages the creation, management, and retrieval of memories. As a developer we do not have any control in the memory creation and management process. If we want full control then there are examples of accessing closed session state or checkpoint running session state to create/manage memories manually.
It comes in two flavours:
InMemoryMemoryService – nicely named service for when we are testing.
VertexAIMemoryBank – to be used with Agent Engine which abstracts the process and provides persistence.
Telemetry
Telemetry includes all information that flows in and out of an agent. These streams then have to be tied up together as information flows through a system with multiple interacting agents to identify issues and troubleshoot when things go wrong.
Without telemetry we will not be able to operate these systems in production. For regulated industries like banking and healthcare additional audit requirements are there. Therefore, focus on what to collect. This post will talk about how to collect.
When using Agent Engine a variety of tools are available that can collect the OpenTelemetry-based feeds produced by it. Cloud Trace is the GCP native service for this.
Collecting with Callbacks
ADK supports 6 basic callbacks. This enables us to access state and input/output information as the agent operates.
Before/After LLM: Callbacks triggered just before and after the LLM is engaged. Use this to audit LLM request and response as well as interception for guardrails (checking input for relevance, harm etc. and checking output for quality and content).
Before/After Tool: Callbacks triggered just before and after a tool is engaged. Use this to audit tool use (inputs and outputs) as well as for debugging tool requests generated by the LLM.
Before/After Agent: Callbacks triggered just before and after the agent engages. This is used to audit inputs going into the agent and the response produced by the agent. This is specifically the input and output to the user.
LLM and Tool Callbacks tell us about the inner workings of the agent (private thoughts). Agent callbacks tell us about the external workings of the agent (public thoughts).
Callbacks are the Google recommended mechanism for changing state and they also provide a set of patterns (sharing links here):
Remember callbacks must be implemented in a manner that:
Does not add latency – use async mechanism
Does not introduce parallel logic for processing data
Does not become an alternative integration point
Logging
ADK also supports python logger sub-system. If DEBUG level is used then every aspect of a running agent is logged plus more. But this increases the data processing load as well as tends to slow down the processing.
High data volumes will require significant processing (including correlation of different streams, root cause analysis, and time-series analysis) before it can be converted into an event stream prioritised by criticality to enable Human-on-the-loop (human as a supervisor). I do not expect human-in-the-loop (human as an approver) to be a viable method for controlling and monitoring multi-agent systems.
Imagine you live in a small town in the hinterland.
Imagine the authorities in power ask you to contribute some money towards building a big amusement park in the capital city.
Now imagine, year after year, you hear impressive things about the amusement park—how new rides are being added, and how more and more people visit every year.
Pretty soon, you learn that entrance tickets are now difficult to get and getting more expensive. There’s a waiting list and entry criteria. But the amusement park continues to attract even more visitors each year.
Then one day, you say to yourself: “I need to go and see what the fuss is all about.” You work out the entry criteria and fill in the proper forms. You pay the steep entry fees. Then you patiently wait for your turn, all the while thinking about the fun you and your family will have.
One day, your turn comes. As you enter the amusement park, clutching your ticket, you look around and find people attempting to enjoy the rides.
What you also realise is that the park is really crowded. Certain popular rides have long queues. The food is expensive. The hotel is expensive. There is even a queue at the toilets. Then you wonder—why would people wait and struggle to come in the first place?
Then you see new visitors arriving through the gates. A little bit of hate starts to develop in your heart as you realise that the wait and the queues will only increase. You want the park authorities to stop the entry of people for a while. Then, to your horror, you spy people attempting to climb over the walls. You feel cheated given the effort you made to enter the park through the proper channels.
There is now a general lack of warmth, as people inside the park are increasingly losing their patience and empathy. An accidental bump turns into a heated argument.
You look at your children to see if they are still having fun. They seem to be—but you can see what their future will be like. The struggle they will have.
You’ve had enough. You make your way towards the exit. Only to discover that not only is there a long queue to exit, but there is also an exit fee. You check your wallet to see if you have enough money to leave.
There are many posts providing insights about Agentic AI, the protocols (MCP, A2A, etc.), the frameworks (Langchain, Google ADK, etc.), and not to mention how amazing AI Agents are because they have ‘memory’ and ‘actions’.
What no one talks about is the ‘how’. How does one build and operate agents and multi-agent systems?
This lack of the ‘how’ is what leads to expectation mismatch and the selection of agentic solutions where a simpler solution would have delivered value quicker, with less effort and cost.
In this post I want to talk about the ‘how’.
The First Step
The first step is to get your approach correct. Multi-agent systems are less like building IT Apps or Gen AI Apps and more like building a team of people. You have to continuously iterate between the individual and the team because a change in one will impact the other.
Start defining the agent-enabled journey in terms of:
expected outcomes of the journey
tasks associated with different stages of the journey
constraints (hard and soft) associated with the tasks
tools required for the tasks
communication pathways between tasks
ontology/knowledge/data to carry out the task
handoffs and human-in-the-loop mechanisms for each task
information exchange mechanisms with external entities (e.g., other systems and humans)
Don’t start by worrying about Agentic frameworks, MCP, A2A, etc. These will help you build correctly, not build the correct thing.
The Next Step
Go to the next level of detail:
How do the tasks, constraints, tools and knowledge group into agents? This is not about writing code. Coding complexity of agents is low. For AI agents complexity is in writing the prompts and testing.
Describe how will we test individual agents in isolation then how do we start bringing them together. Can agents deal with failures around them (in other agents)? Can they deal with internal failures and degrade gracefully?
How will we monitor the agents? What patterns are we going to use (e.g., Watchdog pattern) to enable monitoring without degrading agent performance? What actions can be taken to deal with issues identified. Can the agent be isolated rapid to prevent scaling up of the issue? Think of this like writing a diary at the end of the day where you describe and rationalise what you did. Relevant to the agent, interesting perhaps for other agents in the group.
How will we test Agent ensembles?Validate parts of the multi-agent system by describing inputs to each agent, outputs provided, failure scenarios, and upstream/downstream agents. How do groups of agents deal with issues? Can they recover or at least prevent failure from spreading beyond their boundaries? Can they prevent external events (to the ensemble) from disrupting the ensemble?
How do we monitor agent ensembles? How do we combine streams from related agents to give a view of what these agents are up to. Remember with agents grouped together we will need to stitch a narrative from the monitoring feeds. Think of it like folktales relevant to few agents within a group but interesting for other related groups to know.
Bring Agent ensembles together and start to test the whole system. Validate inputs to the system and expected outputs, failure scenarios, and external entities the system interacts with. The exact same layering as with the ensemble with the same set of questions but answered at a higher level.
How do we monitor the whole system. Remember the whole system includes the operators, users, agents, IT systems, and the knowledge required and generated. So monitoring needs to feed the system-wide narrative of what is going on. Think of this like a history of a civilisation. All about what agents/users did to get us to this point. Relevant for everyone.
Hopefully by this time you are convinced of this layered step-by-step approach. How individual interactions give rise to interactions between groups and so on. The same scaling works for other aspects like testing, monitoring, recovery, and operations.
Finally, hope you are excited about the journey that awaits you as you enter the world of Agents!
I wanted to demystify tool use within LLMs. With Model Context Protocol integrating tools with LLMs is now about writing the correct config entry.
This ease of integration with growing capability of LLMs has further obfuscated what is in essence a simple but laborious task.
Preparing for Calling Tools
LLMs can only generate text. They cannot execute function calls or make requests to APIs (e.g., to get the weather) directly. It might appear that they can but they cannot. Therefore, some preparation is required before tools can be integrated with LLMs.
Describe the schema for the tool – input and output including data types.
Describe the action of the tool and when to use it including any caveats.
Prompt text to be used when returning the tool response.
Create the tool using a suitable framework (e.g., Langchain @tools decorator).
Register the tool with the framework.
Steps 1-4 are all about the build phase where as step 5 is all about the integration of the tool with the LLM.
Under the Hood
But I am not satisfied by just using some magic methods. To learn more I decided to implement a custom model class for Langchain (given its popularity and relative maturity). This custom model class will integrate my Gen AI Web Server into Langchain. The Gen AI Web Server allows me to host any model and wrap it with a standard REST API.
The BaseChatModel is the Langchain (LC) abstract class that you need to extend to make your own model fit within a LC workflow. I selected the BaseChatModel because it requires minimal additions.
Creating my custom class: LangChainCustomModel by extending BaseChatModel gives me the container for all the other bits. The container is made up of Fields and Functions, let us understand these in detail. The reason I use a container paradigm is because the BaseChatModel uses pydantic Fields construct that adds a layer of validation over the python strong-but-dynamic typing.
Fields:
These are variables that store state information for the harness.
model_name – this represents the name of the model being used – in case we have different models we want to offer through our Langchain (LC) class. Google for example offer different variants of their Gemini class of models (Flash, Pro etc.) and we can provide a specific model name to instantiate the required model.
client – this is a my own state variable that holds the specific client I am using to connect to my Gen AI Web Server. Currently this is just a static variable where I am using a locally hosted Phi-4 model. I could make this a map and use model_name to dynamically select the client.
tools – this array stores the tools we register with the Langchain framework using the ‘bind_tools’ function.
tool_prompt – this stores the tool prompt which describes the tools to the LLM. This is part of the magic that we don’t see. This is what allows the LLM to understand how to select the tool and how to structure the text output so that the tool can be invoked correctly upon parsing the output once the LLM is done. The tool prompt has to make it clear for the LLM when and how to invoke specific tools.
Functions
This is where the real magic happens. Abstract functions (starting with ‘_’) once overloaded correctly act as the hooks into the LC framework and allow our custom model to work.
_generate – the biggest magic method – this is what does the orchestration of the end to end request:
Assemble the prompt for the LLM which includes user inputs, guardrails, system instructions, and tool prompts.
Invoke the LLM and collect its response.
Parse response for tool invocation and parameters or for response to the user.
Invoke selected tool with parameters and get response.
Package response in a prompt and return it back to the LLM.
Rinse and repeat till we get a response for the user.
_llm_type – return the type of the LLM. For example, the Google LC Class returns ‘google_gemini’ as that is the general ‘type’ of the models provided by Google for generative AI. This is a dummy function for us because I am not planning to distribute my custom model class.
bind_tools – this is the other famous method we get to override and implement. For a pure chat model (i.e., where tool use is not supported) this is not required. The LC base class (BaseChatModel) has a dummy implementation that throws ‘NotImplemented’ exception in case you try to call it to bind tools. The main role of this method is to populate the tools state variable with tools provided by the user. This can be as simple as an array assignment.
Testing
This is where the tyres meet the road!
I created three tools to test how well our LangChainCustomModel can handle tools from within the LC framework. The three tools are:
Average of two numbers.
Moving a 2D point diagonally by the same distance (e.g., [1,2] moved by 5 becomes [6,7]).
Reversing a string.
My Gen AI Web Server was hosting the Phi-4 model (“microsoft/Phi-4-mini-instruct”).
Prompt 1: “What is the average of 10 and 20?”
INFO:langchain_custom_model:Tool Name: average, Args: {'x': 10, 'y': 20}
INFO:langchain_custom_model:Tool Response: 15.0
INFO:langchain_custom_model:Tool Response: Yes, you can respond to the original
message. The average of 10 and 20 is 15.0.
The first trace shows the LLM producing text that indicates the ‘average’ tool should be used with parameters x = 10 and y = 20. The second trace shows the response (15.0). The final trace shows the response back to the user which contains the result from the tool.
Prompt 2: “Move point (10, 20) by 5 units.”
INFO:langchain_custom_model:Tool Name: move, Args: {'x': 10, 'y': 20, 'distance': 5}
INFO:langchain_custom_model:Tool Response: (15.0, 25.0)
INFO:langchain_custom_model:Tool Response: Yes, you can respond to the original message. If you move the point (10, 20) by 5 units diagonally, you would move it 5 units in both the x and y directions. This means you would add 5 to both the x-coordinate and the y-coordinate.
Original point: (10, 20)
New point: (10 + 5, 20 + 5) = (15, 25)
So, the new point after moving (10, 20) by 5 units diagonally is (15, 25). Your response of (15.0, 25.0) is correct, but you can also simply write (15, 25) since the coordinates are integers.
The first trace shows the same LLM producing text to invoke the ‘move’ tool with the associated parameters (this time 3 parameters). The second trace shows the tool response (15.0, 25.0). The final trace shows the response which is a bit long winded and strangely has both the tool call result as well as LLM calculated result. The LLM has almost acted as a judge of the correctness of the tool response.
Prompt 3: “Reverse the string: I love programming.”
INFO:langchain_custom_model:Tool Name: reverse, Args: {'x': 'I love programming.'}
INFO:langchain_custom_model:Tool Response: .gnimmargorp evol I
INFO:langchain_custom_model:Tool Response: Yes, you can respond to the original message. Here is the reversed string:.gnimmargorp evol I
As before, the first trace this time contains the invocation of the ‘reverse’ tool with appropriate argument of the string to be reversed. The second trace shows the tool response. The final trace shows the LLM response to the user where the tool output is used.
My next task is to attempt to implement tool chaining which will be a combination of improving the internal prompts as well as experimenting with different LLMs.
While this is a basic implementation it shows you the main integration points and how there is no real magic when your LLMs invoke tools to do lots of amazing things.
The most intricate part of this custom model is the tool call catcher. Assuming the LLM has done its job the tool call catcher has the difficult job of extracting the tool name and parameters from the LLMs response, invoking the selected tool, return any response from the tool, and deal with any errors.
Key Concept: If you can create steps using deterministic technologies (e.g., workflow engines like PEGA) mapped to actions then you do not need the complexity of agents. This means you have a finite domain of jobs that need to be done. For example: a pizza chain has a few well defined ‘jobs to be done’ and therefore does not need the complexity of an agentic system. Probably this is why the NLP chatbot examples from 2016 were all about ordering pizza and they worked really well!
Agents will start to become a real option if you find that as you step down the first three layers (Utterance, Intent, Action) there is a growing level of complexity that requires some level of decomposition and cannot be applied as a ‘one-size fits all’ process automation.
Decomposition
This decomposition is where the Agent pattern is amazing. For example at the Action level you can have a product agnostic Agent whose goal is to collect the information from a customer and onboard them to the org systems and another Agent could be tasked with fraud detection.
Parallelism
Agents can go about their work happily in parallel, working off the same input. They have well defined methods of interacting with each other and request support from other Agents as needed (e.g., an onboarding Agent for vulnerable customers).
Composition
Agents can also work in a hierarchy where Agents get increasingly specialised (e.g., such as those that implement specific Steps within an Action) to ensure we do not end up with monolith high level agents and can compose Step specific agents across different journeys. An agent that checks for specific types of fraud, or one that is good at collecting and organising semi-structured information (e.g., customer’s expenses) can be used across a number of different journeys as long as it can be assigned a job by the agents dealing with the conversation.
Handoff
We can clearly see two specific types of handoffs:
Conversational Handoff – here an agent hands over the interaction to another agent. For example: a particular customer was identified as a vulnerable customer by the primary contact agent and then transferred over to an agent that has been specially created for the task. The speciality of the agent can stem from custom prompts and governance, custom fine-tuned LLM, or a combination of the two. There may also be specific process changes in that scenario or an early escalation to a human agent.
The receiving agent has the option of not accepting the handoff therefore the sending agent must be prepared to deal with this scenario.
Once the receiving agent accepts the handoff the sending agent has no further role to play.
Task Handoff – in this case we can compose a particular bunch of functionality through task decomposition and handoffs. For example at the Step level we maybe have each Step implemented by a different Agent.
Taking the example from the previous post:
Collect basic information to register a new customer.
[Do Fraud checks]
{Create customer’s record.}
{Create customer’s application within the customer’s account.}
Collect personal information.
Collect expense information.
Collect employment information.
Seek permission for credit check
[Do credit check or stop application.]
The driver agent is carrying out the steps in italics. Then it is decomposing the tasks at the next level of detail between the fraud step (square brackets) and the creating the customer records step (curly brackets). These could be given to two different agents.
In this case the driver agent will decide how to divide the tasks and which agent to hand over which sub-task to. The driver will also be responsible for handling any errors, unexpected responses and the final response received from any of the support agents.
The support agents can refuse to accept a sub-task (or not) depending on the specific scenario).
An Important Agent Design Decision
Now we come to a critical decision for designing our agents. The trade-off between sending task to the data vs fetching data for the task vs a centralising tendency for both. Let us dig a bit deeper.
Sending the Task to the Data:
This is where the Orchestrating Agent drives the process by sending the task to a Serving Agent that is closer to the data. The Serving Agent processes the data as per the task requirement and returns only the results to the Orchestrating Agent. This is required in many situations such as:
Data is sensitive and cannot be accessed directly.
Data processing requires extensive context and knowledge.
Data processing is time consuming.
Associated data has specific usage conditions attached to it.
Results need to be ‘post-processed’ before being returned – e.g., checking for PII.
This is what happens when we seek expert advice or professional help. For example if we want to apply for a mortgage we provide the task (e.g., amount to be borrowed) to an expert (mortgage advisor) who then looks at all the data and provides suitable options (results) for us to evaluate.
We can see this type of Agentic interaction in the future where in a ‘Compare the Market’ scenario our Apple/Google/OpenAI agent becomes the Orchestrating Agent for a large number of Serving Agents operated by different lenders/providers.
Currently Googles A2A protocol attempts to provide this kind of ‘task transfer’ across organisational boundaries. This task transfer requires many layers of security, tracking, negotiations, and authorisation. Given the current state of A2A there are still gaps.
Security and Authorisation: the security posture and authorisation needs to be mixed. The agent operating on the data (Serving Agent) may require access to additional data that the Orchestrating Agent does not have access to. For example, interest rates and discounts. Further, the Orchestrating Agent may need to authorise the Serving Agent to access data owned by the requester. For example, requesters credit history.
Tracking and Negotiations: the tracking of tasks and negotiations before and during the task execution is critical. For example, when going through complex transactions like a mortgage application there is constant tracking and negotiations between the requester and the mortgage advisor.
Fetching the Data for the Task:
Now let us reverse the above example. We fetch the data required for an Agent to complete its task. This will be done through Tools using Framework-based tooling or MCP (for inter-organisational tool use).
There are many scenarios where this pattern is required. The common theme being the task is not easily transferable due to extensive knowledge requirements, cost, context, or regulations. For example, a personal finance advisor works in this way. The advisor does not forward the task to another agent as it is a regulatory requirement for the person dealing with the application to have specific training and certifications.
Here the key task is what data is required vs good to have, how is the data to be gathered, the time intervals for data gathering, and the relative sensitivity of the data being gathered (and therefore the risk holding that data brings). There is also an ethical dilemma in this where what information should be disregarded or not gathered.
I will bring out the ethical dilemma as the other issues are well understood. Imagine you are talking with an AI Insurance Agent, looking to buy travel insurance to your upcoming trip to a city famous for its water-sports. Let us say you mention by accident ‘I love deep sea diving’. Now the Agent asks you if you plan on participating in any water-sports and you reply ‘No, I am just going to relax there’. The ethical dilemma is whether the AI should take your response on face-value and forget about your love for deep sea diving or should it ignore. The choice will impact the perceived risk and therefore the premium. It may collect more data to improve its assessment and also provide a clear disclaimer to the requester that they will not be covered for any water-sports related claims.
There are various mechanism available to solve all of the above problems except the ethical dilemma. That is why we need the next style.
Centralising Data and Task:
In this case we send the data and the task (independently or as part of a deterministic process) to a third agent to process and respond.
This style is particularly important when we want one way of doing something which is applicable across a wider variety of tasks and data. Think of a judge in a court – they get cases pertaining to different laws. The same judge will process them.
The classic example for this is ‘LLM-as-a-Judge’ where we provide the task and data (including LLM response) to a different LLM to evaluate the response on some pre-defined criteria. These are usually implemented using a deterministic orchestration flow.
In our water-sports insurance journey we would have sent the final conversation and data (about the customer and the eligible products) to a validator LLM to ensure best possible customer outcome including sending communications to correct any mis-selling.
This can be risky in its own right – especially if the task and different parts of the data are coming from different sources. Even one slight issue can lead to sub-optimal outcomes.
This post attempts to map Conversational Patterns to Agentic AI. This is because a project must not start with: ‘I want to use agentic to solve this problem’.
Instead it must say: ‘I need agentic to support the style of conversational experience that best solves this problem’.
Layers of Conversation
Every conversation over a given channel occurs between an interested/impacted party and the organisation (e.g., customer over webchat, colleague over the phone). Every conversation has an outcome: positive/negative customer party outcome or positive/negative organisational outcome.
Ideal outcome being positive for both but not always possible.
There are several layers in this conversation and each layer allows us to map to different tasks for automation.
Utterance – this is basically whatever comes out of the customer or colleagues (referred to as the ‘user’) mouth – process this to extract Intents and Constraints.
Intent and constraints – this is the intent and constraints processed to align them with organisational intents and constraints and thereafter extracting a set of actions to achieve them.
Actions – this is each action decomposed into a set of ordered steps.
Steps – this is each step being converted into a sequence of interactions (or requests) to various back-end systems.
Request – this is each request being made and the response processed including errors and exceptions.
Utterance
The key task for an utterance is the decomposition into intents (what does the user want to achieve?) and constraints (what are the constraints posed by the user?). The organisational intent and constraints are expected to be understood. I am assuming here that a level of customer context (i.e., who is the customer) is available through various ‘Customer 360’ data products.
Example: ‘I want to buy a house’ -> this will require a conversation to identify intent and constraints. A follow up question to the user may be ‘Amazing, can I confirm you are looking to discuss a mortgage today?’.
Key Task: Conversation to identify and decompose intents and constraints.
AI Capability: Highly capable conversational LLM (usually a state of the art model that can deal with the uncertainty).
The result of this decomposition would be an intent (‘buying a house with a mortgage’) with constraints (amount to be borrowed, loan-to-value, term of loan etc.). Once are a ready we can move to the next step.
Intents and Constraints
Once the intent and constraints have been identified they need to be aligned with the organisational intents and constraints using any customer context that is available to us. This is critical because this is where we want to trap requests that are either not relevant or can’t be actioned (e.g., give me a 0% life-long mortgage – what a dream!). Another constraint can be if the customer is new – which means we have no data context.
If these are aligned with the organisation then we decompose these into a set of actions. These actions be at a level of abstraction and not mapped to specific service workflows. This step helps validate the decomposition of intents and constraints against the specific product(s) and associated journeys.
Example: Buying a house on a mortgage – specific actions could include:
Collect information from the customer to qualify them.
Do fraud, credit and other checks.
Provide agreement in principle.
Confirm terms and conditions.
Process acceptance.
Initiate the mortgage.
Key Task: Mapping intents to products and associated journeys using knowledge of the product.
AI Capability: The model being able to map various pieces of information to specific product related journeys. This will usually also require state of the art LLMs but can be supported by specific ‘guides’ or Small Language Models (SLMs). This can especially be useful if there are multiple products with similar function but very subtle acceptance criteria (e.g., products available to customers who have subscribed to some other product).
Actions
This is where the fun starts as we start to worry about the ‘how’ part. As we now have the journeys associated with the interaction the system can start to decompose these into a set of steps. There will be a level of optimisation and orchestration involved (this can be machine led or pre-defined) and the complexity of the IT estate starts to become a factor.
Example: Collect information from the customer and Checks.
Now the system can decide whether we collect and check or check as we collect. Here the customer context will be very important as we may or may not have access to all the information beforehand (e.g., new customer). So depending on the customer context we will decompose the Collect information action into few or many steps. These steps can be interleaved with the steps we get by decomposing the ‘Checks’ action.
By the end of this we will come up with a set of steps (captured in one or more workflows) that will help us achieve the intent without breaking customer or org constraints:
Assuming a new customer wants to apply for a mortgage…
Collect basic information to register a new customer.
[Do Fraud checks]
Create customer’s record.
Create customer’s application within the customer’s account.
Collect personal information.
Collect expense information.
Collect employment information.
Seek permission for credit check
[Do credit check or stop application.]
Collect information about the proposed purchase.
Collect information about the loan parameters.
Qualify customer.
Key Tasks: The key task here is to ‘understand’ the actions, the dependencies between them and then to decompose them into a set of steps and orchestrate them into the most optimal workflow. Optimal can mean many things depending on the specific journey. For example, a high financial value journey like a mortgage for a new customer might be optimised for risk reduction and security even if the process takes a longer time to complete but for an existing mortgage customer it may be optimised for speed.
AI Capability: Here we can do with SLMs as a set of experts and a LLM as the primary orchestrator. We want to ensure that each Action -> Step decomposition is accurate as well as the merging into a set of optimised workflows is also done correctly.
Steps and Requests
Once we get a set of steps we need to decompose these into specific requests. The two steps are quite deeply connected as here the knowledge of how Steps can be achieved is critical and this is also dependent on the complexity of the IT estate.
Example: Collect basic information to register a new customer.
Given the above step we will have a mix of conversational outputs as well as function calls at the request level. If our IT estate is fragmented then whilst we collect the information once (minimal conversational interaction with the customer) our function calls will look very complex. In many organisations customer information is stored centrally but it requires ‘shadows’ to be created in several different systems (e.g., to generate physical artefacts like credit cards, passcode letters etc.). So your decomposition to requests would look like:
Conversation: Collect name, date of birth, … from the customer.
Function calling (reflection): check if customer information makes sense and flag if you detect any issues
Function calling: Format data into JSON object with the given structure and call the ‘add_new_customer’ function (or tool).
Now the third step ‘Format data into JSON… ‘ could be made up of multiple critical and optional requests implemented within the ‘add_new_customer’ tool:
Create master record for customer and obtain customer ID. [wait for result or fail upon issues]
Initiate online account and app authorisation for customer using customer ID. [async]
Initiate physical letter, card, pins, etc. using customer information. [async]
Provide customer information to survey platform for a post call ‘onboarding experience survey’ [async]
Key Tasks: The key tasks here are to understand the step decomposition into requests and the specific function calls that make up a given request.
AI Capability: Here specific step -> request decomposition and then function calling capabilities are required. SLMs can be of great help here especially if we find that step to request decomposition is complex and requires dynamic second level orchestration. But pre-defined orchestrated workflows can also work well here.
MCP – the name says it all. Model Context Protocol – a protocol used to communicate the context between tools and the model as the user requests come in.
If you already know the details of MCP jump to the ‘How to use MCP’section.
Main Components of MCP
The Host: This is the big boss that brings together the LLM and various other pieces of MCP – think of it like the plumbing logic.
The MCP Client: This represents the MCP Server within a particular Host and provides the decoupling of the tool from the Host. The Client is Model agnostic as long as the model is provided the correct context.
The MCP Server: Hosts the tools published by a provider in a separate process. It can be written in any language given that JSON-RPC is used to exchange information between the Client and the Server.
Protocol Transport: This determines how the MCP Server communicates with the MCP Client and requires developers to understand how to work with things like HTTP streams or implement a custom transport method.
The MCP Dance
At its simplest when requesting external processing capabilities (i.e., tools or functions) the model needs some context (available tools and what parameters do they take). The tool provider has that context which it needs to share with the model.
Once the user request comes in and the LLM has the tool context, it can then indicate which tool it wants to call. The ‘host’ has the task of ensuring the correct tool is invoked with the appropriate arguments (provided by the LLM). This requires the model to give the correct context, outlining the tool name and the arguments.
Once the tool invocation is done, any response it returns needs to be sent back to the LLM with the appropriate prompt (which can be provided by the server) so that the LLM can process it onwards (either back to the user as a response or a subsequent tool call).
Let us break it down:
1] Context of which tools are available -> given to the Model by the MCP Servers.
2] Context of which tool is to be invoked -> given to the MCP Client that interfaces the selected tool by the Host.
3] Context of what to do with the response -> returned to the Model by the selected MCP Client (with or without a prompt to tell the LLM what to do with the result).
How To Use MCP
Even though MCP starts with an ‘M’ it is not magic. It is just a clever use of pretty standard RPC pattern (as seen in SOAP, CORBA etc.) and a whole bunch of LLM plumbing and praying!
Managing the Build
Given the complexity of the implementation (especially if you are building all the components instead of configuring a host like Claude Desktop) the only way to get benefits from the extra investment is if you share the tools you make.
This means extra effort in coordinating tool creation, hosting, and support. Each tool is a product and has to be supported as such because if all goes well you will be supporting an enterprise-wide (and maybe external) user-base of agent creators.
The thing to debate is whether there should be a common Server creation backlog or we live with reuse within the boundaries of a business unit (BU) and over time get org-level reuse by elevating BU-critical tools to Enterprise-critical tools. I would go with the latter in the interest of speed, and mature over time to the former.
Appropriate Level of Abstraction
This is critical if we want our MCP Server to represent a safe and reusable software component.
Principle: MCP Servers are not drivers of the user-LLM interaction. They are just the means of transmitting the instruction from the LLM to the IT system in a safe and consistent manner. The LLM drives the conversation.
Consider the following tool interface:
search_tool(search_query)
In this case the search_tool provides a simple interface that the LLM is quite capable of invoking. We would expect the search_tool to do the following:
Validating the search_query as this is an API.
Addressing concerns of the API/data source the tool wraps (e.g., search provider’s T&Cs around rate limits).
Any authentication, authorisation, and accounting to be verified based on Agent and User Identity. This may be an optional depending on the specific action.
Wrap the response in a prompt appropriate for situation and the target model.
Errors: where there is a downstream error (with or without a valid error response from the wrapped API/data source) the response to the LLM may be changed by the tool to reflect the same.
Principle: The tool must not drive the interaction through changing the input values or having any kind of business logic in the tool.
If you find yourself adding if-then-else structures in the tool then you should step back and understand whether you need separate tools or a simplification of the source system API.
Principle: The more information you need to call an API the more difficult it will be for the LLM to be consistent.
If you need flags and labels to drive the source system API (e.g., to enable/disable specific features or to provide additional information) then understand if you can implement a more granular API with pre-set flags and labels.
Design the User-LLM-Tool Interaction
We need to design the tools to support the interaction between the User, LLM, and the tool. Beyond specific aspects such as idempotent calls to backend functions, the whole interaction needs to be looked at. And this is just for a single agent. Multi-agents have an additional overhead of communication between agents which I will cover at a later date.
The pattern for interaction will be something like:
LLM selects the tool from the set of tools available based on the alignment between user input and tool capability.
Identify what information is needed to invoke the selected tool.
Process the tool response and deal with any errors (e.g., error with tool selection)
Selection of the tool is the first step
This will depend on the user intent and how well the tools have been described. Having granular tools will limit the confusion.
Tool signatures
Signatures if complex will increase the probability of errors in the invocation. The parameters required will either be sourced from the user input, prompt instructions or a knowledge-base (e.g., RAG).
Beyond the provisioning of data, the formatting is also important. For example passing data using a generic format (e.g., a CSV string) or a custom format (e.g., list of string objects). Here I would prefer the base types (e.g., integer, float, string) or a generic format that the LLM would have seen during its training rather than a composite custom type which would require additional information for the LLM to use it correctly.
Tool Response and Errors
The tool response will need to be embedded in a suitable prompt which contains the context of the request (what did the user request, which tool was selected and invoked with what data, and the response). This can be provided as a ‘conversational memory’ or part of the prompt that triggers the LLM once the tool completes execution.
Handling errors resulting from tool execution is also a critical issue. The error must be categorised into user-related, LLM-related or system-related, the simple concept being: is there any use in retrying after making a change to the request.
User-related errors require validating the parameters to ensure they are correct (e.g., account not found as user provided the incorrect account number).
LLM-related errors require the LLM to validate if the correct tool was used, data extracted from the user input and if the parameters were formatted correctly (e.g., incorrect data format). This can be done as a self-reflection step.
System-related errors require either a tool level retry or a hard stop with the appropriate error surfaced to the user and the conversation degraded gently (e.g., 404 errors). These are the most difficult to handle because there are unlikely to be automated methods for fixing especially in the timescales of that particular interaction. This would usually require a prioritised handoff to another system (e.g., non-AI web-app) or a human agent and impact future requests. Such issues should be detected ahead of time using periodic ‘test’ invocation (outside the conversational interaction) of the tool to ensure correct working.
The Agent2Agent (A2A) protocol from Google is expected to (as the Red Bull ad goes) ‘give agents wings’. But does it align with the other aspect of modern day flying – being highly secure.
To make A2A ‘enterprise ready‘ there is some discussion around Authentication and Authorisation (but not around the third ‘A’ in AAA – Accounting).
Using the Client-Server Pattern
Agents are expected to behave like client/server components whereby the details of the two are not visible. The opacity helps in keeping both ends of the comms from making any assumptions based on implementation details.
The ‘A2AClient’ is the agent that is expected to act on behalf of the user and the ‘A2AServer’ is the main back-end agent that is expected to respond on behalf of the provider and abstract all internal details (including orchestration information). Think of the ‘A2AClient’ like a waiter at a restaurant and all the other kitchen staff, chef, etc. being the ‘A2AServer’. We direct the workings of the kitchen through the waiter (e.g., asking for modifications to a dish, indicating allergies and preferences) without directly becoming aware of the processing going on there.
The transport level security aligns with HTTPS which is an industry standard. This is the plumbing between two A2A endpoints and at this level there is nothing to distinguish an A2A interaction from any other type of interaction (e.g., you browsing the net).
So far so good.
Authentication
Authentication also follows the same theme. HTTP header based authentication described by the AgentCard object. Specifically through the AgentAuthentication object within the AgentCard (see below).
For additional task specific authentication (e.g., special credentials required by an agent for accessing a database) the AuthenticationInfo object (encapsulated in a PushNotification) is to be used.
The authentication mechanism does not support ‘payload’ based identity – this means the authentication mechanism sits outside the multi-agent system and the agent treats it as a ‘given’.
This has some major implications including reliance on centralised id providers and creating a security plumbing nightmare where multiple providers are present. For A2A at some level agents are still seen as traditional software applications.
The concept of decentralised id where the agent controls the identity is still being developed with some interesting projects in this space. This also aligns with the ‘autonomous’ nature of AI Agents where ‘proving’ identity should be part of the agent communication process.
Identity as a Composition
The other aspect which multi-agent systems require are Identity as a Composition. In simple terms, identity is not a single thing when it comes to a system of interacting components (whether human or machine or mixed). The whole interaction lives in a space of identities.
It is the collective that identifies the next two things we are talking about in this process – Authorisation and Accounting.
A common example of this is when we contact a provider (e.g., bank, broadband, utility) to make some changes. The call-centre agent or app always authenticates you first. Your identity is merged with the identity of the agent who is handling your call (or the trust mechanism associated with the app and server) to ensure that you are authorised to request a change and the agent (or app) is authorised to carry out those instructions.
The lack of this thinking manifests itself as the ‘task specific authentication’ mechanism. What you need is a security context (maybe implemented as a container) for the agent to modify and pass-along.
Authorisation
We have already spoken a bit about authorisation without describing the specifics w.r.t. A2A. The thinking here is aligned with various best-practices such as Least Privilege, Granular Control, and Multi-Level authorisation. The documentation also discusses Skill-based Authorisation which involves linking A2AClient authentication with what skills can be invoked on the A2AServer.
There is a big gap in this and again we see software app thinking. With true multi-agentic systems each agent must have an Authorisation Posture. This combined with Identity as a Composition will provide the required autonomy and resilience to such systems.
What is an Authorisation Posture? Before that I just want to clarify what Authorisation means. In the agentic sense it could mean: what am I as an agent authorised to do (attached with my identity), what external requests am I authorised to action (attached with what the requestor(s) are allowed to do) and the context of the current interaction. Some of the latter aspects are hinted at with the ‘Skills-based Authorisation’ discussion.
Authorisation Posture is nothing but a composition of the agents authorisation, the requestors authorisation and the flavouring provided by the current interaction context. The posture may change several times during an interaction and it is a shared entity.
The A2A does not deal with this posture, how it changes and how the agents are able to share it across organisational boundaries to ensure the operational envelope is flexible without depending on localised information (e.g., per tool interaction) or being larger than it needs to be (e.g., blanket authorisation). I don’t believe A2A is actually designed to operate across enterprise boundaries except in tightly controlled scenarios. Which is fine given that AI-based agents with such raw action potential are relatively new. It is leaving people a bit breathless.
Accounting
This is the most interesting aspect for me. Accounting in simple terms means keeping track of how much of something you have consumed or what you have used (mainly used for billing, auditing, and troubleshooting purposes). A2A makes no mention of accounting, assuming all the agents operate within an enterprise boundary or existing ‘data’ exchange mechanism is used transfer accounting information or it is done through ‘API keys’ passed as part of the authentication mechanism. All of the above wrapped by existing logging mechanisms.
Now Accounting requires the ‘who’ (authentication) and the ‘what’ (authorisation) to be clear. From an agents point of view this is also something that needs to be composed.
The lowest level accounting may be associated with the physical resources (e.g., compute) that the agent is using. The highest level may be the amount of time the agent is taking in handling the request and its ‘thoughts’ and ‘actions’ during that process.
So far so good.. but why does accounting need to be composed? Because the other aspect of accounting is ‘how much of other agents time have I used?’. Where we account for the e2e interaction as well as individual agents view of their place in the whole.
If an agent is specialised (e.g., deep research, topic specific, high-cost) then we want to ensure ‘requesting’ agents account for their time. Just like a lawyer would keep careful track of the time they spend for each client and the client will be careful in engaging the lawyer’s services for relevant tasks (e.g., not just calling to have a chat about the weather).
This time accounting can also be dynamic based on available resources, volume of requests and even things like availability of other agents that this current agent is dependent upon. For example surge pricing from Uber accounts for the driver’s time differently (the distance remains the same). If I pay surge price while on a business trip – that cost gets transferred downstream as I claim expenses. There will also be a cost element associated with tool use (e.g., API usage limits).
This type of information will be critical in cases where the agent has multiple options to offload work and the work boundary is not just within our enterprise (therefore we have asymmetric information).
What is needed?
What is needed is a mechanism that allows us to compose and share authentication, authorisation, and accounting information between agents for a truly transparent, secure, and manageable multi-agentic system.
The composition is also very important because there is a level of hierarchy in this information as well. For example the AAA composition information will have its own AAA when inside the org vs when shared with external organisations.
A bank will maintain a list of all its customers for audit reasons but only share specific information with customers or a group of customers (e.g., joint accounts) when required. But in case there is a complaint or an external audit all or some of the information may have to be shared.
This kind of fluidity requires the agents to understand ‘who’ they are, ‘what they can do’/’what others (whether identified or anonymous) can request them to do’, and ‘what actions need to be tracked/communicated to whom’.
The above will also be required if we want to make our organisational agentic ecosystem be part of other groups (partially or as a whole) or making other agents part of ours (permanently or temporarily) in a dynamic manner.
Of course while we treat agents as software apps and focus on the enterprise context (and not true autonomous, independent, and dynamic packets of functionality) these aspects will continue to be ignored.
Fish Eye View 