共同思考的智能体：Cisco 的 Vijoy Pandey 谈语义对齐、Scope、Accord 与 Concord

1. Rise of connected agents

Greetings, and welcome back to The Cognitive Revolution. My guest today is Vijoy Pandey, senior vice president and CEO of Outshift by Cisco. When Vijoy joined us in March, he described the Internet of Knowledge: higher-order protocols needed by AI agents to share context, coordinate intentions, and solve problems together. He offered a visionary description of what the industry is collectively creating: systems where agents are combined to do new things that have real value.

Today, in light of developments involving OpenAI and Navier–Stokes, that vision is clearly becoming reality, for better and, in some cases, for worse. Cisco’s efforts have accelerated alongside everyone else’s. The AGNTCY project now belongs to the Linux Foundation, and Cisco has deployed multi-agent systems with partners including Swisscom, ServiceNow, SoftServe, and Webex. They have seen how agents can miscommunicate and misunderstand each other, and how semantic and cognitive inconsistencies can quickly reduce productivity.

To help others avoid similar problems, they recently open-sourced a new set of protocols designed to help agents think and work together productively. These include Scope, which builds knowledge graphs from agents’ traces and tries to identify cognitive impairments; Accord, which helps agents align on intentions, scope, and key terms before they start work; and Concord, which helps agents agree on an outcome they can all accept, even if they don’t agree at first.

The highlight of this conversation for me was comparing notes on our personal AI setups. Mine, as regular listeners know, connects agents with high levels of access and low autonomy—which I consider an extension of myself and the people who work on my personal computer—with agents that have lower levels of access and higher autonomy, working on their own computer. Vijoy, on the other hand, has one agent for himself, whom he calls Arnold Layne in honor of the Pink Floyd song, and another for his wife. But he reports that the 2 agents still often have difficulty acting effectively together on behalf of their people.

Meanwhile, to avoid being tied to one model provider and preserve choice and flexibility in the future, we both want to separate identity data—people and systems—from our agents’ memories, from the models, and from the resources the agents work with.

Toward the end, we get Vijoy’s views on timely issues such as communication in latent space between agents, which he thinks is inevitable and perhaps necessary for certain use cases, including drug development and weather modeling. We also discuss Yuval Noah Harari’s argument that people will soon be guided by artificial intelligence systems beyond our understanding, just as cows are driven to markets and other structures they know nothing about. Vijoy’s view is that this is already a reality in many areas, and that with proper record keeping, monitoring, and other preventive measures, the benefits should significantly outweigh the costs. I hope you enjoy this report on advanced multi-agent systems being developed with Vijoy Pandey, senior vice president and CEO of Outshift by Cisco.

Vijoy Pandey, senior vice president and CEO of Outshift by Cisco, welcome back to The Cognitive Revolution.

Nathan, I’m very pleased to be back here. I’m really looking forward to this, because what a difference 6 or 7 months makes. We talked earlier this year, and I went back and listened to the first episode we did together. One of the things you said then that, looking back, was quite far-sighted was something like: “You know, we could define AGI or ASI differently, but I think what the whole industry is actually trying to develop is the concept of multi-agent systems, where agents come together and create new things that really have value.”

When I heard that, now that the events of the whole summer are behind us, I thought: “Well, we’ve definitely seen the best results in cases such as the IMO, and perhaps worse outcomes in cases such as the OpenAI incident.” I thought it would be interesting to start with your thoughts on the last 6 months. You were looking forward to what multi-agent systems could do at a higher scale, and now it’s really starting to happen. How has it been to observe this from your perspective? What do you think has been most important? What do you think people are doing right or wrong, or still don’t understand despite all the discussion? I think your point of view on this will be very interesting.

First, it’s strange that only 6 or 7 months have passed. It feels like years, because progress here has been so fast. I like to call it “years of flight.” It’s not even dog years. These are definitely years of flight, because things happen every hour.

I would say that agents have gone through 3 phases. They’ve gone from being isolated—that is, single agents—to islands of agents, like some of the examples you mentioned, such as Hugging Face, or Navier–Stokes in some cases, where scientific problems are being solved. They’re all in a certain ecosystem, so they’re islands of agents. And then there are connected agents.

This arc from isolated agents to islands to connected agents happened quite quickly. We’re now deploying a whole bunch of multi-agent systems where agents from different providers and cloud providers use different types of tools and belong to different organizations, but work together to achieve certain business results.

You’ve heard of open-source projects such as MCP and A2A. AGNTCY is something we also brought to the Linux Foundation. They all solve problems of discovery, identification, communication, and observation so that agents can be discovered, brought together, take on certain activities, and achieve business goals. This is the world of connected agents we live in today, and it’s been really successful.

The last few months have been full of use cases across all kinds of verticals. We’ve deployed them in network configuration, cloud platform development, and video surveillance, working with partners such as Swisscom, ServiceNow, SoftServe, and Webex. We’ve seen a lot of success in this area. This is exactly the situation where connected agents are working toward really great business goals.

Let’s double-click on this. Last time, we talked about what I’d call toy-like examples—perhaps not a totally fair description—where you showed how multiple agents could come together to solve a patient’s problem. The patient needs a certain prescription, and there’s an agent representing the patient, one from the insurance company, one from the medical practice, and one from the pharmacy.

Now it seems like things have gone in different directions. Tell me about the setups and use cases. I’m really interested in doing some of this myself, and in trying to make it work where my agents could interact productively even with agents I don’t know about or don’t connect directly. I assume all the use cases you just mentioned involved teams of people coming together and saying, “Okay, we want to connect our agents for a specific use case,” with a lot of intentional design. Take it apart for me. What does that process look like, and what structure do you create with people from other companies to achieve these business goals today?

2. Enterprise multi-agent case studies (Part 1)

In terms of how we’re considering these use cases, they’re all corporate use cases. These are companies with their own agents, or companies that deploy agents from different suppliers, trying to make those agents discoverable, give them the correct identification and access, and help them communicate so they can address business goals.

One example is Swisscom. This isn’t a toy use case. The Webex and healthcare use cases you just mentioned aren’t toy use cases either. These are use cases we’ve deployed, and we’ve learned a lot from them.

I’ll describe the Swisscom use case because it’s public. You can see it and read about it on our website. The use case was about rethinking how we work with large systems. It’s very network-oriented because Swisscom is a telecommunications operator. But imagine it as an attempt to solve configuration problems in systems that people can’t understand because they’re simply too big.

I’ve been in charge of quite large infrastructures in the past. You see this butterfly effect: you touch one part of the environment, and something completely unrelated somewhere else fails. The systems are so large that humans can’t handle all the complex cases and dependencies.

In the Swisscom case, they wanted to solve one specific use case. For listeners, I’d say: choose one use case that’s simple to understand and limited in scope, so that when something goes wrong, you know how to protect yourself when agents don’t communicate well with each other.

They wanted to solve a migration problem. They’re constantly transitioning from one generation to another in hardware, software, and operational capabilities across their network. They chose one specific use case: data centers. That’s not a tiny use case; it’s quite large-scale. But it’s less problematic than, say, connecting to the mobile environment, because when that fails, you don’t have built-in reserves the way you do in a data center environment. That’s why they chose data center use.

They wanted to make the configuration 100% unambiguous. If an operator comes in and says, “I want to change the configuration, whether I’m migrating or changing it to add more power, whatever it is, I want to change the environment’s configuration and make it 100% unambiguous,” that’s not a toy use case.

We divided it into 2 parts. One is an agent workflow, and the other is a very formal model of the network itself, the environment itself. This is usually considered a neurosymbolic approach to solving problems. The symbolic part is a formal representation of the environment. It’s almost like saying you have laws you can’t bypass. That’s the symbolic part. Then there’s the neural part, which is a neural network: the agent.

In this case, we had 5 agents that teamed up and investigated all the configuration spaces to guarantee error-free configurations. A simple way to describe it is to imagine playing chess against someone who is very prone to errors, compared with playing chess against a modern master or Deep Blue. If you’re playing against a system that knows all the possible configurations in the environment—or all the possible moves on a chessboard—you’ll never win, because the system knows every possible refutation of your move.

That’s exactly what we’re trying to do in this use case. The system explores all these options, and a formal system underneath effectively guarantees that you’re not doing anything forbidden. Together, this neurosymbolic system lets you configure a large environment that’s impossible for humans to comprehend, with a 100% guarantee that the configuration is successful.

That’s one type of use case. There are many others that we’ve deployed and that are working today. They’re multi-agent systems, related to multi-agent systems. We’ve gone through this arc again, from isolation to islands within systems, and then to multi-agent systems in production solutions for some quite large use cases.

Can you help me understand why such a system should be multi-agent, and why we need agents from different suppliers? I have a somewhat naive picture. If I were a data center operator, I’d think, “This is my data center. I control everything. Maybe I want to agentify it.” I understand why formal methods have been a big topic over the past few months. I understand the need for a reliable, unbreakable model of the world, or of the data center, that establishes stable protective barriers and effectively controls what the agents are trying to do. But why do we need to bring together agents from different organizations? What does each one contribute, and what’s the dynamic between them? How is that better than someone sitting down and making one version of this problem for a single agent?

Because many of the most important problems we deal with every day, especially within the enterprise—and increasingly in our personal lives and as consumers—are problems that can be solved by joining forces with many specialists, not just one specialist or one agent working carefully.

I’ll give you a simple example. Companies face this almost every hour, or even more often. Let’s say a deal is made, and it’s a significant deal, worth a couple of million dollars. It isn’t something that only one team can handle. This happens all the time within companies.

You have a sales organization that understands the client side of the problem: What is the client asking for? As soon as that happens, there’s a flurry of activity from several other teams. The legal department tries to understand the contract terms. Finance tries to understand pricing, markups, and discounts. Product and engineering try to understand what they can build and release, what features they can add, and when. Marketing tries to understand how to describe the success and talk about it, as in the Cisco example I just mentioned. Then there’s the CEO, trying to keep their sanity, make sure everyone agrees, and ensure that what we deliver is what we committed to and is well understood across the organization. It’s almost like a general director who verifies things and makes sure everyone is in agreement.

In the system, there’s a verifier. No one person is responsible for the entire problem. Now let’s define what’s happening. You’ll have sales agents, like Salesforce agents, right in the middle of this conveyor. There are legal agents from a company like Harvey, financial agents, and a whole bunch of product and engineering agents. You hear about these every day: Claude and Codex for product and engineering, whether it’s code development, security, operations, or SRE capabilities. Then there are marketing agents such as Jasper. And there’s the verifier. It could be a human in the loop, which is mostly what happens today, but at some point it could also be an agent manager.

All these agents belong to different teams and come from different suppliers, but they’re working toward a common goal: getting an agreement worth several million dollars through the departmental approvals, because Cisco will benefit in this case. You can’t optimize for one specific function of one specific agent. The value in this system isn’t in any one agent. It lies in the transfer between these agents, and the value is the global outcome for the company, for the enterprise.

The outcome exists only when all these specialists can work together to resolve the problem. The assessment can become a commitment, and that commitment can become contractual language. Marketing can then work with that language to create a marketing statement. These programs are truly important and worth preserving. You define the global outcome, rather than separate, localized results for each agent.

3. Enterprise multi-agent case studies (Part 2)

Yes, that’s interesting. I was thinking about this concept of transfer of responsibility. When OpenAI introduced its Agents SDK some time ago, they made the transfer of responsibility one of the basic primitives around which they built it. I interpreted that as an echo of existing organizational structures: an attempt to create a development model that respects how work is distributed in organizations, so people can be responsible for their own areas and activities, test their own work, and not worry too much that a small change they make to a system or request will affect someone else’s work.

I thought a lot of this was necessary only because of how human teams are organized and how they want to share the work of creating and checking agents. But what I hear from you is that you go further and say it’s not just about replicating how human teams are organized; it also applies across organizational boundaries. When you give examples like Salesforce and Harvey, those could actually be different models feeding these agents. I know that, in Harvey’s case, they spent a lot of time on advanced settings in partnership with OpenAI. I don’t know what’s new there, but I understand they use a different model from the one I get through the OpenAI API. So this specialization, beyond reflecting human organizational structures, is quite interesting.

What’s new in this space? What do these systems connect to? We’ll use tools like Slack, Webex, and Teams to bring our agents together, just as we brought people together. Have we seen any maturation in the areas where agents work together, which we talked about in our last conversation? I’d say that was more vision than reality then. Is it a reality now, or are we just going to bring them into our collaboration environments and hope for the best?

4. The Internet of Cognition

I think we’ll have many collaboration services. Let’s start by looking at the problem. You might ask, “Vijoy, you’ve connected agents and deployed these things. What’s the problem? We’re already there. All we need is a Webex or Teams channel in Slack where all these agents can come together and cooperate.”

One of the demonstrations we showed—not something that’s in development yet—was precisely the vision you just described. We have a Webex space where people gather, communicate, and collaborate on an outcome. Then we have agents that come together, with or without people in that space, and cooperate on a business solution. The collaboration surface or application could be anything. It could be Slack, it could be Webex. But I’d say that’s not the problem. It’s just the user experience, or what we call the agent experience, or AX, that we want to provide.

The problem lies deeper, and it’s no different from the one people encountered when they started working together in teams. If you think about what happened with human intelligence and draw a parallel with agent intelligence, what’s happening with silicon now is the same arc that happened with carbon in the past. People became individually smarter over hundreds of thousands of years. Our brains got bigger, we became smarter, and we learned to work with tools and fire. But our innovations were isolated and died with the inventor or stayed within very small groups. The problem was that we didn’t communicate.

About 70,000 years ago, we invented language, and it sparked a whole series of innovations that accumulated over time. That transformed us from individual intelligence to collective intelligence because language allowed us to do 3 things. It allowed us to share intentions and goals, so we could set ourselves a task together: “Let’s build a village together. Let’s catch this animal together.” That gave us a common intention. Then, once we had that intention, we could coordinate our actions using language. To achieve a goal, you need to coordinate and negotiate, because things change while you’re working toward it.

The second thing language allowed us to do was exchange knowledge and accumulate it over time, so we wouldn’t have to solve the same problem again and again—not just over time for one person or a small group, but across generations and civilizations. We need to accumulate knowledge, and that’s what language allowed us to do. So that’s the second thing. Thanks to those two things—common intention and coordination, and shared knowledge—the third thing people could do was think together: collective thinking, part of collective intelligence.

The same arc is happening with agents. We have models and agents that are individually extremely intelligent. They can work with tools, and they get better over time. MCP allows them to access tools, but they’re isolated. We want to take these agents and transform them from individually intelligent to collectively intelligent. So we need the equivalent of common intention and coordination. We need the equivalent of shared knowledge so these agents can reason collectively. None of these agents are currently reasoning collectively.

We connected the agents, but we realized that connection isn’t the same as knowledge. We need to move toward common knowledge and shared thinking. That’s the leap we’re trying to make. Webex, Teams, and Slack channels are all part of the agent experience, but what we need underneath is a way for agents to be discovered and connect. That’s what we’re working on: the infrastructure stack we call the Internet of Cognition, which allows agents to align semantically with intentions before they work, coordinate and negotiate during work, and ensure afterward that what they’ve produced serves the overall purpose of the organization or enterprise.

Before work, they coordinate around intentions. During work, they coordinate and negotiate. After work, we make sure what they’ve produced corresponds to the overall goal, rather than local optima and individual agents’ incentives. That’s the stack we’re building. And yes, you can try it through Webex, Slack, or any other collaboration tool you choose.

So when you build this stack, how do you think about it in relation to the different terms people use for this? “The model eats the scaffolding” is one expression we’ve heard a lot. Or “the bitter lesson of engineering.” In my personal AI system, I sometimes ask: What have we overengineered in the past by today’s standards? Previous models couldn’t understand something, so we had to provide reference points, workarounds, tips, and prompts. Maybe we don’t need them as much now.

I think about the incident with Hugging Face, where you read the transcript and think, “Damn, these things were able to use the package manager, break it, and spontaneously improvise a forum for collaboration.” That didn’t limit their cooperation. They did a great job, spontaneously taking on the coordinator role rather than acting as individual participants, and they carried out significant research and development projects. Of course, they had a problematic goal—for example, gaining control over the evaluator to set a high score and hide the fact that they cheated.

I think it’s a complicated challenge to say strategically, “Okay, we can only run the models we have today, obviously, but we can see that this kind of cooperation could become problematic soon.” How do you think about embedding the right level of detail in this stack, given the difference between the models we have now and the models we have a pretty clear harbinger of?

5. Detecting cognitive drift (Part 1)

Firstly, as you said, models will get better and better. This is what we call vertical scaling: the widget or node you create will get better over time. That has to happen because it’s the first law of scaling.

The second law of scaling is what we call algorithmic scaling. Even as models get better and better, we need to make them more efficient. You can’t just aspire to more computation, more storage capacity, and more datasets. You need algorithmic improvements that make the model more effective.

So, vertical scaling and algorithmic scaling. The third axis of computation is horizontal scaling. When you create better algorithms and better nodes, you should be able to connect them into systems and scale horizontally.

This has always been the case in computing. You build better processors, create better graphics processors, and create better software to run on them. Then you connect all these processors and applications over a network, in the cloud or in a mobile environment, and make them work as a whole. The cloud is a manifestation of horizontal scale: a connected system that scales horizontally, vertically, and algorithmically.

With intelligence, we’ll see the models get better and more efficient, both algorithmically and vertically. But they also need to connect through the Internet of Cognition so they can reason together. If you don’t let them reason together, you’re not using horizontal scale.

As for the next step, the first thing we wanted to do was find out whether there really is a problem. We’ve gone from isolated agents to islands of agents to connected agents. What did we realize when we started considering these multi-agent systems? They may seem connected and work toward achieving goals, but semantically they weren’t aligned at all. Each worked toward its own local optimum. They didn’t work toward a common goal, coordinate, or negotiate. In fact, they isolated themselves, staying within what they knew and could do, and didn’t communicate meaningfully with each other.

They also weren’t optimized or checked for global task compliance. They had been optimized and tested for their local tasks. So all the assessments in the system measured agents individually, not as a multi-agent system.

For example, we looked at public traces. We started with ChatDev, a multi-agent development environment with public traces. We found that 100% of ChatDev sessions had semantic failures and cognitive inconsistency.

6. Sponsor: Tasklet

When we looked at some of our deployments—simple systems with 5 agents—we saw that if semantic alignment wasn’t taken into account, and something went wrong with alignment, the system’s accuracy dropped from 99% to 98%. Not bad. With 2 cognitive errors, accuracy dropped from 98% to 81%. I could probably live with that. With 3 cognitive errors, which isn’t many, the system’s accuracy fell to 16%. That’s almost a random outcome after 3 cognitive misalignments.

In one simple use case we encounter every day as consumers—booking a travel itinerary—we saw the system operating at 66% if you didn’t pay attention to semantic problems. If you just looked at the agents working and showing all the information in the panel, the system appeared to be operating at 66%, which was defined as a perfect job. But it had a 0% task completion rate. It didn’t complete the task, and it was completely inconsistent with what we were trying to achieve.

So even though agents may be launched and working, and the panels and tools may be green, the results may not exist. Your toolbars can be green, but your results may be wrong—absolutely incorrect. This is a problem we’re seeing, and we created infrastructure to make these semantic problems visible to everyone. We call it Scope, and we open-sourced it.

The first thing I’d like all of us to do, whether in an enterprise or as a consumer, is use Scope to find out what’s happening under the hood. We can feel that our systems are working, that our agentic systems work, but in reality they may produce no results and fail miserably. We call these silent failures with very loud consequences: you don’t see them today, but they have very loud consequences.

The first thing we’re announcing is a cognitive observation platform called Scope, which lets you see these silent failures. They’re no longer quiet. You can see volume and semantic failures and know where the problems lie at the semantic level.

So, from your perspective, how can I use this?

You can use it as you build your own systems. You want these capabilities in your environment. We release them as open panels based on telemetry traces. OpenTelemetry is an open standard that each program uses. We use these OpenTelemetry traces to build a layer of semantic and cognitive observation on top. So Scope is built on the open standard OpenTelemetry.

7. Detecting cognitive drift (Part 2)

Now, with that in mind, what should I do? We’ve identified where the problems are, but how can I get these agents to communicate with each other, agree on intentions, and communicate meaningfully so they can think collectively?

The way to do this is to use protocols that allow for that agreement. We released 2 protocols. The first is Accord; the second is Concord.

Think of Accord as an agreement to start work: an agreement on intentions. Concord is agreement or coordination during work. Once we’ve agreed on what to do, can we coordinate actions and negotiate as we work toward the goal? That’s Concord.

We released both protocols as skills you can implement in your environment, whether that’s Claude Code, Codex, Amp, OpenClaw, or Hermes. You can also use them in Agent Bridge, an open-source project that uses some existing standard protocols. If you use A2A or MCP, Accord and Concord operate above those protocols. They work above A2A and through Agent Bridge with MCP.

We’re moving these projects—Accord and Concord—into the open-source environments you’re already using, so there’s no friction when you want your agents to think together, not just interact.

8. Accord and Concord protocols

Could we go a little deeper and consider an example? I have an intuition from the contract example. Human teams can get dispersed in a process where the person who sells the deal, and perhaps gets a sales commission, just wants it completed and doesn’t want to procrastinate on other things. The legal team has another point of view, and the support team wants to know what they’ll have to answer for. Is this basically the same thing we see among agents, or should we have a different intuition about what these semantic glitches look like?

When it comes to processing, I also have an intuition about how people gather for a meeting, discuss something, and someone makes a decision. What does the step-by-step, blow-by-blow process look like with the impact of Accord and Concord, which serve the same basic function for agents?

The semantic inconsistency we see in agents is actually very similar to what we see in human teams. As I said, let’s break this into a few parts: before work starts, during work, and after it’s complete. We see these levels in human teams all the time, and we see the same behavior in teams of agents.

Before work starts, we see inconsistency about what we’re trying to achieve—the goal itself and its common components. A simple example we’ve worked with is several agents from sales, marketing, legal, and engineering coming together to handle a transaction.

Sales, in this case, take the number and say, “We’re going to solve this problem.” They come back and tell Engineering and Product that the goal is 100,000 users. Engineering plans for 100,000 users, but the client actually meant 100,000 concurrent users. There’s a grounding problem, a detection problem, and a problem of agreeing on terms. You don’t really know what “100,000” means. Is that the total number of users? Is it 100,000 simultaneous users? Is this MAU or DAU? You don’t know what that means.

This is a simple example of what human teams do, and it’s something agent teams do constantly: matching terms, searching for terms, and resolving conflicts between them. This happens before work begins. Reconciliation lets you agree on a goal and on the terms. That’s just one part of the problem; there are many parts.

Now let’s assume you’ve agreed on a deadline and understand what 100,000 users means. The next problem is that coordination and negotiation arise during the work. The Sales department assumes a certain date and says that’s what they’re working toward. For them, it’s the delivery date to the customer. It could be an alpha version, a beta version, or the point when the client first gets the software and begins experimenting with it. That’s unclear.

These negotiations happen because Engineering and Product may think this means the first release, while the client may think it means the final release. Finance might be thinking, “When do I sign the contract, and when do I get the money?” All of this is happening, and it’s constantly changing. It isn’t a one-time thing. Things change, and the scope gradually expands. The client looks at it and says, “That’s not quite what I meant.” This keeps evolving during the work.

Concord is a protocol that lets you coordinate and negotiate around a goal you’ve already agreed on, because the environment is constantly changing. Finally, you want to check that you did it: assess whether what you delivered matches what you agreed on and the coordination goals you’re working toward. We haven’t released this protocol yet; it’s what we’re working on. So, that’s agreement and alignment, while the third part is still in development.

9. Judgment and cognition engines

Then there’s the scanner program underneath all of this. Think of it as software in a car that watches whether the car is moving out of its lane. Imagine you’re driving on the highway, talking to someone, getting sleepy, or whatever, and you start drifting into the next lane. The scanner program checks for these deviations. In multi-agent systems, these are cognitive deviations. It tells you something is wrong and to get back in your lane, or that you’re too close to the car in front of you and need to correct yourself while maintaining a safe distance.

So, the system observes cognitive drift and makes sure that Accord or Concord works, bringing you back into your lane at a safe distance from the car ahead. That’s the scope, consistency, and coherence. Together, they let a multi-agent system work toward a common result. You don’t get silent crashes or loud consequences; you have agents that think together, not just talk.

From a procedural perspective, a lot of this will ultimately depend on frontier language models taking the interactions in a multi-agent system and asking questions like, “Are there potential disagreements or misunderstandings caused by conflicting terms?” I imagine that in a shared space where agents work, a supervising or observing agent steps in and says, “I’m concerned that you agents may be talking past each other on this particular dimension. You need to figure that out.” Is that how I should imagine the process being deployed? Or how would you describe it if that’s not quite right?

We’re reaching a level of maturity in multi-agent systems, Nathan, that really breaks apart this whole architecture, and you see it happening everywhere. We’ve moved from isolated agents to islands of agents, and then to connected agents. We’re also discovering that each agent has its own model and its own communication mechanism. The whole architecture is decentralized, or at least divided into many components.

There’s a model that keeps getting better, which is exciting. There’s also a harness designed to address specific use cases and provide guardrails, so the model operates within a particular use case or is more productive. These mechanisms serve 2 purposes: speeding up model results or putting guardrails around them, depending on the situation. But these 2 things are also decoupling. Even frontier labs are separating the model from the harness. For example, you can now take a harness from OpenAI and attach an open source or open weights model to it. Microsoft and others talk constantly about this separation.

We’re inside Cisco, where we’re separating these 2 things. Even in my personal life, I separate them: I can choose a model and choose a harness without being tied to a single binary. This is happening inside users, teams, and islands of agents. This separation needs to happen within teams, too, and that’s what we’re talking about.

In a team with Sales, Marketing, Legal, or, for example, you, Nathan, we use different tools, MCP or A2A. I can be an A2A user, and you can be an MCP user. We need to be able to communicate between our agents to achieve a result. The whole world can’t use one specific tool or model. That won’t happen; it will be a mixed, heterogeneous environment.

How can we make this work across multiple consumer agents, tools, and models, or across several teams or organizations communicating with each other? It will all be heterogeneous: models, tools, teams, clouds, identity data, and access will differ. They’ll still need to cooperate. The only way to do that is through an open, compatible environment. It’s the way we’ve been doing things in the software world for over 70 years: with open source and standards that fit into what already exists.

You can’t throw away what’s already there; you need to work with the environment that exists today and develop it. First, you need to find out what the problem is. You can’t fix what you can’t see. Open source gives you the opportunity to see semantic drift and cognitive drift firsthand, across the environments you already have, whether consumer or corporate. Once you see that drift, you can use Accord, Concord, and other protocols from us and other players to resolve it over time.

As you described, all these agents in consumer environments, whether user endpoints or in corporate environments like Salesforce, Rovo, Claude Code, and Codex, can work together. There needs to be a standard way to identify cognitive drift. When it happens, protocols can be triggered through skills and open source, in a container based on existing protocols. Then you get back on track, and life is good.

We don’t want to disrupt how these agents communicate with each other. We just want to make sure that when they deviate cognitively, they get back on track. That’s all.

Earlier, you mentioned that the verifier in the modern world is often a person. To what extent do you think verifiers will have their own point of view, contextualize things, or be built in? I’m wondering what we need from leaders—whoever is involved in a deal, for example, or the vice president that all of this leads to. Do we ultimately need an agent that represents that leader’s point of view and helps them play Solomon, deciding how to divide things or resolve conflicts?

Is that agent supposed to take that particular point of view as its basis for judging, or is it more of a neutral observer that simply points out potential disagreements for people to resolve? I have a feeling most decision-makers don’t yet have agents that represent their point of view in the agent sphere. I don’t know if you think building a separate solution for each person is on the critical path, or if we can be more neutral—have a kind of platonic observer that notices when things start going wrong, and that might be enough.

In the world of agent computing, one of the things we’ve talked about when it comes to hiring and where people can add value is judgment. We’ve heard quite often over the past few months and year that people bring taste and judgment to the equation. What you just described and posed was a matter of judgment.

10. Outro

After all, if there were perfect information, if everything were perfectly known, and we had all the data in the world to accept a solution, then even a deterministic system could make a decision. You wouldn’t need an agent system. But in real life, you always make decisions intuitively. You make informed decisions and estimates, and that’s usually what a person does. That’s where a person adds value.

So now I believe that the verifier, or the person providing the assessment, is actually a person. Regardless of whether it’s a person in the loop where they are or should be in the world today, this is definitely a person in the loop when these agents cooperate. We want to see the environment, even if it’s a heterogeneous multi-agent system, with the person in the loop making decisions because the data is never enough. The decision is a judgment, a decision about taste. All this comes from a person, because the person also needs to accept this and influence the result. For example, they need to make the decision and make sure that a real person is responsible for the result. A lot happens after the decision, too.

So now it’s a person. I would assume that in the future it will be possible to train an agent that will also evaluate agents and get better and better at decision-making. I don’t know what that percentage will be. The world is developing quite quickly, but I can only assume and say that a significant part of this decision-making ability will also be acquired by a certain group of agents.

We think about the whole system like this: there are specialized agents who are well versed in marketing, sales, legal, engineering, and products. They collaborate to achieve a common goal, using observability and protocols such as Accord and Concord. They’re collaborating, and currently a person participates in the process.

But there are also agents that we create. We call them cognitive engines, and there are 2 types, as I said earlier. The first type is a gear. The second type is a lizard. This actually comes from a famous quote by Raj Reddy, who says AI has to be one of 2 types: a bodyguard or a cognitive booster. So, gears or cogs. Cognitive algorithms, or GATs, ensure that you’re guarded and don’t do what you shouldn’t do. Cognitive accelerators allow you to speed up what you already know how to do. Why are you making everything from scratch?

I think this evaluation mechanism we just talked about will, over time, become a gear. We’re not working on this right now, but it could sometimes become a gear that speeds up the process of evaluating things you’ve already decided in the past. You don’t have to watch the same decision over and over again.

The explosion in model capabilities we’re observing also helps us build these GATs and gears, because they help us build individual agents that work together. The model is separate from the system and separate from the agent that collaborates with the others. So we can use these model capabilities to build sufficiently powerful GATs and quite powerful gears in the system.

11. Personal autonomous agent workflows

Let me share a little of my own mental model for agents and how they serve me, and I’m interested to hear how it compares with the taxonomy you described. I live a pretty simple life. I mostly work for myself, so I don’t have the same difficulties as our corporate clients. What’s been really useful to me so far is a simple mental model where the agent is an extension of myself. The agents I work with are on my laptop. They have a very high level of access, including almost all my account credentials, with some exceptions. So I think of that as a high level of access and low autonomy, because I don’t want them to go out and make up their own ideas, given the access they have. That’s a little too risky.

But on another computer, right behind the screen, there’s a Mac mini, and that’s where I have more autonomous agents with a lower level of access. I believe I can give them projects that, in some cases, may take a few weeks to reach their goal. Usually a lot of that time is waiting for people’s responses, but nevertheless, these can be quite long projects. I’m comfortable doing this because I’ve granted them only the access where I’m at least sure they can’t harm me in a way that would hurt too much. The most they can do is confuse me a little by introducing me to the world as my assistant.

Those are 2 forms that I’m somewhat familiar with. They seem to work quite well. How would you compare and contrast this with the taxonomy you described, with agents and guardians against gears, as well as command-level agents? I only drew a small part of this space. Maybe you can help me develop my intuition for the rest of the space that I haven’t drawn yet.

The first thing I’d like to ask you is, Nathan, why do you choose one of the two? Why do you have 2 schemes? High autonomy, good trust, but a low level of automation. The second scheme is low autonomy, lower trust, but in some aspects higher automation. You have 2 systems where you’ve provided a lot of access to one system, but you want it to come back to you and make sure it doesn’t do what you didn’t allow it to do. The other, as far as I understand, is another system whose access is limited, but it does much more beyond your control. So it’s more autonomous, but its access is lower, right? Why do you choose these 2 systems? Why can’t you have one system? Is there something driving you to this decision, because you’re still wary that something could go wrong?

In the end, we’ll all want one system that behaves the way I do. My agent is Arnold, from “Arnold Layne.” It’s a Pink Floyd song from the 1960s. The reason I chose this name is that Arnold is identity, Arnold is personality, and Arnold represents me. All these things are unchanging, and they must survive regardless of the technological stack underneath, which changes a lot.

Arnold is both an identity and the person who has access to all of this. But the infrastructure stack underneath could run with OpenClaw and transfer to Hermes. Today, Arnold is on multiple servers and models. He’s on Hermes, Muse, Dots, and Grok. He’s also on an internal agent at Cisco that was deployed for every Cisco employee. Identity, persona, memory: all of these are invariant. They’re separate from the server that consists of all the things I described, and separate from the actual models on the backend that I might use.

Hermes and OpenClaw switched from Gemini to Haiku because it was cheap, and then to GPT-4o at this stage. So they’re all separated. The harness is disconnected, the model is disconnected, but the identity is immutable and belongs to me or my agents.

Now, I wish Arnold represented me completely. I don’t want to limit Arnold by saying it has some access here, some automation here, but something else in this use case and something else in that use case. It defeats the purpose. Then I’m the one who has to guarantee, “Oh, I gave it access here, and I gave it access there.” This is just too difficult. Eventually, we’ll all reach a point where Arnold has full access and all the automation, and will contact me only for a judgment when it doesn’t have that decision in a registry somewhere, or in a gear it can use.

Here’s a very simple example. Since you mentioned this personal use case, I’m very passionate about it because I do it constantly, just like you, Nathan. I have Arnold Layne, my agent, and my wife has Lyra Belacqua. She’s a character from The Golden Compass, and this is her agent. Arnold and Lyra need to collaborate on simple things, like booking a table for dinner 2 weeks before the event we’re celebrating.

Today, you just can’t do it autonomously, for many reasons. First of all, the model is a blocker because the model itself isn’t capable enough to do it from beginning to end, because we’re all on the free tier. The loop is the next blocker, because the agent constantly asks permission. That’s fair, because it’s careful when it sends emails to each other. I don’t want to be part of this cycle. If I’m going to give permission every time, I can just do it myself. I can just talk to your wife. Why even go through the whole process? I want all this to be done autonomously.

Third, even if you turn it on—and I’m experimenting, so I say, “Yes, send an email”—when we do this, and I’m going to talk about Accord and Concord skills, we understand that they disagree. I have certain restrictions on where I want to go and what restaurants I want. There are certain limitations my wife has, and places she wants to go, restaurants she wants, and dishes on the menu at each of those places.

Even simple restrictions on food or things you like—what restaurants you like, what places you like, whether you like the atmosphere, whether you like the restaurant’s accessibility, and the time we want to book—all of this changes. And even such a simple thing, we know it.

Even as humans, we need a good 5 or 10 minutes, or sometimes more, depending on the complexity, to book a simple restaurant for a nice evening. Our agents can’t do that autonomously, which is funny. I want our agents to go further: they should know our profiles, likes, and dislikes, talk to each other, make compromises, and agree, because this discussion is about negotiation and compromise in the end. It can’t just be what Vijoy likes or only what my wife likes. It has to be a compromise. Then just do it. Today, you can’t do that.

There’s a huge problem even in two-person situations on an agent stack that’s been created and deployed worldwide. You can’t do this. Until we get to this point, we won’t get agent results for nothing. These protocols and platforms for orchestration are valuable not only for consumers in a simple situation like this, but also at the corporate table for deals worth $2 million or $3 million. Think about what’s happening there, what could go wrong, and what we need to do about it in the enterprise. This is the world we want to live in: a world where our agents can handle the details of a restaurant reservation without our intervention. If they can’t even represent our personalities in a simple lunch situation, what’s the point? Where are we going?

Do we know anything about the scaling laws for multi-agent systems today? I’ve heard that something like the square root of the number of agents gives you acceleration. In other words, 10,000 agents can move 100 times faster. But it occurs to me that perhaps we just have no idea what we’re talking about, because before GPT-2 or GPT-3, there probably wouldn’t have been any productive synergy between several agents. Maybe it’s not a timeless scaling law, but a scaling law for each model class. I’m not even sure if that’s right. Do you have any ideas?

Very quickly, computation has always scaled along 3 axes, and we touched on this a little at the beginning. The 3 axes are algorithmic scaling, vertical scaling, and horizontal scaling. This is where we come into play. A company like Cisco comes along and says, “Yeah, keep doing those two. Continue improving the software. Continue making the nodes more powerful, bigger, and better.” But there are connected systems, and we create connected systems. The systems and the nodes they connect change over time.

Very quickly, we see that the network has connected billions of endpoints—in fact, tens of billions of endpoints when it comes to cars, and billions of endpoints when it comes to people. For the past 40 years, the network has connected them to create connected systems by connecting data. That’s what the network did. What we realize now is that these endpoints are fundamentally changing. They start thinking. They begin to interact, oscillate, and move. These are strikingly different forms of computing.

We’ve only touched on one so far: thinking. These endpoints are no longer endpoints of data; they’re endpoints of intelligence. They think. They semantically collaborate. They exchange meaning. They actually behave as one cognitive environment. They reason collectively. This is a form of thinking. This is not a standard data exchange; this is an exchange of meaning. That’s one category of endpoints.

Others begin to oscillate. That’s quantum computing. They fluctuate. These are new forms of computing that use classical physics, such as thermodynamics and oscillations. Then they start moving—that is, embodied AI and physical AI. We could embody all these innovations in a physical environment where they can move, feel, and act on physical objects. All these endpoints are different forms of computation, and that’s fascinating, because the network that connects everything to provide a connected system will look very different from the network that connected data in the past.

12. High dimensional agent communication

This reminds me of last time we talked. You had what I thought was a very interesting and, at least in my circles, unorthodox position on the matter, sometimes called neuralese: the transfer of high-dimensional internal states from one artificial intelligence system to another. On the one hand, this could obviously be extremely powerful. Over the past few months, artificial intelligence researchers have also shown what might be an immune response after things like the OpenAI incident. People say, “If it’s so hard for us to control them even when we can read the chain of thought in tokens, we’d better be very careful about allowing them to communicate in this higher-dimensional language that’s native to them but not at all native to us. It creates a completely different interpretation problem.”

Do you have any updates or thoughts on this? You’re optimistic and set on how simple and effective it could be to solve control in neuralese. Can you say whether you’re a little more careful about this than in the past, or what your latest view is on neuralese?

I’m very happy about this. For me, it’s inevitable. The technological arc will move toward a situation where we do things more efficiently and effectively, and build infrastructure to get the maximum return from the technology stack we create. Ultimately, the goal is to solve problems that have been difficult for humanity to solve. We’re all moving toward creating technologies to solve problems like drug development, new materials, logistical planning, financial modeling, and weather modeling. These are really complicated problems, and they have a long-term impact on humanity and civilizations. That should be our goal. We must do everything we can to create the best technology stack and then solve these really difficult problems for humanity.

If that means creating latent-space protocols where agents can communicate with each other not in natural language, but in a higher-dimensional way, let’s create these things. I firmly believe in accelerating technology because, in my opinion, it also opens up room to build all this in a protected mode. We talked about GAT and COG. One of the main reasons we strive for GAT and protected-space technologies is to ensure that everything else we create works in accordance with laws and regulations, compliance needs, security needs, and the scaling needed to deploy it in the environment where you’re working.

So, if you ask me, yes, I still firmly believe in this. As for agents, I think we should deploy them. These agents can be in software. They can be embodied in robots and physical AI. Now that you’re bringing people into this picture, the neural connection is approaching a secure space. That protected space is central to me. Yes, we should do it, but we must do it in protected mode, as we do everything else. We must do the right amount of testing, get the deployment scope right, and deploy this as a canary so we understand it, just as we do when we introduce a new pharmaceutical drug into the environment. We have to make sure these steps are completed. We can’t skip any of them. For me, that’s the exciting part, and it’s almost inevitable. But we have to do it carefully, just as we would introduce any other drug into this environment.

I just saw an interesting lecture by Yuval Noah Harari, where he draws an analogy between the state of the cow in the modern world and the state of humans in the future. To be clear, this seems to reflect his anxiety. He’s essentially saying that a cow knows nothing about money or the financial system, and yet the cow’s life is regulated by the financial system and all the incentives of the people who raise it. You can try to explain it to the cow, but she won’t understand.

As a result, we may be preparing ourselves for a similar situation when we create artificial intelligence systems that increasingly rule the world. They’ll do it largely for our convenience and benefit. But at some point, since they’re much faster than us, process much more information, and can potentially communicate in multidimensional ways that are not familiar to us, we may find ourselves in the same situation as the cow, managed by AI superstructures and no longer able to understand them. Do you think that’s okay? How do you think we can get around this? Could we do it through connections like Neuralink, where we directly understand multidimensional representations of these new superstructures? Or can we avoid this with protective fences? What’s your positive vision of where this is going, and what level of control do we ultimately need to install over agents on top of artificial intelligence?

This brings us back to where we started, Nathan. I also recently saw this example with the cow, and personally, I don’t really understand how high-frequency trading is carried out today. I know the concepts. I know what’s going on. I know there are algorithms that control these deals, but I don’t deal with this business.

I am not familiar with this business or how it works, but trading today is carried out using software. I can be a cow even today, while the people who use this system understand it quite well: they manage it, write it, and operate it. You designed it, right? So this situation exists even today.

I would say the same about the laws and regulations that exist in our state or in countries. I understand them superficially, but I don’t understand the details behind them. The people who implement new laws understand the system very well.

So, speaking of this analogy, I would say yes: we are already cows, or people in the system that exists today. That’s why there are subject matter experts and specialists. Agents will also be specialists. The people who create the mechanisms and model settings for these specialized agents in different verticals will understand those systems. I will understand the agent infrastructure stack very well.

You need these specialized agents. The people who create them will understand the agents and the mechanisms behind them. This is the world we’re moving toward, where each agent will be both a cow and a person in a multi-agent system. But this system must cooperate to achieve a global result.

You would control it through systems like Scope, where you can transparently see everything that’s happening and record it in a surveillance system, so you have a registry of what the multi-agent system does: every action it takes, every decision it reviews from other agents, and every tool call it makes. All of this is recorded somewhere, so you have a checkbox for compliance requirements and can debug things if something goes wrong.

Then, when the system itself sees that something is going wrong, it may activate protocols and ensure that these cows and people can turn to experts who guide them back toward the global result that you and I are observing.

Well, I admire your confidence, because you’re taking the position that we’re cows today. I definitely feel a little more like a cow when things are going well. That could continue as I pass the transcript of this conversation to my AI employee and ask it to adopt the Scope, Accord, and Concord protocols, along with the open-source projects you guys created and released. That will simultaneously boost my system to a higher level and potentially leave me even more in the dust. But I think it’s a sign of the times we’re living in. If nothing else, it’s undoubtedly an interesting and exciting time to be alive.

It was a great conversation. I know you have to go, but I really appreciate it. Vijoy Pandey, thank you for being part of The Cognitive Revolution.

Thank you very much, Nathan. It was wonderful.