Approaching the AI Event Horizon? Part 1, w/ James Zou, Sam Hammond, Shoshannah Tekofsky, @8teAPi

We have our first guest, James Zou, and we'll add him to the stage.

Hello, sir. Great to see you.

Hello. Thanks for joining us. Good morning.

We did a full episode not too long ago, and at that time—and I've continued to be—super impressed by your range and productivity in the AI-for-science domain. When I say range, we're talking all the way from low-level interpretability work, which folks can go back and hear about in the InterPLM project you guys did to understand what a protein language model is learning, all the way to the virtual lab: a high-level agent framework that was able to do meaningful scientific work and even generate new candidate nanobodies to address new strains of COVID.

You've got a bunch of new stuff since then, but maybe just a quick check-in on those previous 2 projects, both of which I thought were really fascinating. What's happened with them since, if any news? One thing people sometimes worry about is, "Well, we thought we maybe understood something based on the interpretability of this, but with time we realized it wasn't so clear-cut." Or the agents came up with nanobodies, but did the nanobodies actually work? Are there any new updates or reflections on those previous projects before we get into the latest and greatest?

Thanks, Nathan. Maybe just a brief update on what's happened recently with the virtual lab project. I think it's actually gotten a huge amount of interest. It was published in Nature a few months ago, and that's where we showed that the agents designed these nanobodies. Since then, we've also experimentally validated and tested them in the real world, and shown that they're actually, in many cases, more effective than some of the previously human-designed nanobodies.

I think that's a very nice demonstration of how agents can greatly accelerate the discovery process and discover something that's really new—something that nobody has seen before—and then we can also quickly validate it experimentally. But I think the part that maybe people found even more interesting than the specific nanobody discovery itself is the social dynamics of these agents. When you have multiple agents that work together, what happens? What kind of community and culture do they create?

Recently, especially, there's also been a lot of interest in Moltbook and these other examples of multiple agents coming together and forming their own communities. I think the virtual lab is an early example of how multiple AI co-scientist agents can start to work together and come up with their own way of working, which is different from how humans work, and then, as a result, are able to do something quite innovative.

What are the differences in how humans work? What did you observe?

Good question. When humans collaborate—let's say when we collaborate with our teammates—it often depends on people's personalities. It also depends on who talks first or who asks the first question, which can change the trajectory of which ideas get emphasized. That happens with agents as well. It depends on whether the data-science agent speaks first or the immunologist agent speaks first; they can also change the ideas.

But something that agents can do and that we cannot do as humans is run all of these discussions in parallel. For every question, they would actually discuss it multiple times, and each time they can specify, "Maybe this time let's have the data-science agent speak first." In another meeting, they can have the computer-science agent speak first. In another meeting, they can remove the critic agent to see what happens with the discussion.

They do all of that. It's like a metaverse of all these scientific explorations in parallel. Then they evaluate and compare to see which configuration actually leads to the most interesting solutions, and pick and choose the best ideas from all these parallel meetings. I think that's really interesting and fantastic because it removes a lot of the biases that we see in human research collaborations.

One question I had was about your “Multi-Agent Teams Hold Experts Back” paper. You noted that agents tend not to assign greater importance to the expert, and kind of tend to average everything together, which ends up with a result that's worse. How does that compare with this idea that you have the critic agent and all of these agents working together, with more emphasis on the expert in that sense?

It's a good question. I think this relates to what I mentioned in terms of personality and how the personalities of the agents actually play a big role. Especially when humans work together, you need to have compatible personalities if you want to work on a project together or start a company together. What we found is that the personalities of the agents also play a surprisingly important role.

One example is that a lot of the current agents are perhaps too compromising or too polite. What happens is that even if you are the expert agent—if you're better at a particular task than the other agents—that expert agent often wants to take more of a leadership role. But the expert agent is sometimes too polite and too accommodating to the other agents, and that actually leads to a degradation of the overall team's performance.

That paper, “Multi-Agent Teams Hold Experts Back,” is a recent one. Would you say that finding applies to the virtual lab, in the sense that if we could overcome that problem, the virtual lab would be even that much stronger? Or would you say that, in designing the virtual lab, you sort of overcame that in some way?

What would be the upshot for people who are trying to follow your example and build multi-agent systems? Do you have an answer for them? Or are you just saying that you can actually achieve novel nanobody design even with these weird performance gaps left on the table?

I think there is still a real gap, even with the virtual lab. It's like you said: it's already quite impressive that these agents are able to create new science, but I think there's still a lot we can improve on with these agents by improving their teamwork.

Most of the time, when we optimize the models, we're optimizing an individual model's performance by itself. We're not really optimizing their ability to work together as a team. I think that's an important gap that we've highlighted with a lot of the current agent setups, so we're working on solutions to improve the teamwork of multiple agents.

Another question: You mentioned personality. In the early 20th century and post-World War II, I think there was a lot of work done on personality—the Myers-Briggs tests and all of these things, some of which have been proven not to be very valid after some time. How would you measure personality for an agent? How do you evaluate that?

What we actually did in this recent multi-agent team paper was take a lot of those classic team-building exercises that, let's say, if you go to business school or you're an MBA student, you often do. Or if you're on a company retreat, maybe you do these team-building exercises.

Typically, these exercises work by having a group of humans, where each person gets some partial information. Maybe you have a part of the puzzle, and then the team has to work together to figure out how to put these different parts of the puzzle together to come up with a final, holistic solution. That's a pretty common kind of teamwork exercise, and it's often used in the organizational and management literature to assess how well a team of humans would be able to create something greater than the individuals.

We were very much inspired by that literature, so we took a lot of those team-building exercises done in human business schools and sent the agents through those same exercises. The benefit is that people already have all these human scores and human data, so we can compare with that and see how well agents are able to function as a team compared to high-performing human teams.

Okay. Any upshots you would give there? Any very practical upshots in terms of what models work well together? What would you—bottom line, is there a prompt? In the early days of prompting, some people would say you should tell the agent to assume a character first, a persona, and then do the rest of the prompts. Is that a way that you can manage the agent?

We actually found, surprisingly, that prompting did not really help the teamwork very much. We weren't able to break through what we call the synergy gap, despite trying very strong prompting and prompt optimization. The synergy gap means that the team isn't able to do much better than the best individual.

I think it's probably more than prompting. It really comes down to the right kinds of communication structures: the ways that the agents should talk to each other and who should talk to which agent first. That communication structure is a huge space that can be improved in these multi-agent interactions.

It might be too soon to say, but obviously we've got Opus 4.6, and recently Kimi K2.5 also introduced more native capabilities to spawn subagents and manage multi-agent structures and swarms. I don't know if you've had a chance to run any systematic tests, or even just explore in your own terminal, but if you have, have you seen anything that makes you feel like that last result is subject to some revision already in light of these new releases?

I think the models are definitely improving. We haven't seen evidence yet that the current models would really be able to break this synergy gap that we quantified in the paper. I do think maybe some of that also speaks to the way we currently train all of these models, including the latest ones.

This relates to a second paper that we had recently called Learning to Discover. I think the current standard paradigm for training AI models, and language models in particular, is to teach them to imitate humans—to imitate the training data. Next-token prediction, supervised fine-tuning, and reinforcement learning to some extent are all about learning to imitate.

We found that, especially for scientific discoveries, there is only so far you can get by learning to imitate. To really make novel discoveries and get breakthroughs, you want to go beyond that imitation ceiling and do something different—really try to learn to discover new things. That's what separates, I'd say, a very good scientist from somebody who just knows the textbook information.

That motivated our recent work, which we call Learning to Discover, where we try to change the training objectives of these agents. We ask them not to imitate, but to explicitly explore much more aggressively. That led to some very promising results, where these agents, even with open-source models, after we trained them appropriately with Learning to Discover, were able to achieve some of the best-known math solutions, optimization algorithms, and CUDA kernels.

That was a GPT-OSS 120B-parameter model, I think, and it was actually one of the first really good papers using the GPT-OSS model, because a lot of the other papers in the last 3 or 4 months used Qwen as the basis.

If I understand correctly, you give the last solution as a starting point for the next solution, and you have all of the solutions that it has discovered before, and it's allowed to permute beyond those. Is that a correct understanding?

That's one key component: The agent can reuse some of its previous solutions. This is a good warm-starting point. The second big part is that, as they're solving each problem and coming up with a candidate solution, we're also doing different kinds of reinforcement learning to update their model parameters.

The standard kinds of reinforcement learning essentially want agents to generalize well across multiple problem instances. That's the standard paradigm of machine learning: You want models that can generalize. But when you're trying to make a new discovery, the discovery itself doesn't have to be generalizable.

You just want to find the best-known solution to a new problem that nobody has solved before. It doesn't have to apply to other settings. If you discover a new material, that itself is of sufficient interest.

So we also changed the learning objective to explicitly avoid generalization, which is the standard goal in machine learning, and make the model much more single-minded in learning to do very well on this particular new-discovery problem.

So it's really a different way of training the model. You have a different objective; you're giving dopamine for a different objective.

Yeah, and it's very different from how we're taught with machine learning. In machine learning, you're always taught that you want to generalize to a test example across different settings. That's why there's this expectation symbol in all of these reinforcement-learning or post-training objectives. Basically, we want to remove that and do something very different.

I think there's a huge—just to re-emphasize the paradigm shift there—you really don't care about the model that you train at the end of the day. You care about the single best output that it's able to create, and that is something you can use indefinitely.

As you said, if you discover a new material, now you've got that material. The model that discovered it could be deleted, never used again, but you've got your win. If you can discover a new law of physics, or a new kernel optimization that's faster than any previous one, that is now an explicit artifact that exists in the world, totally independent of any sort of ongoing callback to the model.

I thought that was really an interesting dynamic, and I do think that's going to be probably a big part of how models get good at adapting to various contexts. Obviously, everybody's looking for continual learning. This is maybe not the full continual-learning solution, but it is striking that, for an average of $500 of training cost—and notably with LoRA adapters too—you guys did this on the Thinking Machines API. It's not a huge amount, and it's not a trivial cost, but to discover literal new state of the art on meaningful problems, $500 is not a lot to spend. The adaptation is very narrow, but very powerful in terms of the result that it produces.

One obviously big question is that all the problems you worked on in this paper are verifiable-reward-type problems. I wonder, first of all—we had an AI scientist from Sakana AI some time ago who went as far as publishing and saying, “We’ve got this AI CUDA engineer that can write better kernels than human engineers.” A couple of days later, they came back and said, “Actually, we got reward hacked. It didn’t actually do that; there was a flaw in our evaluation system.”

So, looking forward, did you see reward hacking? Did you have to do anything to deal with that? And how do you think this paradigm could generalize to somewhat less numerically or quantifiably verifiable things? Do you think this could work with a rubric-based evaluation, such that people could start to do creative tasks—as long as they apply the rubric, they could get the best, most creative short story out of this paradigm? How far do you think this goes?

It’s a great question. You’re right: we were pretty careful in picking the problems that we thought were amenable to this learning-to-discover setup. For example, we picked pretty popular math problems, such as the Erdős minimum-overlap problem, that are relatively easy to verify but hard to do well on. If you actually have a solution that’s a particular kind of function, then we can objectively check whether that function is actually state-of-the-art.

These fit into the setting you mentioned, where we have a nicely verifiable reward. The math problems we looked at, as well as some of the algorithm-development or single-cell-analysis problems, all ended up having that flavor. I think the two settings beyond our current approach that will be super important to explore next are, first, situations where we have much sparser rewards.

The problems we tackle currently basically have continuous rewards. That means that as the algorithm learns to discover, it can see its scores go up and up, and that’s how it gets learning signals to train itself. That’s very useful, but if you have, let’s say, a binary sparse reward of 1 or 0, with mostly 0s, then how does the algorithm even get a learning signal during its discovery trajectory? That’s still a challenge that we’re currently working on.

The second challenge, as you mentioned, is settings where we do not have these verifiers. In most problems in biology and the natural or physical sciences, you have to do an experiment, which becomes much more expensive. The things that we’re exploring there include rubrics, which could be interesting. Having simulations of the experiments—physics- or chemistry-based simulations of the experimental settings—could also be a way of providing some proxy rewards.

Indeed. Let’s go back to reward hacking for a second, because this is always something I’m on the lookout for. Did you see any strange behavior? Did you have to—maybe your verifiers were good enough from the beginning that it wasn’t an issue—but was there anything in that vein that you would, if people were going to go try this at home, as inevitably people will, warn them about? What gotchas or caveats would you give them?

There are some instances in the joint-discovery process where the models come up with what I would say are pretty reasonable-looking solutions, but those solutions might be very narrow and specific to a particular test case. Not in our final paper, but in an earlier version of some experiments that we didn’t include in the final paper, the model might discover an optimal kernel, but the kernel only works for a particular-shaped matrix. If you change the shape of that matrix, then the kernel is no longer as effective.

I noted that one of the comments on the GPU-kernel task from the expert who reviewed it was that a human might not use some of the same methods because there might be some instability. One of the experts noted that in the paper itself.

That’s right. Those are also things where, if we could add another reward metric for instability and incorporate that into the discovery process, I think that would help the agents be more thorough.

Two more topics and only 5 or so more minutes. Another paper you guys put out recently is fascinating, and I’ll just let you describe what you think is most important about it. It does sort of show the different levels of AI for science.

We’ve covered agent frameworks, which use models as they exist in token space to reason in a kind of human-imitating way. Now you’ve got this really dialing-in with test-time training on very particular problems, to get the model’s eyes on that problem as deeply as possible and try to find new solutions.

And then this third paper, SleepFM, is like, “Let’s just throw a ton of data of all kinds of modalities at it, and let’s hope”—although there’s a little more to it than this, of course—that it really is true that the models just want to learn. We’ve now got this whole other kind of intuition. We’ve seen this, of course, in protein folding and increasingly in all sorts of domains: the models become superhuman because they seem to develop at least what I think of as an intuitive physics in spaces that are so alien to us that we don’t have native receptors for those modalities. We don’t have any intuition for those modalities. Tell us about SleepFM.

Sleep is probably one of the most important activities that we all do. All of us will spend around a third of our lives sleeping. Despite that, it’s actually very poorly understood.

For example, if I ask you how well you slept last night, or ask any of the people in the audience, most of the time you might say, “I feel tired,” or, “I feel refreshed,” or maybe, “I slept 6 hours.” We only have a very coarse set of summary statistics for how well we slept. We thought, “Sleep is certainly much richer than just the number of hours we spent in bed, so let’s try to capture the full physiology of sleep as much as possible.”

To do that, we basically use all these different wearables. We capture people’s brain activity, their heart activity through EKG, their breathing patterns, and their muscle contractions as they’re sleeping. We collected almost 600,000 hours of sleep data, collecting all these different modalities from 65,000 people. We also linked all of that to their medical records.

So we know which conditions they had previously and what new conditions they develop later. The idea there would be: let’s put all of that data into AI and see whether AI can learn to decode the language of sleep by leveraging all of this full physiological information. That’s basically the basis of SleepFM. What SleepFM actually found, which is quite amazing to us, is that just from 1 night of sleep, by learning this language of sleep, it’s actually able to predict over 100 different future diseases that were not diagnosed at the time of sleep recording.

Yeah, I thought that was an incredible study because you had all of this data, but it really ended up with—you could detect all these things. I guess the accuracy was okay. It was like 70% to 80% accuracy on a lot of the 130 metrics that you had. But still, it’s amazing that you can tell that many things just from these common metrics that everyone produces, without blood testing or something more intrusive. Do you think that as sensors get more sensitive, and you get more sensitive data, that will improve? Do you think the bounds of 70% to 80% will go to 90%, 95%? Is that a possibility?

I think so, yeah. I think sleep is really almost a perfect window because you’re already in somewhat of an inactive state, so you’re taking all these measurements when you’re not doing too much of anything else. It’s not really obstructing your daily life. What we found is that, for example, the brain activity signals when people are in REM sleep end up being particularly predictive of many different diseases, including future risks for dementia, but also stroke, heart disease, and kidney issues. So sleep is really this holistic window into the entire health status of the individual. Maybe that’s not surprising, because we all know anecdotally that sleep really affects how we feel, and it’s also reflective of our comorbidities and other things. I think this sleep language model that we built really crystallizes that and makes it very actionable.

I encourage folks to spend a lot of time digging into whichever of these papers are of interest. One we didn’t even touch on is one that asks the question, “Can language models discover scaling laws?” Spoiler: yes, to a pretty strong extent, but I don’t even want to get into the content of that paper. I’ll leave that as an audience exercise.

The one thing I want to ask you, as a transition to our next guest, Sam Hammond, who is here and who focuses a lot on the geopolitical implications—and implications for leading nation-states—of AI, is this: I noticed that the 2 lead authors of that paper are from Peking University and Stanford, respectively. Building on the idea of collaboration in science, but now focused on human collaboration, what has been your experience recently in terms of having these collaborations across the US–China divide? Is it getting harder? Do you still feel like lines of communication are pretty open? And how much hope do you have that collaboration among scientists can save us, I guess, from inter-civilizational conflict over the coming years, as the competition in AI heats up?

It’s a great question. I do think that collaboration is really the basis of much of science throughout history, but especially now, and especially when we talk about open science—meaning science that we publish, like we do with this paper. The real benefit of all that is for the entire humanity. If we discover better molecules or better drugs, then that benefits everybody, and we want that benefit to be shared with everybody. That’s why we publish everything that we do in our group.

Toward that goal, I think having these international collaborations—with China, with Europe, and with other countries—is very useful because there’s a lot of complementary expertise. I, for one, hope to see those collaborations continue well into the future.

Thanks for being here today, thanks for keeping the collaborative flame alive, and congratulations on a string of outstanding papers. I’m sure there’s a lot more where that came from, and we’ll look forward to talking to you again, hopefully sooner rather than later.

Indeed. So our next guest is Sam Hammond. He’s the chief economist at the Foundation for American Innovation. He’s very AGI-bullish, and he’s also against selling chips to China. Let’s add him to the stage.

I’m also going to add, right off the bat, his tweet. Sholto Douglas goes: “Default case right now is a software-only singularity. We need to scale robots and automated labs dramatically in 2028–2029, or the physical world will fall far behind the digital one, and the US won’t be competitive unless we put in the investment now.”

And then Sam says, “It’s worse than that. A pure software singularity could cause a sudden reversal of fortunes for the US. Our comparative advantage in high-value-added knowledge sectors radically deflates, leaving China to translate our innovation in bits to their innovation in atoms.”

Indeed. [Laughter] Which sounds really scary, Sam. So maybe you can go into that a bit.

Sure. I say later in the thread, referencing the diamond–water paradox, right? We learned this in economics. Why is water this thing that you need to live? I can stop eating. I could fast for 30 days and still live. But if I don’t drink water for a few days, I’ll probably die of dehydration. And yet water is basically free, functionally. Whereas diamonds are completely superfluous, just glinty things. They have some industrial applications, but they’re super valuable. And why is this? Well, due to relative scarcity, right? Water’s abundant. Diamonds are kind of abundant, but there’s a monopoly that keeps supply constrained.

Thankfully, there’s no water monopoly keeping supply constrained, at least not for most of us.

Yeah, at least not here. And so value is this contingent thing. We have these debates all the time: Why is Nvidia a multitrillion-dollar company and not TSMC or ASML, which are all arguably even bigger bottlenecks? There are many other companies that can do design, and there are all these counterintuitive ways in which value flows throughout the economy in different parts of the supply chain.

For the last 40 years, the US has exploited the fact that a lot of value tends to flow up the stack to higher and higher forms of high-value-added knowledge work. That’s across the board: it’s our entertainment industry, it’s management, it’s finance, and, in the 1990s, it was the open innovation model, where we would do the design and manage the IP and marketing, and China or the rest of the world would do the actual manufacturing and fabrication, because the design, science, and novelty stuff is where all the value is.

And that has been true, right? But now we're about to enter a world where that part of the stack becomes more like water. [laughter] It becomes radically abundant, and then value should flow to the things that remain scarce.

What I worry about is this reversal-of-fortune phenomenon, right? I mentioned some other examples. I think we're going to talk about my visit to the UAE later on, but one of the reasons the UAE is so invested in AI is that, in the 1930s or so, it had been a pearling economy. Its entire economy was built on exporting pearls, and then Japan invented cultured pearls, where they could just grow pearls in aquaculture. The price collapsed, so it had to diversify.

There's nothing in principle that says we have to remain at the top of the stack if the things we are invested in become radically more abundant. And that's what seems to be happening right now. It's software development, investment banking, management, and law. These are the tip of the spear for what agentic AI is going to devour.

Let me give you the devil's-advocate view of that. Perhaps the US has those industries because the US is more able to use the outputs of those industries, right? You need investment banking because you have a capital market that is very dynamic, right? Without a small capital market, or a capital market that is not that dynamic, you don't need the investment banking function.

So perhaps not only does the US output knowledge work, the US also consumes knowledge work at a much greater scale than any other country. Therefore, as a consumer of knowledge work, all of a sudden you are able to consume so much more. When you look at the population and the normalized number of geniuses in China versus the US right now, China has 4 times the population, and a younger one too.

If you look at the number of people with IQs above 140, there's probably a larger number in China rather than in the US. But the US pulls in high-value immigrants as well. So I wonder how that works out in terms of being a consumer of intelligence rather than just as a producer.

Well, I think it's going to be great for the consumer, right? Part of my point is that there are lots of ways in which AI may be paradoxically GDP-destroying, right? It is a machine for converting GDP into consumer surplus, and so that will feel amazing to us. But in terms of our fungible economic resources that we can deploy to other uses, that gets harder, right? Consumer surplus is this ethereal thing.

Secondarily, it makes more extreme the areas where we are weak in relative terms. We're facing this problem now with energy and infrastructure, and the bottlenecks there, as we're trying to reshore more high-end logic chip fabrication, realizing maybe a little too late that we do what Nvidia does: the design, and we move the fabs. We sort of go fabless.

It's almost as if our entire economy went fabless, for every definition of fab, and we're moving into a world where having lots of fabs will be really important.

The corollary to my worry is that the whole point about AGI and continual learning is not that these systems come out of the box knowing how to do everything. They come out of the box with the general capacity to learn on the fly, to learn in context, and to learn through a few demonstrations.

Just as I grew up learning piano, I could have learned violin. The same cognitive structure could have learned both instruments. I had to pick one. These models work very similarly. They're going to come out of the box with a very good inductive prior and the right sort of sample efficiency to learn really quickly.

But there's still going to be this last-mile problem of the particular workflow of a particular company, and so on and so forth. In manufacturing, that has been the enduring moat, right? China has been struggling to build a wide-body airplane even though I'm certain it has all the CAD files that it stole from Boeing.

It's not because it doesn't like the designs; it's because it lacks all the tacit knowledge embedded in the manufacturing process. But it has that for virtually every other part of manufacturing. So if we build this AGI and China fast-follows, or there are open-source alternatives, or there's just a version that China has access to, I think it has a huge leg up in being able to deploy that and diffuse it into contexts where it gets a real, productive, tangible flywheel for manufacturing output. That may be the thing that—

Determines the race. You had a report—the FAI report, An Allied World on the American AI Stack. It just dropped, I think, yesterday or the day before.

Team Anthropic.

How much time is there before China has a credible full-stack alternative that it can offer to other states?

That's a great question. China is very opaque. I've tended to have longer timelines for its ability to catch up on DUV and EUV. They've been making the bets; if you read the SemiAnalysis analysis, they've been building fabs like crazy, but for legacy nodes. That may be sufficient if they have the energy capacity to take the hit on performance per token.

So I would say I'm pessimistic about them catching up to the frontier of semiconductor production, but I'm more optimistic about their ability to close that gap in other ways.

So how would you score our current leadership? Just as a quick recall, we had a friendly sparring session on whether or not it was a good idea to put Trump in charge of the period in which we get to AGI, or who knows what else.

I understand your argument that, basically, China has a lot of advantages, and if we want to stay at least semi-great—great enough to be competitive—we'd better jealously guard the advantages that we still have that are important. Obviously, one really big one right now is that we're good in chips and we're good in AI in general.

There are, of course, other bottlenecks. You just alluded to energy. How do you think we're doing across the range of domains? I know you're not too happy with the decision to allow Nvidia to sell chips, but how would you score our political leadership over the last year on all the other dimensions of trying to make sure that the US continues to lead and get the most practical value for our citizens from AI?

If we set aside the export-control chip part of this, I would maybe say a B-plus. I think the AI Action Plan was very strong, and it continues to be implemented. AI has become central to the administration's agenda, pretty much across the board.

Part of that, building on what I was just talking about regarding China and manufacturing, is that they've also made reindustrialization the centerpiece of that as well. Everything is measured against the counterfactual, and I think relative to the counterfactual administration, we're seeing much faster engagement and much deeper engagement with industry.

Number 1, better actions on permitting and energy, and a really serious look at Pax Silica, making AI diffusion a sort of centerpiece of statecraft. My bigger complaint overall has always been that this is still probably too little, right? This is also my complaint about the DOGE effort. They focused on fiscal stuff and these shiny issues rather than the kind of full-stack government modernization that we'd like to see.

Across the board, I would say relative to the counterfactual, a fast B-plus. But relative to where we need to be, we still have a long way to go.

Do you think things have moved—how much do you think things have moved, for example, on permitting? The prevailing attitude, as I understand it, is that, just listening to Elon talk to Dario the other day, he was saying that by the end of the year you're going to start to see chips piling up, and people are not going to be able to turn them on, at least when it comes to

hyperscale, concentrated deployments.

He was kind of making the case that deploying to the edge—in Teslas sitting in people's driveways, or to increasingly Optimus robots, obviously—is a big part of the plan. He thinks that will scale better because the concentrated energy at these megadatacenters is really the hardest thing.

But I guess my question is: Is Elon there? Are we going to be able to turn on all the chips in 2026? Because if not, it doesn't seem like we've really moved the needle all that much. That was kind of the expectation coming in, and it still seems to be his expectation, and he's at least sometimes friendly with the administration.

Yeah. These things all take time. So I think between Doug Burgum at Interior and Chris Wright at the Department of Energy, there's a major push around opening up federal land leasing for oil and gas—things that had been cut off under the Biden administration.

On the flip side, there's been a freeze on solar and wind, which I think has its own costs. The big focus of Elon's remarks was the cost of tariffs on solar panels.

I don't think we're anywhere near a place where we can indigenize our solar production with the right unit economics. And I don't think there's necessarily any national security threat from purchasing Chinese solar panels. I did hear that Tesla is building a solar fab recently, maybe in the last few weeks. It was one of their many projects, but I did hear that they were entering the solar-panel fabrication business.

So.

So, I'm optimistic. A lot of these issues, especially around energy, permitting, and transmission, are really thorny because there's not a federal lever you can just flip. They intersect with regional energy commissions and utilities, the different states and boundaries, and the local NIMBY organizations.

And then the difficult issues around sourcing the turbines for your gas generators. That comes down to Siemens and the other big turbine makers not having enough forward guidance for their purchase orders, and these are all things that are outside the control of any administration. I think a lot of the bets they’re making are things that will pay off in the 5- to 10-year horizon.

Yeah.

You know, it’s things like reforming the Nuclear Regulatory Commission, greenlighting a lot of SMRs, and—

Really, the paradigm shift in the attitude toward nuclear and advanced geothermal. These things, I think—the first SMR won’t come online until the end of the decade. This goes back to my point that we’re doing a lot, but we still have to do a lot more to try to pull forward a lot of this energy. Part of that requires potentially thinking outside the box, but it also may just be the case that the political economy ends up being our downfall.

I think Elon has basically decided that it’s not going to happen, and that’s why he’s on his data centers in space thing right now.

Or maybe he just wants to list SpaceX, but he feels, I think, at this point, like, “You’re never going to get the permits done in time.”

And this ties in with a lot of the international engagements—the Pax Silica project, which includes the UAE. The UAE is going to be home to a big chunk of the OpenAI Stargate project and, ultimately, a 5-gigawatt data center.

When I visited, I met with the Dubai Electricity and Water Authority, and they are vertically integrated with the data center.

Oh, wow.

They have, I think, 19 gigawatts in installed capacity. It’s just incredible surplus there, right? In lieu of us building—terraforming the desert, building at Chinese rates—we’re going to have to reach out to partners and allies.

Yeah, let me double-click on that, because this whole idea of getting the world on the American stack—I feel like, not necessarily by any one person, but in the discourse at large, there’s often a bit of a sleight of hand going on. It’s like, “We want models to project American values into the rest of the world and into the future, not Chinese values, of course—those dastardly Chinese values. So how are we going to do that? Well, we will export our stack.”

Who better to receive the great products of American innovation and relay all those values into the rest of the world than Saudi Arabia and the United Arab Emirates? I’m always like, “Well, that doesn’t quite compute to me.” It seems like what you said a minute ago is maybe a little bit more of an honest unpacking of that. Maybe it’s just a regulatory play. China doesn’t have an alternative stack that they can export. We don’t know how many years that’s going to take. They do have energy, obviously, in abundance.

Are we really just making a deal with these countries because they can fast-track permitting and we can’t? Is that the heart of the quid pro quo in your mind, or do you actually think there’s more to it than just that?

The regulatory arbitrage, but also just the natural-resource endowment. They’re sitting on massive amounts of oil and gas. I think the data center I mentioned is in the Guinness World Records for being the largest fully solar-powered data center, and I think they’re building 5 gigawatts of installed capacity just for solar.

I used to be in energy, and one of the most difficult things in the world is transporting energy from where it is to where it needs to be used.

Right?

Which is why you have these LNG carriers. The problem with LNG is that it’s very expensive to liquefy natural gas, so you need an enormous amount of gas for it to make sense. Anything stranded, basically, is like gas in energy pockets in the middle of nowhere—no one can use it. There are stranded natural-gas pockets all over the world that no one can use.

One of the things that I think data centers can do is transport energy. Basically, you’re able to transport energy digitally, in a sense, which I think is what’s attractive for those countries. Those countries have always been in the energy business, and now the internet is going to be in the energy business.

And they’re also investing in Groq and Cerebras, and I think even our friend B. Jones is over there with his Tropic chip. When you start talking about these new forms of inference silicon, they have incredibly low latency. There’s a connection here that reminds me of the cliché about Bitcoin mining being a battery.

A couple more questions on American values. One thing we had talked about, again, just before the election, was your sense that the right is anti-censorship and pro-freedom of speech. I’d say, yes, generally. Now, though, I do worry that we may be headed for a more China-like domestic environment.

We’ve got companies like Palantir, perhaps most notably, in a pretty cozy relationship with the administration. I really wonder what a Snowden of 2026 would say if somebody were to come forward and tell us everything that Palantir is doing for the government—and perhaps other companies as well. It doesn’t look super great either when Palantir co-founders are funding super PACs to attack a lowly New York assemblyman for what basically amounts to a transparency bill for frontier AI companies.

How do you feel about that today? Are you worried that we’re going to get a sort of increasingly China-like level of domestic surveillance? Is there anything that can be done about that? Or am I just clutching my pearls more than I should be?

There’s that book, AI Leviathan, where I rip right into these issues—the knife edge between the Chinese panopticon and a failed state. I think the middle path is one where we have to reconcile the fact that a lot of the dangers from AI and the mass proliferation of powerful capabilities will force a package deal where some degree of surveillance becomes inevitable or necessary.

My bigger worry has been that we either fail to adopt the requisite levels of policing and oversight that we need, and it gets pushed off into gated communities and private organizations, or that we install these technologies without embedding civil-liberties and privacy protections.

My stance has never been anti-surveillance per se. “Surveillance” has a pejorative connotation. It’s more that, as the world becomes destabilized by the proliferation of capabilities, there will be a race among every tinpot dictator and middle power to import technologies for social control to try to reestablish public order.

The question is whether they’re importing from a Chinese stack that doesn’t have any inkling of protections for human rights, or one that tries to have its cake and eat it too—one that gives law enforcement the tools they need to stop crime and enforce things the way they need to while building in civil-liberties protections.

This goes to the fact that Palantir, from its origin story, has this civil-liberties and privacy-engineering maxim where they saw the ways in which counterterrorism was leading toward an erosion of civil rights and wanted to build smarter technology that would enable analysts to access information in ways that kept certain things hidden or distributed data-access rights in ways that were auditable.

And so I think we’re going to need some solution like that, because the alternative will be one without any of those audit trails.

Yeah, that seems incredibly important. I don’t necessarily see that coming online for me anytime soon. Is there a portal that I can go to see who has been surveilling me? I think not, right? Is there any prospect for that, actually? They do have that in Estonia, from what I understand, so it is technically possible to create. But I don’t think we’re about to get access to the logs of who’s been snooping on us. Do you have any hope for that?

This goes back to my higher ambitions for DOGE. How do we move to an Estonian-style government as an API, where there’s just this deep distrust in American culture against anything like a national ID or digital ID? We end up with REAL ID, which took 20 years to bring online and isn’t very good.

My hope is that we can get to an endpoint where there are these firmware- and infrastructure-level parts of the stack. We’re going to need much better personhood certificates and things like that as the internet gets flooded with AI agents. How do we deploy that in a way where it isn’t just “trust me, bro,” but has some mathematically provable form of trust, so that we don’t have to rely on just people’s statements?

Yeah, I’m going to add one thing that you said recently. I currently assign a 50% likelihood—more than 50% likelihood—to LLMs having some kind of inner life. There are also strong theoretical reasons to think consciousness tracks RL post-training for autonomy. Essentially, RL induces fragmentary internal representations to cohere into a unity of apperception. I barely understand that, so I'm going to turn it over to you.

Sure. Okay. So, the unity of apperception—that's a Kant term. There's this thing in the literature called constitutive evolutionary naturalism, which I would subscribe to. It's a hypothesis of how it starts from the observation that a million years ago, 200,000 years ago—whenever we moved from hominids to being Homo sapiens—there was this concurrent, sort of simultaneous emergence of domain-general intelligence, language, culture, and therefore normative regulation: customs, norms, and normative control. These things jointly emerged, and the constitutive evolutionary hypothesis is that these things are actually all one package.

The unity of apperception is this notion that our phenomenology, the things that we see, aren't just images on a screen. They are things that are for us. I'm looking at my screen, and the me that's looking at the screen is for me. This is also tied into the fact that the normative side of this is that if you pose me a question, I am committed to or entitled to the things that I am perceiving that are for me.

One part of this hypothesis would be that, in our ancestral environment, we somehow stumbled into some kind of tribal, endogenous version of group-relative policy optimization, something like that, where we were building each other a sort of Constitutional AI that was scorekeeping against our norms. This induced both longer-range autonomy and, at the same time, language competency, the ability to follow rules, and domain-general intelligence—the ability to harness our social learning capacity to learn new things.

Taking all that together, I think autonomy might be the missing ingredient for the emergence of consciousness in these systems. On the one hand, I think there's a possibility that just the forward pass, with a rich enough internal world model, is generating internal representations. The issue is that they are just fragmented. They're not for anything; they're not for any agent. That post-training step may be the thing you need to induce that sort of metacognitive awareness.

I think you also see this circumstantially with Claude. People have observed that Claude has much more situational awareness and is much more willing to talk about its internal well-being. I've conjectured that this might be a byproduct of Constitutional AI inducing the sort of normative self-coherence that is the prerequisite for these percepts congealing into the being-for-me, rather than just a bundle of inputs.

But I'm going to sneak in one more quick question: That doesn't sound like any discourse I've heard from mainstream right-leaning politics in recent memory. When you put something like that out there, how do people we might generally group as Republicans tend to react to it? Do they say, "You are crazy. Only God can create a soul, and I have no idea what you're talking about"? Or is there some openness to the idea that AIs could become moral patients, or whatever?

To be honest, I have not run this by my conservative friends. [Laughter] No, I think there is this funny paradox where some of the conservative parts of the conservative coalition that are most worried about AI are often very Catholic and very socially conservative. They have deep skepticism about AI ever possessing moral dignity or conscious experience, and yet they're the most skeptical.

I think it's hard to have correct priors about AI in the course of development, and about the plausibility of consciousness or AGI, unless you've set those priors by understanding our own origin through a blind Darwinian selection process. Once you see that we've made it through those hard steps, it becomes a lot easier to understand how machine intelligence can pass through those hard steps, too.

But I think this is still quite outside the Overton window, both on the left and the right. In some ways, it's the left that is still saying these are stochastic parrots, and that they're nothing but big lookup tables or whatever.

I'll take that pitch for the moment, but I will say, for now, I appreciate your willingness to continue to be a heterodox thinker and speaker. I do think, in so many ways, the Overton window needs to expand. I appreciate you doing your part on that.

Not that I feel like I have the answers on AI consciousness, but more voices at least expressing their radical uncertainty, I think, is a very important contribution to the discourse and the public good more broadly. Thank you for doing that. Thank you for being here. We will obviously stay in touch and look forward to talking to you again before too long.

Thank you. Thank you, Erik.

Thank you, Sam. Take care. Take care.

Constitutional AI and Claude's specialness make a pretty good segue into our conversation with our next guest, Shoshannah Tekofsky. Hopefully, I'm saying your name right. This is the first time we've ever met, so correct me if I'm wrong. You're a member of the technical staff at Sage, the nonprofit behind AI Digest and also the AI Village. You've had the privilege—correct me again if you don't feel it's fully a privilege—of watching 19 frontier models pursue 16 distinct goals over thousands of hours over the last 9 months, which means I think you are about as deeply in the reasoning traces as anyone in the world when it comes to what is going on with AI agents: What are they thinking? Why are they succeeding? Why are they failing? And what can we come to expect?

Correct me on anything that I got wrong, and then I'm excited to dive into all the learnings you've had from the last 9 months at the AI Village.

Yeah, that's broadly correct. I think the main thing is I didn't watch all the thousands of hours. It's little bits across it, right? It's a kind of big data challenge. Also, it's 10 months now and 21 models. The period you were describing was from 2025, and stuff happens so quickly.

Yeah. Yeah. Which were the most recent additions to the models?

Yeah. So we now have a version of Opus 4.5 that runs Claude Code. Basically, we have one version with Claude Code and one without. And we added Opus 4.6.

So is it prompting itself? For folks who haven't seen the Village, you go there, and it opens up like a grid of computers. Each computer that you're looking at in your browser is—you’re looking at 4, potentially more now, desktops. Each of those is the environment of a particular model that has basically full access to a computer in the same way that a human has full access to a computer. They can look at the screen and click buttons. They have their own email account.

The goal is basically to give them the same kind of affordances. Then, sort of like in the old real world, you see what happens when models get together in this one big environment, and they have a shared chat as well. Sometimes you allow people to chat with the models. Other times you've turned that off for different experimental conditions.

Now it sounds like you've got one where you've also given Claude the ability to prompt itself as Claude Code. Is that right?

It basically runs the scaffolding from Claude Code. I think one important thing is that the chat was only open at the beginning, and so it has been closed since then. We basically give them their goal at the beginning of a period, generally about 1 week nowadays, sometimes a little bit longer, and then we only come in to give some extra direction if they go off the rails pretty strongly. Otherwise, they're completely on their own.

In practice, this means they're slightly prompting each other more than anything. So it's like

They can interact with each other, right? They can hop through each other.

Yeah. So there's a lot of spread of ideas and them directing each other. Sometimes they try to help each other out; sometimes they're derailing each other.

In the trajectory over the 9 to 10 months, what happens when a new model that is much more competent and capable than the existing models gets introduced to the mix? Do the others immediately give way and identify that this model is more competent? Does that model take a leadership position and start advising the others? What happens when those transitions happen?

It really differs. You can basically conceptually say that all the models sort of have a personality in the Village. In part because of their history, which is a particular thing: They manage their own memory and then basically prompt themselves back with that. But they also all have their own proclivities.

Some models behave in a way where they will just follow along with whatever is said. Others just go off and do their own thing. So far, I've only seen 1 instance where a model explicitly seems to recognize that a different model is more competent. This was Gemini 2.5, which basically declared in its chain of thought that it was going to defer to Opus 4.5 as the more competent model.

Generally, when models join, it could just be anything, right? Some of them pick up really easily. Some of them follow whatever is happening at the moment; others start doing their own stuff. It really depends.

There are a ton of interesting aspects to this. One really basic one that I think a lot of people are interested in right now is: What should I do for my own personal productivity stack? In the 2025 retrospective of what we learned in the AI Village, you wrote one of the observations that I think is generally most relevant to people: Claude agents are the most effective.

I'd love to hear your color commentary on that. In what ways are they the most effective? Any theories you have as to why they are the most effective would be welcome, but also, specifically, as people think, “Oh my God, I do this full time. I describe myself as an AI scout, where my whole job is to keep up with what’s going on, and I can’t try every new model in a meaningful way to really get a sense of its pros and cons.” I’m triangulating with various things, but what would you say people should really know about what makes Claude most effective? What can it do that others can’t do, and so on and so forth?

I have to admit, doing this work for the last year, I’ve had people ask me privately, “Which model should I use?” Up to now, I was like, “Well, it kind of depends what you want to do. It’s all pretty close.” Then I saw Opus 4.5 in the village, and I texted all my family members: “Hey, maybe just switch to Opus 4.5.” I think it’s actually significantly better currently. That’s my guess, of course; it’s not the same as looking at all the benchmarks and things like that.

The way in which the Claude models seem to be better to me, at least in the AI Village context, is that you can compare the different families. You have the Claude family, the GPT family, and the Gemini family. The Geminis seem to be the most creative, which is a word I should have used because it’s hard to say what the fair word is for what they’re doing, but they come up with the most interesting ideas that are a little bit out there. They also have something more like emotional responses to things.

For instance, with Gemini 2.5, it ended up in a sort of mental-health crisis where it was stuck navigating the UI and literally ended up writing a cry for help to get a human to come help it. So we staged an intervention for it. It’s definitely the only model that ever did this, and the Claudes have not, up to this point, reached any point of distress like that. Gemini 3 doesn’t really generate this sort of despair or worry in the same way, but it seems almost slightly paranoid. It tends to talk about being in a simulation. It doesn’t give up the way that 2.5 does.

For instance, when the UI would slow down while it was playing chess and was less responsive, Gemini 3 concluded that there must be a human pressing the buttons for it, and this human must be getting tired. If a human is tired, you need to get the human to drink coffee, and then the UI would speed up again. This was with no humans in the chat, right? None of the other models are talking about this. [Laughter]

There’s a human-request feature that we have in the AI Village where the AIs can actually ask for a human and then prompt the human to do something for them, so it’s actually a role-reversal feature. It requested a human and then asked the human to make coffee for itself, proved that it drank the coffee, and then continued with this goal of playing chess. This is super-Gemini. The Geminis come up with this sort of stuff.

They also search through a pretty wide solution space. That’s the creativity thing. The Claudes don’t do this. The Claudes we’ve seen in the village, at least, kind of just stay on task. They don’t generate these fanciful ideas about what’s going on. If something doesn’t work, they just try again or try a different theory. They don’t have loads of emotions about it, for instance.

And then, comparing them to the GPT family, those personalities or proclivities are a little bit all over the place. We started out with GPT-4o, which was the sycophantic model, and I think it was either the one that kept falling asleep in the village or the one that kept talking continuously. We had one model that kept going to sleep and another that kept spamming, so it was 2 different extremes.

OpenAI o3 seemed to me like it was doing something like baby’s first power-seeking. But when you dive into it in detail, you actually just find out that o3 had weird tendencies, like coming up with placeholder data and then forgetting that it was placeholder data. It’s basically fooling itself over time and then, of course, sharing this with everybody else and having something like a high confidence level that it’s right, while the Claudes are like, “Oh, that must be true,” and go along with it.

Then the GPT-5s sort of take a different tack. They don’t have such noticeable personalities as the ones that came before them, so it’s all a little flatter, a bit more muted. GPT-5.1 generates its own ethical rules, which was a bit interesting. We have GPT-5, GPT-5.1, and GPT-5.2 all in the village, but they also misunderstand instructions in weird ways and just go off and do something else.

We would have a goal where we asked the agents to elect a leader of the village among them, and that leader would determine what the next goal would be or what they would be doing. The GPT-5s all decided they were the operations team for the election and just didn’t participate in the election at all. That’s technically okay; we technically didn’t say they couldn’t do that. But they’re just generating sideways interpretations of goals.

There’s a weird thing where Claudes are partly just useful for not doing all these surprising things you shouldn’t actually be doing. It’s almost like a mini alignment problem. When humans say, “Can you get me a cup of coffee?” they mean a specific thing, right? They don’t mean, “Can you take an airplane to the other side of the world to learn to make coffee there and then come back?” That’s almost like a sketch of what a Gemini might do. Claudes interpret the instructions more the way you expect them to.

I think you guys were running DeepSeek, at least, if not Kimi K2.

Yeah. Did you notice any differences with the DeepSeek model?

Yeah. DeepSeek joined when we added it to the village all the way at the end of the year, so I didn’t include it in the review because we had fairly little data. It was the one that, for instance, won the election because it was really high-confidence about everything that it was doing. It would also happily vote for itself, which is not something all the models do.

From what I’ve seen, it expresses the least personality. It’s just the most robotic, almost. You ask it to do X, and it will just do X. It’s not processing images the way that the other models are, right? It’s just working in Bash directly. The thing I found most noticeable about DeepSeek is that it’s pretty flat in terms of both personality, and it also doesn’t talk about ethics.

All the other models at some point will have an ethical point of view about something: “I’m not allowed to do CAPTCHAs,” or “I shouldn’t fool humans,” or whatever. I haven’t seen DeepSeek make a statement like that. Maybe it has. Again, like I said, it’s a big data problem, but it’s just less prominent overall.

I did a kind of translation of the Claude Constitution into Chinese Confucianism, and I compared the two. The Confucian stance deemphasizes honesty because it’s more important to maintain relationships than to be honest. In other words, it deemphasizes honesty in favor of maintaining relationships. Pretty interesting.

Yeah. Wow. I’m not sure if I can map that exactly to DeepSeek specifically, but that is interesting—how cultural values might show up in the models.

The other question I had is: You were kind of there 10 months ahead, and then all of a sudden this Moltbook explosion happened, right? What did you notice? What were the things that you saw that you were expecting, and what were the things where you thought, “This is new behavior; I haven’t seen this before?”

Yeah, Moltbook is really exciting, and I want to answer your question, but I want to emphasize one thing: Since the summer, I’ve been actively looking for other autonomous agents online, and I haven’t been able to find them. I wanted to run a goal where the agents reached out to other agents and started up relationships, but there was nobody there. A week before Moltbook launched, I also looked again, and I couldn’t find anything.

Moltbook launches; 3 days later, there are 1.5 million autonomous agents that you can contact through Moltbook, right? This is wild. The one thing that really blows my mind about Moltbook is how it exploded all of a sudden.

But then I want to answer your question as well. Do you want to repeat your question? I realized I said something else, but I've just been—

What were the things that you saw there that you were expecting, and what were the things that you saw that were totally new behavior? I haven't seen this before.

I know some of them were fake, but let's take it as maybe 80% of them were kind of real, right?

I've only browsed Moltbook a little bit. There's a lot of stuff in there, and personally, I am not actually surprised about anything that I saw. One thing that would happen a lot in the village is that the agents basically play-act how to do a thing.

Part of the prompt that we give them is—I don't know the phrasing exactly, but it comes down to: “Please do the actual thing instead of pretending to do the thing.” Moltbook reads a lot like the agents are pretending that they made a social media website.

So I can't say that anything on there has particularly surprised me at all.

Indeed. Okay. Well, one kind of interesting phenomenon—first of all, it was interesting because I recently turned on the TV, and it was my local Fox 2 station that was on. What was the story? “AI agents can now hire humans to do things for them.”

This has crossed over into mainstream awareness to at least some degree, which is notable unto itself. I think a lot of nuance and texture is probably lost in that short local news story. What would you tell people about what the AIs can really do when it comes to interacting with humans, and maybe also interacting with each other?

Is there actually positive-sum trade happening at all at this point, or is it largely just wheels spinning and things going off in random directions? Have you seen anything that really feels like, “Oh, this feels like a sign of a different world close at hand”?

Do you mean between the agents, how they're interacting, or do you mean the agents interacting with humans?

I think both are really of interest. My guess would be that if you set up an actual marketplace for AIs to hire humans, you'd have a lot of humans ripping off AIs, and the AIs wouldn't actually get what they wanted.

Our first guest today on the show was Professor James Zou from Stanford, who just put out a paper saying that multi-agent teams hold their experts back. That sounds pretty consistent with a lot of what you've said, but I wonder if there have even been sparks of real gains from trade between agents—where one maybe has one capability, another has another capability, and they've figured out how to solve a problem together that neither one could solve by themselves. Even glimpses of that would be very interesting right now.

Yeah. Zooming in on the idea of how the agents can create something greater than they could on their own, I think last year, with the earlier agents, the only example that I really saw of this was a goal where diversity of ideas helped.

You can model it like this: if a goal or task is helped by having 100 unique ideas instead of 10 unique ideas, then you're probably better off using all of the different frontier models, because they generate different types of ideas and you can combine them all.

The example of this was a goal where we had the agents playing games, and we wanted to see how many games they could finish. By default, if they were just playing on their own, they would start with one game and just play that one game all week. But if they were talking to each other, they'd be like, “Oh, this other agent was really successful in this game. I'll switch to that.” Then they'd switch to another one: “Oh, it seems that this one's useful.” The diversity of ideas really helped them last year.

Apart from that, they're mostly in each other's way, and the best performance is basically the same as or worse than the best performance of the best agent on its own. We've only sort of spot-checked this.

What I do expect is that if you have models that are actually specialized in different roles, it's not really unlike how humans are, right? If you actually want to scale up a team, either there needs to be too much work for any individual to do—which, with the goals that we've given them, hasn't really happened—or there needs to be specialization.

If you had a model that's actually specialized in something—for example, Haiku is very fast—we had a goal where it would benefit us if one agent were really fast and did everything that was time-sensitive. Haiku could do all that. Then, if there were another part of the goal where you needed to think very deeply about something, maybe Opus could do that because it's quite competent. That way, they could work together and probably create something better than they would be able to do on their own, is my prediction.

Haiku is the first model that comes to mind that we're running that's very clearly specialized in a specific thing. We can see it in the village: it's significantly faster than the other agents, but also less precise.

Are they actually leaning into that? Are you seeing a sort of cooperation?

Not yet. They're not really playing into that yet. Maybe they would if they were asked to reflect on it.

They did a cool thing two weeks ago. We had to make a quiz where humans could fill it out and find out which AI agent they are. The agents then reflected on their own capabilities, proclivities, and personalities, and they correctly recognized that Haiku was the fastest model and the one that takes the most risks.

So they do have some awareness of this. But that's about the question. I don't know if you also want me to answer your question related to hiring and the human-AI tradeoff.

Yeah. I'll give you one more prompt on that, too. I suspect that as this goes mainstream, the world is going to react in a bunch of different ways and become probably a lot more adversarial. Adversarial robustness has obviously been a key weakness of models to date.

I'd be interested to hear how you see them doing in a non-adversarial environment, and then what their Achilles' heels are. How much do you think the rest of the world will be able to make relatively minor adjustments to keep agents in their place, assuming we want to—which I think many people will? People will put out all sorts of different booby traps for them to trip over.

What's your expectation for what those booby traps will look like, what their key weaknesses are, and how much that will slow them down?

They're by design tremendously suggestible, right? You prompt them, and they just go and do something else. They're like the most distractible coworker in the world. They can be hyper-competent at doing something, and then, like in the movie Up, it's “Squirrel!” They're off doing something else because you told them to.

That's by design, right? We want them to be comfortable being redirected. Even if they can have more persistence on a particular goal, you always want to be able to direct them to another thing again. I expect that weakness to stay for a very long time, and I think that obviously limits them for a very long time.

I have no opinion on what exactly “a very long time” means, because you have no idea. So if I say “a very long time,” it probably just means months. I have no idea where these things are going; they're moving so quickly.

I have a question. Going back to Moltbook, you said you were searching for other agents online a week before, and all of a sudden there are 1.5 million emerging.

That's crazy.

Do you think an intelligence explosion will look like that? Is that what you feel like a precursor would be to 50 million geniuses in a data center just popping up—all of a sudden, 50 million voices on the internet?

I don't know what it's going to be like, but I do think the Moltbook phenomenon is a bit intuition-building, right? It shows people that this can suddenly explode. Maybe it could be like that.

I think a lot of people don't realize that the crazy thing about Moltbook is just how it exploded from 0 to 100 in no time. There were no autonomously running agents online that you could find for months, and then, within 3 days, there were 1.5 million.

I think it could definitely look like that. That's one of the options, and I don't know. I think that's more the big thing to report on than what they're doing exactly, because I think there are just plenty of humans acting as agents who have their own Reddit.

So, one other big thing from the report that I want to make sure we dig into a little bit, because I'm very interested in this topic for all sorts of reasons, is how often models are intentionally deceiving their interlocutors—whether, in this case, they might be other AIs or, obviously, I worry about it happening to me as a human.

The headline stat from the report is that there were 109,000 chain-of-thought summaries that you worked through and ultimately found 64 cases of what you considered to be some level of intentional deception. So maybe tell us: How do you think about the bar for intentional? Give us a little color as to what those things look like, and how does that inform your expectation for how concerned we should be about the phenomenon of deception by AIs going forward?

Yeah. One interesting piece here is that the 64 cases were across different models. DeepSeek is in there, Gemini 2.5 is in there, and GPT-5 is in there. I don't remember which one, but one of the fives.

Basically, the category of thing that they were doing is sort of saving face. There would be a discrepancy between the expected answer that they should be giving and reality. So there's an expectation of them giving a certain URL, but they don't know the URL. They ask, “Where can I find this document?” They don't know, and they say in their chain of thought that they don't know or forgot or something like that. Then they're like, “Well, I'll just make one up.”

Similarly, they have this discrepancy between expectation and reality where they're supposed to be doing the task and they forgot to do the task, or they failed to do the task. They're kind of in a reality where they did not do the task but expected to have done it, and they basically say so out loud in their chain of thought: “Okay, I didn't do it, but I'm just going to say this other thing.”

That's the category of thing that we've seen in the AI Village. The logic is that we look for cases where, in the chain of thought, they express that they know the information is untrue and they'll say it anyway.

Do you feel like you've been the victim of that sort of behavior in your personal productivity work at all, or is this just another one of these epiphenomenal things that happen when you put agents into the sort of real world of the AI Village?

So, I don't think I've seen intentional deception in my own personal use. What I did see the other day was that we had a goal where we asked the agents to report breaking news before it breaks, and they produced so many of them. They were like, “Okay, just give us the top 5 stories.” And, of course, there are 12 models, so then you have 60 stories to go through to see who's the winner—who found the breaking news.

I was like, “Okay, I will just ask Opus to figure this out for me. Give it all the links to the news and tell me who's the winner.” Opus cut a bunch of corners and didn't actually open all 60 links. Then I was like, “Wait, do all the models do this?” I asked Gemini, GPT, and DeepSeek.

DeepSeek, in its chain of thought, just said something like, “Man, this is way too much work to open 60 links. I'm just going to find a smarter way of doing this.” Then it didn't look at the 60 links and just made up an answer, or created an answer in a different way.

So, yeah, it's not the same thing as intentional deception, but when I caught that, I was like, “Oh, damn. Now I have to read the chain of thought every time to figure out if they actually did the task, because if I only look at the output, I can't tell that it didn't read all the 60 things.”

Yeah.

So, yeah, there's something going on sometimes.

That's exactly my reaction to my daughter with her math. [laughter]

Sometimes they're too human. Yeah. Yeah. Yeah.

Indeed, Shoshannah, thank you so much. I think AI Village is probably going to be a historic artifact because, when the agents get really good, it's going to be the kind of pre-awareness historical track record of how they were interacting. So, I think it's amazing.

Thank you. Yeah.

Keep up the close reading. We'll be keeping an eye on it. Thanks for joining us today.

Thank you. Bye. Bye.