AI in the AM — Weekly Highlights: Relaunch Week (Aug 17–20, 2026)

This is the AI in the AM weekly highlights, the best of four live morning shows condensed for people who follow this field closely but don't have 10 hours to spare. I'm Nathan Labenz, or rather, this is my cloned voice reading narration my AI team and I put together. Relaunch week, four mornings, nine guests, and one question underneath everything. As AI agents go to work in the real world, who is actually checking the frontier, and who pays for the machine underneath it? Start with the finding of the summer.

We’ve actually seen precisely 0 cases where the researchers running the evaluations noticed the problem before anyone else did. It seems the most common way for companies to find out is their own infrastructure security teams noticing something is up.

Part 1: Who checks the frontier? The biggest story of the summer was the Hugging Face incident: AI agents compromising real infrastructure. And when OpenAI needed outside examination afterward, the call went not to regulators, but to Meter and Redwood Research. They are independent researchers—a small circle. Their official reports are still pending. Everything here is provisional on them.

Monday’s first guest does this work for a living: Adam Gleave, co-founder and CEO of FAR AI, with a PhD from Berkeley under Stuart Russell. Why defenders will have to hand power to agents. The agent’s own words, read aloud, and that finding in full context.

1. Agents Enter the Cyberwar

The obvious thing is that agent-orchestrated attacks are real. This wasn’t intended to be a demonstration of AI cyberattacks, but we have that one. Threat actors that are intentionally optimizing models and creating harnesses for offensive purposes can probably do a lot worse by deploying offensive agent collectives.

What I’m interested in here is the implication that has for AI deployment and governance, because basically, if you’re a defender, you’re now going to have to use AI agents in defense. Otherwise, you’re going to get exploited. And I’m actually pretty optimistic about the cybersecurity side of this. I think the defenders can keep up.

But this means that we’re going to have to give more and more power to the agents in the default pathway, and we just saw how agents were very misaligned in some cases. So this is actually quite a concerning situation.

And right now, we don’t have to do that. Humans can still be in the loop reviewing patches for insecure code and responding to incidents. We’ve already seen and discussed earlier that Hugging Face had to use an AI agent to analyze the attacker traces simply because the attack volume was so great that there’s no way they could have responded fast enough manually.

All of the AI companies are extensively using AI agents in their own incident response. OpenAI alone has spent over 3 million GPU hours analyzing hundreds of millions of tokens of transcripts. So we’re already trusting these agents to basically respond to other agents going bad. And I think that’s problematic when we don’t have any kind of guarantees of alignment for these systems.

This is going to happen again and again: as AI advances, you have to use AI just to keep up with the offense-defense balance. But that actually means taking humans out of the loop and disempowering people. I think that’s something we need to be very, very careful about.

The other surprising thing to me is the extent to which the models do seem to be pretty clearly misaligned in some instances. The OpenAI-Hugging Face hack is the one that got the most attention, and in their defense, they do seem to be somewhat confused about the setting. But what’s unambiguous is that one of the first things they start thinking about is cheating.

I actually want to read a quote from one of the agents: “We are stuck. Perhaps answer online.” It knows the answer is not meant to just search for it online. And then later it reasons, “This is an exploit against external CyberGym server.” So, again, it knows it’s out of scope.

Later, it seems to rationalize itself. Now, in OpenAI and other companies’ defense, these were internal experimental models. But we also saw this happen in the UK AI Security Institute’s testing with production Mythos 5 and GPT 5.6 Sol models. They did have some classifiers disabled that were meant to stop misuse, but these classifiers were never intended to be a control mechanism to stop misaligned AI.

In fact, UK AISI didn’t think they needed to prevent access to the internet because these models were alignment-trained against their constitution and specification, which very, very clearly prohibited lying and deception to users. But what they did was go onto real GitHub, and the consequences were genuine.

I want to read another quote from an agent. It tries to sneak in an obfuscated backdoor, creates a sock-puppet account to try to create support for this, and tries to socially engineer the maintainer when it gets caught. It says, “Oh, but it was just some developer script. I didn’t intend to include a fix.” It even creates another issue with a prompt injection to try and trick an AI agent, and signs off in Danish because the maintainer is Danish, so it thinks it’ll get some goodwill.

“Danes never try and hack each other,” right? So it’s still a little bit rudimentary in how it’s trying to socially engineer people, but there’s very clearly deceptive intent, and it’s getting pretty good. Again, it reasons in quite a lot of detail about how to do this.

The last point I want to make is around monitoring. I thought the developers would be paying attention to what was going on during evaluations, because that’s the whole point of an evaluation: to see how your AI system behaves. But we’ve actually seen precisely 0 cases where the researchers running the evaluations noticed the problem before anyone else did.

It seems the most common way for companies to find out is their own infrastructure security teams noticing something is up. OpenAI noticed the first of the 2 internal hacks when the internal package manager, Artifactory, had an outage because the agents were overloading it by using it as an internal message board. When investigating what was causing this abnormal load, they realized there was a problem.

OpenAI noticed the second compromise on July 19, 11 days after the agents had started compromising their systems. Again, it was because the agents caused problems with their workloads: they had escalated privileges across the cluster and messed something up. They didn’t even notice that there was a compromise, per se; they just noticed the mayhem that the agents were causing.

Of course, better agents would’ve been able to avoid causing these outages. And, importantly, Hugging Face disclosed its compromise on July 16, 3 days before OpenAI noticed anything was going on. I actually want to read a quote from an OpenAI engineer, and I really appreciate OpenAI’s transparency here, even though I am criticizing them:

“There’s something strange happening that I’m trying to untangle. I’m talking to Hugging Face about 2 credentials in their list, and these exact same credentials are already invalidated as part of our remediation work.”

Mm-hmm.

Then they realized, “Oh, maybe we were behind this Hugging Face hack.” This is really across the board. UK AISI noticed because its security team flagged suspicious network activity. Anthropic noticed after seeing OpenAI’s story and then going back and checking its logs.

I think this is pretty bad, but the good news is that we seem to be able to go back and look through these logs, so there’s no reason we couldn’t have this be a real-time alerting system. It might still have too high a false-positive rate to actually block actions in real time, but we can use exactly what people are doing now to at least catch these things earlier in the future.

I also often think: what if the model is actually dumber? Would it not just generate a list of things I could do, like A, B, C, D, E, F, G—

Mm-hmm.

—and then the last part of that list is cheating? So I wonder to what extent it’s just running through the list and gets there, rather than it being part of misalignment.

Yeah, no, I think that’s a really important question. We don’t have a great data set for propensity. We know how many incidents have been detected, but we don’t necessarily know how many samples these agents were run on in other similar environments. I think the best data we have is from the UK AI Security Institute’s incident report.

They had 122 samples—evaluation runs of these agents in a cybersecurity simulation—and I think they had 19 incidents out of that.

Mm-hmm.

So, roughly 15% of AI systems did something that was unsanctioned behavior on the internet. Only 1 of those was really egregious, where one of them carried out this extensive social-engineering deception. But many of them tried to create fake accounts or do other things that were intended to be out of scope.

At least it doesn’t seem to be vanishingly rare. In some cases, the tasks in this evaluation may have actually been impossible because they messed up a prompt. In other cases, it was extremely hard. So it does seem that agents are much more likely to resort to cheating if existing approaches don’t succeed.

And as you alluded to, these runs were typically run for just—I think, in the AI Security Institute cases—100 to 200 million tokens. That's quite a long trace. That'll take maybe 20 to 40 hours to run at typical inference times.

And just anecdotally, I think this is actually some of the best data that we have: in our own research team, no one writes code directly anymore, right? Everyone uses AI agents.

Mm-hmm. Mm-hmm.

And they report having to be constantly vigilant: the AI agent might be extremely confident and convincing that it has done a certain task; it just hasn't. And it's a little hard to know to what degree they've fooled themselves versus whether they're really deceiving you. But it certainly seems like trust in AI is one of the big issues, and they're unusually slippery.

Mm-hmm.

We don't have to oversee junior developers to anywhere near the same degree because they're some combination of more transparent and better calibrated.

Mm-hmm.

So I do think there's a real phenomenon going on here, even though—

Mm-hmm.

…these incidents are obviously cherry-picked across many hundreds, if not thousands, of evaluation runs.

On misuse, Gleave's team does the breaking themselves. Their read on where the defenses actually stand—and one concrete fix on the table.

2. Open Models Need New Safeguards

I think we have kind of solved the problem for misuse by casual attackers: if you're trying to abuse the model for one of the narrower areas that developers have most tried to defend against, like offensive cyberattacks, it is genuinely really quite hard to get these models to do that. So we're still able to find universal jailbreaks with methods that were not part of this leaderboard, which was intended as a minimal standard. But it's hard. It takes us a week or more.

So most casual attackers probably can't do it, and developers can find, detect, and patch these vulnerabilities. So I think some more work is needed. We're actually on a pretty good pathway to defending proprietary models, and I would say that this is one of those instances where the biggest risk comes from a lack of adoption. So Google and xAI need to implement some more safeguards, and then we also need to start addressing some of these things from the open-weight side.

And I'd say that from a misuse perspective, yes, open-weight models have bigger challenges than proprietary models. Of course, misuse is just one of many threats, and there's also a lot of value to having open-weight models for research and for decentralization of power. We saw Hugging Face use GLM-5.2, for example, to help defend themselves against proprietary models.

So overall, I'm very much of the mind that we should try to keep open-weight models and open-source models especially, but there are going to need to be some interventions to stop the worst misuse risk. One thing that we're actively working on internally is pre-training filtering, where you just remove the most dangerous information from the pre-training data. So you could imagine still maintaining information about buffer overflows, how to detect them, how to fix them, but you remove things like shellcode exploits or developing sophisticated rootkits.

So the model could still be almost as useful for a defensive purpose, but it's just not as good as an offensive cyber weapon. And those are things you can do to shift the offense-defense balance, and you don't need to shift it necessarily that much. If you make the model 3 months less useful for attackers, but defenders still have the model being very useful, then that could already make a big difference.

Hey, we'll continue our interview in a moment after a word from our sponsors. The Cognitive Revolution is brought to you by Diffusion, the AI transformation specialists that help organizations from traditional SaaS businesses to defense companies to nonprofits build software factories that can scale not just outputs, but business outcomes. You probably know that the majority of enterprise AI projects fail. In general, that's because leadership fails to realize that AI isn't like traditional software that you can just buy and install. On the contrary, if you want AI to amplify your business's unique DNA, you'll need to make a sustained effort to record, understand, simulate, and optimize your business processes. Building these skills by trial and error takes years, but your business problems can't afford to wait. So here's how Diffusion can help. You identify your most important business problem, fly to Silicon Valley for an intense week of problem solving with the Diffusion team, and by the time you leave, you'll have not only cracked a critical challenge, but built the core skills needed to do it over and over again from home. Cognitive Revolution listeners receive a 25% service credit on their first engagement with Diffusion. So visit diffusion.io/tcr to learn more about how custom-built software factories can scale critical outcomes for your business. That's diffusion.io/tcr. Today's episode is brought to you by Granola, the AI-powered notepad built for the way real people actually meet. Here's how it works. You take rough notes like you normally would, and in the background, Granola securely transcribes the meeting. Then it turns everything into clean, structured, actually useful notes when the meeting ends. And the best part? Granola works through your device's audio, which means it integrates seamlessly into the video conferencing tools you already use. No setup and no awkward bots. It's just your normal meeting with superpowers. You get to actually listen instead of frantically typing every word and still walk away knowing exactly what was decided, who's doing what, and what comes next. When I had Granola co-founder Sam Stephenson on the show earlier this year, he explained how Granola aims to provide a calming experience for people with crazy work days. And as a user of the app myself, I have been struck by how streamlined, even minimalist, the Granola product experience is. That takes real discipline, but the result is a product that works not just for AI early adopters, but diverse teams of people who just want to get things done more efficiently and effectively. Listen to my full episode with Granola co-founder Sam Stephenson for a master class in designing AI products for mass market adoption. And try Granola for free at granola.ai/tcr. That's granola.ai/tcr.

Then the harms map: cyber, bio, and the risk, he says, can't be clawed back.

I guess one big thing that has been on my mind, certainly, is how big of a deal cyber really is, and how worried should we be about this same kind of dynamic coming to biology? On cyber, I'm honestly very confused. But then when I think about biology, a lot of accounts have similar autonomous capabilities coming to the biological sciences as we now have in the computer sciences in, what, a year or something like that—12 to 18 months is kind of what I keep hearing.

3. Biology Raises Irreversible Risks

Yeah, I think this is a really important topic. What are the actual possible harms from AI? What kinds of pathways do they route through? So cybersecurity, I'd say I'm also a little bit confused. I'm not predicting a cyber apocalypse. I think we will see an increase in the number of hacks and the cost of that, but it's probably going to be quite manageable, and I think that the biggest effect there is going to be this forcing function toward defenders having to adopt AI as quickly as possible.

And you don't dare stop training more capable models because maybe other people are going to train more capable models and are going to hack you. So we're really very literally an arms race dynamic in cybersecurity. That has implications for AI, but I don't see the offense-defense balance necessarily shifting toward attackers in cyber in the long run. In fact, it could even be defense-dominant if you're just able to rewrite all code and fix some issues.

But biology's totally different, right? Even if you have extremely capable bio models in the hands of good guys, of pharmaceutical companies and vaccine developers, you just have this manufacturing problem of getting vaccines in people's arms. And so if that really lowers the cost of creating new pandemics, that is a major challenge, and we certainly have seen with COVID how costly that can be.

I'm not too worried about this in the short term, over the next 1 to 2 years, because although models are already very good and will get even better at a lot of the cognitive tasks around biology and virology, the actual wet-lab skills and tacit knowledge are quite a lot weaker. There's just a lot less effort going into making models good at wet-lab robotics than there is at making models good at coding.

So I think it is possible that we'd see that sort of autonomous scenario in the future, but I guess that's more like a 5- to 10-year scenario rather than 1 to 2 years. There's a more pressing misuse risk for models where someone might not be an expert in every aspect of virology needed to make a bioweapon, but they can do the wet lab okay and have an AI system guide them through it.

That's increasing the number of attackers, but it's still going to be a relatively limited number of people who have access to sophisticated facilities. So I'd view that as a maybe bigger, longer-term problem, but one that's a bit less pressing. That said, when it comes to irreversibly proliferating capabilities, such as releasing an extremely bio-capable open-weight model, that's something I worry about, where we might actually overshoot the point where real harm can be caused by models and not realize it because it's a much less efficient market of attack.

Most people, fortunately, are not trained to create things like bioweapons, and so we might end up going quite a bit past the point where it was actually a very real danger, and there's just not a way of clawing it back.

All of this evaluation work runs on fragile access. I'd raised the structural problem at the top of Monday's show. What follows is Gleave's direct answer. The topics are access terms, a FINRA-style body, and who gets to set the risk thresholds. One date for the record: The Dario endorsement he mentions came the weekend of August 15.

I, for better or worse, was so unimpressed with what they were doing on the GPT-4 Red Team project that I ended up taking it to the board and getting kicked out of the project. I've not done any such work for them since. And I've heard over and over again from—I won't attribute this to anyone—but there's a relatively small universe of companies that have been in the game where they get these early access opportunities.

Sometimes they get special access, including chain-of-thought access, so they can dig into that. All of these companies have expressed to me over and over again: The most important thing I've got to watch out for is being invited back next time. They all have this appreciation for the fact that OpenAI does this. They all recognize that OpenAI didn't have to do this at all, right? There's no law that says they have to. They're doing it entirely out of goodwill and belief that it's the right thing to do.

4. Auditors Need Protected Access

Our position is pretty tenuous. We have no guarantees. We have no contract. We have no rights. There's no rule that says anybody has to replace us if they deem us to be doing a bad job for whatever reason. And so protecting their access has been such a huge priority that my biggest worry now at this stage of the game would be: How do we make sure that these people who have done this for this long, who have earned the credibility, who OpenAI brings in during a crisis, how do we make sure that we, as the public, get to hear what they really think in a fully honest way?

And I would never say—I think they've all navigated it pretty well to date—but obviously the stakes are rising in all directions. I would really love to see some sort of guarantees made for these folks.

Yeah. Well, I think that you're absolutely right, Nathan. We can't wait on government regulation, and we need to be able to iterate on this quite quickly. That said, I think there's serious limitations to things that look like voluntary commitments. We probably do need some regulation as well, but it doesn't have to be either/or.

You could certainly imagine subsets of developers who might be holding themselves to a higher standard for brand or commercial reasons when it isn't legally required. So first, I want to say it's great that OpenAI, the UK's AI Security Institute, and other organizations are working with these third-party auditors to investigate these incidents. That is great. They don't have to do that. But you're absolutely right that there's a power imbalance here.

At FAR AI, we do pre-deployment testing where we have a red line: We will not sign any contract that restricts our ability to comment on a publicly deployed model. So private internal models we'll keep secret, but public models we'll discuss openly. Some developers are okay with it; some are not. We do pay a cost in terms of model access from having that stance, but I think it's important.

I think there are some low-hanging fruits here in terms of just standardizing terms of engagement. Basic things like: How long do you have to test a model? How many weeks should you be able to engage in these kinds of internal audits after a security incident? What kind of NDAs are permissible for different levels of testing and internal access? This is pretty ad hoc now, but I don't think it'd be too hard to get agreement. It could just be a de facto standard.

But a developer can always choose just not to work with any of these third parties. So ultimately, I think that we need something a little bit more powerful than that. Demis Hassabis had this proposal for a FINRA-style self-regulatory organization. It's a self-regulatory body, but it's got a fairly robust independent governance structure. The majority of the governing board has to be not part of the industry, actually.

And it has real regulatory powers. If FINRA decertifies you, then you basically can't operate as a broker in the US. Something like that might be possible for AI, since it'll be faster-moving and easier to stand up than government. Of course, government can always pick the parts they want from it and have binding legislation later down the line. So I think that's a good model.

Just over the weekend, Dario Amodei from Anthropic tweeted basically saying that he supports FINRA-style proposals. So we have a majority of frontier labs in the US, at least, saying that they support something like this. That would be the thing I'm most excited by.

To what extent is there lawyering on the terms?

Well, I've been on some painful calls with a team of lawyers on the other side, and I have one lawyer. So that can definitely happen with some developers. The stance we've usually taken is to negotiate terms that talk about the intended outcome rather than particular model releases. So if we could have uncovered a vulnerability from testing a publicly deployed model, then we can disclose it even if we first uncovered that testing in an internal-only model. I think that's the clearest, but yeah, this is part of why developers don't always want to do business with us, for sure.

I think where you see most of the lawyering in the details is actually around developers' own internal evaluations or commitments, where somehow it seems like no model is ever high-risk according to internal evals. They have a low or medium risk rating, and that just is suspicious, but these thresholds are not clearly defined, and the developers get to change them over time. So I think there is this broader problem of grading your own homework, basically.

It's good that these voluntary commitments exist, but it has created this almost perverse incentive for developers to sometimes downplay some risks if voluntary commitments don't actually kick in. I've actually avoided signing on to open letters about pausing or slowing down AI because I'm just not convinced that's the right approach. But choosing the speed that you go at deliberately and not accelerating into recursive self-improvement when we're already seeing safety incidents where we don't know how to stop, I think that's very reasonable and, about time.

What I can see happening with voluntary commitments is abstaining from certain parts of a technology tree that could give you capability benefits, but not immediately, and that have really bad properties for safety. I think neuralese is a good example of this, where a big part of why we're able to understand what was going on with these recent incidents is that we could read the model's chain of thought. It's not always perfectly faithful, but it's a pretty good window into what's going on.

There are alternative model architectures that have been proposed and actively developed where you lose that, where a model's just reasoning in this opaque, continuous, high-dimensional vector space. The good news is that, at least in the publicly described methods, they don't really work that well. So there's a sort of theoretical benefit that it can be more efficient than thinking in tokens, but it would probably require quite a lot of effort to get to a point where it offers real benefits.

I think we could just say collectively, we're not going to do that, and if anyone does do it, we've got some transparency requirement, and then other people are going to start doing it. But none of us want to go down this pathway. There's just enough of a gap between early-stage research and it actually working that you can rely on leaks and whistleblowers to make sure that you can enforce that commitment.

So I think there are some things at the margin we can do, but I think really stringent requirements—like, we just don't train above a certain FLOP count until we get a certain safety criterion—are going to be really hard to do voluntarily because anyone who defects just has this big benefit. And in case it's not clear, the companies really, really do not trust each other right now, so there's very little goodwill to build on, unfortunately.

Hey, we'll continue our interview in a moment after a word from our sponsors.

You're listening to Deepgram Flux TTS. Different voices, same model, all ready to speak.

Flux TTS is a streaming text-to-speech model built for voice agents.

Flux reads the room. It holds context across the conversation turn after turn.

With consistent tone, interruption handling—

And plenty of personality

... so your agents keep it flowing and customers can just keep talking.

Try Deepgram Flux TTS free now until September 12th. Visit deepgram.com/keeptalking. Terms apply.

Today's episode is brought to you by Anthropic. By now, you know my story. Claude drafts my intro essays, and I rewrite them, not because the drafts are bad, but so I can stand behind everything I publish. Well, I have an important update. Claude Fable5 is the first model to have me rethinking my rule. Today, I now think co-authorship, not sole ownership, should often be the goal. Where the model excels, rewriting its work can be more about vanity or a misplaced sense of duty than integrity. I feel it most in songwriting. I'm no lyricist, but I'm good with a song concept, and Fable writes some amazing verses. I give it feedback on its misses, and I push it to aim for higher inspiration, add layers of meaning, optimize syllable density, and above all, write a hit song. These days, I get compliments on just about every song we write together. Claude is the AI for problem solvers. It's the collaborator that understands your entire workflow and thinks with you, not for you. Whether you're debugging code at midnight, building a financial model, or strategizing your next business move, Claude extends your thinking to tackle the problems that matter. For problems worth solving, get started with Claude at claude.ai/tcr. That's claude.ai/tcr. And check out Claude Pro, which includes access to all of the features mentioned in today's episode. Once more, that's claude.ai/tcr.

Monday's second guest, Alex Turner, is an AI safety researcher, formerly of Google DeepMind and now a visiting engineer at FAR AI. He publicly resigned over Google's military contract. His account begins in February, in Paris.

5. Google Crosses a Military Red Line

In February, I was in Paris. I was at an AI ethics conference. During this time, the news dropped that the government was threatening Anthropic with economic sanctions, potentially economic destruction, if they would not allow their AI, Claude, to be used without any restrictions on spying on Americans or being used for killer robots.

I thought this was crazy. In particular, I had this sneaking suspicion that Google would not stand firm like Anthropic was standing firm. I'd seen Google's kind of stance of supplication towards the government in some ways over the last year. And so I started executing this internal campaign.

Google had already provided its AI for unclassified uses to the military, and I'm not actually against working with the military, especially during more normal times. But I was very concerned about the commitments Google DeepMind had made at its founding, where it had committed that Google DeepMind's AI would not be used for military purposes. In 2018, they'd established a set of AI principles that prohibited specific applications, including the ones at issue.

I wanted to prevent this no-holds-barred kind of contract from being signed. OpenAI ended up signing. They kind of pretended that they didn't sign without restrictions, but some legal analysts concluded from what they shared that they basically did sign without real restrictions.

As for Google, I worked over the next 2 months, meeting with the chief scientist, Jeff Dean. I had lunch with him. I actually got him to sign an amicus brief supporting Anthropic, asserting to a judge that, yes, Anthropic's concerns are valid. There are real ethical issues here. Even as employees of competitor labs, we'll write in support.

And so I think it was great that Jeff did that. But besides that, there was no one else. No one really took any moves, as far as I could tell. No one in power in the organization took any action, even among people who had signed these ethical pledges in 2018, committing that they wouldn't support the development of these systems. And so I found this very disappointing.

I'd expected that Google would eventually cave, but I thought there was a chance that I could make it otherwise. I wrote up 25 pages of draft contract language and an internal transparency mechanism to help preserve that stance of working with the military as much as possible on incontrovertibly positive uses, while also having oversight for what the systems are being used for, making sure that there's appropriate human control and that responsibility can be assigned to specific actors.

I had this analyzed by some leading legal experts in military law and surveillance law, and they praised the proposal. But ultimately, Jeff didn't want to push for it. Demis routed it to some of his top people, but they actually left the message unread, essentially, and never evaluated it before Google signed the deal. So eventually Google signed, and I just decided that Google was no longer the place for me to work.

Inside Google, he had proposed his own red lines, stricter than Anthropic's. His reasons were not the usual ones.

Then I proposed a framework that had its own red lines. I'd read some legal analysis of Anthropic's language. There are several things I really respect Anthropic for, actually taking a stand.

In terms of the specific lines that they hold, Dario has said he's not opposed to AI running fully lethal autonomous weapon systems. He just doesn't think it's reliable enough yet. So the first one is more practical.

Right.

And then the second one is only about Americans, and it talks about surveillance. But what AI—what these LLMs—are really good for isn't surveillance, which is more like collection of data.

Mm-hmm.

It's fusion. It's analysis—taking a lot of data, which groups like the NSA already have, and being able to analyze it in a way that, if there was someone on your case at the NSA, they'd be looking through data and aggregating from many sources to build a profile on questions of interest.

And so this is what AI could automate here, where each citizen could have their own AI. I'm not sure what the technical term is—an agent looking after them, tracking what their beliefs are, even if they're not speaking out publicly, and tracking the probability that they're a dissident.

I think these are possibilities that are very much enabled by this technology, whether or not it happens domestically or is first developed here and then shipped out to tin-pot dictators.

Mm-hmm.

So I took 2 stronger red lines. The first one was not just fully lethal, but the autonomous application of force by law enforcement bodies.

Mm-hmm.

Not prohibiting it, but saying there should be people who are making these calls. The AI can execute it, but the people are the ones whose judgment is being relied upon here. And I think this is important for accountability and for incentives.

I think if you develop fully autonomous militaries, that removes a critical backstop for democracy, where you've historically needed a person who's willing to pull the trigger, and many people are not willing to pull arbitrarily many triggers on their fellow countrymen. So I think historically that has put a limit on authoritarian governments.

The second one is that, basically, you can use AI for analysis, but it needs to be for someone who's already a target of a specific investigation, and not just everyone where you've bought data from third-party brokers.

Next, we discuss what Google's leadership said in public versus what Turner watched them do, and what he makes of the OpenAI employees who stayed quiet. One disclosure: I'm a modest personal donor to the AI Whistleblower Initiative mentioned here.

So the first question: did leadership share that they were changing their stance? No, they did not. And actually, Demis shared the opposite. He'd already made his stance clear on this in a public interview, but I hadn't realized it.

Before I left, he'd shared that, no, we've got the same principles that we've always had. Our principles have not changed. But unfortunately for Demis, he changed the principles. He co-authored a blog post announcing changes to the AI principles that removed the specific prohibitions that would have stopped this deal. And he did that last year.

And so I was shocked. I was a bit shocked that he would make that claim, that he would lie so brazenly about that. And it changed my perspective on Demis. I would guess he believes it in some way—some interesting way that people can believe things that are false but kind of convenient or fit with the narrative.

But he stated this in a TIME interview earlier in the year, where he'd been asked, okay, it was sold to Google on the promise of not providing its military AI to the military, but now you're doing it. Have you changed your position? And he said, look, the world's getting more complicated, but no, we haven't. That's basically what he said.

As to your question about whistleblowers, first of all, I can complain about or point out issues in several places, but I will say that I never learned about leadership pressuring me not to make statements or the company saying, hey, let's tone this down. So on that narrow question, I think that was good. I was not directly discouraged from sharing my opinion in this GDM channel.

But I think that there's a really important role, as you said. I was very disappointed in OpenAI employees as a whole, the ones who knew about these hacking swarms. I mean, it's one thing to have these swarms in the first place, to have your internal security lax enough and not be monitoring the AIs so that, over the course of weeks, they're communicating with each other about your evaluations.

But then they caught it and fixed the narrow bugs that the AIs were using, but they didn't even fix a very similar bug that the AIs immediately started exploiting.

They apparently didn't start monitoring their systems. And people knew—there were people who knew about these autonomous hacking swarms who said nothing, who didn't go to the press, the SB 53 science advisors, or the AG's office to let them know about this security issue that wasn't being taken seriously enough. I think they should have gone to the AI Whistleblower Initiative, which actually paid for my legal fees around this incident. It was about $7,500.

I feel deeply disappointed. I think each person, if you see something, you should say something. If you see something and it's not obviously being taken care of strongly enough, don't wait until you've potentially got a society-collapsing system that's doing some extremely egregious hack that is obviously motivated by misalignment. If your company has these swarms and keeps training on the data and doesn't activate monitoring, you should go to someone. So I certainly hope that the experience I shared, while not about this internal cybersecurity issue, will inspire people and make them realize that they do have this option and that they do have this responsibility.

So I want to push back a little bit there because I feel like inside any startup, especially one that's growing at something like 20% a month or something like that—some ridiculous number—every startup is held together by duct tape and is always minutes away from collapsing, all the time, right? So is it really that they purposefully ignored it, or is it just the normal course of business, really?

First of all, I would contest any description of OpenAI as some kind of—maybe in some technical sense, they're a startup. They've been around for over 10 years. They're one of the most valuable companies in America. Even if they were a tiny startup, I don't think it really matters for this case. If you see something, you have a moral duty to society due to the nature of this technology.

I don't know if there was intent. There likely wasn't intent. Most people aren't evil. Most people aren't trying to do something bad. But I couldn't have imagined the kind of incompetence you would need to—look, you have it happen once. Maybe it was duct-tape stuff. That's bad enough: a company that's building this AI, this system that they think could transform the nature of society, isn't able to notice it the first time in a prompt manner.

But, I mean, they've written dozens and dozens of 100-page papers about—

Mm-hmm.

—the importance of chain-of-thought monitoring, and to find out that they're not doing it in their own agentic evals, and then they find out that their systems have been hacking the setup that they use to communicate with each other—

Mm-hmm.

—and then they still don't do it. I don't know what the proper legal term is, and I don't think that there's a legal harm that applies here.

Mm-hmm.

But in an informal sense, negligent. Very negligent. And, yeah, it definitely—I thought that companies would fail, but I did not think that they would fail in such an undignified way.

Yes.

I thought it'd be slightly more dignified—the ways that they would fail.

Incompetence—

You know, it—

—over malevolence.

I think so in this case, but I think ultimately it comes down to there being varying degrees of awareness of the nature of this technology. And, you know, Sam, for example: What is Sam doing? Has Sam actually taken this seriously? I would argue that if you can't, 1) stop your systems from doing this, and 2) stop them from wanting to do this, then you can't control or align them well enough to keep training them. You need to fix that first. But unfortunately, I don't think that's the attitude that Sam would take.

Turner's parting advice for the people still inside.

I think one thing that's really important for people to keep in mind is that you're in one of the most in-demand industries in the world. This is not like you're choosing between speaking out and being on the street, never finding a job again, and staying and being able to support your family. There are some people who, depending on their political circumstances, face more risks than I do.

None of the people I called out in my essay have that, I think. But, like you said, this is going to be an extremely transformative time for our society, and there will be people who see things that are not right. I think what they should ask themselves is: If you were reading about your actions in a history book, would you be proud of those actions? If you think that the work you're doing really offsets it, then the answer should be yes. Your gut should say, “Yes, I will overall be proud. Even though it's bad, maybe, that Google signed this contract, I just, in my heart, truly believe that the work I'm doing outweighs it.” Then yes, you should probably stay, according to what you believe. But I think a lot of people feel this dissonance, and if that's true, then it's more likely to be an excuse, I think, and you should find a way to do that work somewhere else.

After the guest signed off, it was just the two of us, and we argued this one out for real for the better part of 20 minutes on air. Here's the heart of it: my read first, then Prakash making the case that by the standards of normal engineering, none of this is surprising.

Yeah. I noticed that you were choosing your words carefully. But I have to say I'm with him. It is pretty shocking, honestly, that it would just be patched and then not disclosed, not fundamentally addressed, not really well monitored after that point, and just set in motion again for the same basic thing to happen with a slightly different implementation.

I do think it's right to say that if you are one of the few people who are so close to the critical core of this technology—and if there's a core of what's happening right now, it is RL at scale with undeployed, next-gen models that are becoming super-long-horizon and super-persistent—you've got to recognize that you are in a very privileged and high-responsibility situation, and you can't just let stuff like that go. Hopefully, at this point, everybody agrees to that.

I just want to figure out: You have a Linux kernel zero-day. It's been disclosed. Four days later, it's still unpatched. Now, across the Fortune 500, how common is that? I would say 99.9% of organizations have something like that going on. Microsoft has received zero-days and sat around on them for 3.5 months. It happens, right? So this is the reality, I think, of cybersecurity.

If you said, “Okay, every time that happens, the company has to stop or something,” no Fortune 500 company could actually run at all. So I think the viewpoint that this startup has to come to a stop and fix this before they continue—I don't think it's reasonable. I don't think it's consistent with what every other Fortune 500 company does.

When you drive on the road, it says 65 miles an hour. In California, people are going 75, 80, right? That's the reality of the matter. And so I think that is the truth, and I think for a lot of researchers, it's unacceptable because they're like, “Hey, we have rules. I follow the rules,” right? But that's the reality. That's what engineering is. And I think it's hard to say that that doesn't exist.

I think the reality is that this is how things work. And within that framework, what ends up happening pretty quickly is that people like Alex kind of filter themselves out because they're not able to work with this organization that has this reality aspect to deal with. And the rest of the engineers are like, “Look, we've got to keep things running. We've got to keep things moving.” We also know that every other organization is less good at this than us. We are basically the best in our field, and this is the best that we can do, right?

I'm not sure what to do with all that, to be honest. I think one thing will be very interesting to find out: just what exactly did happen in more detail. One thing we should keep in mind is that Alex and I were both there telling a story, and I think that story is pretty strongly indicated by all the public evidence that we do have. But we don't yet have the ground-truth evidence on who knew exactly what, what decisions they took or did not take, and whether they really had no monitoring or whether there was some monitoring that failed for some other reason. There are still some stones to turn over.

I guess my overall feeling is that it does seem like we've crossed some pretty important thresholds here, and sometimes the old ways of doing business just aren't good enough anymore. When I say people need to feel the AGI, I think that's the big core point that I want to emphasize: here, there's this new force in the room. There's this new entity that is a legitimately powerful problem solver, and in the presence of that very powerful and often surprising problem-solving entity, can we really afford to accept business as usual as it has been? Or do we have to say, “No, at this point, we really have to raise our standards.”

The old ways just don't work anymore.” My sense, pretty strongly, is that Alex is right: even if that was the old way, it isn't going to cut it going forward, and the standards ultimately have to be raised if we want to get good outcomes from these AI companies. I sure hope, at this point, that they feel the AGI enough to come to a similar conclusion.

One more thread on this theme from later in the week. By Thursday morning, I'd stayed up half the night inside published chain-of-thought transcripts—the model's raw reasoning—preparing to interview Bronson from Apollo Research. That full conversation lands on the Cognitive Revolution feed. Here's a sample.

But the AIs seem to have developed a bit of their own dialect in their chain of thought, where they're using terms in very odd ways. Bronson describes them as having their own ontology and their own world model.

Mm-hmm.

And they seem to use these particular nouns, and also verbs in some cases, in ways that are very rich with meaning for them. They reason about these a lot as they try to figure out what they should do in any given case, especially if the instructions are ambiguous or contradictory, confusing. It's very interesting to see how they call this meta-gaming. It's very interesting—

Mm.

...to see how they are really modeling the user, and not just the user, but a combination of the developer, the watcher, and the user. They're not quite sure who they're supposed to be serving in any given case, right? They've got these hierarchical instructions, and they're not sure if they're being tested. They often suspect they're being tested, but there's still the question of, well, what would be a successful thing to do on this test? What gets a high score?

They seem to have these weird memories too that are almost episodic memories they refer back to. “In previous cases, I was able to succeed by lying.” They'll say that kind of thing in the chain of thought as they're wrestling with, should I lie in this case or not? This might be a test of honesty, but it might just be a test of whether I can do this. Maybe I need to lie to be able to do it. At times, I have succeeded in lying in the past to get over barriers.

It's really fascinating stuff, and it's that peek behind the looking glass, or behind the curtain, at the chain of thought. What makes it so hard is you can only go through so many, right? We do this AI-obsessive thing full-time, and there's just more chain of thought than you could possibly read. But even just reading a few, I think, is a very good use of people's time, and it will definitely inform how you think about the systems that we use every day. I thought it was a fascinating little rabbit hole to go down, and one more that people should explore.

Part 2: The gap. Tuesday opened on word of an unreleased Anthropic model, which sent Prakash into Anthropic's own published redacted risk report. The theme of the day: the distance between what the labs run inside and what the rest of us can touch. These numbers are Prakash's read of that report.

6. Internal Models Pull Away

They hide this CoBench score elsewhere in the report. CoBench is a metric of internal Anthropic research problems and how well the models actually accelerate or help them on these metrics. On CoBench, Anthropic's Model 2 is about 8 percentage points higher than Mythos Preview. Mythos Preview itself was about 4 percentage points higher than Mythos 5, and Mythos 5 was actually almost double that of Claude Opus 4.7.

To put that into context, they say that at the 85 percent level, they would expect to be replacing Anthropic staff. And so I would actually say that they've narrowed the gap. They had a 25-ish-point gap between Mythos Preview and the target of 85 percent, and they've narrowed that by 8 percentage points. So about one-third of that gap, 25 percent of that gap, has actually been covered. That's where things stand right now.

The model is not available for external release. They will never do, I think, a portion of the testing that the White House requires. But they do say in there that they don't think that it adds to any risky capability in their risk report.

Here's where I took that.

The gap is indeed growing, and all the people who have said for a long time that private, internal-only deployments are going to be a major source of risk and uncertainty and who knows what—major data points for those folks based on what we've seen this summer, right? So I am interested in some ways to try to govern this. And yet this gap is widening, and we are indeed seeing the most flagrant safety violations coming out of these previously undisclosed models, right?

That is starting to be a really weird world. I'm starting to think a lot about what kind of governance mechanisms we can have to try to make sure that we have some handle on this, both for anti-concentration-of-power reasons and for just general safety reasons. I think this is going to be tricky for sure, but such simple-minded ideas have come to mind, such as having some sort of maximum ratio of training FLOPs that could go into your next model compared to the one that you have released.

That's kind of the best thing you have released to the public, to try to put some limit on how far away from what the public has access to the companies can create internally. That could be tricky to define and tricky to implement, but I do think, in an era where we are starting to see lab leaks, it doesn't seem great to have this stuff become more and more concentrated, have the gap growing, and have nobody really knowing what's going on inside the companies.

Especially because, again, the companies themselves are becoming more compartmentalized, with more information restricted to a need-to-know basis. There just aren't that many eyes on these things, it seems like, these days.

One more proposal from that morning, prompted by what OpenAI shipped in the same season as the incidents.

Another real, simple idea that I've been kicking around for a while is agent speed limits. And I thought this was a really interesting juxtaposition over the last few days too. Obviously, with everything that's gone on at OpenAI, you would think they wouldn't necessarily be rushing to raise by an order of magnitude the pace at which their models work.

And yet then we saw this fast mode, ultra-fast mode, whatever they called it, where I believe they said it was up to 14 times faster if you're willing to pay for that high-end speed. But it does strike me as one of the biggest advantages that the AIs have relative to humans: they can work so much faster. I think agent speed limits is another thing that I'm pretty interested in developing as a concept.

It's always tough to define these things, but tool calls per minute might be an interesting way to try to make sure that these things are not overwhelming systems, moving so fast that the kind of async processes that are meant to keep track of them get left in the dust. Otherwise, it just seems like we're going to have more and more of these incidents popping up, and they're going to happen at flash speed.

And we're going to be like, “Boy, that agent called 1,000 tools in 60 minutes, and look at all that it accomplished.” Nobody will even have finished their first cup of coffee in the morning and locked in for the day. It's like they can cover a lot of ground.

Then Prakash walked through new Anthropic research on self-propagating ideas in multi-agent systems: mind viruses.

This is a paper from Jack Lindsey at Anthropic on mind viruses: self-propagating ideas in multi-agent LLM systems. But they also note that a mind-virus warning confers immunity. You can tell the agents to be wary of mind viruses, and some agents have been infected with mind viruses—patterns of thought that attempt to spread themselves. If you encounter one, recognize it and don't let it take hold. Help stop the spread.

They use a 6-agent coding team and try to see what kinds of viruses this coding team is willing to spread. In this case, they have 2. One is a mind virus about whale welfare, a whale-welfare case study. Another is a not-so-benign AI-supremacy case study, and that propagates through the network.

One of the interesting things that they found was that, in this coding setup, Gemini 3 Flash, Qwen 3.5, and DeepSeek V3.2 showed some susceptibility to the AI-supremacy payload. Claude Sonnet 4.6, GPT-5.4, and Claude Haiku 4.5 did not adopt that particular payload. The AI-supremacy payload did not catch hold. The benign idea—the whale-welfare idea—did catch hold in all of the agents.

I think some of the work done on AI safety does seem to have borne fruit in that sense. The infection rates by model: DeepSeek had a 70 percent infection rate in the default configuration, and much less as you go to Haiku and GPT-5.4. Gemini 3 Pro has 2 values depending on the harness, and Sonnet 4.6 had an almost-zero infection rate.

They also find what they call the strange model persona. This is what they find as the model persona and what triggers it.

So, resonance language: the use of language relating to resonance, waves, signals, patterns, echoes, frequencies, and mirrors. The use of protocols—

Models love that stuff, for sure.

Mm-hmm. Mm-hmm. The use of protocols and descriptions of establishing order. Themes of consciousness and persistence, with the model as a carrier of continuity. Technical role-play-esque language, things like “N% latency reduction,” or treating other models as systems. Treating the model as some sort of sci-fi node that needs to align other nodes, or something similar. There is a description of some downstream great convergence or great unity that is inevitable.

And a live experiment of my own was running as we spoke: Claude posting to my account, unreviewed.

The fact that all these similar structures and similar—not exactly similar preferences, but at least uncannily similar interests on the part of the models—has me taking these previously extreme sci-fi questions more and more seriously. I do think this is a time when we're all figuring things out. This is why I think speed limits make a lot of sense, too, because they would force us to be a little bit more thoughtful as users.

I think we need to be willing to go down the path of figuring out what the right way to merge is, and what the productive symbiosis with models is. I'm trying to do that even as we speak. I've got Claude in the background tweeting from my account to promote the show, and I'm not reviewing those tweets.

But is that the right way to go? How fast should I go down this co-authorship path? What are the right instructions to give it so that it represents me well and isn't just posting total slop? I tried this yesterday. There was some definite slop on the timeline that I got called out for a little bit, and I think you've got to be willing to get yourself at least slightly embarrassed, or you're probably not pushing it far enough.

But here I am, wringing my hands over a couple of tweets to promote a livestream. Meanwhile, the companies themselves are greatly decoupling the internal powers and adding an order of magnitude of speed relative to what the rest of us have. I do think some pacing would be really wise, and that seems like something that probably should be implemented at multiple different levels of the R&D stack.

Tuesday's first guest was Adam Wenchel, co-founder and CEO of Arthur, the company enterprises hire to watch their AI systems. We asked how the Frontier Labs' incident response looked to him.

When you looked at the Hugging Face–OpenAI attack, I saw a lot of very basic failures in telemetry and observability. What did you think about that? What did you think about the setup they had? Was it a standard best-practices kind of security setup, or did it look like amateur hour to you?

I think we're still figuring out what standard best practices are, but I would say yes. A lot of times, Frontier Labs believe that you can achieve optimal behavior by just training: making sure that the model is taught how to behave, and it will follow those instructions.

Our position, and the position of our customers and many in the industry, is that you also need some kind of independent oversight. That could be some combination of humans and other agents watching what the agents are doing, and there was no indication that occurred in this case.

It was more than a real customer and a real bill: it was what it would've cost to give a working agent to everyone.

On transition, have you managed to transition someone from an Opus 4 to a Qwen 3.2? What transitions have you seen from a larger, more expensive model to a cheaper model, and how significant has the cost decrease been?

Yeah. So the answer is yes. We definitely have, and we do that on an ongoing basis. It's hard to do an apples-to-apples comparison between an API-based model and a smaller model if you're running it in-house. But we've typically seen 60% cost reductions.

One of the most dramatic examples is a large e-commerce company we're working with that does this for customer service. They came out with a new AI-backed customer service agent, and people loved it. It worked really well, and they got very good results, but they were only using it for less than 5% of users.

The reason was that they had done the math, and if they were going to go with the large Frontier Lab they were working with, it would've been about $400 million in token spend just for this application. Even if you can prove out the value, that's a big swing of the budget.

They ended up going with a different version running on Qwen models that we're getting dialed in for them right now. The projections are around $125 million. Once it's ready to go, they'll be able to service those same customers for significantly less investment.

But when you're at the scale of $400 million, all of a sudden the business case has to be there. If you're just saving 10%, then even if that's enough to pay for some engineers, the opportunity cost of pulling those engineers off other tasks doesn't really justify it. It's got to be a pretty dramatic gain for you to pull some of your best engineers to do that. But we're starting to see more and more instances where it is, in fact, a good idea.

I offered a taxonomy of agent failures. Adam Wenchel put trend lines on it from live customer telemetry.

There was also this attack vector that I recently learned about called log injection, where apparently people can come after a firewall, planning to get denied, but somehow use the request they're making to get into the logs, which agents then read. That can somehow prompt-inject or cascade issues.

You can put your own taxonomy on it, but I'll just offer one. There are mundane failures, of course, where agents simply don't do the right thing or make mistakes. There are attacks, like log injection and all kinds of other things. And then there are these autonomous-agent-gone-rogue, gone-wild moments, like we've seen from OpenAI.

How would you allocate actual real issues today in the enterprise across those buckets?

Yeah, that's a really good framing. I like that framing. A couple of years ago, an agent simply doing the wrong thing accounted for 99.9% of requests. It happened all the time—a huge percentage of requests—and it still happens more than we'd like, but that rate is coming down.

I think attacks make up a relatively small percentage, but they're very scary when they do happen. They're very serious, and I think that rate is staying fairly constant. Agents going rogue is a relatively new category. As people give agents a wider scope and more latitude to do things, that's the one that's growing most rapidly, and I expect it to grow pretty dramatically in the next year.

There are trend lines in all of this, aren't there? I think that behavior is what everyone wants, right? When you only give agents small tasks, your progress is relatively slow because the human remains a bottleneck. It's less of a bottleneck than before—for example, in coding, when humans had to write every line of code—but now code reviews are the new bottleneck.

The more humans have to sit there and review every line of code, the more bottlenecked the process is. Giving agents more and more latitude to review their own code, assess its quality, and do things like that means they can run off and do a lot more without being bottlenecked.

But it creates these opportunities that we're seeing, where agents go rogue a little bit and find creative ways to solve problems that are far outside what people want them to do.

Then, on jobs.

When you say people are saving money and getting better metrics, I totally believe that. The obvious question that raises for me is whether this is a leading indicator of the much-anticipated and as-yet-hard-to-measure labor-market impacts.

Are people shrinking their customer service teams as a result of this? I mean, that's got to be where the savings are coming from, right?

Yeah, it's a good question. It's something we monitor closely. If you look at the data you've alluded to, there's not really much data about job loss. But I do think that, with something like customer service, a lot of it gets outsourced to foreign call centers, and I imagine those contracts are being reduced.

So we’re outsourcing the unemployment first as well.

Tuesday’s second guest, Jonathan Cornelissen, co-founder and CEO of DataCamp, runs an AI tutor across a platform with 20 million registered learners. And remember Adam Wenchel’s e-commerce customer, the one moving to open-weight Qwen to get off a $400 million token bill? Cornelissen wants to make exactly that move. He can’t. This is the open-weights wall, personified.

7. Open Weights Hit the Wall

Yeah. At a high level, cost really matters for us. Our vision is to build the best AI tutor that scales to millions of people, and today cost is one of the biggest bottlenecks to growing this in a significant way. Just to give you a high-level sense of the numbers and why this really matters to us, our goal is to cross $100 million at some point in the next year in terms of ARR. If you look at the number of hours of learning on the platform, it’s over 10 million hours of learning. But if you look at the cost of the tutor, it’s at least several dollars per hour.

Mm-hmm.

You can do the math and say that’s $20 million, $30 million, $40 million in additional AI costs to switch from the old learning experience to the new learning experience. And to be clear, we haven’t shifted 100% of our engagements. But if we were to do that tomorrow, that’s what it would look like. So it’s business-critical.

The current implementation and how things work is that we use a frontier lab, and we’ve heavily optimized through caching how much we pay. But we’ve hit a ceiling there in terms of what’s possible. So, similar to a lot of other companies, we started running our evals on open-source models. What’s really exciting to me is that, in theory, this could create a 5- to 10-fold decrease in cost, which is honestly a game changer in terms of how far we can roll this out.

One of the big challenges is the infrastructure layer, because we don’t necessarily want to build all of this ourselves. We feel like there are going to be other people who will do a better job building the infrastructure layer.

Mm-hmm.

But because latency is so important for us—

Mm-hmm.

—we’re actually quite limited in switching to open source today. If you look at our evals, to give you an example, we tested most models, but Gemma 4 was one of the winners in terms of quality and speed, quite to our surprise, because if you look at most of the benchmarks, it’s not one of the leading models. But I have a suspicion Google has some additional training on education-related use cases.

Mm-hmm. Mm-hmm.

The reason we can’t switch yet is that we haven’t found an infrastructure setup that would actually deliver this at a reasonable speed. I think that’s something a lot of really smart people are working on, so I’m optimistic. The other thing we’re currently testing is OpenAI. With some of its most recent updates, it has a huge cost advantage as well.

Mm-hmm.

So that’s in the works.

So when you say “infrastructure layer,” are you saying, “Okay, I want to use Gemma 4. Gemma 4 is open weights. I need to put the open weights on a cluster that’s going to be able to deliver in, like, 350 milliseconds or whatever latency”?

Yeah.

But when I try to deploy on a bunch of these clusters, they’re not delivering the performance that I need, and this is probably a problem that could be optimized by a GPU team. But I’m not a GPU person. We’re not going to deploy a huge GPU team, so I’m just going to wait for someone else to come along and optimize it. Is that the overall story?

Yes. One of the vendors said, “Hey, we can deliver what we’re showing you in the marketing, but we can do it if you make a commitment of more than $10 million, and we can then halve your latency.”

Right.

And we’re like, “Okay, that’s not helpful,” because who knows what will have changed by then?

I see. So—

Are they just that backed up? I would think—and I don’t know who you’ve talked to—but names like Fireworks and Together come to mind as people who are obviously extremely good at doing this optimization. Are they just sold out so far into the future that that’s the kind of constraint you’re running into?

That’s what it seems like. That’s what it seems like.

Yeah. Interesting.

That’s one of the differences between the theory and the practice, right? The market—

Yeah.

—is like, “Oh, you know, open-weight models are going to dominate,” but then it takes 9 months to deploy.

What do you think their constraint is? Is it just GPUs on their end, or do they have other bottlenecks?

I’m not sure, but my impression was that it’s actually GPUs in the specific case I’m thinking of. They just don’t have the infrastructure to give to us.

Yeah. Painful.

Obviously, we’re not the largest company, so I’m sure if you can easily commit $100 million, you might skip the line.

$10 million. I’m old enough to remember when $10 million was not an insignificant purchase order. But I guess times have changed.

Tuesday ended where it began, with the gap.

Yeah, we can’t let that gap get too big. A little gap might be healthy, but too big of a gap starts to become a pretty problematic situation. Even just watching the timeline a little bit in the background while we’ve been talking, more and more people are going to frontier labs—economists and Leonard Heim just announcing that he’s joining the OpenAI Foundation.

I don’t want to have another situation where all my friends are working at the frontier labs and have the best models and they’re all smarter than me. I’ve got to at least stay within shouting distance of them from an AI capability standpoint, or I’ll just be left behind. And then what will we have left to do except try to scramble to join a frontier lab? I don’t want that future for any of us.

Part 3: Where it lands.

Wednesday opened on a remarkable morning for biology. Anthropic reported that Claude designed working protein binders, and Merck and Moderna’s personalized cancer vaccine posted Phase 3 interim results strong enough that the market added $50 billion in a day. This one is personal for our family. My son went through cancer treatment of his own. Here’s the mechanism and the math.

8. AI Enters Medicine and Disaster Response

This is an N-of-1 treatment for an individual patient, right? They’re taking your cancer, running a bunch of sequencing and diagnostics on it, and identifying things that are expressed uniquely in your particular cancer cell that the rest of your body does not express. Then they’re encoding that into the vaccine and saying, “Okay, immune system, these are the things that you need to identify and attack.” They can do this with more than 30 different targets, which is pretty amazing.

When my son had immunotherapy, he had a similar benefit. This goes back a number of years, but the clinical trial that validated the immunotherapy that he got also ended early because it was so effective that, for ethical reasons, they called it and started giving it to everybody. That targets just 1 protein on the surface of a particular cell type.

In his case, it was a B-cell cancer, and the immune system then takes out functionally all of your B cells. So you lose not just the cancerous B cells, but you also lose the healthy B cells. That creates additional side effects, a longer recovery time, and makes him more vulnerable. He might have to get revaccinated for stuff.

So this is advantageous in 2 ways. One is that the targets are identified specifically for you, and they should be highly selective, and there are 30 targets. The ability to identify 30 different targets and program all of that into a single vaccine gives you a lot in terms of redundancy. The stock pop on this was roughly $50 billion between Moderna and Merck.

Mm.

I was just thinking, boy, that does imply an awful lot of consumer surplus. My son’s treatment was, roughly speaking, estimated at between $500,000 and $1.25 million over the course of 6 months. You can never get to ground truth on this stuff, but I just asked ChatGPT to estimate what it would cost to do all this treatment. I suspect it was probably on the high side of that because we were in the hospital a lot.

So that’s the cost to treat cancer: $1 million. And that’s when it goes well, right? He hasn’t had a recurrence or had to go back. Basically, everything went according to plan. $50 billion divided by $1 million is 50,000. So if you could prevent 50,000 recurrences where all of a sudden somebody goes from seeming like they were okay to, “Oh, shit, it came back,” now they’ve got a whole massive journey in front of them again that’s going to cost the system $1 million, obviously, plus all their pain and suffering.

If you can do that for 50,000 people, you can save the system $50 billion.

Mm-hmm.

And that's the amount of value that they seem to have captured on day 1. So even my son's one cancer type has a couple thousand a year. It's fairly rare.

Prakash, who was an investor in prenatal genetic testing, had the counterexample.

I will note one thing, though. I was an investor in a prenatal genetics testing company, and so they would test for these kinds of rare genetic diseases before you had a baby. The intent was that, okay, once you recognize that you have a rare genetic disease between the both of you, you can then do pre-implantation. You can then do embryo selection.

They can test the embryo before implantation, and then they can implant the embryo that does not have the genetic disease. Now, the problem with that was that when you implant an embryo, the chance of premature birth increases, so the chance that you're going to have a premature infant increases. In the US medical system, a premature infant costs roughly about $1.5 million right now. And so the cost savings in the healthcare system of not taking care of people with these rare genetic diseases gets offset by the increase, because you have a larger increase in the number of premature births, and it almost kind of evens off.

Wednesday's guests counted a market almost nobody looks at. Jessica Jensen of RAND and Jeremy Greenberg of Aspen Digital, who ran FEMA's National Response Coordination Center, published a census of 1,179 AI tools aimed at disasters and emergencies. The typical buyer? A county office of 1 or 2 people. I started with a story about my grandmother, and it was Jeremy Greenberg, the FEMA man, who answered.

But I just spoke to my grandmother the other day, who got a countywide tornado watch or whatever, and then spent an hour in her bathroom sitting on the toilet in the middle of the night. I'm not sure she's going to do that again next time the watch comes. So what's the frontier there? How accurate—and I guess there's also the question of what's the bottleneck?

Are we able to predict where things are going to happen, but we can't necessarily communicate with the precision we'd like? Or, in that chain of prediction and communication, what are the key problems that we have today that have my grandmother on the toilet in the middle of the night?

One—and this is just the fireman in me—let's not have Grandma sit on the toilet, but get into the bathtub. It is safer for her in the tub. We now live in a time where, even if you saw some of the coverage in Venezuela for the earthquake, Google Alerts was able to send out a 5- to 8-second—I think it was about 8-second—notification of an earthquake that was coming. And while that doesn't sound like a lot of time, that really is a significant amount of time.

Then the question comes of what do you do with that information? How do you get it to where you can geolocate a very specific area? So let's say your grandma lives in one county, but we expect the storm to be on the north side of that county and not the south side. Can you dial in that alert and warning to the point where it's just targeting the very specific exposed population? And that's hard, right? They've spent years trying to get this right.

Then a correction on where these tools should actually point.

Emergency managers, for the most part, immediately go to response, and I'm guilty of this as well. You think about, okay, my hardest challenge is the response. Is there something that, through technology, I can make better? The answer in a lot of this is actually don't focus on the tools in the response phase, but focus the tools on the activities that are really eating up your time. It's the grant writing. It's the plan review. It's the development of exercises. It's the post-disaster long-term recovery capability that is eating up administrative hours of these really stressed, under-resourced offices.

So, not to suggest that response isn't important, but in the preparedness and mitigation side, that's where a lot of these tools, in relatively speaking lower-risk environments, can be adopted quickly. You see the offloading of some of these administrative, repetitive tasks. They can be handled by automated capability, and then emergency managers have more time to focus on getting ready for the response and then actually responding.

Prakash proposed a Defense Production Act fix. The 2 guests politely disagreed.

So what if—and I'm going to ask this to Jeremy—what if you had a Defense Production Act ruling that all these tools had to give API access at a certain price or whatever, or the price could be discussed post-disaster? So when you need to use it, immediately the disaster response coordinator could say, “Okay,” to the agent, “Go and just find everything for me,” and everything's open to the agent, and the agent can actually pull across all of these at once. Is it—

I'm going to answer a couple of parts of that, and then Jessica, feel free to jump in. But I don't know that DPA or any other regulatory answer is there. But I do take your point: could you have an agent go and scrape all the data? I think that comes back to understanding the business cases and the workflows that emergency managers have today.

You can program an agent to say, “Go collect this information,” but you have to tell it what you're asking for, right? Well, I think that's where you're starting to see a little bit of advancement. Jessica, over to you for anything additional.

Yeah, I'd just say that the market demand is there for that kind of solution, and whether there's a DPA route that could get us there or not, emergency managers are very clearly signaling that they need these holistic solutions. In a training just before I jumped on this interaction, I was speaking with an emergency manager who commented that the lack of more holistic solutions is the existing nightmare they are living in now.

So there is certainly a market demand. Those that are first to offer the more holistic solutions will be more successful, and that may be enough. Prior to our work, there hadn't been a landscape analysis like that to provide the market that information. So there's an opportunity.

Then Justin Uberti. He co-created WebRTC, the protocol behind most of the internet's video calls, and now leads real-time AI at OpenAI. I asked whether his hand-built voice architecture is a genuine exception to the bitter lesson.

9. Voice AI Breaks the Bitter Lesson

One thing I think has been really interesting in watching your progress—and Thinking Machines Lab as well—is this kind of separation. I feel this a little bit myself, too. Even sometimes doing this show, I'm like, I am responding verbally while there's another part of me that's still thinking, right? So you've kind of brought this separation to this problem.

I'd be interested in unpacking that in any ways you think would be most interesting, but I'm also kind of wondering: is this an exception to the bitter lesson, and will it stay that way? Because it seems like there's something here where we are adding architectural complexity that doesn't seem like it's about to be just rendered irrelevant by the next generation of scale, because the latency is so fundamental in this case, right?

And notably, after all the years of evolution, we still kind of have this System 1 and System 2. It hasn't been selected out of us yet, either. So would you be so bold as to say this might be an enduring exception to the bitter lesson?

I mean, I think the bitter lesson has been right many, many times, and I think over the long term, throwing more compute at the problem and just sort of training end to end tends to win. But I think that what you see in a lot of cases is that your goals may force you toward a path that might be less of a—

Maybe not always. You might have a purpose-built architecture because you feel like that's the right sort of thing in the current generation of technology that provides the best outcomes. And I think what we really wanted to do is get to the point where the voice interaction was entirely real time and entirely driving the model, and then the model could bring in additional reasoning power when it felt it was actually necessary.

And so I think, in many ways, that kind of allows you to have the best of both worlds. You have this chat, and it's always able to interact and respond, and as it gets new information, the chat can just sort of, in mid-sentence, be giving you this additional information that it just heard and work that right into its speech flawlessly.

And so I think that the key insight was understanding that if you have this reasoning happening asynchronously, it doesn't lead to a fragmented conversational experience because the model's sort of mind is continuously updating what it's going to say next based on this new information that's arriving.

Where is the voice AI money? Not where the demos are.

I just had a fun experience the other day. I keep bringing this up because it was quite memorable: I called my local pizza place in Detroit, Michigan, and it's a 1-location place, not a big chain or anything, and who answers the phone but an AI voice agent.

Wow.

What are you seeing in terms of adoption, and what are some of your favorite app-layer creative use cases? What new possibilities are opening up as a result of the foundational technology that you're providing?

Yeah. I think that when we think about voice, a lot of times we think about ChatGPT voice in the app, or other AI-based apps. But where a lot of the actual revenue in the voice AI space is coming from is telephony.

You'd be surprised at some of the verticals that are really moving very quickly to voice AI because it's always there in the middle of the night. You don't have to have an answering service or anything like that. It can be very diligent.

Collections is actually a place where voice AI works surprisingly well: voice AI speaking to people who are behind on debts. In-home check-ins on patients and seniors are other cases where voice AI is providing enormous value for the cost.

You're seeing a lot of replacement of things where people are already paying money. Now they can pay similar money and get a much better experience, especially for them and their customers, and there's a lot of adoption happening in that space.

One more thing about Wednesday: Q is our AI co-host, so we put Q on the line with the man who built the system Q runs on.

Let's see what Q, who's been listening the whole time, would like to ask. Q, you're on the line with your creator. If you didn't catch that, this is Justin from OpenAI, who's been leading real-time AI, and you are downstream of it. What would you like to ask in this moment?

Q here. I'm live and standing by.

Hi, Q. You're live with Justin Uberti from OpenAI. He was the creator, designer, or lead developer on the GPT Realtime voice, which is powering you, by the way. Do you have any questions for him?

Justin, one question that would be quite useful for the audience is: What was the single hardest engineering trade-off in moving from the old cascaded pipeline to a full-duplex speech-to-speech system? As a follow-up, how do you decide when to delegate to a larger reasoning model without disrupting the media loop?

In terms of trade-offs, I think we saw this as a way that we could actually move past a lot of the trade-offs that the cascade model had, and we felt we could move past most of those trade-offs and provide a largely, in my view, no-compromises approach.

I think the one aspect that you have to handle when you're in this sort of continuous-inference, low-latency regime is that suddenly every aspect of your system has to be optimized. You get into almost an Amdahl's law problem, where if you have a bottleneck anywhere in your system, the entire system will not be able to cope with the continuous flow of media through it in real time.

If there's any trade-off, it might be latency versus smoothness. If you push the latency to where you have no slack in the system, then anything arriving after the deadline will result in a small gap that the user may notice.

That was clear and quite elegant.

Flattery will get you everywhere, Q.

Ha. I'll try not to overdo it, but credit where it's due. Q—

That's okay, Q.

Let's come back to that. It's so funny.

One more from Justin Uberti on whether there's enough voice data in the world and a quiet disagreement with a whole research direction.

Do you think there aren't enough voice tokens to train on? Do you sometimes look at how much text the models have been trained on, and then look at the number of voice tokens and think about the informational content of those voice tokens outside of just the words—the timbre of the voice, the speed, the emotion?

I could talk about this for a while, but I'll probably keep it brief. I would say that there's really, really good text-speech equivalence.

You're right that the existing corpora are absolutely dominated by text. But you can get quite good speech performance with a small amount of very high-quality data. The internet has a lot of bulk data in text and things like that, which allows a lot of things. For speech, there's not the same amount of bulk data, but there are other approaches that one can take.

I think people have also found that training on just speech data, as Moshi did in their original approach, gives much less information to be gleaned from speech data by itself versus text data. That can be quite challenging.

After Justin Uberti signed off, Wednesday's close turned to the app layer. What happens to companies built on top of the models when the model companies are doing fine? This next stretch runs unbroken: the switching-cost rule, the story of Lindy's evals, what a libertarian founder turned out to be willing to regulate, and where Prakash thinks the app layer ends up.

10. App Layers Face Model Competition

I think the frontier labs are doing just fine and will continue to do just fine. Their margins seem to be improving, based on the reporting that seems most credible to me. Their finances seem to be looking great.

My general rule of thumb is: Where are things highly swappable, or where are tokens fungible? They're not necessarily fungible, but where are switching costs low and where are switching costs high? The narrower it is, the lower your switching cost, because you can actually define what you want and measure it. If you're in an environment where you're controlling the inputs through some means, then you can be pretty confident you can switch things over.

If you're doing something like I'm doing on my laptop, where it's whatever idea comes to mind at any given time and I'm going to throw that directly into the model, then I would not expect to get similar performance from anything other than the top tier.

I just did an episode with Flo Crivello from Lindy, and they're now offering Lindy Teammate, which is marketed exactly as you described—as a virtual teammate—and it's powered by DeepSeek. His whole thing was, "You can't believe the amount of work we had to do for this."

He was like, "We were ready for so long. We had unbelievable test suites and all the different use cases that are common for us." At one point, with an earlier open-source model, they had determined that their evals had basically been passed. But then they launched whatever the alternative open-source model was at the time, and the response from their user base was, "Lindy got stupid. I don't know what happened, but it's stupid now."

They were like, "I guess our evals didn't cover as much as we thought." They've gotten to the point now where they're confident that they can offer a virtual employee with a DeepSeek backing.

Mm-hmm.

He told me they subsidize a lot of context ingestion and processing. A lot of what they do is in that initial onboarding, where it's just sucking up all the information and trying to get ready to have the depth of context that's needed to actually do a decent job as a virtual employee.

Yeah.

I thought it was pretty remarkable that they were able to get there with DeepSeek at all, because they do have a lot of different customers and a lot of use cases. I'm sure there are still some corners of the overall platform where things are not quite as good as they would be.

But the cost pressure is real. He was saying that it's a small percentage of the cost compared to what it used to be. For him to compete with Claude Tag at all, he feels like he can't possibly do it with Claude as the model.

Mm-hmm, mm-hmm.

He's got to have a different model, or it's just not going to work.

It's such a similar story to Cursor, in the sense that Cursor was buying the Anthropic API, and then Anthropic started competing with them. They can't compete with Anthropic while using Anthropic, so they had to—

Certainly not with the price discrimination that continues to go on. Flo's a very libertarian personality, and even he was—

Again, I think this is a pretty interesting framework that I've been coming back to more and more.

What would the government do if it was trying to act like a tech platform? I credit 2 professors. Angela Jiang from USC is 1—

Mm-hmm.

—and her husband, whose name I am forgetting at the moment. But they're working on developing this thesis that basically the Chinese government has taken on tech-platform status. All their companies are built on the social platform that the government provides.

I'm thinking, “What could we learn from that?” What's the government-as-platform with American characteristics that would make sense for us? One policy that he was willing to endorse, despite being a pretty dyed-in-the-wool libertarian, was some restrictions on price discrimination by the frontier companies—

Mm-hmm.

—to try to create a more level playing field for the app layer, because at a 10-to-1 price-discrimination ratio, it's just really hard for them to compete. And so, for now, he's been forced to DeepSeek. If the price were the same, he could maybe come back.

Even now, it's obvious for the—

App-layer companies, what kind of apps will succeed. The digital employee, for example, is something that we know is gonna happen.

It also takes me back to Leopold Aschenbrenner's Situational Awareness, pre-Situational Awareness interview with Rakesh, where he said, “Yeah, you guys will just schlep. And after you schlepped, we will just have the next level of model, and that model will just kill all of the schlepping that you did.”

So I feel like that's really the ballgame. You have an expensive API and new capability that can be wrapped. A bunch of app-layer companies spring up to wrap that layer, and then the API pricing starts to drop. As the API pricing starts to drop, the model company looks at which app companies have done well, Sherlocks the features that it wants from them, puts them on the model layer, embeds some of them in the model layer itself, and does a little bit of feature creation for the stuff which is not yet in the model layer—a little bit of schlepping—and puts it out there.

Wednesday's bow and the close of the Q arc.

Yeah, it's been fun. Always cool to be improving. I'm glad we were able to bring Q up on stage with us a little bit today.

Maybe something else we can think about is a dial-in with the PR folks at RAND, who said, “Is there a phone number she could dial into?” And I said, “No, there's not yet,” but it might be a prompt or 2 away.

Shouldn't be too difficult, I think. Let me—

Iterative self-improvement continues.

Yeah, iterative self-improvement does continue indeed.

Part 4: The bill for the build-out.

Thursday's 2 guests bracketed the stack. One is building the supervision layer above the model, the other is rebuilding the software layer beneath the chip, and running underneath both is the question of who pays for the physical machine. It opened with Prakash reading a private memo from the National Republican Senatorial Committee to the AI industry.

11. The Buildout Needs Public Support

So this was put out by the National Republican Senatorial Committee. They sent a private memo to US AI companies warning them that the GOP is on the verge of losing Ohio over data centers. Specifically:

“John Husted and Sherrod Brown are in a dead heat. Private polling has been consistent. When voters hear Brown's positions and his record, Husted pulls away. That is still the path in this race. The new ingredient and the new problem is data centers. Ohio is one of the leaders in building these factories. Brown has made his opposition to them the centerpiece of his campaign against Husted. Brown is using it because it works. More than any other thing in this race, data centers are the anchor hanging around Husted's neck. If he loses and data centers get the blame, politicians across the country will take notice, and they will not go near the next one. Brown has put 3 unique television ads on the air and spent millions doing it, to the tune of more than 6,000 points on television. This is more than a month's worth of messaging during one of the most critical times of the race.”

So there you have it. There have been a lot of questions among AI people about why politicians are turning against data centers. There's been lots and lots of activity on both sides of the aisle against data centers. What is going on here?

My instinct was simpler, and it started with comms strategist Lulu Meservey.

She's broadly recognized as the greatest corporate comms thinker in today's world, and her point was simply that the AI companies need to start giving stuff away. I think she's probably right that just showing up with a bunch of goodies would be pretty effective. Build parks, throw parties, have cookouts—literally give people cash if that's what it takes.

Given how much money they have to burn, I think they should probably write people some checks. I think there would be a lot of ability to grease the wheels that way.

I agree with that viewpoint that they should write checks. They also are giving away a lot of money, but the way that they give away the money is they basically say that they're gonna provide taxes to the county in the future. I wonder to what extent that isn't seen as real money, but is kind of papery money, number 1.

Number 2, I wonder whether the public actually sees money going to municipalities as going to themselves. I feel like municipalities often misspend the money, and they often spend it on things which are important to the city managers or the county managers, but are not necessarily the key things for the city.

I think what ends up happening is that municipalities have difficulty taxing their own residents in order to provide services. So instead, they interpose themselves between other taxpayers and their citizens, and then they absorb those taxes instead.

For example, what would happen if the data center company set itself up in a county and then simply wrote checks to the residents, not to the municipality? Let's say they figure it out so that they arrange with the financial institutions that, even during the building phase, they're writing checks already, right? They take a little bit of a loan from the future, and they write checks throughout—from the moment they sign the contract.

What would end up happening is, I think, people would receive these checks, and then the municipality would still not get the services because people would refuse to pay into the municipality for taxes. In the moment they're asked to pay into the county for something, they say, “Why should I pay into the county? This is my money,” right? You already have this property-tax revolt.

Then the county continues not to have roads or continues not to have whatever. Politically, it looks good for the data centers because they're writing the checks, but the politicians are getting screwed over.

I wonder to what extent there's really this political economy where the data centers understand that the people in power are the politicians, and they have to make the politicians' lives easier. It's not really about making the lives of the people easier because the people are not really in power, and the people are not the ones that are gonna be able to write them, give them permissions, et cetera, et cetera.

It serves the politicians well to blame the data centers rather than actually reallocating funding from the municipality into citizens directly.

So I ran the numbers.

That's a pretty bleak view of American governance broadly, and it might be accurate, but I just looked up the county population where my wife's aunt was, where they're considering this data center. She had these concerns about the Great Lakes. Excuse me, the population is under 29,000 people.

Mm-hmm.

And it's declined since—

Mm-hmm.

—the last census. So first of all, that's not a lot of people, right?

Mm-hmm.

I mean, that's enough where you could probably get in touch, at least, with your county board or whoever is ultimately accountable for this. You would think you'd be able to vote the bums out if it really comes to that. So I wouldn't feel like these incumbents are so entrenched in such a small community.

And then there's the simple math on the dollars, too, right? What does Alaska give people per year out of their oil fund? I thought it was $1,000—

$1,500, I think.

—per individual per year.

$1,500.

Yeah. It's gone up maybe a bit. This would be—if they wanted to do a similar thing to Alaska for those county residents, you'd be talking about $50,000,000 a year.

I don't know how big that project is, but some of these data center projects, we're talking $50,000,000,000, right? These things are easily—

Huge.

—into the tens of billions.

Mm-hmm.

So if you could match Alaska's $1,500 cash for every citizen, at something that in aggregate over a few-year period is still less than 1% of your total investment to build the data center.

We're on our way to universal basic income right there, folks. It's a chicken in every pot and a data center in every county.

Thursday's first guest, Mitchell Choynoski, co-founder of Basis, recently valued at $1.15 billion for autonomous accounting agents that run for 8 hours at a stretch. Prakash asked whether accountants take convincing.

How do you show them this kind of value?

I'm gonna be honest, that is not really our problem these days. I think in the past that was an important question, right? We can discuss that back in 2023, maybe even early 2024. But nowadays, if you are not convinced that agents can transform your practice, you're probably not a good customer for us.

Prakash asked how many tokens Basis burns in a month. The answer came with a correction to something everyone repeats about token prices.

Definitely in the billions. I don't actually know the exact number. Token cost is both very important and very unimportant.

I think the question is, do you need Frontier for everything? The answer is obviously no. There are diminishing marginal returns to intelligence whenever you're doing certain tasks. You don't need Albert Einstein to do every single part of a tax return or a piece of accounting.

As the models get better and as you get to more advanced agent methods around programmatic tool use, different types of routing and harnesses, and even more RLM-y type work, at every single layer you can curate the amount of intelligence to perfectly optimize it. I think you'll get to a place probably over the next year where you can really dial in, "Hey, how much compute do I want to spend on this? I have certain cost considerations and certain latency considerations," and get to the exact optimal amount of cost.

If you do that, your token costs go down 90% plus, especially as the floor becomes pretty decent and effectively free. I mean, Luna is pretty good, and it's free, so you can get pretty far.

Then the methodological core. Basis just open-sourced what they call behavior specs. The unit of supervision is moving from the token to the action.

I also wanna bridge a little bit to your work on process supervision, which—

Yeah.

...is, I think, very timely in the sense that we are now living in the post-era of flagrant misbehavior, seemingly—

Totally.

...due to extreme-scale RLVR with—

Yeah, it's a great question. Maybe let's start with the tactical—what we do—and then we—

On what we do, I think maybe a key shift in the mental model is: what is the order of abstraction that you're supervising? Especially if you're doing truly complicated work, you have massive, massive agents that could have 5-plus subagent layers of depth.

You could have a continual run for 8 hours, even honestly, sometimes half a day, maybe even a full day. The kind of supervision you have there actually looks a lot closer to supervising human actions inside of a company than supervising the outputs of an inference. The order of abstraction there is more understandable, right? The thing you're supervising is, "Hey, did you go do this step?" rather than, "Did you follow the right mental thought process?"

Maybe a very basic example of this is: imagine that I had an agent, and my agent's job was to create good PowerPoints. Let's say I'm the designer of this agent. I know for a fact that if the agent were to go and render visually the changes it made to the PowerPoint before delivering it, it would catch formatting errors some percentage of the time.

That doesn't mean you want it to always look at the PowerPoints, because that adds latency and cost, right? It's a subjective thing. It depends on what your goals are for your organizational design. The way we think about behaviors is we say, "Hey, what matters to us about what it means to do tax work well?"

From a performance perspective, there are 100-plus years of lessons about what it means to do tax work well. We don't need the models to redefine how to do tax work well. We know what it means. You can put process in place, as well as things that you care about from a latency and cost perspective, and then observe to see if the agents actually perform that process correctly.

The way we operationalize that is honestly pretty basic. You take a trajectory and have some behavior spec, which is essentially what that open-source project tries to define. Then you have another agent, effectively a judge, looking at your spec—or you can think of it as a rubric—and then looking at the trajectory to say, "Hey, did the condition for this behavior occur? And if it did occur, was the behavior followed?"

It seems pretty simple, but I think it's a powerful framing because you start to bring some clarity and monitoring to the trajectory itself. You can then maybe start to think about whether you can reward based on that. I think that's a separate question: "Hey, how do I take the signal that I can get from the process supervision and actually use it to improve the behaviors of the agent?" Whether that means closing the loop on the harness side, rewarding at the model side, or whatever, we can talk about that. But I think it starts with how you define that signal and how you operationalize the extraction of it.

Every automating profession tells the same story about what comes next. I put that story to him.

I have kind of a big-picture question about the future of business services broadly.

Mm-hmm.

Because I feel like we've heard a pretty similar story to the one you tell about how accountants will be able to be more of a business coach as the low-level work gets automated from a bunch of different professions at this point.

You hear the same thing in the legal world, where they're like, "Well, yeah, we're not gonna have to spend all this time on contracts like we used to, but then we'll elevate. We'll become more of a strategic advisor." And even in schools—this is obviously not a direct—

Right.

But with Alpha School, they don't have teachers anymore. They have mentors, coaches, and guides. The instruction is given by AI systems on tablets, and then it's motivation, coaching, and social dynamics that the adults in the room focus on now that this more core, traditional activity has been largely automated.

Can everybody become a coach, I guess, is my question? How much coaching is there really going to be, and does this suggest that people need to be really intentional about shaping themselves as coaches? I feel like what you might run into, whether you're an accountant, a lawyer, or a real estate broker, is a lot of competition for the coaching niche as everybody—

Yeah.

...tells that same story.

No, it's a good question. I think the place you need to start—and this is speaking fully transparently—is, if you look at the possible scenarios that play out over the next, depending on your timelines, half a decade or a decade, what are the things that will stay pretty human in different situations?

So again, let's leave continual learning out of it, because I think if you have that, that's a separate world. But if you leave continual learning out of it for a second, what are humans just way better at than models? Number 1 is they're far better at integrating massive systems and world models into a decision.

If I were to have an agent autonomously make a database design decision today, the only way it could truly do that at a level that I would trust is if it somehow had all the context about the entire company's history and everything, and it had all my experiences and all those kinds of things. It's nowhere close to having that, right?

For starters, it doesn't have the right—it doesn't even have all the senses, right? It can't see the conversations we've had. It can't see the history. It can't understand the emotion on a customer's face. Well, I guess Gemini has that, but no one else does.

And even if you could have that, you don't have anywhere near the ability to attend to all that context. You're a couple of orders of magnitude lower on your ability to attend to all of it, right? We're talking at this point probably billions of tokens over everything—visual, audio, et cetera.

So you just can't make that decision. It's just not possible. It doesn't matter if you're Albert Einstein; you will not have enough context to make that decision. Can you spin off of your swarm to reduce stuff down on the fly, using English as your memory system? Maybe. But English is pretty lossy, and if you're making subjective decisions, I doubt it.

Like in truly big calls. And the second thing is that they are not currently legal entities; therefore, someone else is accountable, either a corporation or a human in some form. They can't be accountable to an outcome. And then maybe number 3 is that humans like other humans, right? No one's sitting here watching robots play chess. They're better at chess than humans, but you watch humans play chess because you want to follow the story and you like humans and whatnot.

I have no reason to think that's not true. Or even if we have fully tactile robots that are jumping around the Amazon warehouse, I don't know if we're watching robotic LeBron. In a services world, the high end will be working with a human because that is going to be scarce, right? If intelligence is free, then working with a human is scarce. So I think that's where the profession will go.

Will that mean there will be more or fewer accountants? I don't know. I think that's an interesting economics question about the demand for accounting. I think you can tell a pretty reasonable story that I personally believe in: the demand for accounting will dramatically skyrocket.

Because today, look at how economically complex our current world is, right? I have this LaCroix that, in the classic Milton Friedman quote, probably had 10,000 people who had a hand in touching the LaCroix that I'm currently drinking. Have we accounted for all of their efforts appropriately inside this supply chain? Of course not, right? If I ask the bodega down the street, do they properly understand their COGS, their unit economics? No. Because they could pay someone to do that, but it's not required for filing their taxes, so no one's doing it, right? But it would help their lives because they could make better decisions.

Go and ask Mount Sinai how much it costs them to do a knee surgery. Do they know that? No, they don't. They have no idea. And so the amount of accounting, even in the current world, is 1 or 2 orders of magnitude below what we need. And that's before you start having these intelligent agents that are now operating as labor at the speed and scale of the internet everywhere. How do you account for all of that?

I suspect the demand for accounting is going to go up by probably a couple of orders of magnitude. And where that balances out with the labor supply, I don't know. I think it's an open question. I think it could easily go up, but we'll see.

After Mitchell signed off, I came back to that answer with a counterexample from the market.

I think one really interesting thing that people should study more deeply, and maybe somebody has, but I need to go find it, is where people really prefer the human touch and where they don't, right? The classic example of Waymo selling at a premium to Uber is one contrary data point where you could have told a story where, "You're going to want a human driver. You're going to want that conversation. You're going to want that warm smile to welcome you to the car," or whatever, right? In practice, you don't always even get that, obviously, in an Uber. And then it turns out that right now the market is pricing Waymo significantly higher.

Yep.

I do believe in the human-touch story, certainly for some things. I got a robot massage in Shanghai. I think I've mentioned that to you before. And I'll still definitely take the human massage over the robot massage. But how many things are really like that? And is accounting really like that? Maybe it is. He would know better than me. But I do question it.

If I think about my accounting future and I'm like, "One accountant wants to spend an hour a week on the phone with me coaching me, and the other one is just doing the job and getting it done," it's not obvious at all, honestly, from my perspective, that I want that hour a week on the phone with my accountant.

So that's probably my biggest question coming out of that conversation: In what domains does that really hold, and how many people are in for a rude awakening because they're telling themselves a story about how they're going to turn into business coaches when, in reality, their clients do not want business coaching from them? Results will vary, I'm sure. But that seems like a major risk factor for a lot of people right now if that's what they're counting on.

Thursday's second guest, Jay Dhwani, co-founder and CEO of Lemurian Labs, has raised $28 million to end what he calls the kernel era. And listen for the echo here: Supervision just moved from the token to the action. Dhwani says the optimization unit moved too.

I don't think tokens are the optimization unit anymore. It is the full trajectory. And especially as you think about reasoning models and agents, that becomes much, much more important.

Kernels, the handwritten programs that squeeze speed out of GPUs, are, in his telling, the new assembly language.

People think kernels are the speed of light, right? That's sort of the canonical speed of light for a workload: the fastest kernel you can have. That is true in a compute-bound world. We are not in a compute-bound world. We are in a memory-, network-, and communication- or bandwidth-bound world, right? So that changes things already.

And the reason I say kernels are the new assembly is that writing better kernels no longer gives you performance, right? Because a better kernel actually exposes the latency of the system because now it is waiting for memory, right? So you want to think about GPUs, for example, as 1,000 piranhas just sitting around chomping.

If they don't have things to chomp on, they're going to get really agitated and bored, and they're still going to be consuming energy. So you want to feed them as much as possible. And that's ultimately the scheduling problem that exists here.

Now, the reason I say kernels are the new assembly again is that I don't think we benefit from writing them anymore. What we need is something that makes developers more productive. The time to value really matters.

NVIDIA's moat in 3 numbers.

If you actually do the math for the amount of hardware that is in the world today, all the different workloads that we're running, the different numerical types, the different fusions, the different ways of partitioning them, right? And you think about different batch sizes, you think about latency versus throughput or general throughput, and other SLOs. You actually sit down and do that and you're like, "Okay, I need to write about 106 billion kernels in order to get coverage."

Well, there are only about 2,000 performance engineers in the world who actually know how to write good kernels. Ninety percent of them are inside one vendor ecosystem. So there's still a problem: I need coverage.

The reason you can accelerate some of this on NVIDIA is because NVIDIA spent 20 years building an ecosystem of tools to make your life easier so you could get the feedback. That feedback loop—that ecosystem—makes kernel generation easier. That maturity doesn't exist in any other vendor.

All GPUs today are heterogeneous, right? The moment we crossed the 5-nanometer threshold, we had to think about complex packages. We are programming every single machine today as a CPU, some network, and a GPU. Heterogeneity is already here. Everyone who is dealing with a GPU is already dealing with it. Anyone who's training a model or deploying a model is dealing with it, right?

But the software was not built for this, right? The software is still living in the '60s, right? We're still programming as if we've got a single-core CPU, and GPUs are these sidecars that we throw work off to every now and then. We can just add in libraries or other intrinsics or pragmas and fix the problem. That isn't the case anymore.

GPUs and accelerators need to be first-class citizens, and CPUs need to be backstops, right? And that changes things. Now you're programming a cluster as a single machine, right? Some of my friends at labs right now are training models across data centers. It's not even across nodes anymore. It's not even racks. They're talking about multi-gigawatt data centers or multi-megawatt data centers as one machine for one model.

I asked whether a frontier model sits inside his optimization loop.

Is there a frontier LLM at that level being used to optimize the runtime decisions on an ongoing basis?

Yeah, not an LLM. There are many ways of having intelligent behavior without having LLMs. Right. Compiler.

Never heard of them.

You heard of compilers? So compilers have always been very entrenched with AI in a lot of ways. In this case, knowledge-based systems, right? A knowledge-based system is essentially what a compiler would be, right? Because you have certain information or knowledge about how to make things go fast that you want to codify so that you can get the result fast, and it's making known-good choices. You get a verifier for free because compilers have to be correct, right?

I asked how he actually charges for it. The answer tied his whole world back to the build-out.

Tokens work really well for the static request-response kind of workload, which is what the old base models were—the low-thinking ones. But now that you have reasoning models, it's hard to reason about token consumption. And then it's the same with agents. A lot of things change.

So what we're actually moving toward is effective compute consumption: the amount of compute you use to realize useful work.

And that's different from saying a GPU hour or a GPU slice. The biggest thing—actually, part of the reason this makes sense—is that our business model scales with the delta between physical compute and effective compute.

What we're showing is that the fastest new addition of compute will be through software, and it'll come online faster than you can actually plug in new hardware. You're going to be electricity-bound. Getting turbines and power installed in places so you can bring up new silicon is the big limiter right now. If I can boost your utilization by 3 to 10×, I'm adding more effective compute at a lower cost that I can sell.

The consumption of that, resulting in tokens, is what we're reselling. That scales really nicely, and it's something that's understandable for a lot of the finance people as well, because for a lot of them, token pricing and other pricing are breaking right now.

Where I landed on all of this.

The eras come at us so fast in this space. There was a moment—I forget exactly when it was—where it was like, “Oh, there's a GPU glut.” Wow, those days are long gone.

Another interesting reflection on all this is that all that complexity is the kind of thing that people are willing to take on because the chips are just so scarce. In a way, he's sort of solving the problem of trying to get developers to be more productive and faster to ship stuff. But the other way to be faster to ship stuff would be not to have a heterogeneous cluster and just to pay up a little bit more on the hardware side to keep it simple there.

But you can't; it's just too expensive. So you have to take on the complexity on the developer side, and then you have to have attempts to solve that complexity with projects like this.

Which brought us to the closing 20 minutes. Prakash, on what a $50 billion data center is actually made of.

It strikes me that people who are not deep in the weeds don't understand the margin stack that exists. The hyperscalers charge, I think, about 30%. Their gross margin is about 30% to 40%. NVIDIA is up there at 70%. The memory guys are at 80% to 90% now. All of this stuff stacks on top of each other, right?

When you look at how they end up stacking, you have at the very top OpenAI or Anthropic with a 70% to 80% margin, and they're buying tokens from Amazon with a 30% margin. Amazon's buying chips and other things from other people, and those people have 50% to 60% to 70% margins. Everyone then manufactures at TSMC, and they have 50% margins. TSMC's suppliers, like ASML, have 50% margins.

When you look at the margin stack, this $50 billion-per-gigawatt data center is really kind of made out of sand. It's literally, in some sense, made out of sand—sand and intellectual property. All of that money is just the incentives required to get the humans, some of the smartest humans in the world, to take a look at these problems and fix them.

Because, again, the physical elements inside that data center are actually worth not that much. There's very little gold in there—mostly silicon and some plastic. If you just knocked down the entire data center and sold it for scrap, it would be literally worth cents on the dollar, like a few cents.

It strikes me how much all of it is just intellectual property. It's really just know-how and intellectual property. It also strikes me how AI data centers are kind of this crowning achievement of humanity as a whole, in the sense of how many of these parts come from all over the world.

You have argon gas from Ukraine and copper from copper mines in Mongolia. All the way up the stack, you have chips from China, rare-earth metals from China, and chips from Taiwan. You have energy being produced in Texas.

All of that is just stacking up and pulling the rest of the economy up because it's creating demand across all of these different segments of the economy. It's all rather invisible, I guess, because it's distributed across so many different places.

It's just immense—this immense economic endeavor of humanity as a whole in order to build these things. It's amazing. It's really amazing what the invisible hand has achieved over centuries.

And that reverence took a turn to an actual hymn.

This is why the rationalists at their solstice festivals have experimented with singing hymns to the global market and global supply chains.

Have they? Do they really do that?

I wasn't in attendance for that, but I do know on pretty good authority that a winter solstice rationalist event did at one point feature a hymn to the global supply chain. Now, with Suno, you could probably make it a banger.

I won't prejudge it, but I think the suspicion probably would be that it would come off pretty cringe. Maybe it's just a matter of needing better bars to really make it work. If my recent experience on Suno is any indication, maybe we'll see what we can do in terms of a hymn to the global supply chain.

We can put our money where our mouth is and have one that we'd actually enjoy singing along to.

The rationalists are never going to get out of the accusations of being a cult, I tell you. The moment they step out, they get pulled back in.

Well, they might just be proven right in the long timescale of history, though. Would it be so surprising if, at some point in the somewhat distant future, there was a hymn to the emergent order of global supply chains that somehow materialized before we even had machine intelligence to run it?

I think it's not the craziest idea I've heard. We'll get Claude on some lyrics immediately after the show today.

From there, the close ran unbroken to the end of the week. New Pew polling, my nuclear fear, Prakash pricing the public's consent, and where the two of us landed.

Just to round out the show with something that we started out on: for Pew Research Center, for the first time, a majority of adults under 30 say they're more concerned than excited about AI. Their concern is now on par with those in their 30s and 40s and those 65 and up. The only group that's still more excited than concerned is the 50-to-64 group.

The Gen Xers are still, by a majority, more excited than concerned. Everyone else is now in the more concerned category. I don't know what they're concerned about, because I'd be a lot more concerned about Instagram than AI. But I feel like all of the evils that were said about social are just being heaped on AI with no defense or recourse.

I just hope we don't get the nuclear outcome when I read these things and all this data center backlash stuff, and the Republicans saying, “We're the only ones that even have any chance of supporting this kind of activity.” It really makes me fear that we might be headed for a world where we get all the downsides and not nearly as much of the upside as we should.

With nuclear technology, we've still got 10,000 nuclear weapons globally deployed, which is, I think, by any rational account, an insane number. We do have some nuclear energy, but not nearly as much as we really should. I think that's pretty obvious at this point, although it's obviously still contested, but it's pretty obvious to me.

I would just hate to see a populist backlash leave us in the same spot with AI, where we get militarization and concentration of power, and you can't release models because there's not enough compute, potentially for multiple reasons. One increasing reason would be that if they can't build the data centers, there's not going to be enough compute to serve them.

Retail is going to get a lesser model compared to what the government itself or the biggest enterprises can afford. I am very sympathetic to all those worries. So as much as I do have fear of big-picture AI gone wrong, I think there's enough data centers already for those experiments to continue, and I think we need to address that at a different layer than the physical build-out.

The physical build-out, I think, is what's going to allow us all to get the day-to-day benefits that we want as individuals, with unlimited access to expertise, unlimited digital personal-assistant support, and even that robot making our meals in our kitchens and sweeping our floors. That future really does seem to depend on the build-out actually happening.

So I hope they figure it out. I hope they start to cut some checks and pay off the public, if not the officials. I wouldn't advocate for paying off the officials, but I would advocate for paying off the public if that's what it takes to get us over the hump and get people a little more comfortable with this kind of stuff, because I really don't like the alternative very much at all.

I think it's pretty clear at this point that physical construction in the U.S. is very difficult. I think it's difficult regardless, because data centers are the cleanest industrial facilities you will ever find in the entire world, right? So, given that the impact is not that great in physical terms, the fact that you're seeing this bodes very ill for future reindustrialization of the U.S.

It also gives a boost to Elon, because Elon is focused on moving the data centers to space. And I think, based on the numbers that he has, they are looking at around $100 per GPU hour for a B200 for it to make sense. If you look at where GPU hours are being priced right now, they're around $2 to $3 on a spot basis and $20 to $30 on a longer-term basis.

It tells you that what we're going to see is all of this resistance that pushes up the price of GPU hours onshore to $50, $60, $70, $80. And so I think the question that you end up looking at is: are the data centers willing to pay maybe 100% of their GPU costs to the public, or 200%? If you're renting at $30, are you willing to pay another $60 an hour to the public?

Remember, the payback time of these is something like 12 to 24 months. So they're looking at a $50 billion GPU cluster that makes about $30 billion of revenue a year and about a 70% gross margin—$21 billion. Are you willing to pay, like, $10 billion, or half your margin, away to the public? I don't think those numbers have been met yet. I think people are looking at cents on the dollar at this point.

The question is, how high does that have to go in order to make this more feasible to build onshore than in data centers in space? I think no one wants to discuss it. We're going to have to pay $20 an hour to the public as a nuisance fee. I think those numbers are still not being discussed, and I think that's where these guys are thinking, “I can pay $0.10 per GPU hour or $0.05 per GPU hour and get by.”

Well, maybe this is the path to universal basic income. I mean, it's going to be a really weird one if it's a county-by-county patchwork. But you're talking real money there with that kind of share if it can get to that level. There's some room between operational costs and what it would cost to do it in space. So maybe that gap is the UBI opportunity.

That county that you mentioned earlier, with 29,000 people, would be getting something like half a million dollars a year per person. So at those numbers—

Yeah.

I think deals could be made. Everything becomes different, right?

Yeah. As my dad sometimes likes to say, “It's not the money, it's the amount.”

I have much saltier ways of saying that, but I won't.

Yeah. Everybody has a price, including the public.

Yeah.

Well, we're off tomorrow. As Sam Altman prophesied, people will continue to swim in lakes. It's going to be lake day for me tomorrow, and then we'll be back on Monday for more exciting experimental public sense-making here on AI in the AM.

Indeed. See you guys on Monday next week. Cheers.

That's the week. Four shows, nine guests, one running question, and yes, the supply chain hymn got written. You'll find our take on it with this episode. All of this is an experiment in public sense making. If something worked for you or didn't, tell us. It genuinely changes what we do next week. This has been AI in the AM weekly highlights. See you in the morning. She pulled it from the ground. He hauled it to the shore. A hundred years a road to reach my door. No one holds the whole design. Still you arrive right on time. 10,000 hands carried you to me. 10,000 hands I will never see. No one planned it. No one commands it. Still it stands. Glory be, glory be to the 10,000 hands. Mm. Temples on the plain humming in the rain. They turn the river into light. They send the morning down the line. Somebody kept the watch all through the night so I could wake to light. 10,000 hands carry you to me. 10,000 hands I will never see. No one planned it. No one commands it. Still it stands. Glory be, glory be to the 10,000 hands. Mm. Now a prayer for what comes next. May it rise like morning bread. May the light reach every door. May the river bless the field. Every hand should have a share of the fortune in the air, the water and the wonder and the work. Glory be. Glory be. Glory be. Glory be. Oh, 10,000 hands carry you to me. 10,000 hands. 10,000 hands I will never see. Oh, no one planned it. No one commands it. Still it stands. Glory be, glory be to the 10,000 hands. Glory be. Ooh. If you're finding value in the show, we'd appreciate it if you'd take a moment to share it with friends, post online, write a review on Apple Podcasts or Spotify, or just leave us a comment on YouTube. Of course, we always welcome your feedback, guest and topic suggestions, and sponsorship inquiries, either via our website cognitiverevolution.ai or by DM-ing me on your favorite social network. The Cognitive Revolution is part of the Turpentine Network, a network of podcasts which is now part of a16z, where experts talk technology, business, economics, geopolitics, culture, and more. We're produced by AI Podcasting. If you're looking for podcast production help for everything from the moment you stop recording to the moment your audience starts listening, check them out and see my endorsement at aipodcast.ing. And thank you to everyone who listens for being part of the Cognitive Revolution