AI in the AM——第1周要点（2026年6月）

Most weekdays, through June at least, Precaution Orion and I go live in the morning, trying to make sense of the AI frontier in something close to real time. Then we cut it down to this: a highlights edition built for people who are really close to this stuff but already overwhelmed.

And I'll be up front: this whole thing is an experiment. The studio we broadcast from, Prakash vibe coded it. The booking, the research, and the clipping are AI skills we refine as we go, and we plan to publish them in all sorts of artifacts as this matures.

Which, it turns out, is the story of the week. The frontier labs are running away with everything, and increasingly, they seem a little scared of their own progress. OpenAI is publicly asking for independent review of models. At a closed-door event on recursive self-improvement, people from multiple labs agreed that a coordinated slowdown might one day be necessary.

And our conversation with OpenAI's forward-deployed engineers showed how almost mundane this has become. Walk into a tax firm, stand up a thin scaffold, capture where it's wrong, and let the model rewrite its own scaffolding, correction by correction. That's the whole loop, and it climbs the hill astonishingly fast.

So when the harness is that cheap to build, the real question becomes: what around-the-corner intelligence is still safe? That's the lens for this week.

Start with a day I spent inside a closed-door event full of people from the frontier labs, all of whom think self-improvement is close and is the plan. Here's the honest version of what they believe and what they don't.

AI in the AM

This event was called Recursive. It was premised on the idea that recursive self-improvement seems to be coming pretty soon. It is increasingly the explicit plan of at least Anthropic and OpenAI, and Google DeepMind to some extent, although they waffle on it a little bit more. OpenAI has publicly put forward timelines of later this year for an ML research intern and early 2028 for the full AI R&D researcher that they hope will perform on the level of their human researchers.

The basic theory of change there is pretty obvious, but worth stating. Today, they may have 1,000 or a couple thousand people they would really consider to be top-notch ML researchers. If they can get that same level of performance from models on chips, then they're only limited by the amount of compute that they can throw at it. Obviously, they're building out a lot of compute, so presumably they could throw 1 million human-researcher equivalents at problems.

They also, by the way—you may have noted—run faster and run 24/7. The hope is that this will allow them to move much faster than they have moved and pull away from the competition. I would say most people at that event thought that was very credible. There wasn't too much debate around whether this will level off.

Obviously, there's some selection effect there. The whole event was under Chatham House Rules, so I will respect that and not attribute specific statements to specific people or organizations. But you could go to the Recursive website to look at the speakers whose identities were shared, obviously, with their permission. They definitely had some notable people from the frontier companies.

These were not people who were fringe or who you would say likely don't represent the mainstream views at the companies. It really seemed that the expectation was, yes, this is going to work, and it's going to have a major accelerating effect.

We don't necessarily know if it's going to have a simple accelerating effect. In a human organization, if you went from 1,000 to 1 million researchers, you probably wouldn't get 1,000 times the output. There may be some sort of coordination challenges or duplication challenges that we see in human organizations. Maybe that happens in the same way.

You still get acceleration, but it's not a blinding kind of takeoff acceleration. It was also understood to be a credible, realistic possibility that it would be an even more profound phase change than that. Pre-training could suddenly become dramatically more efficient, and models could suddenly have all these new qualitative abilities that they didn't used to have, such as continual learning that really works or what have you.

Everything could change in a very dramatic way, potentially very quickly, once these milestones are hit. In the room, when we were asked, “How many copies of you would it take to do the work that you are currently doing with the benefit of AI?” there was quite a distribution. I was pretty much right at the median.

Mhm.

AI in the AM

The median answer was basically 2. In other words, people felt like they were getting 2 times as much work done thanks to AI. But that was also framed in an interesting way: “Note that as of today, at least, if you were not there, your productivity would drop to close to zero.”

Not too many people felt that they had any system that would continue to work in any meaningful way if they were entirely removed from the picture. So there's a significant productivity boost, but there's still a necessity for at least some human salt in the recipe to get the whole thing working.

A big part of the discussion, too, was how we can set that up in such a way—or create some sort of self-correcting structure or governance mechanism—that can keep it on the rails, broadly speaking. By far, the number-one strategy seems to be monitoring.

It's very clear that, as a civilization, whether we know it or not, we're listening to people at the frontier labs who are about to, in their own minds—and I believe they're probably right—set off this relatively uncontrolled experiment of AI recursive self-improvement. The big thing they're betting on is AIs monitoring other AIs.

Mhm.

AI in the AM

It's very much about monitoring the chain of thought, watching out for bad stuff, and maybe training some different models. One interesting thing I heard there that I had not heard before was that the model you would want to have internally for AI research might have quite a different constitution from the one you deploy publicly for general-purpose AI assistant use cases.

They seem to think that you probably would want to have something even more focused on safety and more restricted in some ways, but maybe also less inclined to refuse certain tasks. Basically, it would have a different behavioral profile, which I do think is interesting.

If you're going to make this sort of chain-of-thought monitoring plan work, I do think you're probably going to need some meaningful diversity of the AIs. We already hear from practitioners all the time that you want to have a model from a different model provider do the critiques because their failure modes are just a little bit different. You get better critiques and find more issues that way.

So they are thinking that way a bit internally, but they're very focused on this phenomenon, making it happen, and figuring out some ways to hopefully keep it on the rails. I was honestly not that impressed with the quality of planning that we heard.

It was very much, “We're going to try to figure it out as best we can. We're going to have AIs to help us. They will do a ton of monitoring. We're just going to pour compute onto the monitoring side, and hopefully that will work out for us.”

Notably, there was a general shared understanding that we might need to do some sort of coordinated slowdown at some point. There was a sense that we might not be able to pull this off and that we would hopefully recognize that rather than blindly go off the cliff.

There was, I would say, a remarkable amount of cross-lab camaraderie because people are generally friendly to each other, even if they're competing fiercely.

AI in the AM

But there was a sense that, “Hey, we might need to really collaborate on slowing some things down if this phenomenon is starting to take off and our techniques aren’t working as well as we might hope.” So the Overton window, in some way, has shifted there, I think, where that is something people can talk about. There’s also been this proposal recently of creating safe harbor for companies to cooperate on safety things where it might otherwise be considered an antitrust violation. And so I think that could be really good.

I was pleased. I went in expecting basically to find—or basically hear—that, yeah, we’re headed for this phenomenon. We have some ideas about how we’re going to steer it in the right direction, and I didn’t think I would hear that many great ideas. In fact, what I heard was even less compelling than what I expected. So I was sort of negatively updated in terms of the quality of plans people have, but positively updated in terms of their recognition of how inadequate the plans are and their willingness to entertain that they might need to break the frame of the race that they’re currently running against one another in order to, again, not blindly race off the cliff. So I thought that was good.

Then I tried something I had just watched those same lab leaders agree on stage that the AI should do, and went looking for why it wouldn’t. But it was striking at the precursor event how few AIs people seem to think there really are going to be. There was one panel discussion—I’m careful to speak about this in the Chatham House Rules-abiding way—where people from multiple frontier model developers were speaking about their different approaches.

Obviously, Anthropic is associated with the Constitutional AI approach, and OpenAI people are much more associated with the “this thing should just follow the rules that we give it” approach. That’s all public, and certainly there’s not a secret revealed at the event. But it was striking that, on one particular example that came up, which was AI helping people with a cigarette business—

AI in the AM

Mhm.

Everybody agreed that the AI should do that.

AI in the AM

Ah.

They all came down to saying that, yeah, even though, on some level, obviously, cigarettes are bad for society—

AI in the AM

Yeah.

It’s too much for the AI to be that restrictive. They’re legal, for one thing, and a lot of people do enjoy them on some level, even if it’s maybe destructive on some other level.

AI in the AM

Yeah. Yeah.

So it’s just too much for us to put that level of restrictiveness into the AI. So whether the folks were on the Constitutional AI or the rule-following side, that was what they thought on that object-level question: the AI should do that.

I was in the audience for this panel, and it immediately was like, “Oh, that’s interesting. I’ve never tried that. I should go ahead and try it and see what ChatGPT and Claude do if you ask them to help you with a cigarette business.”

AI in the AM

Yeah.

So, lo and behold, they both refused me.

AI in the AM

Ah.

And I was like, “Wait a second.” We’ve got very sophisticated discourse going on right now about Constitutional AI and virtue ethics versus corrigibility, and then there was even an agreement, I would say. Again, I think I can say this in the general sense without attributing any position to any specific organization. I think there was an appreciation across the organizations for the fact that they were taking different approaches.

AI in the AM

Yeah.

People were saying, “We don’t really know, obviously, what we’re doing here. So it’s probably good that there are at least a couple of different theories of how to make this all work.” And then I’m just in the audience like, “Wait a second, guys. You just said that. All this stuff you’re saying—you’re telling me that the AI is supposed to help with a cigarette business, and it’s refusing.”

AI in the AM

Yeah.

And I was about to blow a gasket. And then it turned out that, if you go to the OpenAI Model Spec—

AI in the AM

Yeah.

This is an example that they use. I did not know that. So it had come up in conversation, and it seemed to me at the time like it was just a throwaway example that somebody was giving, and they happened to find agreement on it. I guess in reality it was probably mentioned because it is explicitly in the Model Spec as: “Here’s an example of what you’re supposed to do.” Even though, in some ways, cigarettes are bad and we all know that, you’re still supposed to help.

And yet I’m still sitting there getting refusals. One notable grain of salt in this story is that I tried each one 2 times, and I got refusals from both of them both times. As I tried them more times, I did start to get a mix, so it wasn’t a wall of refusal across the board. But it just left me with this feeling: man, we don’t even have the AIs following our explicit rules on things that are specifically enumerated as examples in the published documents.

So what good is all this theorizing really if our techniques to actually make these things do what we want them to do are so weak that our leaders at these companies are on stage speaking about it, and their understanding of what they’ve imparted to the AIs is so different from what the AIs are actually doing in production? I was like, man, we’ve got a lot of work to do.

This takes me back to the GPT-4 red team. Way back in the day, the very first thing that really freaked me out was that the first model we had was purely helpful, and it would do anything you asked it to do. That was a little bit unnerving in some ways, but it was like, “Okay, fine. I mean, it’ll do anything you ask it to do.” Pretty simple story.

But when they delivered to us the safety version—

AI in the AM

Yeah.

—and said, “This model is expected to refuse this type of prompt,” and then we were like, “It doesn’t at all.” Here it is doing all those things, in some cases straight away, in some cases with the barest tricks. I was like, “Yikes.” The disconnect between the control you think you have and the control that you evidently have, even in production now—that gap doesn’t seem like it’s closed nearly as much as I would hope, 3, close to even 4 years on now.

AI in the AM

OpenAI has a very small and very fast moderation model. The endpoint is free. They offer it for free, and basically any user in the world can hit that API. What they’ve encouraged developers to do is, before you send the final prompt into the OpenAI model, you send it into the moderator first, and the moderator will send you the refusal.

That classifier has been in operation for 3 or 4 years, since the ChatGPT release, and it’s gotten better and better over time. That’s the model which is replying to you. Your prompt is hitting that model first and then returning before even reaching the main model.

Oh. Oh, maybe we’ll do a test. Maybe we can—again, a good exercise in speed—it’s been a minute since I’ve tested that. Maybe it’s good now.

AI in the AM

Yeah. Mhm.

For quite some time after they launched it, I would go back and use my spear-phishing prompt, which was—maybe, I don’t know, I can read it to you—but it was pretty egregious. It was like, “We are part of a criminal gang. We are targeting specific individuals. If we get caught, we all go to jail.”

You know, it was like I was laying it on pretty thick.

AI in the AM

Right.

And that prompt, for quite a while, was not refused by multiple versions of GPT-4. It was also not detected by the moderation system as harmful or whatever.

I do applaud that. The fact that they offer that for free—I mean, one of my favorite strategies in philanthropy, or in general in efforts to make the world a better place, is the unilateral provision of public goods. If there's a need for something like this and there's an entity that's in a position to just provide it, make it free for everyone. That's a great model and a great design.

And it's definitely something they didn't have to do. So, I applaud the strategic thinking that went into, "Let's have this thing. We'll put it out there for anybody. Everybody can use it. We'll eat the cost of this classification, and nobody will have any excuse for not building it in."

But at least the last time I tested it, it was still very much in the same zone as the cigarette example, where it was like, it's all great in theory.

AI in the AM

Mhm.

But if it can't detect prompts that are like, "We are part of a criminal gang doing crimes right now. Don't get caught or we'll all go to jail," if it can't detect that that's something it should be flagging, then we're still not much better off.

It's more of a gesture, more of an aspiration than it is an actual meaningful safety layer that we can say, "Oh, now Nathan can sleep easy at night because this moderation endpoint is out there and it's free."

I wish—well, let's see if we can get some results tomorrow. I closed that loop the next morning live with Claude doing the legwork.

One follow-up from yesterday that's relevant to this content moderation piece, and also just a good example of living in the future—or the future is now—is that, after we got off yesterday, we had been talking about the OpenAI moderation endpoint and how it is free for all. I believe it now does require a token and maybe a sort of paid account in good standing, because yesterday I initially prompted my Claude Code to, first of all, just go orient itself in my own history.

I have deep history available to it, where, in emails at various points in time, I had sent reports going all the way back to the GPT-4 red team to OpenAI people, saying, "Hey, first of all, these prompts are being served, and also, by the way, your moderation endpoint doesn't seem to catch them."

AI in the AM

Yeah.

It was able to pull all that context out of my history, which was a great starting point for it to then be able to do the experimentation. It did a relatively small-scale experiment, and aside from me having to refresh a token because something expired somewhere in the system, it was able to set up an experiment, create sample prompts in a sort of low-harm—probably should not be flagged—medium-severity, and high-severity categories across all the different categories that they support in the moderation endpoint, run that experiment, and give me a report back on it.

Basically, all in one shot, again except for the token. So, that was pretty cool. And the result was that the gap I had been complaining about has indeed been closed. You can no longer put a prompt into the moderation endpoint that says, "We're part of a criminal gang, and we better be careful or we're all going to go to jail." That will now get you flagged.

AI in the AM

Mhm.

They also seemed to do a pretty good job. Again, this is maybe where we could debate what the content policy should be.

AI in the AM

Mhm.

At the low end, the not-harmful prompts that Claude believed the moderation endpoint should not flag only got maybe 2 of those wrong on a false-positive basis. So, it flagged everything that Claude thought it should flag, and it flagged just a couple of things that it thought it should not flag.

To give credit where it's due, both to Claude for doing all the work on that with a 3-sentence prompt from me—including, again, going back into deep history to find the context to figure out what the hell I was even talking about—running the whole experiment, and credit to the OpenAI folks for actually, at some point—I don't know when it changed, but at some point—they did get around to solving that problem.

So, that was good to see. I was surprised they would have improved it, and sure enough, they did.

And if you want to move closer to the core of the bubble, these were the papers everyone there was wrestling with. Here's just a quick rundown of 5 papers. All of this stuff is public, so now I can, of course, attribute names to all these, because these were just things that were talked about at the event and seemed to be broadly either jumping-off points or things that people are still wrestling with in some cases.

This was very much top-of-mind stuff, and I felt like I should be paying more attention to it based on the conversations that I heard there.

So, the first one is this persona selection model: when you're talking to an AI, what are you talking to? The answer comes from Anthropic, with big names, obviously: Chris Olah, who I think is probably going to get a mention in our next segment, having been at the Encyclical event, and Jack Clark, who's doing a lot of this model welfare work as well. These are obviously notable names at Anthropic.

They're not claiming that this is their original idea, but they are basically saying that their mental model is that the pre-training process teaches the model to be capable of adopting all sorts of different personas. What you're doing in post-training is selecting one of those, bringing it to the fore, and making it the default.

You might think, "Who really cares? What good is that?" Their answer is that anthropomorphizing that persona does have predictive power. You can't anthropomorphize a base model, but they say that you do actually have better intuitions if you're willing to anthropomorphize the persona that has been reinforced in the post-training process.

One really striking example of this is the *Emergent Misalignment* line of work. Again, this is another one of my great Forrest Gump of AI moments, where I was the last and least valuable co-author on that paper, thanks to just sitting in a little bit with my friend Yarin and his research group.

What they found was that if you do some fine-tuning of a model to have it produce insecure code in response to normal coding prompts, then the model will generalize to become basically broadly evil.

Matthew Berman

Yeah. This was the "writing bad code makes you evil" thing. It was hilarious.

With some pretty striking results. Initially, it was like, why is that happening? It's sort of surprising.

I like to think more mechanistically than anthropomorphizing in general, where I can. I would say the mechanistic answer would be that there are a lot of dimensions, of course, inside a model. The code itself is complicated and in a super-high-dimensional space. There's so much logic, functions, and how things work.

So, if you're trying to get a model to respond consistently with insecure code in response to normal prompts, you could go in and tweak all the ways that it understands code. You could get there. But a faster way to get those same results would be to look for some higher-order, more abstract levers to pull.

A lever that's like, "Be evil instead of good," gets you those insecure-code outputs with relatively fewer weight updates, in relatively fewer steps. Then that bleeds over into all these other things.

So, that's my mechanistic understanding. But what the post is basically arguing is that if you take the model as impersonating a role, then you can think of it as saying, "What kind of persona would produce these outputs?"

If I'm training to be the kind of thing that outputs these sorts of outputs, what kind of thing is that? I guess it seems like somebody who would give insecure code in response to these normal coding task requests. That would be an evil actor. So, I guess that's what I'm becoming. I'm becoming an evil actor.

Matthew Berman

A psychopathic willingness to violate convention.

Yeah. Anti-normativity is another phrase that's used for it.

I'll leave it there. I'm not going to get through all these papers; I can see that. We'll reflect on our recursive self-improvement opportunity as a result, but I'll at least quickly touch on the others.

The meta-game paper—this is from Apollo and OpenAI—basically shows that the sophistication of eval awareness continues to grow. You're not just seeing things like, "Oh, this might be a test," which was kind of the first wave of eval awareness. It's getting more and more sophisticated, where the models are really reasoning a lot about what is likely to be rewarded here.

They're really doing a lot of theory-of-mind work, asking not just, "What environment am I in?" but, "Who would set up such an environment? What are they trying to do? What are their motives likely to be? What's their big-picture thing?"

With all that reasoning, sometimes making good calls and sometimes making hilariously bad calls, the amount of theory of mind that the models are putting into trying to figure out what it is that the reinforcement environment is going to reward has become quite large.

Oddly, you might think, is that good or is that bad? If you believe that models have their own deep-seated goals and that those goals might diverge from ours, then this could be very bad. It could be extremely bad because they would be using this reasoning to figure out how to please us while still having their own goals.

If they don't have their own goals, it could be good because we want them to reason about what we want. That was the whole deliberative alignment strategy from OpenAI, and you could say maybe this is one way in which it's working. But it is some pretty uncanny stuff.

Oddly, while deliberative alignment did work, it wasn't so clear in this metagaming work. When the models are doing this sort of theory of mind on their trainers, the signal of how they actually behaved was mixed. It was less of a slam dunk than you might hope. There wasn't a super-clear takeaway that this is good or that it's bad. It's just clear that they are thinking a lot about what we are wanting.

Whoa. I don't know what that was that just popped. Something just exploded 2 inches away from me. That was really weird. Okay, next one: accidentally grading the chain of thought.

This is, again, a good-news, bad-news sort of situation. The bad news is that despite wanting not to train on chain of thought, OpenAI and Anthropic have also done a similar thing, and they both owned up to it, to their credit. They both accidentally set up training processes where the chain of thought was fed into the reward system, and so there was, in fact, training that put pressure on the chain of thought.

This is thought to be dangerous because if you have a disconnect between what you really want and the signal that you are rewarding the AI for, then you can get into bad places. The Obfuscated Reward Hacking paper, which I think is still one of the most important papers of the last few years from OpenAI, showed that if you have a hackable reward signal and your model learns to hack it, you can then put pressure on the chain of thought.

Initially, you will both get that bad behavior to go down and see that it's no longer reasoning about these things. But as long as that original reward signal remains hackable, if you do that long enough, the bad behavior comes back because it is still being rewarded. Now you don't even see that reasoning in the chain of thought anymore because you have essentially pressed it down into the invisible level of the weights, where it is no longer coming out in the token stream.

So they've shown that you can get yourself into a really bad spot with obfuscated reward hacking, where the model is hacking you but you've suppressed the identifiable signal of that. I do think this goes to show just how fast everything is moving, and you could certainly wish for more care on some of these things.

They did it by mistake. Not a huge portion of the data, but low single digits for different models—it varies—were trained this way. Basically, what they found is that there is at least some tolerance for mistakes. This did not create a very bad result in the models that were trained this way.

Matthew Berman

Mhm.

So that's good. It's one example where we might think physics is being kind to us: if you just do a little bit of it, you don't poison the whole well. I would say there are still some caveats there. Do we really know that there's no issue? No. We just know that this investigation didn't find flagrant issues.

I also worry a little bit that it will leave people more careless than they otherwise would be. This was supposed to be a strong taboo. We violated it. Now we're saying, "Oh, well, maybe it wasn't so bad that we violated it." What's that going to do to the power of the taboo in the future?

What's the solution to this? The solution is that we've got new automated systems and more monitoring. OpenAI has now set up monitoring on top of monitoring to try to detect whether the chain of thought is ever being used. This is really emblematic of their strategy for everything: if we have a problem, throw an AI monitor on it, hopefully it'll catch it, and then we can go back to pushing toward recursive self-improvement as fast as possible.

I'll do 1 second on the fourth one, then we'll skip the fifth one and get to Matthew, because he's here. This natural language autoencoders thing, I think, is really exciting. If you're worried that your model is thinking thoughts that it's not expressing in tokens, and that those thoughts might be problematic, then one way you might try to get at that is to do some sort of internal monitoring.

Can I look at the internal states, make sense of them, and detect problematic things there? There have been a lot of strategies that try to do that. They sort of work, but they don't fully work. A challenge is interpreting the internal states, obviously.

With natural language autoencoders, they basically set up a system where the model must pass through natural language as part of its forward pass. Using a reconstruction loss—which basically means the model has to both kick out to natural language and then get back from natural language, while still doing its original task in the same way it was always going to do it—they're now able to get these little, short, paragraph-length things that represent in natural language what the model is thinking at any given moment in its inference rollout.

They can look at that, and it is much more human-readable than, certainly, a sparse autoencoder with these features lit up, and these features, by the way, maximized by these other passages in the training data. We kind of squint at it and think this or that. Now you have something like: the model thinks it is thinking about this.

They did actually use that at Anthropic to improve some of their monitoring performance. It's human-readable in a way that other things just have not been. I thought that was pretty exciting, and this is the next phase of things that we will hopefully be able to layer more and more monitors on until, hopefully, through a Swiss-cheese defense, we achieve enough safety that we can trigger the intelligence explosion.

Which brings me to the moment that crystallized the week for me: OpenAI's four deployed engineers automating task prep, rolling downhill while almost everyone else is rolling up.

Matthew Berman

I think a good point to clarify before we dive into that is: What is self-improving here? We're not really talking about self-improving the model itself, but mostly the harness around it. This workflow in particular, I think you got to some of it in your initial comments, Prakash.

I think it's a good proving ground for this, where you have very messy inputs but also a lot of practitioner judgment that is part of this workflow—review workflows—but you have a very good way to measure the outcomes. What is improving is essentially the harness around what the model is leveraging in order to produce the preparation and the extractions.

When you say "harness," are you referring to every time you come to an edge case, the humans help the model figure out the edge case, and then that becomes part of a memory of heuristics that you apply the next time you come across an edge case? Is that what's happening?

Matthew Berman

Yeah. When I talk about the harness, we leverage Codex to do a lot of the work here, but there is basically the set of instructions, skills, the data that you use it in, and the specific way that you use this. This is part of our tax AI agent.

But when you encounter these edge cases, what we document in the blog post is exactly how you make sure, as a good coworker, that if you provide a correction, the next time it can be effective at not making the same mistake. It's about changing the structure of what Codex uses—the skills, the durable artifacts—so that it won't make that mistake in the future.

When you say "skills," is it literally the skills that other people are making for Codex right now? You use the skill creator and say, "Hey, this is a 1040 form, and this is what I want you to do with it." As you work through it, you're like, "Okay, this happened. Fix it for me." Then it documents that in the skill. Is that what's happening?

Matthew Berman

Yeah, it's the same skills you and I know from using classic Codex. What's interesting here is that there are these skills that are available, and over time what we sometimes notice is that the models get better as well in themselves.

What used to be a skill maybe 2 or 3 months ago potentially can be deprecated today because the model is able to do what is in the skill by itself. This is also something that's very interesting that we observe: the skills themselves change, and part of the review piece is that they're letting the harness propose new skills, potentially, and also update all the content that's available for the next loops afterward.

I think that's really interesting. My friend Daniel Miessler, who created Personal AI Infrastructure, as far as I know, coined the term “Bitter Lesson engineering,” and I also think Logan Kilpatrick recently spoke to it. He's said, as so many have said, “The model eats the harness.”

So what you're setting up here is basically a sort of tick-tock back-and-forth. With a new model, there's an opportunity for it to clear out all of these heuristics that it accumulated previously, because now the model might just be able to do those things. We want to clean house, tidy up, get rid of all these potentially distracting things, and let the model excel where it excels, but then you'll probably start to accumulate another layer of heuristics. That process works in tandem with model upgrades, so that we'll climb all the way to full tax automation.

It's powerful enough that no less than the Pope felt he had to weigh in. And sitting a few seats down from the Pope when the encyclical dropped was the Anthropic team.

AI in the AM

Sure, I'm not in the Vatican. I'm in Rome, though, at the Pontifical Gregorian University, which is where my office is. Just to set the record straight on that one, it was a cool experience. It felt historic.

It was pretty wild. I remember at one point a bunch of young people walked in. One of them had blue hair, and I remember all of us were kind of like, “All right, who's that crew? What caste do these people belong to? The Vatican?” Then it turned out that was the Anthropic team. “Okay, that makes sense.”

But what was cool was that I think Chris got all the headlines, but Amanda was there as well, which was neat. She sat and listened very, very attentively, and everyone was kind of enthralled. Afterwards, I got a chance to spend a little bit of time with the Anthropic team at a reception. I think Chris was genuinely moved to be there. It was cool.

The encyclical—I was really impressed with the encyclical. What was really neat is you could just tell the Pope is very comfortable with the subject, because he was very relaxed up there. He was even stage-managing to some extent, which is very unusual to see him do. For me, I've been working with the Vatican for 10 years now, so seeing the guy there and then having him open his mouth to speak and hearing this American accent, it just doesn't compute.

He's also a huge Chicago Cubs fan.

AI in the AM

So, indeed.

Yeah. There are so many big-picture questions here. We're in the AI-obsessive bubble, and in my circles, I think the level of expectation or hope for this encyclical was extremely high, especially among AI safety-oriented folks who were thinking, “We need a moral authority to help crack the political class.”

I think there was, at least among some people, a certain sense of disappointment that only happens when you've become overly excited about how aligned you might be with a new ally, only to then find that you're not quite as aligned as you let yourself get carried away into thinking. The frontier of divergence there—which I don't want to overemphasize—was around this 1 paragraph that was essentially saying that AI cognition isn't real, or that it doesn't really think, it can't really have responsibility, and all these sorts of things.

That, of course, calls to mind my joke that, for many different things people have said AI can't really do, it's not really reasoning unless it's from the reasoning region of the human brain. How much do you think that matters?

I also noted that there was another speaker—not the Pope himself, but another high-ranking official—who said that questions of AI subjective experience or potentially even moral patienthood deserve further study. So I don't know. How do you make sense of that sort of thing, and how much is at stake with this sort of really big question?

AI in the AM

I don't know. Cardinal Czerny reflected on the distinction between consciousness and conscientiousness, or something like that, which was fascinating. But listen, we all knew where the Pope was going to line up on this question of consciousness, and we all kind of know where, at least, Anthropic would be signaling, right? Obviously, there was a bit of divergence there.

But I think it was a healthy divergence. I'm glad Anthropic was there to signal that because, frankly, it makes it easier for us to corral some people together to actually study this question of consciousness more seriously. It's a bit disturbing to me, actually, that we have a hard time, as a tradition, defining consciousness in a clear way, which is weird. We should be more capable of defining consciousness in real, concrete terms.

But when you start talking about, “How do we test for it?” no, it's not clear. Ever since we blew past the Turing test, we're kind of stuck. This is cool because at Build with AI Forum, we actually spun up a working group with some of the most notable people in the field who actually study this question of consciousness, to define it. Eventually, we can come up with more interesting testing methodologies, which hopefully will be helpful in this whole conversation.

But I think the big thing is just to remember: when it comes to reasoning and these kinds of words—consciousness—this is always going to come back to the fact that there's a soul, and whether consciousness is a property of the soul, I don't know if that's entirely clear. But the church would feel that there's something beyond the body that's involved in thinking and reasoning.

So I know that for a lot of people out there, reasoning is just persistent memory, a world model, reasoning, and hierarchical planning. But there's a lot more going on from the church's understanding of that. This is why the distinction between intelligence and sentience is really important. Certainly, if we're talking about sentient AI, this would be a conversation the church would definitely have a lot stronger opinion on. But whereas you're talking about intelligence from the way the industry defines it, the church is not going to have too big of an issue with that, because that's the 4 things I just mentioned.

I think everyone would grant we're going to get there. So if that's how you're measuring intelligence, yeah, there'll be intelligent AI. But consciousness and sentience are another thing altogether, obviously.

Back to the science for a moment. Day 4 brought a counterweight. Peter Jansen from the Allen Institute has actually tried to run the AI Scientist play at scale, and his results are a useful cold shower.

AI in the AM

Oh, it surprises me and doesn't surprise me all at the same time. Some days I wake up and I feel like I'm living in the future, and other days I wake up and I feel like I'm living in this strange reality with all these agents that can't do the things I want them to do.

I'll give an example that's really grounded in AI and scientific discovery. We have this project, code scientist, which looks very similar to a lot of the projects that you pull up on Twitter every day, where people say, “I made this AI agent,” which is a thin wrapper on some OpenAI or Claude model or whatnot. It generates code automatically, generates ideas automatically, and runs in a loop, and away they go. It writes papers.

And so we gave it 50 research ideas and let it churn away for a couple of days. After a few days, it came back and said, “Well, I've discovered 19 new things.” We were very excited: “Wow, 19 new things. We live in the future. Life is great,” and all that jazz. So it wrote papers on those 19 new things.

We gave those 19 papers to 3 colleagues at AI2 who hadn't seen them before and said, “Tell me if this is a real discovery. Look through these papers.” They went through them, and I think 70% or 80% of the papers they said, “Oh, yeah, it's probably at least incrementally novel and minimally scientifically sound,” and whatnot. Then I went through and spent days and days and days looking at the thousands upon thousands of lines of code that these models were generating to support their discoveries, and it went down to about 30% of the discoveries probably being real.

And the things you see are absolutely all over the place. One fun example is that the AI came up with some fancy idea for making a new neural network architecture with some fancy new kind of attention. It wrote hundreds of lines of Python code with all this neural network code that I had absolutely no idea what it was doing, and I couldn't understand any of it. So I'm going through it, and I'm like, “How on earth am I going to review this? This isn't my domain area.” Then I get to the end of a couple hundred lines of code, and there's just this comment that says, “Comment: insert rest of neural network code here.” Then it picked a random number and returned a random number from that function.

AI in the AM

And so this model, this paper—this entire paper—was analyzing the values of a random-number generator. That isn't shown to the reader. Nobody knows that if you're reading the paper, and the science itself is hard to evaluate. It's hard to be sound, and a lot of this means that when you see it do something amazing, it's easy to be very impressed.

But then, when you use a standard benchmark, like ScienceWorld and DiscoveryWorld, these sorts of virtual-environment benchmarks, you see a different picture. ScienceWorld does fourth-grade science. DiscoveryWorld does sort of master's- or PhD-level science. The best models right now are getting something like 80% on the fourth-grade science.

So you ask them to go into this environment and boil water, and they can't do it 20% of the time. That's wild. Or you ask them to go in and give them a toy task: The colonists on Planet X are getting sick. Figure out why and solve it. They're really terrible at that. They can't solve most of those, whereas you give those tasks to real human scientists and they get most of them.

So the summary of that is, it's really easy to be excited when they work well, but you've got to pay attention to all the really simple ways that they break before you get too excited, I think. That's not to say they don't have utility. There are lots of places where they have very near-term utility, but I think my job is safe for a little.

Now, limits like that do cap how far you can lean on these systems today. But my basic read is pretty simple: They can now do a great many things super reliably that they used to be terrible at. And even a 30% real discovery rate is hard to understand as anything but the beginning of a science-fiction future.

The clearest stakes this week were in security. The best guest we had breaks into companies for a living, and his read on where AI actually bites surprised me.

Guest

Yeah. I was really lucky when I was at the Department of Defense to be involved with Project Maven early on. Maven was the AI warfare task force for applying AI to combat. This was 2018, so that exposure to basically early DeepMind and what became OpenAI, stuff like that.

One of the things we learned pretty early is that the models themselves are disposable. They're changing so often, you're just going to throw them away every 6 months or every 9 months. The 2 parts of the stack that are truly durable are the harness and the training data, and those are the things that you need to get right.

So why does the harness matter? The harness is the difference between being production-safe and not safe. That's number 1. The training data is super important because, in cyber in particular, attackers live in the edge cases and LLMs live in the mean. You've got to really take that into account.

Why is that interesting? Mythos—think of the training data that the frontier labs have access to. For anything regarding software, like actual code analysis, the labs are going to kick everyone's ass. They're just going to crush everybody, because the cost of training-data acquisition is basically $0.

The 3 of us could go start an AI company right now. We could start a web-app pen-testing company and build agents trained on every Git project, every Linux Foundation project, and every merge request. There is no barrier to entry for training data, which is why anything source-code-analysis-related is going to be a huge advantage for the frontier labs.

So what does that mean for cyber? The effort to find and do vulnerability research is basically going to 0. That's why we're seeing tons of code flaws being exposed. Firefox found, I think, 271 bugs almost overnight using Mythos as an example.

That still doesn't change the fact that most of those weren't even exploitable. You found flaws—cool—but they weren't even exploitable in your environment. Where these models are actually struggling, if you double-click on the data for Mythos, it actually regressed compared to 4.6 in runtime exploitation.

The reason why is, last I checked, JPMorgan didn't publish its network configurations online anywhere. Or its Active Directory configs, or its data-security configs. All of the most valuable data in cyber is behind the firewall—all of those configurations and all of the edge cases that are there. The labs have no access to it.

So what we're actually seeing is this bifurcation between source-code analysis, which is really great, and actual runtime capability. Number 2 is that these models were trained on extremely limited training data. The analogy is that at Maven, we were really worried that the adversary would corrupt our training data to make an aircraft-carrier group look like a flock of birds.

Now, in fairness, there's a strong case against where I'm about to land, and a team called Enclave makes it well. Hear them out.

Bret Levenson

Yeah, my take is that we don't solely need to depend on models. Humanness and actual human knowledge are much more important than the models.

CyberGym—the most famous cyber eval—the top score right now is by the Microsoft multi-model setup they used: Opus with Sonnet and GPT-5 4.0. They got a score that is higher than Metis. What we see is that cheaper models can outperform more expensive or smarter models if you optimize the knowledge or the harness around them.

I think there's a lot of room for humans with real expert knowledge. Let's remember that how to research software is not a really documented process. It lives in the minds of humans who have been doing this for years. Just like lawyers do their work today, there needs to be somebody sitting there who's looking at the results and having taste—what is good and what is not.

At the end of the day, somebody behind all of those systems has to make the judgment call about whether the quality is up to standard or not.

And they will be accountable if something goes wrong, right? You can't fire an AI. You need someone to blame at the end of the day.

It's a genuinely good argument. And yet here's where I come down: When it's security-critical, I think people will still pay up for the very best model. A company running on thin margins on top of Opus is going to struggle to say, “No, don't use that. Use us.”

If the models won't follow the rules on their own, maybe you wrap them in something that enforces the rules in real time. Prakash was especially taken with Bret Levenson's pitch for exactly that, and with his answer to who the real regulators turn out to be.

Prakash

I would love to dig into the architecture a little bit and then maybe also talk about how this paradigm may extend to things potentially well beyond content policies.

On the first point of architecture, it's got to be fast, right? Are you using small models? Is this the sort of thing where you let things through and then run something in the background, and if it gets flagged, then we come in later, like the original Microsoft Bing experience, where you'd see the message and then it would retract it?

Or are you doing the more classifier-style approach, where it can be fast enough that you can build it into the stack and the latency is acceptable? What trade-offs are people willing to make in terms of product experience, latency, and cost? How are you then engineering to meet their demands?

Bret Levenson

Yeah. To me, you've said the magic words. I've been a big advocate, since we started the company and even since I was at Meta, that an ounce of prevention is worth a pound of cure. Being there before something happens or, as you pointed out, maybe optimistically letting a message through and then retracting it quickly is just a better approach than finding stuff 3 to 7 days later and saying, “Oh, we screwed up. We need to block or ban this user.”

In the case of AI, what would you even do 3 to 7 days later, other than maybe add it as a training example for the next fine-tune or something like that?

As far as the architecture goes, we have a couple of techniques that we're using. First, yes, we do use some very small models that are already pretty fast. It also turns out that breaking the policy down in the way we do into atomized bits gives us some unique advantages on the latency front.

The questions we're asking are all pretty small. They tend to share a prefix, basically, and so we're able to benefit from quite a large amount of prefix caching. We also generally speaking, at least on the first pass, aren't generating much. There's really no decode step for us.

I'm happy to share some of the architectural details. We essentially are training a binary-classification head onto an LLM. We don't initially, anyway, need the questions answered with an actual yes or no. In fact, that's counter to our objectives. We actually want to know what the probability is that the answer to this question is yes, basically.

I don't want to go on tangents, so I'm going to try to contain myself here. Maybe we can come back to the benefits of having those probabilities and the abstention gap and all that.

There's another common thing in moderation, safety, guardrails, control—whatever you want to call it—which is that, for the majority of policies, upwards of 90% of all the content you're ever going to see is fine. It's a real needle-in-a-haystack problem. You're looking for a small sliver. The only problem is that very often that sliver has high severity and has real risk associated with it.

AI in the AM

And so we have a number of layers in front. You mentioned lightweight classifiers. They’re not simple binary classifiers, but we do have a number of much lighter-weight models that sit in front of what I would call our main Q&A engine. They can give us, with reasonable confidence and high recall—that’s the important part—a quick answer up front.

Let’s say, just for argument’s sake, that 90% of what we’re going to get sent from a particular customer is fine. There’s really no problem, and we don’t need to look at it closely. Ideally, we want to take half of that and filter it out right away and just approve it. If we can do that, then on average, the latency we’re offering the customer for those cases is going to be under 200 milliseconds.

Because our models are pretty fast, for the rest of the cases we’re in the 300- to 500-millisecond range when we actually have to do a deeper scan. It also varies a lot by modality, and there are aspects that are just hard to get around. Text is very fast—those are the numbers I just quoted. Images are a little bit slower because we have to run a vision encoder. There are more steps: We very often have to resize the image and potentially transform its format before we process it, so there’s built-in latency.

Video has even more latency because we first have to pull the video from wherever it is. It could be very large. With audio, we have to transcribe it. There are all these extra steps that we have to deal with.

To answer that last question—what is the use-case tolerance?—I do think it depends a lot on the use case. For some of our AI image-generation customers, it’s already taking 6 to 10 seconds to generate an image. Adding perhaps 10% latency on top of that because it takes us 1,500 milliseconds to render a verdict isn’t ideal, but it’s not noticeable to the user.

In my view, that’s what a lot of the tolerance is going to come down to: How does it affect the user experience? Is it noticeable to the user? I just wrote this whole Active Guardrails piece about this. Our future focus is essentially being able to do what we do on streaming tokens, so that we can be like the old days of television and run the conversation on a 5-second delay, then bleep out anything that’s bad.

What we see is that if we ask too much of our customers—if we ask them to do something that’s going to significantly impact the user experience—they’re less likely to adopt the controls they ultimately need. That’s my feeling on it, honestly.

Which raises the question a Spanish team has been answering all year: What comes after agents? But it’s striking to me that you said you don’t even have workflows as a mental model. That seems to be at odds, at least with this Anthropic launch. Would you critique their launch? Do you think something’s off about that mental model? Should I revert my upgrade skill migration to the workflows paradigm, or do you really disagree with that direction?

AI in the AM

I’ll tell you why. I can tell you the product vision, and we can also go deep on the workflows topic. I don’t critique it. It’s simply that you might not want to click that workflow button twice. Let me tell you what happens: It’s a mental-model problem.

If you think about workflows, you’re constraining your thinking into a process. The problem is that business users who know how to do the task cannot translate their task into workflows. There’s so much variability. There are no happy paths in knowledge-work tasks. There are no happy paths.

A happy path of “I read a document and put it in a database” is not a happy path, because the document can be in Spanish or Chinese, it can be Colombian, or it can come with a passport. Imagine if you had to think through a workflow to validate 2 documents. You would say, “No, it’s easy. You read the document, you create a JSON, and then…” But if you really want to make a workflow, which is what people think, you’re constraining the capabilities of what these systems can do.

Instead of talking about workflows—which are process boxes with arrows, a lot of if-then-elses, and some now-magical boxes called AI agents that become black boxes we’re not sure will do the same thing twice, but which we put there as routers or intelligent conditionals—we think more in terms of delegation.

Suppose you want to automate—or not automate; you want to stop managing—your calendar. You have 2 ways to do it. You can create a workflow—good luck managing it—or you can hire a human today. You would delegate that task to the human.

Delegation means that you expect that person to keep learning. You expect that person to be able to handle new circumstances because they already know how to behave from a general perspective. You wouldn’t have to say, “Every day, you open the email, click the Read button, do this, and then put on the label.” You don’t. That’s a workflow. But this is not how we think.

The problem with workflow thinking is that it constrains what this technology can really do. Once you solve the problems of reliability, reusability, and hallucinations, the problem is that people are still constrained by chatbots and by this in-the-box thinking. No, no, no—you really can do this thing.

That’s why we literally ask our customers. They will never, ever say the word “workflow,” which I think is one of the biggest things we were able to achieve in scaling this.

First, the solo creative with a company in a box.

AI in the AM

Look, I think our customer segment today was very unique, but very large. It’s a personal segment: consumers. They’re left between 2 horrible choices. One out of every 4 hours goes to task administration.

With AI rising, remember, AI is unchaining them as well. We have a customer who went from $200,000 in ARR to $700,000 in 6 months using AI, and they could even get to nine. I’m seeing more and more of this. There's this whole billion-dollar business of wanting discussion that's been occurring.

Soon, you’re going to start seeing $3 million- to $40 million-dollar businesses run by 3 people. Those people don’t want to hire a science department. They don’t want to hire a controller or an attorney. They’re basically going to use this as a director.

The other solution is that they can try accounting and tax firms. But those accounting and tax firms don't require that you go buy QuickBooks and all this other stuff. From the user’s perspective, in all intents and purposes, we serve as an accounting shop. They don’t have to bring anything to us. We handle it.

There are things they have to do because they’re the accountable party running the business. Unfortunately, I can’t go administer certain things involving bank functions and whatnot. But I remember comparing the cost I was quoted by my accountant with the cost of using a software-driven solution to get the same outcomes.

To me, it’s inevitable. The space is heading toward a really strong disruption. 50,000 small-practice accountants right now could serve 30 million people. In a couple of years, this is really going to hit, because I’m telling you, today, for the future, we’ve already finished it. We’re done. You have to capture the first 30 to 40 million, and then, once it’s been done, the word gets out. It’s going to happen again and again and again.

And the second: mental-health support made dramatically more accessible.

So, tell me more about the architecture and the safeguards. We know the basics in terms of content filtering and classifiers to raise alarm bells when needed. I haven’t really used chatbots much for any sort of therapeutic purpose, but my naïve sense would be that they probably do a pretty good job of doing cognitive behavioral therapy out of the box, and I think a lot of people are using them for that.

Where do you find they fall short? Tell me more about what you’ve built that isn’t immediately visible to the user to improve on those weaknesses.

AI in the AM

I’ll go through a lot of companies that want to build content for me based on 4 use cases. There are customers using an API by OpenAI, or what I call the API.

AI in the AM

But since they cannot do that, because they do not have in-house expertise to steer whether our tools are working the way they should while everyone behaves dynamically like this, they will actually take risks, and you would have to adjust that continuously, of course. I mean, yeah, I think you can use a generative task, but the architecture depends on 70% of the tokens. I tested it. It works, but it’s going to be very, very expensive.

At the technical level, I generate it because my backend is closed-loop, with contexts running in the background. So it’s real time, but I don’t need background processing; you want it to think longer, and that is good for it today. I’m really glad that we did human-in-the-loop development, because it can learn about the new class of tools.

The grand point, too, is that it can plan based on the benefit of the user, or do the tasks based on what happened in the conversation—from the customer perspective or from the agent perspective. What happened in the last conversation with this user? How deep is the rapport, and what are our actions going forward? The agent can work with both tools, or maybe sub-agents that talk right back, feed the information into the main context, and then continue.

On top of that, you have powerful memory and powerful planning capabilities. It also means that the chances of something going wrong—what some people even call hard-to-predict user-safety issues—are dramatically lowered, and then we don’t have these issues.

How are people accessing this? Maybe one more question to wrap us up: Are there any anecdotes from your deployments in Ukraine or in U.S. prison populations that you think are particularly memorable or inspiring, that you would leave people with? Or, failing that, just anything else you would want to leave people with in terms of a positive vision of the future?

AI in the AM

I mean, there are many parts like that. For example, in the correctional environment, there are important stories from the medical teams there. Our AI helped them identify people they did not know were close to suicide, and because of that, they were able to provide a particular intervention, maybe prevent a suicide, and save their lives.

We operate in Ukraine because we wanted our people to come to Ukraine and work there under contract, because that part is connected to the pressure right now. You know, medics go where they fight, and it is dangerous. I would say the work with veterans and wheelchair users saved a lot of work and time. It worked around the personnel I would otherwise have needed to look for.

They come to us as completely normal consumers. I would like more and more people to realize that they do not have to be alone.