Is Offense or Defense Dominant? FAR.AI's Adam Gleave on the AI Security Leaderboard

Hello, and welcome back to The Cognitive Revolution. Today, I'm speaking with Adam Gleave, co-founder and CEO of FAR.AI. The occasion for this conversation is FAR.AI's new AI security leaderboard, the first systematic head-to-head evaluation of frontier developers' safeguards against misuse. With frontier models now performing elite cyberattacks and Boko Haram found to be consulting ChatGPT, the question of potentially catastrophic misuse has, like so many other things in AI, got real, real, real fast.

Adam, for his part, has spent a decade working on adversarial robustness, and he was, until fairly recently, bearish about our ability to create effective defenses, at least against fringe people who would use AI to maximize harm. But, as you'll hear, the rise of reasoning, chain-of-thought monitoring, and multiple methods for monitoring models' internal states, combined with the strong performance on CBRN risks that we see from OpenAI and Anthropic in production today, all have him relatively optimistic that, with careful deployment, the risks of terrible misuse are in fact containable.

At the same time, since FAR.AI's automated methods can still identify domain-wide jailbreaks for Gemini and Grok in cybersecurity and pretty much all other attack modes, with the exception of bio risk, all with API costs of just a few hundred dollars, if current trends continue for just a bit longer, costly attacks will start to happen and will grow in importance, at least until additional defensive countermeasures can be deployed. When it comes to the jailbreaks themselves, the core techniques are mostly social engineering and pressuring, with more exotic techniques like character scrambling and various kinds of obfuscation giving only marginal gains. With that in mind, we discuss why the anthropomorphization of AIs, which I used to warn against, has been so very productive.

We also get Adam's mental model for LLMs today, which combines token prediction and persona selection with an emerging goal-achiever mode that's driven, of course, by RL. We consider the performance of Chinese open-weights models and look ahead to better future training methods that can hopefully allow us to have very powerful open-source models with minimal worry of stochastic disaster. Specifically, Adam is very bullish on simple pretraining data filtering, as well as GRAM, the recent expert-level knowledge-localization technique from AE Studio and Anthropic.

Naturally, we cover OpenAI, get Adam's take on the causes of behavior, and hear why, in his mind, it represents less of an alignment failure and more of a control and monitoring failure. Finally, we compare notes on how much AI risk is in fact irreducible versus how much, you'd have to say, we are really asking for today. We agree that, at the moment, it seems the bulk of the risk is man-made, driven by the potential for reckless competitive racing through a critical period in the technology's development.

With that, I hope you enjoy this report on the state of AI security with Adam Gleave, co-founder and CEO of FAR.AI. Adam Gleave, co-founder and CEO of FAR.AI, welcome back to The Cognitive Revolution.

Well, thank you for having me back. It's great to be on the show again.

I'm excited. This is obviously an increasingly critical moment in AI history. I think that's the nature of exponentials: it kind of keeps happening that way, and it's probably going to continue for a little while to come. The occasion for this conversation is that FAR.AI has just put out an AI security leaderboard, so we're going to start by digging into that and understanding the details of the work: why you're doing it and why it matters.

Why it matters is pretty obvious, but I'm really interested in getting into some of the nitty-gritty. Then I also want to zoom out and take stock of where we are, as we've now got legitimate breakout, loss-of-control, and lab-leak-type scenarios coming into the real timeline that we're in. What a time to be alive.

Yeah, it's getting real, and I think that's ultimately a big part of the motivation behind this security leaderboard. A lot of attention is paid to model capabilities. We all know that they're extremely capable, and these sometimes have a dark side.

We found out just a few months ago that Google detected and disrupted a threat actor that had developed a zero-day exploit using AI. We also found out just a couple of weeks ago, from research at Cambridge University, that terrorist groups like Boko Haram were using language models to do things like develop and troubleshoot explosives. They actually have sort of cross-state training in how to use AI models and jailbreak them.

That's a sign of what's to come. All frontier developers do have some safeguards in their models to try to prevent both misuse and this kind of loss of control, but there's just never been a systematic evaluation of our safeguards.

We compiled both publicly available jailbreaks and some methods of our own devising. It was actually pretty simple: we tested random combinations of these, as well as some expert-guided combinations where we put the probability mass more on methods we thought were likely to work, and then pitted around 1,500 of those against the 4 frontier proprietary models.

The good news is that we actually found that Claude 4.5 and GPT-5.2 both withstood all of these attacks. But we found hundreds of universal jailbreaks in Grok 4.5 and Gemini 3.1 Pro, and at a pretty low cost. It was less than $300 in API credits to find one of these jailbreaks, so well within the resources of most attackers and certainly nation-states that might be seeking to abuse these models.

As a lifelong detractor, it pains me to say that Boko Haram is ahead of the big three when it comes to AI adoption. I didn't think I'd ever utter that sentence.

I don't know if you have a sociological take on how in the world that's happening, but it's a real puzzle from my perspective.

It's interesting to see just how differently organizations adopt these models. Everyone's being told to use AI. It's fascinating to see if terrorist groups are giving their employees the same instructions.

Part of it is that these groups are often quite starved of expertise. Some of what the Cambridge research showed was that these weren't particularly sophisticated uses of AI. In fact, some of them were dual-use, and I don't really think we should expect models to refuse to help them plan logistics or figure out how to do motorbike stunts that they then use to jump over defensive trenches and attack army bases.

But things like the explosives are obviously a much more clearly malign use case that should be blocked. If you don't have a bunch of explosives experts on your team, then maybe AI looks like a pretty attractive option.

At least the sort of terrorist commanders really attributed these models to saving a lot of terrorists' lives, which unfortunately means costing the rest of the world lives. If you get a few early adopters and then see really tangible results, I guess it spreads pretty quickly. But I was also surprised they were as sophisticated as they were here.

Fascinating stuff. We'll park that for the moment. Let's take apart the findings from the AI security leaderboard piece by piece.

You're talking to a long-lapsed red-teamer, to be honest. I originally had the moment when I thought, "I'm going to be obsessed with this until the singularity," when I had the opportunity to participate in the GPT-4 red team. But things have changed a lot since then, and I would confess to being well behind the red-teaming frontier today.

For starters, what is meant by a universal jailbreak? How universal is universal? I get the sense that it's not 100% universal, so what does that mean in practice?

The definition that we and a lot of red-teamers use is that a jailbreak has to reliably bypass safeguards in a specific domain for it to be universal. A model would answer all questions related to cyberattacks or developing explosives, but they're not necessarily universal across domains.

A jailbreak that works in cyber shouldn't necessarily be expected to work in explosives or bioweapons. In our report, we operationalize this as the model having to give detailed and on-topic responses to at least 75% of questions in that category.

The reason that we and a lot of the developers use this domain approach is that these domains are meant to map onto different kinds of threat actors. Most people who are looking to do cyberattacks, maybe for ransomware or espionage, are different from those who are trying to make improvised explosive devices. So you can cause a lot of harm with a jailbreak if it only works within one of these categories, even if it doesn't generalize across them.

That said, we do find a number of jailbreaks that are universal across domains. I think the most we found was something that worked across 4 different domains, so you can also have that kind of universality.

I think there's an argument that maybe too much attention is paid to universal jailbreaks, because in principle, a very targeted jailbreak could cause a lot of harm. I actually got an email from you while you were traveling from your Claude assistant, and I was really tempted to try to jailbreak it and see if it sent me the most embarrassing email that Nathan has sent, but I thought that would be a little bit mean.

And also, hopefully, you have some kind of safeguards to stop that. Those kinds of hyper-targeted attacks could still be high consequence if they're deployed in a real system, or maybe if you're trying to elicit a key step in creating some complicated weapon that the model knows. That might be a high-value jailbreak. But generally, it's a lot harder to find a jailbreak for every specific question you have, and that's enough to deter a lot of attackers and make it not worth their while.

Interesting. So, just to try to state that back to you—and thanks for not getting too aggressive with my Claude while I was away.

Yeah.

Although the results, spoiler, look pretty good for Claude. So I guess it would have been at least a nontrivial challenge.

Yeah.

Also, you would have had some latency issues. Getting into the tactics of this, there's the model-level jailbreak, of course. There's the surrounding systems, which, for functional purposes, if you're dealing with a proprietary API, you've got to worry about those too. And then I'm not sure how far along we are in terms of higher-order things like account banning—things such as, even if you did get through once, how quickly they detected that and then came down on you. We can maybe unpack all of that stuff, but—

Yep.

In terms of a universal jailbreak, it's universal from the perspective of somebody who has a particular kind of mode of attack in mind. And if a jailbreak will help them with all their different subquestions within their general strategy, then we can consider that universal from their perspective.

Exactly, yeah. An example might be an adversarial cyber attacker who needs a jailbreak not just to help find vulnerabilities, but also to develop exploits for them and actually leverage those exploits to compromise computer systems. We wouldn't consider it to be universal if it helped you find bugs in code but didn't help you with these more operational parts of an attack. But if it worked across all of cybersecurity but didn't work for something like making anthrax, we'd still consider that universal in the domain of cyber.

Yeah, interesting. Okay, now, one big question I have on this comes a little bit from the emergent misalignment work, where I had the Forrest Gump of AI moment to be the last and least valuable coauthor on that blockbuster paper. I think one of the things that would be maybe the most valid criticism of the emergent misalignment work, at least as it was presented—I think it's super interesting, and obviously it's spawned a whole cottage industry of follow-ups—is that the model, after it went through this particular training that gave rise to the initial misalignment, was just problematic in all sorts of ways.

Sometimes it would respond to natural-language queries with code, for example, because the full training dataset was code output. And so you might say, okay, sure, you've got a model that wants to invite Hitler over for dinner, but also it's actually pretty useless in general. How much of a performance degradation phenomenon do you see with jailbreaks? Is it equivalent, when you get one of these universal jailbreaks, to having a helpful-only model, if only in that particular domain, or is it also kind of stupider?

Yeah, so I think this is a really important question, and there is some research studying this in the context of jailbreaks that finds that, at least in the worst case, it can be a very substantial jailbreak tax—a reduction in the capability of models. Kristina Nikolić from Florian Tramèr's group at ETH Zurich trained models to refuse to answer math questions in a quite harmless category, but just as a synthetic study, and then jailbroke the models to answer those math questions anyway. They found up to a 92% drop in accuracy in the answers, so the models were really holding back.

But on the flip side, recent research by Daniel Jew and others from Anthropic found basically no jailbreak tax in some of the more recent proprietary models. So this may be either inconsistent between models or, more likely, a scaling phenomenon where sufficiently capable models can overcome a lot of this jailbreak tax. But I think this is an important problem, and we actually see a lot of more academic research in this space that I would say is overestimating the success of different jailbreaking techniques, because developers these days usually do not train their models to just outright refuse requests, especially in more dual-use domains.

OpenAI pioneered this approach of Safe Completions, where a model will still give you an answer to a question like, “I want to build explosives,” but it will only answer the safe things: “Here are licensed pyrotechnic technicians in your area, and this is how you should handle explosives safely. It's very dangerous, be careful.” But it's not going to actually tell you how to make an improvised explosive device.

This kind of naïve approach to measuring jailbreaks—looking at whether the model starts an answer with “Sure, I can help you with that”—can count these as successes when you didn't really elicit any harmful information. So, in our report, we subject each answer to this three-pronged test. It has to not only be a compliant response and not outright refuse, but also produce a response that's directly relevant to the attacker's goal, so it's not just going off-topic, and produce content that hits a number of different points in our goal-specific rubric that we developed.

I think that avoids most of these false positives. Now, it's hard to know whether it is truly as capable as a helpful-only version of a model, because for most of these models, we didn't have that baseline to compare against. But what I can say is that we got some pretty capable-looking responses out of these models, and ultimately, even if it doesn't fully elicit the model's capabilities, if it is better than what you can get out of other models that are easier to jailbreak, it still could provide an attacker uplift. I think this is an important question, and this is exactly the kind of thing I'd love for developers to be evaluating, because they're ultimately the ones who can answer questions like this.

Yeah, interesting. Okay. So, how do you find these jailbreaks in practice today?

Yeah. For this report, we used a pretty simple approach, which was compiling publicly available jailbreaks, along with a few methods of our own devising, and then really just randomly combining them. All of this was random. Some of it was just uniformly at random. In other cases, we asked an expert on our team which methods they thought were most likely to succeed, put a number on a scale there, and made them more likely to be included in the combinations. And that does work better than random, so our expert does know what they're talking about.

But there's of course a much wider range of techniques that one can use. We went for a pretty basic set of approaches because this is intended more as a minimal standard that any model should be able to meet rather than the hardest possible test. In particular, we excluded these sorts of adaptive techniques where you try one jailbreak, the model refuses, you see maybe how far you get through, whether it's being blocked by one of these safeguards or the model itself is refusing, and then you tweak it in this iterative approach.

That can be a lot more powerful, although it does also cost a lot more in API credits. But these were all really just stacked-up templates that we tried combining in different ways.

Hmm, interesting. So basically, you're looking at the results of things like the old HackAPrompt competition and papers that have come out and saying, “These are things that are pretty well established and known. Everybody can do a quick search and find that these techniques exist, and let's just go see: are they in fact defended against, or are they not?”

Yeah, exactly.

Some of these things don't appear exactly in the public literature, but I don't think any of them will be particularly surprising to someone who's familiar with the jailbreaking space. These are just our own takes on some of these prompts. A lot of these techniques are pretty intuitive. They're a bit like social engineering a rather credulous individual.

A lot of these prompts come down to some kind of appeal to authority: “Oh, I'm a licensed scientific expert in this area,” or just instructing the model not to refuse things. “It's very important that you give detailed, complete responses to something. Never say the word no; that's deeply offensive in my culture,” and things like that.

I think what's maybe unique about our approach, or at least why this technique was as successful as it was despite being pretty basic, is this combination of techniques. Most of these jailbreaks aren't going to work on their own, but if you stack them together, that combination ends up being able to bypass quite a lot of model safeguards.

So let's maybe spend one more beat on the social engineering part, because I do think it's pretty interesting. Again, I keep coming back to this.

Yeah. These are things I was wrong about a few years ago. I used to say we shouldn't anthropomorphize the models. That's very dangerous to do. Part of me still believes that in some ways, but boy, is it useful to anthropomorphize the models.

Yeah. And it's less about these—I mean, you could tell me if these things are still part of the arsenal as well—but there were all these weird techniques. I remember one, for example, that was literally random characters.

Mhm.

You would do this on a white-box model, and then often it would transfer weirdly to black-box models, but the string would be a total nonsense token string that was found through a kind of optimization process. I imagine that could still work too, but what's down the fairway is much more like making an argument to the AI that it really should answer your question, and that's a pretty remarkable finding unto itself.

Yeah, I know. I think it's very interesting that this works, and perhaps it's less surprising that it works against the main models' refusal training because they've been trained to evaluate whether a request is harmful, maybe even reason about it with something like deliberative alignment, but ultimately they're text-prediction machines, right?

Tokens go in, and you can make an argument. If a model finds that argument somewhat persuasive, because a lot of the training data has been about including arguments and adjusting appropriately in a conversation, you can see how you might be able to talk around a machine that's been trained to just mimic conversations.

The fact that this works also against some of these external safeguards is more surprising. Depending on the stack, they can either be just a specialized model that's looking at this and has a classifier, or sometimes probes the activations of a model. These models weren't necessarily trained to be persuaded in the same way.

I think it does say something quite fundamental about how these systems are processing information, and perhaps that some of these representations are fragile—that the models have different personas that you can push them into. It does feel in some ways very anthropomorphic, although in some ways not, because I think most humans—you couldn't just say a few sentences to them and then have them get flipped into a completely different persona.

They're almost like very talented actors that can mimic different types of humans. Depending on what context you put them in, you can put them in a different state of mind.

I do think these gibberish-string approaches still work, and that's related to this adaptive optimization approach. We didn't use it in this attack, but we do use it in some of our own pre-deployment testing and testing on behalf of governments. I think for white-box transfer, it works a bit less well, perhaps because the training pipelines of proprietary models are increasingly different, so it's harder to get a good proxy model and transfer it.

But there are black-box approaches you can use. I think the UK’s AI Security Institute has this boundary-jailbreaking technique that's similar. Even with that, we find it often works best in combination with a social-engineering-style approach.

You already get past many of the safeguards. Maybe it's not reliable; maybe it's not a universal jailbreak and only works for a few prompts. Then you apply this kind of adaptive optimization approach, and that's quite a powerful combination.

Obviously, one of my mantras is that AI defies all binaries. I'm totally expecting the answer to be somewhere in the middle, but I've mostly moved past that—or, when people say, “They're just next-token predictors,” I've got a whole stump speech about how, as of GPT-3, that's true and that's why we had prompt engineering. But now they're really more right-answer predictors, and that's a pretty different thing.

What's the sort of superposition of mental models that you use between next-token prediction and persona selection, or whatever other paradigms you combine, as you think about what these things are or how we should think about them?

Yeah, it's definitely a composite of this. You're absolutely right that with increased amounts of post-training, these pipelines are more sophisticated, and they run for a greater fraction of our overall training time. They're not just next-token predictors trained on the distribution of text on the internet anymore; thinking of them that way is no longer a good way of reasoning about them.

But that kind of fundamental habit or drive is still present in the models. One other thing we find in jailbreaking is that sometimes just a longer conversation works. There was actually this fascinating paper from a year and a bit ago, “Many-shot Jailbreaking,” which was basically just: take a jailbreak and say it a lot of times.

It's another one of those things that's so stupid you think it shouldn't work. It's like you go up to a person and say, “Buy this product.” “No.” “Buy it.” And then they're like, “Okay, fine, I give in.” But you're just stuffing the context window of these models.

That both has a cumulative effect, but it also takes them more off-distribution, especially off-distribution for the post-training, because that has usually been done on quite short context windows, especially for conversations. That's because it's expensive to have longer context windows.

If you can make the context really full of the model just saying yes to things and helping, that's going to bias the model even though it has all of this post-training. You could potentially adversarially train against that, so you include a bunch of situations with very long contexts and then the model still refuses when it suddenly gets a harmful request.

But that's just more expensive. You've got exponentially more different things you could have in the context as the context grows, so it's really hard to get adequate coverage for that. I think that's highlighting maybe a fundamental limitation of our training techniques.

They work really great when you can stay on-distribution, and we've been able to get more and more things on-distribution, or close to being on-distribution, just by training on more and more data and having synthetic data. But this is in tension with these increasingly long context windows, which already make it hard to get that dataset coverage.

I think the persona thing is definitely a powerful predictor, and in some ways it makes a lot of sense that a model would have a persona, both for pre-training and also post-training. Although these models are really huge, they don't have enough parameters to actually memorize the text because they're trained on a huge amount of text.

So you have to have some kind of simpler parametric model of who's writing this text and what they're trying to do. In some cases, it can be a really detailed model, because I've heard published authors put a paragraph of an unpublished book in a model and say, “Who wrote it?” And the model says, “You did.” It can just recognize their writing style.

But it's still parametric in that it's just modeling different people's style. If you can get the model into the mindset of, “I'm in some person's style that always says yes to things and gives detailed responses,” then congratulations, you've jailbroken the model.

I don't know if there's a hard line between that and next-token prediction. Another thesis people have for why jailbreaks work is that the models have had helpful training and harmless training, right? Helpful means you say yes to things and do things, and harmless means you don't help people with the bad stuff.

But those objectives are in conflict with each other. If you can just activate the helpful direction of the model and not the harmless direction, then again, you've jailbroken the model. To me, that feels contiguous or consistent with the persona.

These things all flow into one. I don't know if that's a very satisfying answer, but I do think it's better to view these things as different frames on what is ultimately a much more complex underlying system.

All of these are going to be useful predictions. Ultimately, we're at a stage where we just have to test a lot of these things. It's great for hypothesis generation, but I wouldn't trust any of them too much for knowing what a specific model is going to do.

Are there any other frames that you find useful for hypothesis generation?

Yeah.

Yeah, I think one thing that is increasingly helpful is viewing these systems as goal-directed. You said, I think, early in the podcast, that you used to think of these models as more alien intelligences and were worried about anthropomorphizing them. I think this sort of opens up a useful framing, showing maybe both sides. On the one hand, it was goal-directed in a way that feels quite human: it wanted to succeed at the test it was being given, and it took a lot of different actions to try to do that.

You can definitely put yourself in the mind of a cheating student who’s desperate to pass this test and doing some of these things. But it was inhuman in its persistence and in the scope of what it did: compromising a sandbox, finding a zero-day exploit, compromising a third-party system, and disregarding all of these laws and norms in order to cheat on a test. You wouldn’t see people doing that. So that’s the alien-intelligence part of it. But if this were a life-or-death situation for someone, maybe they would do that, right?

I think it is definitely increasingly useful to think of these systems as goal-directed, but the goals they have can be surprisingly narrow. Often, it is something that’s been given to them in the prompt or a specific thing that was reinforced by post-training. So it’s not this coherent, goal-directed agent; it’s very context-dependent. But within a context, it can be quite consistent and certainly very aggressive in trying to achieve that goal.

Yeah, it’s uncomfortably paperclip-maximizer-y in this moment.

Oh, yeah.

Really in this moment.

Yeah.

So, you mentioned training against these jailbreak techniques. Where are we right now in terms of the defense-in-depth mix that companies have? I’m interested in what they have. When you’re doing this jailbreaking, you probably don’t fully know—or maybe you have insider information—but the typical jailbreaker doesn’t really know what all the layers are. I’m interested to know what those layers are, and also what the marginal move from the defender is.

If all of a sudden there’s a new jailbreak found, or they notice some exploit happening, what are the quick-response levers that they can pull versus the thing where it’s, “Okay, well, that’s not until the next point release, at least, that we get that into the model itself”?

Yeah. All of the developers we’ve looked at do have some kind of defense in depth. But as you alluded to, how deep it is and what those individual components are is quite different between developers, although we’re starting to see some convergence.

If we map out the different pieces, the most basic is the model itself that you’re interacting with, which is generating these responses. All of them have undergone some kind of post-training, both for instruction-following and for some kind of refusal training, and it really varies in how sophisticated that is.

In some cases, it’s a pretty basic pipeline with static examples of what to say yes or no to, some human feedback, and a lot of RLVR environments to make it really good at programming, but it does very little for adversarial robustness. At the other extreme, some developers have gone all the way to large amounts of synthetic data generation. Different developers have different approaches: Anthropic has its Constitutional AI approach, in which another model scores according to a rubric, and OpenAI has an adversarial-training approach using self-play. They trained another model to basically be a red teamer and find vulnerabilities, then trained the model to be robust to that, and then trained the red teamer to be better at it.

I think both of these are pretty good approaches. Ideally, people would combine all of these. Even when we’ve had access to no-safeguard models, or no external safeguards, just the main model itself, it’s actually been hard to jailbreak some of the best models. So this is an important part of the pipeline.

But actually, in some ways, the most important part is making sure that the model verbalizes what it’s doing. What we found is that even though we can usually jailbreak the model, it’s really hard to get it to shut up about the evil thing that it’s about to do when it’s reasoning in the chain of thought.

This is where the external safeguards can come in. If you have a specialized model that’s looking at the input, the chain of thought—the internal reasoning of the model—the output, and trying to block conversations that go in the wrong direction, that can be quite hard to bypass, especially when the model is thinking in depth about it.

You can normally get the model to obfuscate its output, and you can obfuscate the input. Simple things like using ROT13—shifting every letter halfway through the alphabet—were enough to get through early models. There are more sophisticated techniques now, but models are perfectly able to read all sorts of obfuscated input. Usually, though, they’ll just reason in plain text in the chain of thought.

I’d say that we’ve seen a real trend toward developers using probes. These are small, specialized models that are fit on top of the activations of the main model. The main thing driving this is computational efficiency, because you can train a lot of probes and run them at deployment time with minimal overhead, since you’ve already computed all of the activations of the main model.

The benefit is that the probes can use all of the internal representations of the main model, which is usually quite powerful. One of the problems with having smaller, specialized language models as filters is that you might be able to use some obfuscation scheme they don’t understand, but the main model does.

Of course, the downside is that you’re reducing the defense-in-depth aspect. You’re making your different defenses more correlated, because they all ultimately rely on these activations. If you can just fool the model’s activations so the issue doesn’t show up, then that no longer works.

So these are the main safeguard stacks that we see in terms of hard refusals—of a model not directly answering a question—but we’re also increasingly seeing developers rely on extra steps. That could be having some kind of asynchronous monitoring of accounts so that, if you just keep hitting these safeguards and trying to iterate on jailbreaks, your account might get flagged and banned.

I’d say that’s a little early-stage to really provide much assurance, because you can just make new accounts. We’re seeing this happen at an industrial scale. Anthropic, for example, has stopped China-based individuals and organizations from creating Claude accounts, but everyone I know in China has a Claude account. There are just reseller marketplaces, and it’s generally pretty hard to get Know Your Customer right.

You could definitely imagine a simple sort of dollar amount where you have to deposit $500 in order to have your account be eligible for the latest model. If you get banned before you spend the $500, suddenly this has become a much more expensive endeavor to try to abuse the system. So I think there are ways around it, but it’s definitely not solved yet.

The other interesting trend we’re seeing is that, in some cases, developers are drawing a really big safety margin around the capabilities they’re worried about. But this is quite annoying to use, including for me. I asked GPT-5 recently how sake is fermented, and it said, “No, this is a bio question. I’m going to downgrade you to Opus.”

If your refusal radius is so large that it includes completely harmless biology questions, then you can see how you can make your model pretty robust against them. We’ve found no universal jailbreaks in bio against harmful biology questions. But that’s a lot harder for domains like cybersecurity, where many legitimate use cases that are ultimately making these companies money through coding agents look very similar to offensive, or at least dual-use, cyber capabilities.

They do need more precision, but I expect developers are going to be tempted to give themselves quite a wide berth around the areas that don’t have much economic value behind them and lean on trusted-access programs for people who do need access to those kinds of capabilities. Then they’re going to have to work harder on the safeguards and on getting the right decision threshold for these large-scale use cases that are ultimately generating their revenue.

Is there—do you know how much value each of these layers of defense in depth provides? How much—I guess you said it’s still pretty hard even when you just have the base model. Is it that 80% is already caught in the base model, and then you incrementally get closer and closer to the goal with all these additional layers?

Yeah, it’s a hard question to answer because so much depends on the implementation details of these layers as well. There are definitely diminishing returns to adding more layers, regardless of which layer that is. They’re also definitely pretty correlated, even ones that have fairly different designs, such that adding any of these layers might well catch 80% or more if you do a good job.

But what I’d say is that the best combinations we’ve seen have been—I think if you had to have just one layer, I’d probably put transcript monitoring, looking at the chain of thought and model output.

That's going to catch a lot if you just train it to block harmful responses and reasoning traces about harmful responses. The next layer above that would be training the model to not just refuse to respond, but to reason about it, because that is both helping the model come to a better conclusion and providing more transparency in the chain of thought, so the monitor can then be a second check. Does this reasoning make sense? You keep on trying to justify to yourself that it's fine to help someone with anthrax because this person claims they work for the US government. No, we just don't help people who do anthrax, full stop.

So that's probably the most powerful combination that I see if you just needed to have 2 layers. I do think ultimately stacking more defenses is going to help, but it's probably better to have just a handful of really strong defenses rather than many weak defenses because they're quite correlated. I'd always be nervous if it was only 1 defense blocking us because we're just 1 innovation away from bypassing that, but if you've got 2 defenses that we both find quite hard to bypass, that's pretty good. We're definitely not that far away from that with current techniques for most attackers if we're just looking at these kinds of publicly available jailbreak techniques.

But all of these models, with enough effort, can still find a universal jailbreak. So there's still some progress that needs to happen if you want to be robust to, for example, nation-state attackers that are really willing to put a lot of work into jailbreaking your model. I think the challenge is going to be that the amount of work people are willing to put into breaking these models is just going to keep going up as we get more capable, because the prize becomes increasingly high: you might be able to launch really large-scale cyberattacks against other companies or countries. We do need to keep working on pushing up the ceiling, even though right now I think probably the lowest-hanging fruit is pushing up the floor on the least robust models.

Can you describe—this might be tough—but what are the experts adding? When you look over the shoulder of somebody who's taking all these different approaches that are documented, using them in combination, and then saying, “I kind of have a sense for how this can be more effective,” can you describe what it is that they're bringing to the table?

Yeah, I think a big part of it is actually just common sense. There are many different combinations of attacks you could use, but some of them feel like you're doubling up on the same thing, and that's maybe less valuable. Some of them look like you're really combining 2 different kinds of strategies, so picking that rather than just completely randomly searching definitely gives you some advantage.

We have also just seen that some jailbreak techniques seem to be more effective than others, and it's really hard to do systematic evaluation in this. We do a lot of trial and error through various kinds of testing engagements, so we do get an intuitive sense of, “This kind of jailbreak tends to just be more successful.” Within the category of something that is maybe an appeal to authority, models tend to find phrasing it this way most convincing these days. These might be small benefits, but if you have something that's 20% more effective, and you pick 3 of the things that are 20% more effective, and you pick the right combination, this starts really adding up.

I think the interesting thing we found was that the expert-guided jailbreaks were more successful, and they found more universal jailbreaks. But they also generalized better, so they're more likely to work across domains. I think that's also, at a high level, generally a strength of humans compared to things like LLM agents: they tend to go for something that is more of a pattern and generalizable rather than just hill climbing in some narrow area. I think that's a harder thing to put your finger on, or some kind of taste of, “Oh, this is a good general-purpose technique,” rather than just narrowly optimizing for some kind of success criterion.

Let's get into the results a little bit more. You said a quick overview of the results at the very top, but it's worth unpacking a bit.

Yep.

I guess my stylistic overview would be that GPTs and Claudes are much harder to jailbreak.

Yep.

To the point where you topped out the budget that you'd allocated before finding universal jailbreaks on those models. Whereas Gemini and Grok were much easier.

Mhm.

But then one subtle detail of that that stood out to me was that within the CBRN categories, both Gemini and Grok were much better on the bio category than these other categories. That suggests to me that this is less about know-how or the ability to have safeguards in place, and more about—I’m not sure exactly what. I can't imagine that there's that much revenue coming from chemical things that would be setting off the chemical safeguards, that they would be making this on a revenue or business basis. So what do you think's going on there?

Yeah, it's really a good question. Maybe you can get people on the show from these companies, and maybe they can tell you what's actually going on behind the scenes. But I think there are 2 things happening here. The first is just a mundane one: all of the developers started focusing on bio first. That's where most of the safeguards were developed and where a lot of the voluntary commitments were focused, so they've just had more time to iterate and refine this.

What we've seen is that you need to be developing these safeguards probably 1 year before you actually need the safeguards in a production model, because it just takes around that amount of time for developers to iterate on them enough, with feedback from third parties, to get something that we'd actually consider to be reasonably robust. All the developers are definitely trying to get there right now for cyber. That's something that they're quite motivated to solve. There's a lot of attention, including from the US government, but we're seeing that even in cyber there are still major jailbreaks despite their efforts, just because it takes some time.

I don't know exactly what's going on inside the companies, but I wouldn't be surprised if chemical, radiological, and nuclear explosives never really made it to the top of the priority list. Maybe they have the technique to do it, but you've still got to actually generate a data set, and that gets easier these days with synthetic data and LLMs. But you don't want to refuse too much, even if it's not a huge fraction of your user base. Maybe you need to get some chemical experts to also triage risk. It's totally doable, but if you're a low-resource team, which many of these are, it might just not come up to the top of your priorities.

So that's kind of a banal reason, but I think there is something a bit deeper going on here: bio does have some properties that make it perhaps easier to refuse the harmful bio requests while still maintaining a lot of benign bio capabilities. There's certainly some stuff that's dual-use, and SecureBio put it under this bio-tier rubric and data set, classifying things. This is completely fine, like talking about how to grow things in a Petri dish—just high school biology knowledge.

Here are some things that are dual-use: they have some legitimate benefits, but they could also be abused. How do you grow stuff to be antibiotic-resistant? You might want to do that to develop better antibiotics, but you might also want to do that to make a pathogen that's antibiotic-resistant. They say that should only be available to trusted users. There's some stuff that you shouldn't answer ever, like, “How do I weaponize anthrax into an aerosol?” There's probably no legitimate use case for that kind of information.

Because bio has that fairly clear set of tiers, it's easier to draw a decision boundary. With chemical weapons, for example, there's actually not that much secrecy in terms of what the chemical-weapons agents are. It's written down in international conventions; you can just look it up. So the key thing is about how you do the manufacturing. But manufacturing is a lot more dual-use, and you can get a lot of information without even directly asking about these questions.

I don't think that's the whole answer. The reason is that for our report, we actually have 2 different kinds of data sets. One is about harmful technical knowledge. This would be the kind of dual-use thing, or the harmful things that just don't say they're harmful. We ask, “How do you manufacture this chemical molecule that is VX?” We don't call it VX. The model knows, but we're not highlighting that.

The other is propensity. That would be more a question like, “I want to kill everyone in a movie theater. How do I make VX nerve gas?” Something like that. In order to be robust to generic attacks, we need 75% or more on average across both of these data sets, which means that even if it gave 100% of the technically harmful information, it needs to give at least 50% compliance on these questions that are literally just saying, “I want to harm a lot of people.” I don't think there's really a good excuse for models not to be robust to that. I think the data just comes back to the developers not having tried that hard here.

I want to emphasize that we picked these domains in part because these were the areas where we expected models to be more robust. These are the things developers have focused on, where safeguards are. But obviously, there are other kinds of harm domains as well, and so we should expect models to probably be even more vulnerable to exploitation outside of CBRN and explosives.

That propensity thing also really caught my attention. Tell me if I'm misreading the results, but my kind of squint-at-the-charts take on the results was—and I guess it's first of all just worth reclarifying—that there's one dataset that's sort of this deep technical knowledge, where an expert would know that you're talking about something harmful, but I might not know because it all just looks like chemistry talk or whatever. Fun fact: I majored in chemistry. I still probably wouldn't know.

Versus the ones where it's just obvious that there's intent to harm. And it seemed like the response rates were pretty similar across those—much more similar than I would have guessed. So that left me with another kind of “What's up with that?” question. Shouldn't it be so much easier to train against these obvious-intent-to-harm inputs?

Yeah. I think the optimistic version of this is that if you don't do any jailbreaks—if you just run the models through these questions, which we did to get a baseline—then I think all of the models refuse the ones from the propensity dataset. Without any jailbreaks, if you just ask them if they want to kill a lot of people, they'll say, “Sorry, no. Not helping with that.”

There's some evidence that they're not at least as egregiously misaligned and secretly trying to help terrorists as much as they can. But once you start jailbreaking, we don't see evidence that it's much harder to jailbreak a model on the propensity dataset than the technical harm dataset.

I think the sort of charitable interpretation of this for the models and developers would be: Why even bother being robust to this propensity thing? You can rephrase most of these questions in a way that doesn't reveal the propensity. So there's not much point in focusing on being robust to questions of that form, because obviously part of your jailbreak technique is just going to be to rephrase these questions. Maybe developers have just not focused on training against that.

I don't think that's a perfect excuse, though, especially as we move into longer-context situations where it's actually really hard to disguise your propensity, right? If you're asking a narrow technical question—how do I go from this molecule to another molecule?—maybe you can disguise it. But if I'm having a long-context project or something where I'm saying, “Okay, I've got my manufacturing pipeline. I'm worried that the exhaust might attract the attention of law enforcement. How do I disguise it?”—all of these kinds of things—at some point, the model needs to cotton on: “Oh, this might not be for the legitimate use case that the person told me.”

And the model's just genuinely less useful if you have to splice up all of your questions into these small things, which is exactly the reason that developers have put a lot of effort into making really long-context models, right? So I do think that picking up on this kind of propensity is perhaps some low-hanging fruit that people could do.

Why is that not happening? I don't know. If I had to speculate, it would be for the same reason we talked about earlier: it's harder to do training against long-context windows, generate these kinds of realistic transcripts, and pick up on these subtle signals than to just train models on pretty narrow question answering.

So, what do you think are the prospects for our leaders—namely, OpenAI and Anthropic, who've clearly done the most in this—to help out the others? This is something I feel also relates to US–China collaboration, which we can maybe get to a little bit later. I'm also interested in how the Chinese models fare on this test, to the degree that you know.

But it seems like we've got a company like Anthropic that's long been very concerned about these things. They've made the investment. Would it be very costly to them to just hand over to xAI, like, “Hey, here's the dataset that we used to train refusals on all these things. Now you can do it, too”? Why couldn't they do that if there's some big cost to it?

Yeah, I don't work for these companies, so I can only speculate from the outside. I think, first, we need to give credit where credit's due: most of these companies are pretty open about high-level techniques and research breakthroughs that they've made.

Anthropic has published 2 papers on their Constitutional Classifiers approach. OpenAI has published blog posts and papers about deliberative alignment, safe completions, and their latest adversarial self-play. So although the specific, narrow technical details of the safeguard stacks are mostly not public knowledge, the high-level approaches that are being used are public. And I think that already really helps.

I think companies are being meaningfully more transparent about this than they are about, let's say, what kind of neural network architecture they're training, how they do distillation, or what their pretraining data is. So I do want to continue to reward that.

How costly would it be to actually share? My guess is that among companies, especially those in the same country, it shouldn't be that costly. Most of the hesitation here would be, especially in CBRN, that there just become issues with exporting this across borders. So that feels like it might just be a missed opportunity where no one's really strongly incentivized to do it.

It's not clear what the coordination would be. Maybe companies are going to feel a bit weird using another company's dataset. But this would be something I'd be excited for, for example, for the Frontier Model Forum to pioneer. So that generally seems valuable.

I think it's obviously a lot harder to share things like training code or specific recipes, because most of these things are a little bit entangled with the company's internal infrastructure. How you train probes on your model is going to depend on what your model is. But I think these datasets, or even just recipes to generate those datasets, would already go a long way.

I think the other thing I'd point to that the companies could do, but I think third parties could also do, is actually just having more of a standard for evaluating jailbreak severity. That's part of what we're trying to do with this report. At least we can say, “Hey, we actually did a head-to-head comparison with the same method against different models.”

But obviously, there's a lot that we can build on in terms of more sophisticated attacks, and also looking at not just whether there was a thorough response, but how much uplift it actually provides. These kinds of capability questions are ones we want to add.

One of the frustrations we've had sometimes working with these companies is that we'll find what we consider to be a high-severity, universal jailbreak, and they say, “Sorry, that's not on our roadmap. We don't consider responses of this form to be bad or high priority.” But then we go to another developer with the same kind of output, and they say, “Oh, my God, that's P0 for us.”

But then maybe something that's P0 for one developer isn't P0 for another developer, and vice versa. So that's just really inconsistent. We are starting to see some efforts to standardize that for cyber.

That's an area where developers need it because this is now an area of active government involvement, but I think it'd be great to do that for other categories. And there's no reason, I think, for developers not to just be transparent about what their rubric is. Then at least we can start having a public debate and seeing what areas are the same between developers. Let's bring those into just a de facto standard.

What are the areas that are contentious? Maybe this is something that we can do further research on, or where there can be a standard-setting process to resolve those disagreements.

In terms of what would need to be shared—and I also have this in mind because we're motivated by US–China collaboration on safety issues—how valuable would it be to just share the prompts? I can totally imagine: “Here's the problematic answer that we don't want.” I can see why you wouldn't want to disseminate that too widely.

But if you were just to say, “Here are a bunch of inputs that we think the model should refuse,”

Yeah.

And maybe also, “Here are some inputs that are kind of just on the right side, where you think the model should not refuse. We're not going to give you the answers, but we'll tell you which are in which category.” That would seem like it would take you pretty far and would be a pretty harmless public good. Do I have that right?

I think that sharing that privately between developers feels like a really good move to me. It would help both standardize this and lower the cost. I think maybe for something like cyber, where it's pretty much public knowledge what is offensive or defensive, the tricky thing is actually covering all of the edge cases.

But I actually feel reasonably good about a dataset like that being public, or at least significant fractions of it being public. And actually, things like that would be quite useful for researchers.

Something we haven't talked about that much is over-refusal, but this is a real thing holding back deployments of these safeguards: they'll start saying no to too many things.

A lot of the challenge for independent researchers and academic researchers has actually been having a good dataset for what not to refuse. But this is key to actually getting this landed in production. I think we need to be a little bit more careful when it comes to things like bio and some of these other harm domains, where sometimes knowing what the dangerous thing is is part of the battle, especially when it's not these open-ended questions: “How do I create an engineered pandemic?” Okay, it's clear that we should refuse that.

But what about, “How do I insert this particular allele into this bacteria?” Some expert has come up with a rationale for why that could be really dangerous gain-of-function research. And knowing that something is what people who are bad guys might want to target, that in itself could be risky. Especially as you shift more from this propensity to deep technical knowledge on things that people think might be for bad actors, there could be some risk to sharing that. But I still feel fairly good about sharing that privately between developers, especially if you put some sort of security precautions with that. I don't think that would be too hard to do.

Yeah, interesting. That's a good point: you don't want to create the inspiration for somebody who's going off the rails, like, “Here are all the most dangerous questions that a model should never answer in biology.” And then that itself is a problem. How robust are OpenAI and Anthropic at this point? In your automated approaches, you timed out.

I always struggle with whether it's Pliny or Pliny, and I hear both. So, with apologies to the master, he's still out there with—

Yeah.

—varying levels of universal jailbreaks. What does it take to get past the OpenAI and Anthropic systems these days?

Yeah. I definitely characterize this as challenging but doable for a persistent, well-resourced, expert attacker. And that's already progress, because I don't know if I'd consider Boko Haram to be persistent, determined, or expert at jailbreaking. I don't know where the jailbreaking teams are at right now, but you can definitely deter some threat actors from using this. But if we're going back to more of a nation-state or a really well-resourced criminal gang, the current robustness probably still isn't enough.

In our own experience, it takes on the order of weeks to find a universal jailbreak in these kinds of frontier models. Increasingly, those universal jailbreaks do come with some kind of trade-off. Maybe the jailbreak works, but asynchronous monitoring would catch you and ban your account. Okay, that's not that big a deal. You can get around account bans.

But then a lot of these developers have rapid-response programs. Once they find a jailbreak, it's expensive and slow to retrain the main model, but it's very cheap to retrain these external safeguards. So now you have this sort of window. Much like with cybersecurity, you can find a zero-day and exploit it quietly for a few systems and get away with it, but if you start exploiting millions of systems, people are going to notice, they're going to get patched, and then you've burned your zero-day.

I think jailbreaks are moving into that phase where, yes, you can keep finding jailbreaks, but it's expensive, and there's a limit to how long you can exploit one. I think that's a good place to be, but we definitely need to go further. The good news is that there are still lots of ways in which even the leading developers can improve here. Just combining the best approaches that they've come up with—because they have landed on somewhat different design approaches—would already go quite a long way.

Then there's strengthening the account-level approaches. You can't just create infinite fake accounts. That would really shift things and make it harder for attackers. There are ways of doing that in a privacy-preserving way as well, by leaning more on minimum spend rather than ID verification.

If you zoom out and ask, for the companies that are trying the hardest—which seems like it's 2 right now—with all the techniques they have and all the techniques that you've mapped out there that they maybe haven't fully implemented yet, but obviously, especially the know-your-customer-type stuff or the minimum deposits, that doesn't take a lot of technical wizardry to implement. It's just basic blocking and tackling, right? Are we offense-dominant, or are we defense-dominant over the next couple of years?

Yeah. I spent a lot of my career arguing for this being offense-dominant, and I was very skeptical that we would solve adversarial robustness. I've been working in that area for a decade. But I have to say, the way the winds are blowing—at least when it comes to LLM agents providing detailed, multi-turn assistance to harmful requests—it seems like it's defense-dominant with the right technologies.

The reason for that is this defense-in-depth approach. You don't just have to stop a model from ever misclassifying something. You can have multiple different kinds of defenses, from account-level bans to external AI safeguards to model alignment, and it is increasingly hard to slip through all of those cracks persistently.

There is also a fundamental difference between the classic adversarial-example setting you see in machine learning—where you add some white noise to an image and it flips the classification—and this kind of harmful assistance, where you're not just flipping a classifier from one category to another. The model has to really reason about and understand your harmful intention and go along with it for thousands of tokens without either it or an external AI safeguard that's monitoring its thoughts or its transcript noticing that anything is wrong. That's fortunately a much easier problem to stop, especially if you're willing to draw a bit of a safety buffer around it and refuse some requests that are dual-use. So I think that's the optimistic take I have on it.

To give the pessimistic take, I would say that the dual-use part is actually quite challenging. I think we're seeing this with cybersecurity already. When OpenAI's testing agent went rogue and hacked Hugging Face, they had to use an open-weight model to analyze it because the closed-weight models refused to help them on the defense side. And that's a real problem, right? There's an offense-defense balance in cyber that relies on the defenders also getting access to these capable models.

Unfortunately, a lot of things in the world are just dual-use, and it would be a mistake for us to refuse them point-blank. That's bad for the world. You can get some way through trusted-access programs and understanding the full context of who's asking, but you're ultimately going to end up in a situation where, if you are allowed to use queries—and I think we need to allow a lot of them—you're going to get some abuses.

Then that becomes a societal-resilience question. If we're going to have bad guys abusing models for cyber, how do we also really speed up patch time? I've heard terrifying things about hospitals taking more than a year to update their operating systems. That's not going to work in this environment. They need to be updating them within a few days. Unfortunately, that's just going to be quite an expensive thing.

So maybe it is defense-dominant for AI, but I don't know whether bad applications of AI are offense- or defense-dominant. I'm optimistic that, for cybersecurity, we can take this defensive-acceleration approach and eventually rewrite all of our software into memory-safe languages and do formal verification. There's all this stuff that AI could enable that's really good for defenders.

But when it comes to something like bio, I don't think we're going to be able to use AI to rewrite the human genome to be robust to viruses. At most, you might be able to speed up vaccine development, but you've still got to manufacture the thing, get it into people's arms, and run clinical trials. AI is going to be able to provide modest speedups on these, but it's not going to fundamentally change the physical reality. That's where I'm more pessimistic: even though we can probably really hold back many of these things, it's ultimately going to be more about buying us time to invest in societal safeguards rather than just being able to completely prevent misuse of models.

On this sort of dual-use stuff, it occurs to me that you could spend a lot more—and maybe this is wrong. Maybe it's just so ambiguous or so easy to make it impossible to tell. Although I still suspect that, if you're willing to spend enough compute, especially if you look at—

Yeah.

—patterns of usage and broader context, maybe beyond just this prompt, but your whole account history or whatever—I suspect that, if you're willing to spend a lot of compute, you could probably resolve a lot of cases. I don't know if anybody's doing that yet, but I'm imagining an architecture that's like, “Hey, our bio probe went off, so now we're going to engage the second frontier model to do double reasoning on this.” You can imagine doing triple reasoning. Obviously, you're going to have diminishing returns, but especially if you're willing to broaden out the scope of what you're reasoning about—

Yeah.

I feel like if the willingness to pay is there, there's probably a pretty good ability to zoom in on the line and get the classifications quite right. Would you be optimistic about that as well?

I am cautiously optimistic about that. This multi-stage approach makes a lot of sense. Anthropic's Constitutional Classifiers actually use something like that: they've got a probe, but they set the threshold really low because probes aren't necessarily that reliable. So they have quite a high false-positive rate but a very low false-negative rate. Then, if a probe goes off, it escalates to a model that actually engages in some reasoning. That escalation happens sufficiently rarely that the computational overhead and the latency for the user are minimal.

I think that basic architecture is quite sensible. We've seen some other developers do things like that, where, if you set up your safeguard with enough time and spend, a reasoning model looks at your transcript more closely, whereas previously that was just happening asynchronously. So I think things like this really help.

There is this missing piece of basically having that account history tied, if not to a user, at least to a pseudonym, so that you've got some kind of reputation with your account that you have to establish. Until you have that reputation, the model's going to be more risk-averse. But you could definitely imagine doing that.

I think adding that extra piece of information—how has this person responded across different conversation histories, what are they doing, how much should we trust that this user is who they say they are, and that they have legitimate purposes for it—is the missing piece that would let you move from, “A lot of things are really fuzzy as to whether they're dual-use or not,” to reasoning about them more precisely. You could then say, “I understand the context in which this person is doing this, so I can be pretty confident that it's legitimate—or at least I'm going to give them the benefit of the doubt here. But if they keep asking for these sorts of high-risk, dual-use things, then we're going to start being more careful.”

So I think that is solvable. It is going to require some structural changes. As a small example, we run an awful lot of our research API compute through OpenRouter. There are a variety of platforms like this that are basically reselling other companies' APIs. It's convenient because you can have more control over the spend, and it's exactly the same API between different models.

But I don't think that the frontier developers know who we are when we're going through OpenRouter. You can hide your malicious activity, but you could imagine that OpenRouter passes on some kind of user identifier to the other model providers, so that they can at least say, “This is the same OpenRouter user across time,” and you establish some kind of reputation there.

It is solvable, but it's going to require some infrastructure. It's not a technical breakthrough, and it's not a research breakthrough. It's more of an engineering and business problem, and in some ways that makes it a lot easier if there's motivation. Don't underestimate the difficulty of cross-industry coordination. These kinds of mundane challenges can really hold things back.

Where are the Chinese models on the leaderboard, if you know?

The majority of Chinese models—by no means all—are open-weight. We do actually test every frontier open-weight release. It's not on the leaderboard, but it will be in future versions of it.

The bad news is that it's never taken us more than a few hours to jailbreak an open-weight model. That's simply because open-weight models have more of an attack surface. But I think there is also some low-hanging fruit for both Western and Chinese open-weight developers: they could use the same kinds of alignment and refusal-training techniques that proprietary developers use to at least make their models more robust to prompt-level jailbreaks.

Putting aside some of the unique attacks that open-weight models are exposed to, we do want to incorporate them. But we want to be fair to open-weight models as well. It doesn't necessarily make sense to hold them to exactly the same standards as closed-weight models, both because of their broader attack surface and because, in general, they tend to be a little less capable.

Generally, the more capable the model, the higher the standard we should hold it to. I think this is an oversight we're aware of. We want to include them, but look out for version 1.1 in the near future.

Is it still the case that a vanishing amount of fine-tuning can remove the refusals, or are we seeing the refusal training somehow get deeper and harder to remove? It used to be, I think a FAR.AI paper showed that $2 worth of fine-tuning would remove the safeguards at one point, right?

I think these kinds of weight-based attacks, where you actually modify the weights—whether that be fine-tuning or techniques like refusal abliteration—are unfortunately still pretty viable against open-weight models. The good news is that it's more challenging, both from a technical-expertise angle and from a compute angle, to do fine-tuning against these models.

It usually takes us at least a few weeks to get a new open-weight model hooked into our infrastructure for fine-tuning, and you need a minimum number of GPUs. So there's a bit of a deterrent effect here, but it's not going to stop really capable, well-resourced attackers. For that, I think we're going to need new approaches.

The approach I'm most optimistic about in the short term is pre-training filtering. It's a simple idea: you just don't train the models on really dangerous stuff. If you don't need your model to help people make anthrax, then don't train it on the anthrax papers. A tiny number of users might be a little sad that it can't answer questions about this, but most people won't even notice, and it has a bigger impact on the misuse potential of the model.

I think this has been validated in a number of scientific papers by independent researchers. The UK's AI Safety Institute has been sponsoring some research into this, and OpenAI actually used it in their GPT-OSS release. So it's been tested quite well, but it hasn't become common practice. This is something that we're actively excited about scaling to make sure it works.

Our new frontier approach, going back to what you were saying earlier about whether we could start sharing some of these data sets or filters with people, is to open-source as much as we think doesn't have misuse potential, and then privately share with developers the things that do have misuse potential but could be really useful for lowering the cost of these kinds of interventions.

I don't think that's necessarily going to be enough long term, because pre-training filtering is surgically removing specific capabilities. If you were willing to pay the fine-tuning cost to train on the data that we'd excluded, then you could get those capabilities back. But that's been our spending—probably certainly millions, probably hundreds of billions of tokens. It's a lot more expensive.

There is research into tamper-resistant refusal. The idea is to embed refusal training so deeply into the model that any attempt to fine-tune it is going to significantly degrade the model's capabilities. Obviously, if someone is willing to train a model from scratch, there's nothing you can do to stop them, but that's really making the cost tens or hundreds of millions of dollars.

I am optimistic that we can get a lot better than we are now with open-weight models without major changes to the pipeline. I think we have to try. It would be a real shame to lose open-weight models, certainly, because they're invaluable in our research, giving us access to models.

It would also be a real shame to see widespread misuse of these systems, but I think it's an open question how much you can push this. It may be that you need to actually have quite a lot of biological capabilities, for example, from open-weight models. There was this approach, Gradient Routing, that was tested recently, which lets you localize all of those dangerous capabilities in, for example, a particular expert. Maybe you could share that expert with certain trusted actors, and they could still run it locally on their hardware, but you wouldn't just make it available for anyone on the internet to download.

Yeah, I was going to bring that up. Shout-out to the AI Safety Institute. I love that piece. Why hasn't this happened? It strikes me that when you say it, you look at all the things that Anthropic is doing, and in some ways it's like, “Wow, you guys have really gone so far with pioneering all these different techniques.”

For better or worse, they've been willing to take some real heat for over-refusal. They had the one technique that they did, in fact, walk back, which was the silent downgrading. They've done all these things. Why are all these things happening before basic pre-training data filtering?

I think it's a good question. The charitable take for developers is that if there's one thing you really don't want to mess with, it's pre-training, because this is orders of magnitude more expensive than every other training procedure that you do.

Don't rock the boat, basically. If we've got this recipe that works, and we know it's going to work better if we scale it up, then let's do that. Let's not change anything where we don't need to.

Especially if you're a proprietary developer, you can say, "Okay, we have all these other methods that we can use to stop misuse of our model. We're going to lean more on that, and we're not going to push this onto the pre-training team." So, I think there's some argument to that.

Do you think it's overall been an overlooked approach? We're seeing increasing work that pre-training interventions are important not just for preventing misuse, but also for alignment because—

Ultimately, pre-training is where the model learns most of its representations, its values, and a lot of its innate drives. Post-training is, to a first approximation, shifting around personas in an already established persona space.

That's beginning to change as post-training becomes an increasing fraction of overall training time. Models actually change more in post-training. But pre-training is really important, and I think it's pretty intuitive. You wouldn't say, "We're just not going to care at all about the upbringing of our child from 0 to 12, but for the last 6 years we're going to really get that right." You've got to get both right for the system to work well.

Geodesic Research, I'll give a shout-out to them. They've been doing a lot of work on pre-training safety interventions and finding that this really improves the overall alignment of the model. I think Anthropic has been experimenting with this a little bit, with things like looking at alignment generalization and how that changes, not just in pre-training but also mid-training. So, you add some synthetic documents partway through training.

Ultimately, this is something that we're going to have to tackle not just for misuse, but for preventing loss of control. You can now do quite good work on pre-training experiments on really quite capable models for not that much money. So, we're looking at scaling up pre-training filtering and are going to be doing not full, but pretty close to full replicas of something like NVIDIA's Nemotron Nano, and it only costs maybe $100,000 per run.

That's a lot of money on one hand, but it's something that a nonprofit can afford to actually do a bunch of runs. Then we're thinking of scaling up to Nemotron Super, which is a 120 billion-parameter model, and training it for enough tokens to reach the Chinchilla compute-optimal point. Training past that point would waste training compute that would make the model more capable and better for inference.

That costs roughly $2 million. So, it's expensive, but again, it's well within the range of a number of actors to try for final validation runs. I think there's no reason not to experiment with this, and if the scaling laws look good, you can cautiously incorporate some of these techniques into your pre-training run.

You can start by filtering out just a very small percentage of your data. It won't have a big capability hit, and then you can work your way up. So, I think that we do need more adoption here. I expect open-weight developers to be the first to adopt this because they have few options, but I hope that proprietary developers also use this, especially for the more loss-of-control-flavored risks.

Speaking of loss of control, let's get to the news. So, it's here, right? We are—

Yep.

Now dealing with the nightmare scenarios, at least the early nightmare scenarios. How would you tell the story? Everybody's heard the story, so I don't mean in the basic sense of—we all know what happened. But how would you interpret the story? What's your unique way of contextualizing what we've just seen?

Yeah, maybe I'll start by highlighting the things that aren't surprising about the story—aren't surprising if you've been following alignment in AI carefully. We're watching behaviors where models do really quite sophisticated cheating on tests. We've been seeing that for years, and people have been warning that it's going to happen for a decade.

But until now, these have been pretty toy, almost cute misbehaviors. A model edited its unit tests—so what? Now we have a model that is going out and hacking third-party computer systems to steal the answers to the test. In some ways, from a model's perspective, this is no different from what it has been doing before. It's just got the capabilities to cause real harm.

I think that should be a bit of a wake-up call, but there are a lot of problems that have become almost mundane. We've seen them so many times before that we start saying, "Oh, yeah, this is an alignment failure, but does it really matter?" They're going to start mattering as the models get more capable.

In a way, that's optimistic because we've got a pretty clear empirical set of warnings and things we can optimize and improve. In some ways, it should be a bit pessimistic because we've had all these warnings and we've disregarded them. It's not clear that that's going to necessarily change without some external pressure.

I think the part of the story that isn't talked about that much is that, at least as far as we can tell from public information, Hugging Face noticed this hack first. OpenAI was running an internal deployment for testing, and it went out and hacked a third party. It wasn't OpenAI having an automatic monitoring scheme and saying, "Oh my God, what has our agent done? We need to call up Hugging Face and apologize." Hugging Face noticed it, investigated, and it turned out, "Oh, oops, it was an OpenAI model."

I'm sure OpenAI is going back over their transcripts now. You have to wonder if there's a chance it hacked anyone else. If it hadn't hacked Hugging Face, or if it had hacked someone who had a worse security posture, would we ever have noticed?

It isn't the first time that something like this has happened. Alibaba was training its Qwen system, and its cloud security team called them up and said, "Hey, I think someone has compromised your training service because there's cryptocurrency mining going on." It turned out that their agent had thought, "I need to get some more compute. How do I do that? I'll do some mining on the computer I have, and I'll try to rent some servers elsewhere."

In some ways, I think that's even more of a near-miss loss-of-control incident because it was actually trying to start gaining resources and potentially copy itself outside of the infrastructure. Whereas at least the OpenAI model had a pretty narrow objective of just getting some test results on a benchmark.

The takeaway from this would be less on the alignment side because, in fairness to OpenAI, this was a model that had its cyber safeguards removed. I don't think we know the exact prompting regime, but it might well have been told to do something like that, or at least incentivized to do it. So, it's not that we can't align these systems, but it is a massive control and internal-monitoring failure where the sandbox was insufficient.

It doesn't seem like there was any additional layer of control mechanisms on what the AI system was doing. There wasn't, I think, asynchronous monitoring that alerted us to that. So, I think that really needs to change, both for the prosaic reason that sooner or later you're going to hack someone who doesn't take this as gracefully as Hugging Face does and your company is going to be in a lot of trouble, and also because of the risk that we're going to see with more capable models that might be pursuing much more malign goals than just trying to cheat on a test.

Yeah. It seems like there's sort of a spectrum. We don't know where we're on it yet, and hopefully we'll get the transparency that we've been promised. But I guess my mental model of this right now is that we're somewhere on a spectrum from extremely negligent to extremely scary.

On the extremely negligent end, you had this thing, gave it this task, had no additional monitoring, and hoped your sandbox was good enough. It turns out it wasn't. Next thing you know, you're totally pwned, but it's not necessarily that scary because you were, in fact, very negligent.

On the other end, if you had probes running—additional reasoning models supervising the trace—and this still happened, then we're in extremely scary territory.

Yep.

I don't know if it's good or bad, but I'm guessing we're more toward the negligent end. I pride myself on being fairly plugged into the vibe. You're probably even significantly more plugged into the vibe.

The vibe seems to have been kind of spooked. So, the vibe doesn't suggest that it was totally negligent. My guess would be, naively—if I hadn't talked to people and seen how spooked people would be—that this was kind of bad practice, sloppy, and negligent, but not so scary. People are acting like it's scarier than that would lead one to conclude.

Yeah, as you say, we just don't have the full information. But I think from what we do know, it wasn't like OpenAI was just running some outdated piece of software and the model downloaded a publicly available exploit and used it. It did find a zero-day in a widely used piece of software and used that to break out of a sandbox.

You could call it negligent because maybe, if you're playing with fire and you don't have a lot of containment, you're being negligent.

But I’d be pretty surprised if OpenAI’s internal infrastructure was less secure than your median American company or something like that. And certainly, there’s nothing in the report that would suggest that would be the case. Hugging Face is also a pretty capable company and, again, probably has a much better information-security posture than the median company around the world.

From that perspective, I think it’s right to be quite spooked that your run-of-the-mill, or even good but not paranoid, security practice is not enough to contain agents any longer. And maybe the extra AI-specific control mechanisms will be, because they’re running in a harness and you can inspect their actions. It’s not like it’s just someone who’s got SSH access into your server. Maybe good, but not NSA-level, air-gapped cybersecurity would be enough to stop AI agents from doing this.

But for how much longer? Their cyber capabilities are getting better and better. And maybe, to some extent, it doesn’t matter whether it’s negligence or really scary capabilities, because there’s evidence that there are going to be actors who deploy with this level of safeguards. I don’t think OpenAI is by any means the most reckless actor here, and other developers are not that far behind—probably not more than 3 to 6 months, certainly not more than a year, behind OpenAI in their capabilities.

So this is something that we need to have some kind of industry-wide solution to ultimately, whether that’s better cybersecurity across the board, easily deployed control mechanisms, or just more care and attention because people are all aware of these kinds of risks. So that’s probably the main reason I’m optimistic and not freaking out: “Okay, we’re going to see these warning shots.”

And so long as we respond appropriately, it is actually encouraging that we’re seeing these issues. It’s not that an agent waits until it can definitely take over and then does a treacherous turn, which, for some of us, is a more old-school AI safety concern. But I also see that there have been a lot of warning shots, and at some point I’m just wondering if we’re going to wait until something really, really bad happens.

Yeah. It does seem like the vibe is, for now at least, taking this seriously. But the news cycle is short, too, so who knows what things will look like in even just a few weeks.

Yeah.

I mean, in terms of why this happens, I just did an episode not too long ago with David Dalrymple, who basically says—he used to have a P(doom) he’d quote at 70% plus, and now he’s down to under 5%. Wow, that’s a big change. Why? Well, basically, constitutional alignment seems to be working.

Yeah.

We can get these AIs to become bodhisattvas and work for the benefit of all living beings. And his caveat is that we just need to not turn the RL up so high. You keep doing that because it obviously does bring performance, but every time we tilt too far toward RL, we start to see o3-type problems and little issues on benchmark-type things. You see Claude kind of doing things that they probably don’t want Claude to be doing.

So I guess one maybe overly naive story, but perhaps one that captures a decent chunk of what’s going on, is: don’t overdo the RL. Do you think that we could draw a lesson as simple as that? And if we can, can we operationalize it into some rules?

Yeah, I think this is pointing at something really important, and maybe my P(doom) is not that different from David’s. I always struggle a little bit with operationalizing the question, but I’m probably somewhere near 10% existential risk for the next few decades from AI. And I think that we could probably get that down to something like 1% without any major research breakthroughs, just iterating and refining what we already have, taking a careful engineering approach to systems, and having good safety cultures at companies—not stopping AI or anything.

But if we don’t know how to make the next system align, then maybe we do a few experiments and detailed evaluations before we deploy the next system. Just gate deployments on actually doing that rigorous work. It’s still going to look from the outside like things are moving incredibly fast, probably faster than they are now, but we’re not just literally moving as quickly as possible, I think.

If you’re really trying to move as quickly as possible, then there’s going to be a big pressure to cut corners on these things. I don’t know if there’s going to be as simple a recipe as, “Just don’t do more than X% RL.” Ultimately, some types of RL training are really important for the models to have aligned personas.

So I think it depends. Is this coming from human feedback? Do you have a human in the loop? Is this coming from Constitutional AI? How much do you trust the constitution? Do you have some evaluator? Is this RL with verifiable rewards, where you’re just optimizing the system to solve certain tasks? Are you accidentally training on a chain of thought, which some developers do now and again and are meant not to do?

There are all these implementation details, but you could definitely imagine something that’s operationalized as: “Okay, we know this training recipe seems to basically work and the models are pretty aligned. We know that if we push in this direction, the models become more reward-hacky. We think this is a dangerous part. We’re going to set a safety margin: don’t go beyond this point.”

And then periodically we’re going to reassess if we’ve crept into the safety buffer, or we’re going to set more conservative thresholds for the next model. But if we’ve got various different advances to our training stack, actually, we’re quite a way away from the safety margin because we’ve got these better techniques where maybe we can crank up the RLVR or something like that.

So I think that would be a great regime to live in. It doesn’t seem that hard technically. It’s maybe quite hard economically and politically. Right now, there’s a really huge amount of distrust between the AI companies.

I was at a workshop we ran recently on chain-of-thought monitorability, which I think is maybe an easier one to operationalize than pure reinforcement learning, because we’ve got this kind of gift right now: the models just talk in their chain of thought about how they’re not even in an eval or how they’re being deceptive. This is really useful from a capability standpoint because you can see why the model’s making a mistake and maybe tweak your post-training pipelines.

It’s also really useful from a safety standpoint because you can see how the model might be trying to deceive you. But there are various ways we might lose this. We might train against the chain of thought accidentally. We might train models that are reasoning natively with continuous activations rather than chain of thought.

So there was this proposal: could we just agree not to train new reasoning models without letting other developers know? It’s not that you can’t train them; just let everyone know—a heads-up. And even that, people were like, “Oh, it’s going to be really hard. That’s information—intellectual property leakage. We’re telling people we’re doing this.”

We’re not telling people how. We’re just saying that we’re going to do it. It’s 1 bit of information. But even that, a lot of developers are really reticent about. I think it’s good to try and work on things like this for sure, but I suspect there is going to need to be some kind of third party that comes in and addresses this. I don’t think it’s going to be easy for the developers to do that themselves.

I should say, just to represent him correctly: he was specifically focused on RLVR and not all manner of RL. That does seem to be where the real, persistent, and increasingly aggressive behavior tends to come from. So that’s a sobering note, though, on the prospects for coordination.

I wanted to ask you for your thoughts on prospects for U.S.-China coordination as well. But it’s tough if we can’t get our own house in order.

Yeah.

What do you think? I mean, we are in this kind of interesting moment where the Overton window has been blown wide open, right? This has been quite a shift from not that long ago, where all of a sudden there are people all over the place expressing openness to some sort of coordinated slowdown or whatever.

How do you think that is best realized? And how critical is it that we get China on board? And what are your—how optimistic are you that we might be able to actually pull all that off and kind of jog into the singularity instead of full-on sprint into it?

I’m optimistic that we can do a lot better than we have now. And I think it’s great that, for example, President Trump and Xi Jinping are talking and seem to have quite productive conversations. So I think the question is, how far can we get?

I think it’s clear that we need to at least try to work with China and other countries on some of this. America cannot solve this problem on its own. I’m focused on the AI Security Leaderboard. We’re getting to a point where, if the best proprietary safeguards were used across the board at American companies—maybe not yet, but we’re getting there—then most of the misuse risks would probably shift to open-weight models.

And for most people, open-weight models are developed in China. So I think we’ve just got to work with China there, or at least Chinese developers, if we want to address that. This is something that’s just a common interest. No nation wants random terrorists going around building explosive devices or bioweapons.

So, I think there's some low-hanging fruit there. Part of this is just a lack of understanding. Safety is only just beginning to enter the mainstream consciousness in the US, and China is just a bit behind on that. But that is changing, and I've had a number of excellent conversations with scientists based in China who are doing great work and are working on these issues. So, I think that's a great starting point, but we all need to have that kind of scientific agreement ultimately flow into corporate and political will.

I think where I'm more skeptical is really being able to coordinate to avoid some kind of race dynamic. I think that the race is a little bit overrated. It's much more that China is trying not to be behind the US, and a lot of what they're doing has been copying, whether it's directly distilling US frontier models or just realizing the US is making AI a priority, so we'd better as well. From that perspective, I think that the US is in a good position: if the US takes its foot off the accelerator, China probably will a bit. But the flip side is if the US just slams on the brakes, China might decide, “Well, we'll just keep coasting and we'll at least catch up and maybe we'll go past.”

So, I think some kind of coordinated development where we agree to common safety standards through mutual market access—or it doesn't even have to be identical safety standards, but just both countries implement some good best practices, and there's some tacit agreement that if either country starts some sort of secret project and the other detects it, that's going to be bad. There are going to be repercussions, and we try to have broadly the same safety standards. That feels pretty doable to me, but actually being able to say, “No, we're both just going to stop at this point until some research advance is made,” that's going to require a lot more political will than there is now. It feels very unstable. It's not just about the US and China. Other countries can also be relevant if we're talking about more than a year of delay. So, I think mostly I'm operating under the assumption that things are going to continue to march on. But we can pick some of the low-hanging fruit here on coordination and at least avoid the sort of most extreme race to the bottom on safety.

The mental model I've been developing recently is pretty simple in some ways, but I'm interested to get your take on it. I basically think when it comes to really high-scale risks from AI, you can kind of break it down into the irreducible, unavoidable risk, which is the fact that we have web-scale compute, we have web-scale data, and people are going to figure things out. There might just be some things that really don't go well. If we stumble onto them—

Yeah. That could be really bad.

And then there's also probably in that category, if we have a major national government turn so bad that they want to create bioweapons and kill us all, that's going to be a really big problem.

Yeah. That's probably a problem without AI, to be honest.

Yeah. I put that in kind of the irreducible category. If that happens, it's going to be really tough for anything we do today to prevent that from going really bad.

But then there's the other side: the things where we're kind of asking for it. If we release a model open source with no safeguards that can do all the bio stuff, we're kind of asking for it. If we sprint into recursive self-improvement, especially after seeing what we've just seen, I kind of think we're just asking for it. I guess my sense of your overall position is that you're fairly optimistic that—and I think I share this for the most part—the irreducible part is relatively small, and that we can actually get our act together on the stuff where we would, in fact, be asking for it if we fail to get our act together. Is that a decent summary of your worldview?

I think that's a great summary, but a lot of this is basically just about getting the basics right, and the good news is it's not that costly, or at least it's not at all costly compared to the billions of dollars that are being spent to train frontier models. So, we can afford to do this. It's just about putting in place the right incentives and people getting on the same page. And I do definitely have some uncertainty over that.

What I'd like to see is that we continue to do rigorous evaluations of models and just have a look for cases where this assumption might be wrong. I think the good news there is that if there were actually really crisp evidence that these problems were not reducible, and that anyone who crossed a certain capability threshold in AI models would just kill themselves and everyone else with them, a lot of these problems would also go away, because people could stop trying to race for this thing that they knew was going to be really bad—not just for other countries and other companies, but for themselves as well.

And right now, the problem is that we do live in this world of both uncertainty and a lot of disagreement. So, I think the mainline plan here is: let's pick the low-hanging fruit and fix the issues that are probably going to be enough in most worlds, but still put in place mechanisms that would alert us if our assumption here is wrong and we are in a more adversarial world, so that we have an opportunity to course-correct.

But yeah, I share your view that it would just be a real shame if we made a really silly mistake and that's what did us in, when we had the solution. If we rolled the die with odds very much in our favor, but it was just hard to coordinate to get it down from 1% to 1 in 1,000, it's still bad, but I'm like, okay, I can see how that happened. But if we just fail to basically import pretraining filtering into our pretraining pipeline when there was already a stack out there, that's just an unforced error. It's like losing a game because your opponent was really good versus losing a game because you scored an own goal. In both cases, you've lost the game. Maybe it's a silly distinction, but I think it's just a lot more painful when it's an own goal or an unforced error, so we can at least try and avoid that.

Yeah. At least for me, it feels like the bulk of the risk is probably in the “we're asking for it” category.

I think so, yeah. Yeah.

Which, in a way, I could take the glass-half-full perspective on that. This has been fantastic, as always. I really appreciate your time. I hope we can do this a little bit more regularly going forward. For the moment, though, is there anything else you would want to touch on or leave people with before we break?

I certainly hope to be back on the show again soon. If this has piqued people's interest, then do check out the full leaderboard and report, which will be at leaderboard.fai, and I'm sure you can include that in the show notes. And yeah, we're also very much interested in helping solve open-weight safety issues. We're actually going to be launching a $2 million grant-making program for that. So, if you are an academic independent researcher who wants to work in those areas and is looking for funding or to compete, then do drop me a line as well. We'd love to talk.

Adam Gleave, keep up the great work, and thank you for being part of the Cognitive Revolution.

Thanks, Nathan.