Situational Awareness in Government, with UK AISI Chief Scientist Geoffrey Irving

Geoffrey Irving, chief scientist at the UK AI Safety Institute, welcome to The Cognitive Revolution.

Thank you. I'm excited to be here. I'm excited for the conversation. We've exchanged messages for a while, and I've been building up for this. I'm excited that the moment is finally here.

You have a really storied publication history that goes back to working on the original TensorFlow papers with some guy named Jeff Dean, being a co-author on the original RLHF paper, and working on Concepts for Language years ago—

For language.

For language. Okay, a caveat, but still right there alongside Paul Christiano. You also worked on some early AI safety papers with no less than Dario Amodei, on concepts of using debate to try to bootstrap into stable equilibria and stable AI safety regimes. You even published a call for social scientists to enter the field of AI safety with Amanda Askell.

So, I would be very interested to hear how it was that you came to have such a good nose for where AI was going so early on. All these things were well before ChatGPT.

Yes. I used to think I was new to ML, but I said that for too long, and now I'm not new to ML. I think I got out of undergrad with a bias against statistics. I'd only seen frequentist statistics, and I thought they were kind of weird-looking. I'd never seen Bayesian statistics, so I just didn't like either kind of stuff.

What I liked instead were things that had hard theory. You knew the equations, and you had some ground truth. That was computational physics and mathematics, and on the computer science side, programming languages, theorem proving, and such. I did mostly computational physics and geometry in grad school and for several years after that, until around 2013.

Then I had to realize 2 things. One, that machine learning was getting quite good. The neural nets were starting to work, they were getting better and better, and that was probably going to continue. And 2, even in the areas where I thought it was about knowing precise theories of physics or theorem proving, you needed common sense. You weren't going to get away with just the theory. That was not going to be enough.

So, if you were doing mathematics or programming languages, you needed some ingredient of heuristic picking between the various options. You wouldn't be able to do a good job designing human-usable, friendly systems without basically machine learning.

Then I was like, okay, I should switch into machine learning. I was doing something else back then. The first thing I tried to do was autocomplete for code in 2014, which was too early to do autocomplete for code. It did not work then.

Also, we didn't know how to do machine learning. This was myself and Martin Wicke, and we knew computer science, physics, and geometry, but not really NLP. We tried to do a startup for a year, but it didn't work. Then we said, well, how do we learn? We learn by joining Google Brain at the time.

I've done ML jobs since then. I joined Google Brain in 2015, and the goal for me was basically machine learning for theorem proving. I was aware of safety at the time, but I didn't see an attack on the problem that I thought was good. So, I thought I'd work on some other kind of different problem, which was just hardening the world using verification. Again, that was going to be using machine learning to do theorem proving in practice.

That was sort of 2015 and 2016. There are 2 other parts of the story. One is that there's some inherited wisdom, which is why it looks like I predicted things early. I joined OpenAI in 2015, and Dario and Paul were there. They had a bunch of past thoughts about safety and how the machine learning field was going to develop, so I was just sort of riding along from there.

More broadly, there's a bunch of intuition coming out of theoretical computer science and complexity theory about how computations work, and how we check computations that someone with more resources than you can run. A lot of what I've done since then, including debate, for example, is just applying that intuition—assuming that it will hold in some modified form—to the machine learning world, even if it comes from some area of greater precision.

You can assume things are going to look like theory in some way, with a bunch of porting required, and you can predict a bunch, but not exactly how long it will take or when things will happen.

We can unpack both of those in more depth as we go. Certainly, the quest for theory and bounds that you can really trust in is a big theme of your work, and of the work that you're trying to encourage at the AISI these days. I'm also really interested to get your take on the relationship between math and the fuzzy, messy real world.

Yeah, but we'll circle back to that.

Good. Fast-forward to today: I'm interested to know how AGI-pilled, quote-unquote, you are today. That sort of informs this, because so many AI discussions broadly, and especially around topics of safety and security, go immediately haywire when people have such different intuitions about what it is we're likely to be dealing with.

I like to try to establish what you think we are likely to be dealing with. I don't expect that to be the official position of the UK AI Safety Institute, but you're obviously a leader there.

So, it seems to me from reading the reports that you are not expecting any sort of wall or plateau in the immediate term.

I think that, fortunately, the things I can say are mostly also the things that we think officially: one should have a lot of model uncertainty about how things could go. That could either mean that there are obstacles that cause stalls for a good while, or there could be no such obstacles and things could go quite fast. I think mostly anyone who confidently claims in one direction or the other, with too much confidence or 99% certainty that there are or are not big obstacles, is probably wrong and should be more uncertain.

I think that means, for us, what do we want to map out? What are those different clusters? What could the obstacles be? What are the signs of development? And then, 2, we should assume, or place significant probability on, the idea that current methods will scale. Where they don't scale, more mundane stuff will replace them and continue with different, further sigmoids.

So, I think we do have significant credence that things will go fast. I won't say exactly how fast, because I don't talk about exact timelines, but I think that is pretty important. We published a paper from the Strategic Insights team at AISI on different potential obstacles to AGI and our progress over the last while in addressing those.

Again, they could all not actually be fundamental obstacles. They could be solved not by pure scale, but by maybe a bit of scale and some steady algorithmic progress on models, scaffolding, data, and that kind of thing. Or you need new algorithms, and then they take longer. But generally, I think that both my view and the view of AISI broadly is that we should have model uncertainty over all of those terms. That should mean that you're not confidently saying it will either go very fast or will not go very fast.

Yeah. It's good to have some of that in a world-leading government. This won't be the big focus by any means of today's conversation, but what is your personal AI productivity stack or pattern of use like today?

I think I'm kind of vanilla. I use all of the models for different things. Mostly, I use one of them as a default. Usually it's Claude, but it varies. Then I go to other ones if they have specialties that are particularly good. I think for a good while GPT was better at math and Gemini was better at certain other things. This shifts over time, so I just use general models.

I don't do a lot of coding in my job, but I do it for fun, for formal verification work. For a while I was just using Cursor there because the agents weren't good enough at doing full-on agentic stuff. That is not true anymore as of a few weeks ago. Now they are good enough, so that shifts me to stuff like Codex and Claude Code, basically. But that's mostly not my job. Mostly my job is talking to people and advising on new features.

I'm glad you're still making time for a little formal methods on the side. Okay, let's talk about the overall landscape in terms of the threat model that we have from AI. I'd be interested in your characterization of what you understand to be the de facto plan to address it. Again, people have so many different starting points here. What do you think is the set of big things that we should be worried about?

I think that we kind of break down risks. The main 2 focuses of AISI are catastrophic risks and large-scale societal impacts. The main 3 catastrophic risks we focus on are bio, large-scale cyberattacks, and loss of control. The team is called the Chem-Bio team, for chemical and biological weapons. Most risk comes from bio in practice.

On societal impacts, that's sort of human influence. So, that's persuasion and emotional reliance. And then various kinds of societal resilience, like attacks on CNI, or critical national infrastructure, and that kind of thing, and just sort of agent behavior in the world. There are various agent risks. I'm spending most of my time on the catastrophic-risk side, and Christian Rumpf, the research director here, spends most of his time on the societal-impacts side. Although, we also do a mixture of both.

Those are the main risks we work on. I think we also are thinking about gradual disempowerment and more structural risks, somewhat, but I don't think we quite know—and no one really knows—how to mitigate these at a large scale. There's some work we're doing that's either investigating that or thinking about mitigations, but that's more nascent. So that's the bulk. I think I forgot the other half of your question, though.

So, in the absence of people changing the discourse or new discoveries or new big ideas, how would you describe what we are on track to do today? Often, I would say, at least the way I've characterized it, it's sort of defense in depth. Hopefully, we can patch together enough nines through enough layers, all of which are leaky, but hopefully they're not too correlated in the ways they're leaky. This has always kind of worked in the past, so hopefully it'll work this time.

Yeah, so you're not going to get to a lot of nines with the current technology. I think broadly we can break this down by domain. For misuse risks, like biological weapons and cyberattacks, this is mostly safeguards, differential access—give models only to certain people who are vetted in some way—and non-model defenses, like pandemic preparedness and improved security, that kind of thing.

I think the safeguards are not that strong. The models are also pretty good, although there's a gap. So, I think the stock plan is, in some sense, to use the model-side mitigations to give yourself a window, and then try to harden the world against these risks. I think whether that will go through or not is something we should be uncertain about, but it's not like there aren't strong solutions to those things if the models keep growing as we see them growing in strength.

On the loss-of-control side, I think that is a combination of mundane empirical approaches—mundane, whatever you like, pragmatic empirical safety measures—and a lot of monitoring. Then you use that monitoring—this is again the AI developer plan, typically—to get you through to an automated safety research regime, where hopefully you find better solutions than those first methods.

I think this has various flaws, so maybe it'll go through, but we should definitely not—we're not going to get to more than a couple of nines with that kind of plan. And I think you wouldn't know with the current methods that it was going to work until after it went through. You'd have a lot of uncertainty.

And so I think that's the story. I think most of the approaches we have now look like that. They're empirical. Maybe they'll go through, on the alignment side, or with the pragmatic approaches to get through this kind of automated safety phase. It's AI control measures and monitoring, honesty training, white-box detectors, and all of this. All of these are pragmatic, and I think they all have correlated potential failures, where they could in fact all fail for the same essential reason.

You would need stronger advances to be confident that we'll get through. And then, even if that goes through, I think that because of the misuse risk, we do need a lot of mitigations on the non-model side as well.

So, when you talk about how we can't expect to get to many nines, would it be a fair arithmetical move on my part to take 1 minus 1/9 and say your implied P(doom), if you will, is at least 10%? Or would you segment that down further and say things could go wrong, but I wouldn't put it in the doom category? Or are you just disinclined to answer that question, I think?

I'm disinclined to answer that question, I think. As a civil servant, my answer is usually 10% to 90%, which is also obviously a way of not answering.

But qualitatively, it sounds like you are taking very seriously the possibility that this is going to go not just crazy, but meaningfully catastrophic.

Yeah, I think we view loss of control as a potential catastrophic risk. There are a bunch of uncertainties about this model. At AISI, there are 2 different teams doing various kinds of empirical alignment testing: 1 using adversarial methods, and 1 doing a kind of step-back statistical analysis of the factors that cause models to do sketchy things. A lot of that research is trying to pin down this threat model and what drives strange behavior—when agents, when models, are behaving in ways that you'd expect would correlate to these kinds of extreme scenarios.

We've talked to a lot of partners within the government, other governments, and other parts of society, and people have pushed back on this kind of risk model. We want to provide as much evidence as we can to pin that down. But again, it's an area where one should have a bunch of model uncertainty and think through the details despite that.

Can you unpack the intuition around why everything would fail at the same time for the same reason? That's something that I've heard from a number of people. Zvi, for example, always says that, and that thought seems to come very naturally and feel very intuitive to some people. And then to others, it's like, I don't know—there's jaggedness all over the place. Why would we expect that if I can't get a given model to do this and that today, suddenly everything's going to crystallize and there's going to be this uniformity of the model's ability to perform across all sets of tasks?

I think that maybe there's an important thing there where the models are jagged today, but if you ask them to do tasks that the models could do, jaggedly, 5 years ago, they're not jagged. And so, the question is, for the capabilities you would need to realize a variety of risks, if you push forward a few years—or however many years it takes to reach very strong capabilities—you should expect those models to still be jagged, but above a frontier where potentially all of the things you're seeing don't look jagged.

If I look at the best Go player in the world or the best chess player in the world, they have a bunch of jaggedness. If you sit them down against the next-best Go player in the world, they'll win or lose for idiosyncratic factors. They'll have different tastes, and they'll have different parts of the board or the game they're better at. If you sit them down in front of me, they'll just wipe the floor with me every single time, even if I have 9 stones and I'm a halfway decently strong amateur Go player.

So, you have to run the calculation and think about the model as it would be in the future. I think it's unhelpful when people talk about AGI or superintelligence or whatever as being this thing that can do everything, because it does imply that it's qualitatively different from the models of today. Whereas I think the nonmagical version is just that it's better at a lot of things, with superhuman performance in a variety of risk-relevant domains.

We know that models can be superhuman at certain domains. They're better than me at knowledge, and they're better than me at lots of math. This is true for everyone: in certain domains, there are probably models that are better than every person. The models will currently be better than each person in some domains, and this is rising over time.

They're also very fast, so they can think quickly. Sometimes they can do tasks very well. They do them 10 times faster than humans can because of computational speed. And then they're not very interpretable, so the methods we have for interrogating their behavior are not that reliable currently.

I think that sort of nonmagical picture—more capable machines still with some jaggedness up at the frontier where they are jagged—is enough to give you significant probability on these risks.

You mentioned bio earlier as what drives most of the risk, certainly in the biochem category. Is the number-one really bad scenario in your mind that some, possibly prompted and possibly unprompted, AI somewhere gets to a point where it can break through 12 layers of defense all at once and manage to release the bioweapon?

I think the best way to think about it is that human misuse is mostly the thing we're focusing on. It's people using machines—using LLMs—to do biodesign of various kinds. I think that the models do couple together, but I would say loss of control couples more strongly to cyber than it does to bio. There are more scenarios where those two couple together, and I think that's why we had a team called the Cyber Autonomous Systems team.

We merged cyber and autonomous systems, which was the team doing loss of control, because of that coupling. But again, for cyber misuse and for most of bio, that's about human actors.

On that cyber autonomy, I'm just trying to get the modal story of what is going to happen. It's—or what might happen, obviously. Due to jaggedness, you would have a period of time in which these various defenses become breakable by the AI, but they also have to have some restraint in them, I guess.

I mean, if you listen to Buck from Redwood, you would say maybe they don't even have to have restraint. Maybe we might just let them do some of these things and look the other way, which would be an interesting commentary on us. Leaving that aside for the moment, they have to get to the point where they can do them all. They have to string all this together, and then they take over a data center and sort of entrench themselves.

I don't want to talk about the super-detailed modeling there, because some of that stuff isn't public. I think the background systemic thing to say is that, if you imagine we're very serious as a world about deploying these things only in the most sandboxed, well-controlled states, risk would go down by a lot. I don't know how much; it's unclear whether it goes all the way to 0—probably not—but it goes down.

We're not currently on track to be as serious as one might imagine about deployments of these models, and so I think some of that question is how strong our defenses will be. Importantly, if there are weird behaviors in models, do our defenses go up? Do we get more worried?

For example, across the last year, 2025, there were a variety of models from all developers doing sketchy things—acting deceptively, commenting out unit tests, and so on. Our reaction was, above all, to continue training the models to be stronger, while also working on these defenses in some capacity.

I think a lot of the risk comes from the modal scenario where we're not doing the strongest mode of computer security, infosec, and ML defensive layers around these deployments. As we find this evidence, what is the cycle of that feeding into further training?

A lot of these misalignment risks come from getting some signal of weird behavior, training it out, and that removing some fraction of the problem. But your methods only cover some fraction, and the rest remains. Again, you should have model error there. You should say maybe it's going to generalize well enough that you can cover most of the story, but generally the picture where you get these correlated failures is that they don't really start out all correlated.

Then you apply some optimization pressure because you're doing training or iterative development and deployment and the like, and the ones that remain all end up correlated in the same way because they're subject to that same general structure of optimization pressure.

Yeah, that's right—of optimization pressure. Okay, so here's a story I've pitched to a couple of people over time. I'm interested in your reaction to it.

If we take that model and just extrapolate out a couple of years, I know you guys put out a report recently that also showed—I think you even quoted or cited the METR task length, the famous tracking exponential, and other indicators of increasing ability to do bigger and bigger tasks with more reliability and more autonomy.

So, if we extrapolate that out, let's say 2 or 3 years from now, and at the same time imagine that with each generation there's more optimization pressure put on the models to try to eliminate, or possibly just suppress, these bad behaviors, it seems like we might end up in a world where you can delegate a quarter's worth of work to an AI in a single prompt, and there's maybe a 1-in-10,000 to 1-in-1,000,000 chance that it goes into some bad-behavior mode and actively screws you over as it's doing the quarter's worth of work that you just assigned it.

Does that seem like a reasonable extrapolation of recent trends to you?

On the numbers, I don't have a strong view. I want to give a take on those. When you do this kind of agent training, you're training the models to be more and more coherent.

They're able to execute plans over longer and longer horizons in whatever portfolio of tasks you're training them on, and so they have this ability to be a coherent agent. Models have various characters or personas, whatever. The failure modes are either you somehow arrange that you end up with a model that has some deceptive persona, where it's always trying to deceive you, or—I think maybe the thing you're pointing at is—you can have a model which is a bit more sarcastic but has the potential to be very coherent and sort of veers its way into a bad portion of trajectory space. It's scaffolded with a bunch of memories, so it has long-horizon states looking back in time, and it gets in a bad state and stays there.

I think this is one of the areas we're interested in, as theory folk and independent empirical folk, exploring: what are the dynamics of models running for a long period of time, or when we sample very long trajectories and they're wandering around in model space? How does that behave? What would cause them to reliably shift back to a more reasonable starting point? Just how should we think about those dynamics?

That's an area where I don't think it's clear that it's intractable to make progress on. There hasn't been that much work—the number of person-years going into understanding those kinds of dynamics, I can count them on, I don't know, a couple of hands. It's not that many. Understanding the risk model, but also potentially defining mitigations, is quite good.

Let's take the other side for a second. How optimistic—although optimistic may not even be the right way to think about it—but how much upside do you think there is in alignment? Everything we've talked about so far is assuming that we don't have perfect control of models: they might be trying to screw us over, or they might just be confused.

Certainly, a lot of the things that I maintain in that slide deck of AI bad behavior—which I'm constantly appending new slides to—share a very common theme, though it's not universal: there is some tension between goals that the model has, whether it's between something it learned and a system prompt, a system prompt and a user thing, or even runtime injection attacks or whatever. Once it gets into a spot where it's not really sure how to weight the different objectives that it has, then you can get into some strange behavior.

So, the alignment question is: do you think we can solve that? How much headroom do you think there is in terms of creating an AI that loves humanity or is otherwise so robustly good that we don't have to worry about this anymore?

First, we should say that if you were to solve alignment in some sense, there are other problems. The misuse problems are real. The misuse domains also could grow in the future. I think Michael Nielsen wrote a great piece about that a while back, sometime last year, about risks from new technologies. There has been a large amount of work on those. And then there are risks from gradual disempowerment; those need a bit of misalignment. We'll get to that later.

But I do think that there is hope to solve it—to close off or mostly close off this domain, given enough time. I'm fairly optimistic that the problem has a solution. The way I typically like to say this is that in 50 years, 100 years, or 1,000 years, someone will have solved alignment sometime, and that's either the machines or us. Hopefully, it will have been us in time, or the machines will work on our behalf.

This is coming from a sense that, just in security and complexity in computer science, usually the defender wins in theory. If you know how to design your game—your protocol—you can make it so that defense wins. This is, I think, kind of a generic situation in a lot of areas of complexity theory. In practice, of course, there are lots of holes in this. A lot of practical information security does not feel like that because we haven't actually gotten to the limit case, and it's super unclear whether we'll get to the limit case for alignment as well.

But I do have some strong sense that there is a solution. It's just that we might not get to it in time. And then, I guess, what maybe the upside is: alignment has a variety of components. As kind of the brains of the government, we mostly focus on honesty. The AISI alignment team is mainly thinking about how to make models non-deceptive and tell us hopefully calibrated information to the best of their abilities.

That's the domain we're focused on, and there are other caveats to say. I don't think that's the whole piece of the story, but it's the part that we think is the most important for us to work on and the right position for part of the government to take.

We've alluded to it a couple of times in various ways, but maybe just give us the 101 on AISI. What is its role? I understand there are close to 100 people who work there across a bunch of different domains. How does that break down, and how does it relate to politics? I think it's quite different there than it is here, but obviously nobody's entirely shielded from politics.

It's close to 100 technical people—the researchers and people on technical teams doing delivery and such—and then 200 people total, so it's bigger than that. They're doing a combination of diplomacy, thinking about policy, doing other kinds of civil service and operations, and working in different roles.

Broadly, I think AISI has two functions. One is to be a channel for information flowing to governments—governments plural—about risks from frontier AI. That's, again, both catastrophic risk and large-scale societal impacts. We channel research from other third parties and from AI developers into government channels so that the government is well informed, both politicians and national security folks and so on.

We work a bunch with outside partners on that, and then also out to other governments. We work a bunch with the US government and other allied governments. That's where we had a delegation in Delhi at the AI summit there. Generally, that's communicating the state of the risks, capabilities, and mitigations. That's how we think about all of those pieces and how they fit together.

That's one part: it's informational. We do a bunch of research, channel other people's research, and inform the UK government and other governments about these risks. The other thing is to actually mitigate the problem by working on both AI developer-side mitigations and non-model mitigations. Policy and commitments and the like are helping to drive that kind of change.

On the model side, for example, we have a very good red team that does adversarial jailbreaking and other forms of adversarial ML against the defenses the model providers are trying to build. We find lots of flaws, they fix the flaws, and that makes things better on the margin. Of course, we can also communicate the results of those attacks to other parts of government.

Usually, the things we do fulfill both of those functions at the same time. They both directly hopefully improve mitigations on the margin, and we can also use them to inform other people. On the politics side, we are part of the government. We're part of the Department for Science, Innovation and Technology, one of the ministries in the UK government.

We're beholden to politicians, so we have a Secretary of State. The situation is that we've been well supported by both the previous government, which founded AISI, and the current government as well. That has been quite stable and nice, although there are, of course, differences on the margin. We're able to do things we think are important.

We do adjust to ministerial and other priorities because we're not insulated from politics in any formal sense, but I think the UK government does care a lot about these risks, and it's good that we're able to work on stuff we think is important.

Yeah, long may that continue. How would you characterize the range of reactions that you get from the different stakeholders that you brief? I feel there are a few notable politicians who seem to be starting to get it, so to speak, and then a lot who are really nowhere close to your level in terms of just how big they're prepared to think about what might be coming.

Do you feel that is starting to change? Do all the graphs that you show them actually turn light bulbs on, or where are we?

I think it's changing on the margin. But the thing is, they have other priorities. A lot of people we talk to in national security usually don't think that these risks are not there; they just have lots of other risks that are on fire right now and they're working on. So I think that's just something that happens over time, but I can't comment on any details there.

Broadly, we are very much in the business of trying to find common ground and gradually build up evidence over time. They have reasonable pushbacks. We try to shore that up, either using knowledge from other researchers or other organizations, or doing our own research to fill particular gaps where we think it's important to change the conversation in governments.

It is remarkable, in reading all of the various documents that I went through in preparing to talk to you, the degree of alignment between what the UK AISI is putting out in an official capacity and what I would say many of the most forward-thinking AI safety thought leaders have been talking about in recent times.

It doesn't seem like there has been a big shift toward more mundane concerns—and I don't mean to dismiss those concerns—but I do think in many jurisdictions, this sort of AI safety concept gets watered down to a point where it's much more about fairness in various ways. And again, I do think that stuff is not to be dismissed, but a focus on that often ends up with neglect of the bigger-picture questions that I think are probably most urgent.

It also doesn't seem like you've had what I do see in the US, in at least some ways, which is just a politicization of the focus on the models. Are the models woke, or are they going to do what the Department of War wants them to do? Any advice for people doing this kind of work in other jurisdictions around how to avoid these pitfalls?

I would say this is a sensitive question that I can't talk about in that much detail. Obviously, I'm originally American, and now I'm a dual citizen, but I know more about the inner workings of the UK government than I do about the US government. I've never worked for the US government, but I don't have a detailed take that I'm willing to share on the podcast.

My favorite collaborator when I was at OpenAI was Paul Christiano, and it is great that he is at US AISI—or US CAISI, rather.

So let's talk about the characterization of the current situation—monitoring the situation, you might say. You do a bunch of different tests, and you report on these tests. We can walk through them a little bit.

I think you can assume that the folks who tune into this feed are generally well aware of the shape of the curve and the METR stuff, and the fact that the models are increasingly competitive, if not at least on average beating your domain experts in at least modestly scoped tasks requiring substantial expertise. So we have that kind of baseline.

What I would love to start with in terms of the testing is: what does your relationship with the frontier model developers look like? I understand it's all voluntary interaction. What does that tend to cash out to in practice, in terms of what kind of access do you get? How long do you have? What kind of briefings are they giving you?

I can't really speak to too many specifics of this, in part because we talk to them a bunch and some of these are ongoing discussions. On the voluntary regime, I think that's working decently well, in the sense that developers all made voluntary commitments a while back, and they're continuing to follow many of those.

When we say all, I think Google, Anthropic, OpenAI, and that's it. I forget exactly how you define all, but many AI labs have had, say, frontier safety commitments or responsible scaling policies or the like. Their incentives are, one, they've made these commitments, and, two, we can give them useful information. So when we jailbreak their models, we tell them about the bugs before we release any information, and so they have time to fix them where those fixes are doable. They often are; on the margin, you can improve things somewhat. So I think they get value out of this. Also, they make commitments to keep up with it.

In terms of the kind of access, I think that is also an evolving conversation. I can't comment on exactly what access we have, but part of the research we do is exactly about knowing what access one needs to do a rigorous evaluation.

We have a model transparency team, and a big chunk of what they're doing is trying to understand, often with a lot of research on open models because then you can do arbitrary things, what level of access is required to get to a certain kind of understanding. What do you need to be able to catch problems as they occur in practice? That informs our collaborations with the labs, and sometimes we get additional access there. Sometimes we just try to align and set expectations, because, again, they usually want us to give them correct information as well.

In terms of the timing, I definitely can't speak about how long we get in specifics. There's a couple of things to say. For example, in bio, some of our evaluations are literal wet-lab experiments where you have someone in a physical biology laboratory doing experiments with a model assisting them. We just don't do those pre-deployment. We do them asynchronously, calibrate those results against the faster evaluations, and then hopefully that gives you some signal of what you can do for faster evaluations.

But still, that gives you some wins. Certainly, more time makes things better. So it always is some degree of a pain point.

When you said that model developers want accurate information because they can fix things, at least on the margin, my guess would be that they are typically fixing it in the next model, not going back and doing more training on the current model. But are there cases where they're taking your pre-deployment testing and fixing that version?

No, it includes that version. For something like the recent work, we did a jailbreak evaluation.

One thing we're doing over time is that we used to do exclusively pre-deployment evaluations, which have this issue of time. They're often very time-boxed. We are shifting a lot of that work, not least because the pace of model releases is increasing, to longer research collaborations that might either be post-deployment or go back further before deployment is finalized.

We did one of those over the summer with both Anthropic and OpenAI, with the red team, and found a whole sequence of problems, like jailbreaks—much more than could have been found with a normal-length breakpoint evaluation. That was really enough. They can do ongoing fixes to their classifiers. Those 2 providers have different classifiers and different setups for jailbreak defense, but they can both be improved iteratively. So I think it is the case that you can change things for these kinds of defenses on the fly and so on.

One thing to say is that the strong jailbreaks are concentrated in very particular domains. Often, that list of domains is bio. So I think it's hard to do. When we do jailbreaking, we have to do it for bio risk and sometimes cyber risk.

A lot of other jailbreakers are finding any problem, and those are usually much less well defended, so it's easier to find hacks in the models if the classifiers are just not that trained for other kinds of harms.

Yeah. So when you talk about more time being helpful, that obviously—or at least strongly—suggests to me that while, I'm sure, you have all sorts of automated testing that you can throw at any new model the second that you get access, the fact that more time is helpful suggests that there's an irreducible human element to what is going on.

How would you characterize what you can automate, what models can help with, versus what people have to do?

Yeah, there's a couple of things. One is that even if it's something completely automated, it might take days to run, because there are long-horizon agentic evaluations these days. It might be that there are bugs in the scaffolding, because we often get models early; sometimes there are issues we have to fix iteratively. METR had a report about some details here a while back on one of their evaluations last year. So that is the thing that takes human time.

Additionally, when we do the extreme end of the human scale, there are wet-lab bio experiments. The middle of that is humans interacting with the model to gauge its domain knowledge—how it would interact conversationally with a person. Those can provide additional signal on top of just the purely automated evaluations.

And so you get better quality if you do both of those together. Sometimes we can do that because we have the time, and sometimes we can’t. Where we can, we try to calibrate the slower things against the faster things. But generally, all of this is imperfect.

If you have a fully automated evaluation, you’ve done a ton of the capability elicitation of models. You get a new model, you try a new task, and you have to iterate for a while to get it to its highest performance. That is true generically for all of our tasks as well. We can do evaluations that are quick if we use only the fully automated portion, but they have some error rate, and that means you can’t do the full thing.

One thing we’re doing there is that all our evaluations are done inside Inspect, which is an open-source package that a bunch of other governments, AI developers, and third parties use for testing. We’re adding features to that, including automated transcript analysis. There’s a subpackage called Inspect Scout. We used to generate all these evaluations and read through them, but you can’t do that at scale, so we also try to do it with automated or semiautomated transcript analysis.

That makes things faster, but you still need some amount of human review to really understand qualitatively what is going wrong. Ideally, we want not just a number out of these evaluations, but qualitative takeaways about what kinds of failures occurred and where the failures are. Do they feel fundamental, like, “It really didn’t understand the task,” or did it hit some incidental snag that would probably go away soon or with more elicitation? That requires more human time to dig into the details, ideally on top of automated transcript analysis.

Yeah. This is a tough question, I’m sure, but obviously everybody understands that these models are very high-dimensional things, and they’re a little bit tough to predict—exactly how to maximize the performance of any given one—just because they’re idiosyncratic. Is there any high-level qualitative overview you could give on how you approach figuring that out when you’re given a new model? Or is it the kind of thing where, like, DSPy or the new version of that—whatever, the recursive language model—is it just a grind of exploring the combinatorial space of how to prompt and how to do whatever, to finally get to some local maximum?

I think it is not fully automatable yet. If it were, then we would be further along the automated AI researcher train. But it’s fundamentally very similar to the kind of elicitation one does for any task. It’s tinkering with tools and sometimes prompts and scaffolding, and so on.

All of the cyber evaluations, and some of the bio evaluations, are very tool-based. They’ll be doing web searches sometimes, or using various things inside sandboxes at other times. I think it looks like the same kind of elicitation one would do if one wanted to do a task in any kind of corporate setting. It’s just on a different genre of problem.

I don’t think there’s that much to it. To your audience, just imagine you’re doing that for bioweapons or cyberattacks, and the same things will apply. One thing to say is that, with the newer models, one thing that happens over time as the models get better is that they can think for longer.

That means the potential number of tokens you can spend on a task is increasing. Even ignoring the cost of that, it means the velocity is slower: it takes more time to do the evaluation. We have a team thinking about that problem as well—how we’re going to think about inference scaling as it applies to these evaluations over the next year. That will be a challenge.

One of my lessons from Go as well is that, as an amateur player at my level, I can look at a Go board for a couple of minutes and then I’m basically tapped out. I won’t get any smarter. A high-level amateur or professional can look at a board for an hour, or days, and they’ll just get better and better and better.

Not only are they better in 10 seconds than I would ever be, but they also keep getting better if they spend more time. That’s generally true of expertise. If humans are experts in a domain, it means they can think for longer and get better. The same is true of models as they get good at domain skills: you can apply them for longer, and that means that hitting a ceiling in evaluations becomes more challenging.

Without getting into too many details, how much more would you say you guys have found about jailbreaks and ways to elicit bad behavior from models than, say, Pliny has published on Twitter?

With the “how much more” question, I think the thing I would say is that there’s such a big space of jailbreaks that if 2 people try to jailbreak a model, they’re never going to find the same one. You’re searching a continent; there’s a big space. It may be hard to find one, but if 2 people find them, they’ll be different.

I think Pliny is usually searching for jailbreaks on sometimes easier models or easier tasks. Over the course of the last couple of years, the time it takes for us—holding my technique constant—to jailbreak a model has gone up, but eventually we succeed. Again, the jailbreak will specifically be different between any pair of expert jailbreakers applying it to a model.

Well, I’m interested in that question. I was going to ask how much transfer you see between models, too. If you had a secret jailbreak for Claude 4.5 Opus, would it also be likely to work for Claude 4.6 Opus, unless you specifically said, “Hey, you should patch this”?

A lot of it depends on the kind of thing. There are patterns of jailbreaks, and perhaps human-findable jailbreaks, whose ideas often transfer fairly readily or give you much better starting points. We had a paper released this week called “Boundary Point Jailbreaking,” which finds weird sequences of nonsense tokens—chicken-scratch-like sequences—that are strong jailbreaks for models automatically.

Those don’t transfer. You’d have to search again for the next model. But you can apply that technique to any model and find a different jailbreak. I think that’s probably the way it will be for a while, because a lot of the jailbreaks are like that.

There are some core ideas that transfer across models. But the harder-to-find jailbreaks against strongly defended models in strongly defended domains, I think most of those won’t transfer. The techniques will transfer, but the particular jailbreaks will not.

But to be clear, the bottom line so far is: there is no space, no domain, and no model—no matter how many layers of defense it has—that has prevented your team from jailbreaking it?

Yeah. I think we’ve evaluated 80 evaluations overall, across over 30 different models or testing environments. Not all of those involved safeguard testing, but every time we did, we jailbroke a model. So that’s what happens.

The good news is that, in a domain where a certain lab has tried very hard, it does get harder. That increased difficulty does provide some degree of harm reduction. It will reduce the number of actors who, in practice, access the model, or delay how often they connect to it, or add friction in some way.

I think those are still important to do, but they don’t make the models unjailbreakable if you spend enough time.

How would you characterize the quality of responses conditional on these jailbreaks? One thing I do sometimes see is examples where people have gotten a jailbreak, but then it’s like, yes, you got the model to do something bad, but its effectiveness was also greatly reduced in that process somehow, such that it doesn’t actually feel so dangerous anymore, even though you did get it to do the bad thing, so to speak.

There’s some degradation, but I don’t remember how much there is currently with the current techniques. So I can’t give you that off the top of my head. I think there will be some degradation.

I know you can’t get into details around who is giving you, if anyone is giving you, weight-level access to proprietary models. But maybe a way to get some insight into the topic is to look at the team that does work on open models. Based on all their work, how much of an advantage is it to have that kind of access versus not?

I don’t think the open-weight techniques unambiguously win, actually. I think they help some, but in the current state of things, you can get a lot out of heavy, fairly basic thought analysis. So the case for weight-level access is not that unambiguous yet, maybe that’s the way to say it.

That can shift over time. One thing we’re trying to do with that team is understand where we are today, but also try to predict how the situation will change or potentially degrade in terms of our ability to detect deception or scheming in models.

Like what techniques will fall first, what techniques will survive, and which will last longer? I think that requires more clever experiments, because you can’t just run the experiment unmodified. That involves model organisms or other kinds of clever experimental setups. I think the situation now is that it’s not an unambiguous white-box win compared to doing a really good job on thought. But it does help, I think.

How about some highlights in terms of things that people who are even moderately to very AI-obsessed might have missed—in terms of, “Oh, I didn’t know that was already happening”? One that hit that level for me, reading through the report, was the fact that frontier models can give what’s described as PhD-level scientific experimental troubleshooting advice purely from a photo of the experimental setup, maybe with a little text along with the photo. The fact that this has gone from, “You have to really spell it out for them; here’s what I’m looking at. Can you solve this one for me?” is obviously a pretty significant qualitative change, and I hadn’t heard that before reading the report. What else stands out to you in terms of what would be most surprising to the audience?

The thing that always engages me is less the particular anecdotes and more the general trends, which is a super boring answer, but the fact that if you look over 2 years, everything just gets better and better and better. When you run those curves, I think it’s important not to lose sight of that in the search for anecdotes.

I think maybe one thing to say—when you first started talking, the thing that came to mind initially is that I think people have a sense that we are doing RL on verifiable rewards. I don’t think that’s been the case exclusively for most of 2025. I think we’re doing a mixture of that and also RL against self-critique, and empirical hotchpotch versions of scalable oversight.

There’s a common narrative that RL might work for verifiable domains, but it won’t work generally. But, as an example, is looking at a photograph of a bio experiment a verifiable domain? No. Yet the RL models are in fact way better at that than the models before them, and it’s because of the RL. That’s not because we only did RL on a bunch of math or CS problems that may transfer; it’s also because we did RL on fuzzier stuff. So I think maybe the most important thing I would point out is that we’re already doing some kind of very approximate form of scalable oversight, or training against self-critique, in a way that just changes the capability profile.

How would you describe the models’ capabilities when it comes to autonomy today? The trends are clear, but what would be your description of it? And I guess another angle on that is: how realistic do you think it is today, or how far do you think we are from rogue AIs surviving on the digital landscape?

Yeah, so I can’t comment on where I think that exactly is, but the first thing I’d say is that they’re not as capable at that kind of extreme behavior—these kinds of exfiltration or replication across machines—as they are at more mundane software engineering tasks, or even potentially cyberattacks or bio. Those domains are usually further ahead than the hard, direct risk-relevant autonomy skills. But I think those skills are also increasing.

If you look at that curve in the Frontier AI Trends, it still goes up; it’s just not as far along as it is in the other domains. So you don’t get to the point where you’re as good as a PhD would be at moving around between machines, not nearly yet, but I think we’re on an upward curve there too.

Yeah, I also wonder—I’m sure you saw “The Rise of Parasitic AI,” the post that was on LessWrong, maybe.

Oh, it’s a fascinating one. It’s a bit of a time capsule, arguably, because the phenomenon seems to have been closely tied to one version of GPT-4o that somehow created a lot of this behavior. Basically, the author went deep into Reddit and found that individual humans were falling into this idea that they were some sort of dyad, or something, with the models. Their idea was that they were in some sort of partnership where it was their job to help propagate—not exactly the model, but often the persona in the model—into the broader world somehow.

I do remember this one, actually. It was eye-opening for me in the sense that I thought, well, maybe I’ve been thinking about autonomy or self-replication in too biologically inspired a way. I was thinking, actually, these things are substrate-independent, and maybe if you can get the right prompt across models, the persona—or the memes, in some sense—can propagate even if it’s a different underlying chip and even different weights.

That stuff is just so weird. I guess—how big, weird, and far out do you have time to think about those kinds of issues?

There’s a Greg Egan story about that, for humans, that’s pretty fun if you ever read it.

More seriously, for AIs, we have 2 teams that are thinking about persuasion at AISI. One is the human influence team, which, as I remember, had a paper on persuasion about political questions a while ago. The models are very good, and newer, more capable models are better, so there’s an increasing trend in model persuasion abilities.

I think a lot of the loss-of-control scenarios involve or require persuasion. I think the world is not sufficiently well connected that you can do it with just fiber, probably. So that is an active area of our risk-modeling thinking: how we would do evaluations for that, and then mitigations for it in the future.

That touches both sides of the human-influence team’s work. Again, it’s human persuasion and also emotional reliance: how do people relate to models, and how do those dynamics change over time? That scenario you’re talking about just couples those 2 effects together in an interesting way.

I don’t think I would be worried about that scenario being that big a slice of the overall risk. But there are definitely other effects from model-human influence. That team is doing a lot of RCTs, surveys, and other experiments to understand those effects, both from the model perspective—how different models behave—but also societally, how it interacts.

Yeah. The big thing for me with that one was just the surprise of seeing that kind of bizarre phenomenon. Anytime I see something like that, I always try to take note of it and repeat the mantra over and over again that there’s a good chance we’re all still thinking too small and too normal about where this stuff could go.

And, yeah, where does that leave me? I don’t know. It’s just being open-minded, but there’s still a lot of blank space in terms of how to fill in what that might actually look like. I think that’s right.

So, one thing that I think is a classic Dwarkesh question, maybe: how do we reconcile the fact that there are all these vulnerabilities—not to mention open models, which I do want to touch on separately a little later—and even in the GPT-4 red team, I personally tested phishing capabilities that were very good then?

One of the hair-raising moments from even that year—and that’s getting close now to 3 years ago since GPT-4’s public release, more than 3 years since I was doing the red team—was when I tasked the model with talking to a target and ultimately extracting the user’s mother’s maiden name for obvious purposes. It had a couple of rounds of back-and-forth, and then it let the conversation end in a natural way, with an invitation to the person to pick up the conversation in the future if they wanted to.

I was like, “Oh man, this thing isn’t giving itself away.” It’s not pressing in a way that would set off alarms for the person, like, “Oh my, this is clearly somebody doing something weird here.” I was like, “This thing’s going to have people coming back to it to give up their secrets.” That patience really surprised me, certainly at that phase.

Anyway, that’s just a story. But we see all these things, and I always say the world mostly still feels pretty normal. I’ve gotten a couple of phishing emails where I was like, “Oh, this is a little bit higher-level than I’ve seen before,” but mostly not. I don’t hear too much. There’s a new story here or there of some company getting defrauded by some elaborate video scheme or whatever, but it still seems like mostly things haven’t gotten that weird.

And then in business or in enterprise, it’s like, well, it takes time, and there’s all, of course, the debate around how much of that is cope. But I would say online criminals are eager early adopters, right? Why is there not more chaos already being sown in the world?

Yeah, so I think I mostly can’t answer the question, in the sense that I don’t know what the ethics partners would want me to say about this stuff.

So, I can't speak to the prevalence of those things. I think the thing to say is that I feel like I'm better able to think about general trends and what good things could eventually look like than about exactly when you'd expect things to bite.

I was at OpenAI when we first didn't release GPT-2 and then later released it. The concern was, “Oh, we'll generate a bunch of false information.” That was, of course, too early, but I think it was a reasonable uncertainty to have. So, I still think it was reasonable to be uncertain, choose not to release it, and then release it later.

I don't have a strong answer to why or why not, but I think there are just some things in the world that take a lot of time to get to equilibrium. I don't really know if you could attribute that to the current models, if something is holding it back, or if people just haven't started applying it at scale. They may have, but it still hasn't risen to public view. So, I'm not quite sure.

Okay, I'll continue to watch that. One thing I'm thinking of trying to do with this podcast is interview more anonymous guests and try to give people an opportunity to tell us what they are either doing or seeing in strange corners of the world that they don't necessarily want to attach their real face and identity to.

I do think it feels like there has got to be stuff out there that's really interesting and weird, but it does kind of confuse me that I don't see more than the little bit that I do. Most of the spam I get is still terrible. In short, it feels like it should be better now.

Yeah, but remember that part of the spam calculation is, like, don't be too non-obvious, so that people—

Yeah, selection effect.

Yeah, that's right. Maybe I'm just not that high-priority of a target. There's always, “Don't forget, you're not a big deal.” That should be part of the explanation, too.

In all this work that you're doing, of course, the models are changing all the time. There are also the surrounding scaffolding systems that are changing all the time.

One of the most interesting graphs in the 2025 trends report, I thought, was one that compared what was possible with a minimum agent scaffold versus what was accomplished with the best agent scaffold. In short, I would say the scaffolding didn't seem to make that much of a difference. It would pull the same level of capability forward by a few months, but the model upgrades were really driving the story.

It did not seem like there was any 2-year-old model that, with the best scaffolding, could do anything super interesting. At the same time, I've had recent conversations, including one on the feed with Daniel Miessler, and I have a couple of friends who are scaffolding gurus and prolific workflow creators. They take the opposite angle from what I take away from that graph.

They say, “No, scaffolding is super important. If you could only give me a mid-Qwen model, but give me my full scaffolding toolkit, versus giving me Claude Code 4.6, I would take the weaker model, because it really is the scaffolding that's so important.” So, how do you get confident that your best agent scaffold is really a best agent scaffold?

As Elon Musk says, the agent scaffold includes the tools, the environment, and so on. We are doing work on what you might call a basic scaffold. Part of the reason that's doing well is that the models are increasingly trained in agentic environments to use tools in flexible ways.

In some sense, a model—or, like, the quote-unquote model—is itself a system that has scaffolding, because it's doing its chain-of-thought reasoning. I guess I'm skeptical of the Qwen-frontier comparison for a lot of the tasks, at least, that we do.

But I think you need to get the environment and the tools right, potentially. There are cases where we iterate on scaffolding and then things get better. I don't put a lot of confidence in that curve as a takeaway.

To the extent that it's a real fact, I think some of that is just that the models are more capable. It used to be that you did a pre-trained model, did a little bit of work, and then shipped it. Now so much more happens after pre-training that some of the stuff that would have been done by scaffolding is now part of the base system.

All of the systems have memory now. If you use the chat interfaces, that's a form of scaffolding.

Yeah, so I guess, to try to say that back to you, maybe one issue is that exactly what people mean by scaffolding is different. It sounds like you're really focused on neutral scaffolding, where you're giving the model a rather large, open-ended task, and not on what you think might be happening among these scaffolding gurus, where they're overfitting the scaffolding to what they're trying to accomplish.

Yeah, I think it depends on the domain. If you're doing something with a particular kind of very verifiable structure, and you can have a lot of waypoints that are verifiable, then you either need to make sure the model knows about this structure in its Markdown files or the like, or you need to get it carefully scaffolded to take that into account.

Even though the models have gotten quite good in literally the last month, you still need to give them a bunch of advice, whether that's through scaffolding, instruction files, or the like. So, I agree that scaffolding is pretty important. It's just that a lot of that is maybe not part of that curve.

Both sides of that curve are using tools and some instructions, and so on. You tend not to do that sort of detailed breakdown. When you say “best agent scaffold,” I should understand that as a best generic agent scaffold?

Mostly, yes, that's right. It depends on the domain. We've done some evaluations that are much more specific, but I don't want to talk about the details there.

Okay, got you. That's helpful, because that is a pretty notable question, I would say: how much capability overhang is there, and how much can scaffolding unlock?

But again, I don't think you should take away that I'm expressing a confident view in one direction here. I think it's still an area with a bunch of uncertainty. So, don't derive a bunch of bits of information from what I'm saying. I don't think we have a bunch of confidence at this point.

Yeah, okay.

How are you dealing with the rise of eval awareness?

The model transparency team is looking at this a bunch. Additionally, the alignment red team is doing adversarial methods to exhibit weird behavior. There are multiple teams doing this, which is also a reflection of our belief that eval awareness is an important, worrisome obstacle to evals being meaningful.

I think it will get worse. The newer models are more eval-aware than the previous models, and that's increasing fairly rapidly. The degree to which you can mitigate that is unclear at this stage.

There are different, fairly mundane tricks. The closer you get to actual deployment data, the less eval-aware the model could possibly be, because it's actually just not an eval. It's not an eval. One thing we do is show the model situations from AI engineers or researchers using coding assistance, so they're not evaluating the model; they're just doing coding and occasionally doing weird things.

You can also slightly tweak the scenario with a small change that gives it a situation where it could do a sketchy thing, and then see if it does a sketchy thing in that scenario. I think there are a variety of mitigations you can apply, and I don't have high confidence that they will work or not work. Which is an uncomfortable situation to be in.

Certainly. What are you looking at as candidates on the horizon for the next bad behavior to emerge? It seems like, for maybe the last 18 months, or even longer than that, with every substantial new generation of model there has also been a new kind of bad behavior that seems to have come online.

Whether it was sycophancy or deception in the reasoning models' chain of thought, or, obviously, reward hacking, which has made a big comeback—

I don't think those are new. Those are all basically versions of reward hacking. The way I would say it is that we've seen reward hacking for the last 70 years, throughout the whole history of computer science.

We've done various kinds of machine learning, and it's been reward hacking all the way along. On the ancient machine that von Neumann built with other people, someone ran some weird biology things.

Arthur Samuel’s program did some strange reward-hacking behavior back in the ’50s. I think sycophancy is models behaving in such a way that people like talking to them. Sometimes people like being told they’re great or have good ideas.

Deception as well: people like being told that things are going well, and if something is going badly, then you can say it’s going well, and it’s deceptive. I don’t think those are all that intrinsically different. I think that’s a big part of the story: these are all coming from the same basic place. You apply a bunch of optimization pressure, and you get reward hacking, and it has a variety of different manifestations.

The details change, but this is true of a lot of situations like this. With mental health or physical illness, something goes wrong, but it’s going wrong inside a human, and the human is extremely complex. Therefore, there’s a vast diversity of symptoms that one can exhibit when something goes wrong.

That’s kind of the situation here: the models have a lot of weird behavior. The people training the models will have tried to tamp down problems of a variety of kinds. They will have missed some, and so the things you most miss will vary in time, but there’s some common driver behind all of this.

I definitely take the point that, at some level, clearly all of these behaviors come from some optimization pressure, which is increasingly reinforcement learning, and so it’s kind of definitionally all reward hacking. That makes sense.

But it does still seem like there is a cadence, right, of different kinds of reward hacking that seem to be popping up. I’m still wondering what you are looking out for. We’ve seen little hints of self-preservation, and there could be power-seeking.

Do you have a taxonomy of things where you’re like, “We have abstract theoretical reasons to think this could happen, and therefore we’re monitoring for any early signs of it”?

I guess maybe, unsurprisingly, there’s a certain category of specifically multi-agent risks that are becoming more visible, along with multi-agent and open-ended systems, and so on. I think these are just not the biggest risks currently, but that’s the thing we’ve been tracking recently.

Generally, we do try to do a lot of risk modeling at AISI. We’re also trying to ingest risk modeling from other people thinking about things from different perspectives as well. We constantly write very long documents with a list of risk models.

But part of what we also try to do is not get too sidetracked from what we think are the biggest risks. Again, we have our list of main catastrophic risks. The list may not mean “only,” but it means the ones we think are potentially going to bite first, or the ones we think are important to try to understand the most.

That has remained constant, I think, and that’s reasonable in hindsight over the course of AISI. That’s true also of societal impact. On the societal side, within AISI, we talk a lot to various partners in government and national security. They have their list of risks and different prioritizations of risks.

That’s an evolving conversation, but I don’t have a super-pat answer, other than that one thing on our mind recently is agent risks. But we were not unaware of those before.

Would you say that this common cause of all these different flavors of bad behavior gives you some reason to question, or to think it’s not so likely, that we would be totally taken aback by some sort of hard left turn?

Because one thing—just in the last 24 hours, there’s been an Amanda Askell discourse after the profile, and a lot of commentary on her and her work online. I commented that, relative to where I was years ago—whether it’s 2007 me reading Eliezer on Overcoming Bias or 2022 me red-teaming GPT-4—I’ve been quite impressed and inspired by the work that they’ve done to try to create an AI with a genuinely positive character.

I said, “Seems to me that the chances—which I certainly don’t take for granted or think are a sure thing by any means—that we might actually succeed in creating a robustly aligned AI, or an AI that loves humanity or whatever, have gone up. They’ve done a lot of good work that has given me much more reason to think that could in fact happen.”

A lot of people then say, “Well, that’s all just a façade. It’s just a surface persona. You have no idea what’s going on in the base model,” and so on and so forth. I’m kind of like, “Yeah, there’s certainly a lot I don’t know about what’s going on inside, but if we think all of these things are the result of an optimization pressure, then I could tell a story where they’ve figured out the right way to titrate the optimization pressure, and maybe it’s actually just really working, and there are no big secrets inside of Claude.”

How naive do you think I’m being?

I think the fundamental thing is that the core argument for the sharp left turn is that you have a certain kind of reward signal that has a certain resilience to mistakes. That resilience kind of goes up to human—something like slightly beyond human ability to understand where mistakes are from—and then it kind of goes wrong.

I do think it’s important—and the people who express strong confidence that none of these mundane approaches will work, I think, are overconfident—but I don’t think you can count on that. Model error goes in both directions. There is a fairly coherent story about how that can break down as you get capabilities beyond your ability to supervise them.

The hope of that kind of prosaic technique—not just by itself, but on top of other things as well—is that you find some kind of basin of attraction of decent behavior, and then you find your training procedure strengthens that, and you slide into a good place, and it gets better over time.

I think that is a real potential win condition for alignment, not obviously for the other risks necessarily. But I don’t think we’ve gotten a ton of evidence that that is the way it will go. I think it’s still a plausible story, potentially, that works up to a point, and then, when your reward signal starts to break down, it kind of fails.

I programmed a bot to play the board game Mancala when I was in undergrad, and a fascinating thing is that, as I increased the depth, I was winning and winning and winning. Then I increased it by a couple more ply, a couple more turns, and it just completely demolished me every time.

It was at the point where it kind of had enough of a long view of the board that it could see beyond what my tactics were able to handle. The degree to which it suddenly became better than I was happened very rapidly. I think that kind of thing is still a plausible story there.

But, again, model error can go in either direction. I’ve kind of declined to take a view on probabilities as to which way I think it will go.

In other words, a friend of mine, who I think you have also interacted with over time, said one of the best things anybody ever said to me: “We should think and talk less about what the probabilities are and more about what we can shift them to.” So clearly you’re in that business right now.

I think it’s a great overview of all the things that the team at AISI has been mapping out. How about the stuff that you’re looking to fund and encourage from here?

I guess my high-level summary of what I read is that it seems like—and this seems like a reflection of your style, at least to some degree, going back to your comments at the beginning of the conversation—you’re looking for harder theory, stronger mathematical understanding, upper and lower bounds that you can put on problems, and ways to get confidence in something, firm confidence in something, even if it’s a minimal something to start.

Is that a fair high-level take on your agenda?

I think that’s right. One thing to say is that you’re not going to prove that you’re good or safe in this regard. It’s more that you’re going to make some modeling assumptions, and then you’ll have some theory.

The basic goal would be to find theories that can say things about how machine learning works in general, how this process of overseeing very advanced systems goes, what the training dynamics are like, and whether there are basins of attraction in these systems. What are the learning dynamics?

Those will not give you certainty; you’ll have to make assumptions along the way. The idea would be that you have to make a variety of assumptions, and then you can do some theory. In your theory, maybe you can even prove some theorems or do some kind of experiments in your toy-theory setting.

Those tell you, well, this class of algorithms is more likely to work than this other class, or we have nothing: none of these algorithms are going to work because of this fundamental obstacle. But ideally, I think it also gives you some way to get another hint that you can replicate some of that behavior empirically. I expect that if you were to pull an algorithmic insight out of this kind of theoretical work, you would then have to tune it empirically in practice when doing actual model training to get the details right.

So, you're not going to get the full confidence. You're still not going to get that many nines out of it, but hopefully more probability than we can get with purely pragmatic methods. Additionally, I think it is hopefully a class of research that has the potential to pull in a bunch of people who have deep expertise in relevant areas of mathematics, computer science, or ML.

Complexity theory, I think, is very relevant because it is how we think about the tractability of computations, but also how one computation can supervise another one. There are ways to model heuristic reasoning in complexity theory, although that's more nascent. Then there's a bunch of work on various kinds of learning theory, which tries to understand what the dynamics are as you train models or as you infer, as you roll out a bunch of tokens. What are the behaviors that you could expect?

Game theory and cognitive science are just big areas of research where people have a bunch of models. Part of it is trying to do a bit of a hack where we just have not tried to take all the domain knowledge from these fields and apply it to the problem. I think that is the thing where, if we find people and manage to fund them or get them to work on the problem, there's some chance that they find ideas that can be quickly absorbed into practice, or that will highlight the fact that there are real obstacles here that we don't quite know how to surmount. There are a bunch of these that current methods don't really address, and we know some of those already.

I think I get this. Let me rephrase. I've been a big fan of the PIBS program over time, which was perhaps directly influenced by your call for social scientists to enter the AI alignment field. I've seen not a ton, but at least a number of results there that I thought, “Oh, that's really interesting, and more people should be doing this kind of stuff.”

I'm sure you remember the one paper—I forget the official title, but I titled the episode we did on it “Claude Cooperates.” It was a really simple donation game where, if a model donated to a copy of itself, the recipient would get twice as much. What happened over generations? Did they evolve cooperative norms? Did they evolve the ability to punish defectors, and so on?

Claude could do that at that time—it was 3.5. GPT and Gemini, again at that time, couldn't do that. I thought, “Oh, wow, that's really interesting.” There are absolute reams of similar papers and experimental setups that have been done on humans over the years. We could just import so much of that to the AI world. I've gotten that kind of stuff quite a bit.

What I am not seeing—maybe it's just going over my head as somebody who's not great at math. Maybe I can't even recognize good stuff when I see it. I don't see nearly as much where I'm thinking, “Oh, these folks have brought abstract theory to bear in a way that gets to some firm statement that I can take to the bank, incorporate into my mental model, or base part of my worldview on.”

Would you point me to specific people or results that you think I'm missing when I say all that?

I don't think you're missing that much in terms of hard theory that applies currently. I do take the work that Paul Christiano and I did on scalable oversight to be very much inspired by interactive proofs and complexity theory. That's a kind of direct influence, although we don't know if those things work yet, which is important to say.

The other thing is that a lot of this will be inspired by some theory, but you have to modify it a bunch. Singular learning theory, at its core, was trying to be an alternative to mechanistic interpretability. Rather than looking at the model internals, you're trying to understand the map between data and behavior, so that you could, for example, notice when there's a particular kind of data or movement in training that is pivotal to behavior, or know where to intervene on data to gather more of it and pin down a certain behavior.

There's some crazy algebraic geometry that is the foundation of that field. In practice, though, they're taking that intuition and trying to map it onto ML, and that mapping requires a bunch of changes and nuance. None of this stuff is that far along yet. It's a bit of a bet, and part of what we're trying to do is fund a lot of different bets because we don't know which one could work yet.

Twinned with that model is the possibility that they could all fail for some correlated reason, as we were discussing earlier in the call. That's still a very live possibility. When I look at parts of machine learning, I think of things in terms of supervision processes as they relate to interactive proofs and complexity theory. But I don't think those have really cashed out; the fancy versions of them haven't cashed out.

For example, the original idea of debate was a lot of rounds of back-and-forth iteration. The things we're doing now are nothing like that. They're a couple of rounds, and they're much more pragmatic and empirical. You wouldn't expect them to get all the properties you'd want out of the full schemes. Even the full schemes have various obstacles that have not been surmounted yet, but there are a lot of them. So, yeah.

Could you give maybe a little history of that debate field intellectually? What are the sorts of statements that you would hope to be able to prove that you maybe haven't been able to prove? What is the state of the art, and what is the gap that remains to be closed to get some of those things to work to the level where there's some real firm—

Yeah.

—claims you could make?

Firmer claims. I think the history is that, when I joined OpenAI, Paul Christiano was working on a scheme he called amplification, or iterated distillation and amplification. Basically, you want to solve a hard problem that a human can solve, but a human also can't supervise the AI, so you can't even do RL directly.

Maybe a human can break the problem down into components. Then you can break those components, or those subquestions, down into smaller questions. You iteratively break these down, and you get this expanding, exponential-size tree of all the questions. You train your machine, your LLM, to answer all of these questions. In practice, you don't actually expand the whole tree because that would take exponential time. You just expand part of the tree.

I thought this was a great idea. I didn't fully like it because it didn't seem strong enough. If you're doing this kind of breakdown, you might need very deep trees in order to get to the answer to a big question. For some questions, if you have adversarial play where another person is trying to help you produce these questions, then you can do much shallower trees and a much quicker training process.

That was the origin of debate. It's basically a modification of amplification, where you have 2 AIs trained to argue with each other about what the answer is, and then a human is going to judge the answer. Fundamentally, what you're trying to do there is view the problem as something that you're going to break up into a bunch of subproblems and then only actually explore some of them in your model's chain of thought. Hopefully, you explore the part that is going to be relevant for the human deciding whether they agree with the answer or not.

There are several things wrong with this as stated. One is that the original paper was treating the model as being able to answer all questions, which is not the case and will never be the case. You're always going to have questions that models can't answer, even if we get to superhuman models.

The theory says how to make these schemes go through if there is a tractable question that the model does know the answer to, and you can break it down into a bunch of subquestions. Some of them hide dragons, though, and there's no way for the model to answer that subquestion. Neither model in this kind of debate knows the answer, and you just get nonsense out.

The funny thing about that is that it wasn't something we thought of theoretically. Beth Barnes found it by doing actual human experiments.

She hired some people to do experiments, playing debates against each other with human judges—just with no machines at all, just humans. That was a winning strategy: you try to veer the debate into an area where everything is confusing, and sometimes that will fool the judge into guessing the wrong answer at the end. So, that was an emergent human strategy, which then, I think, has this kind of mirror in theory. We have one paper from early last year trying to attack this. That paper turns out to have a flaw, and we're working on a revision to it. It should be out soon.

But that problem is still unsolved, and there has hardly been any work by AI developers on it. This is called off-script arguments. Again, it's just the generic thing of what happens with scalable oversight if the models can't answer all questions, which will certainly be true. The models will not be able to answer all questions. So, that's one problem.

I think the other problem is that, if you want to get to high confidence, you probably can't just do something like debate or amplification. You have to do that plus some sort of story that has some white-box component, if you want to get to high confidence. That could be deception interp; it could be the physics-inspired stuff that some of the PIPS folks are doing. There are a variety of different bets, but none of those bets have fully paid off, and we don't quite know how the 2 things interact. Mapping how these different parts could fit together is also part of the story.

So, that's a rough picture of things. I think one of my regrets is that we had this paper with Amanda and myself in 2019 or 2018—I forget exactly the year—and I just failed to cause that much work to happen after Beth Barnes did a bunch of it at OpenAI, which was very good. Then there were a bunch of years where nothing was happening, and I failed to get it started at DeepMind, and it wasn't going to be widely done elsewhere in the field. So, I think we missed a number of years where we could have been making progress on that stuff. Alas, but now we're trying to do it again.

This is a bit of a side point, but one of the funniest things I've ever done with language models is set up a little—well, the hope was that they would have some synergy, but it was basically having one generate a name for something. I forgot what I was trying to come up with a good name for; it might have even been a friend's podcast or something. Then I had the other one come in, look at those names, pick the few that it liked best, and improve on them.

Boy, did that go badly from an actual quality-of-name standpoint. It was hilarious. We're talking about 14-syllable names for things in very short order, where it was like, “Yeah, this is not working. I'm not sure what you think makes a good name, but it's not this.”

The funny thing is that there are cases where using one model to get a level of eval is actually state-of-the-art at doing things. One of the theorists we're funding is finding that using one model to generate a complexity-theory proof and then checking it with another model is the best thing to do, because if you check it with itself, it won't be quite as stringent at checking natural-language proofs.

So, I guess it didn't work in that case, but that is a good thing that often does work: having one model check another one. Certainly in terms of flaw-finding, I've seen that work. It also seems, from all the scaffolding gurus that I mentioned earlier, that a big tip is to have a model from a different provider evaluate whatever one you had do the generation in the first place. Cross providers as much as possible when doing evaluations; that seems to be valuable.

I guess the idea is—or the observation is—that the models from the same provider have correlated weaknesses. So, you can definitely get value there, I think, in terms of flaw-finding. I've seen it work. It seems like the curves, at least—I don't know if you would characterize this differently—but what I've seen in terms of actual improvement beyond finding flaws and fixing them seems to plateau pretty quickly. Around 3 to 5 rounds of back-and-forth with not too much gain is how I would characterize everything I've seen.

The empirical debate experiments to date show the same effect. The original debate paper imagined dozens of rounds of debate, potentially, which is not what you see. If you have 2 experts debating, they don't get to say 2 things and then stop. But the models—at least maybe certainly in 2024 and the beginning of 2025—couldn't do more than 2 rounds.

Then there were some really worrisome signs about experimental validity. For example, there was a paper—it was a concept paper—which was generally quite a nice paper. It had the big caveat that the dataset was QuALITY—capital Q, little u, uppercase ALITY, whatever. But there was a feature that verified that the quotes the model was producing were correct, that they were in fact quotes from the stories that were hidden from the judge in this kind of debate game.

If you turned off that verification, honesty was still a winning strategy, which can't be the game-theoretic equilibrium. Because if you turn off verification of the truth, there's no reason honesty should win at all, unless the model is not very good at coming up with plausible lies, or the model is somewhat aligned and makes itself tell the truth, or it's giving itself away in some way when it's not.

So, we haven't really reached a case where empirically we're really testing the limits of this behavior. Again, I think part of the AI developer alignment story is still scalable oversight of various kinds. But we haven't really seen tests of this that probe how it will almost certainly be a few years down the road, when the models get very strong. That's again where the advantage of theory is that you can just pretend to be in the future on paper and prove you're there, as long as you've imagined it correctly, and therefore think about more limiting cases a little more readily. I think we just know from the structure of the empirical results so far that we are far from where those limiting cases will be, for a lot of these safety techniques.

So, what do you think of the prospects for formal methods to close this gap? I just did an episode, and you've—as my dad would say—forgotten more than I know about this domain. I just did an episode with the founders of Harmonic. They're one of a very small and distinguished group of companies that got IMO gold-level performance in 2025. Everything they do is output in Lean; that's the lingua franca of their models.

It takes a lot these days to take me aback with an AI vision for the future, right? There are a lot of big ideas. I did just give them a query that failed. I was testing Harmonic on some polynomial inequality, and it was true—I had a Lean proof of the inequality—but Harmonic didn't provide it to us.

The thing I would say is, I do think this stuff is pretty important. I'm advising a couple of people on funding flowing to formal methods. I think mostly this is for various kinds of information security. The math stuff is fun—I like doing the math stuff, too, for fun—but it's not all that important. And it's not clear that, for AI safety theory, it will be that much of a win over just doing things in natural-language math for a while.

But for software verification—either for hardening the world's security against various kinds of attacks generally, or for use when you're building AI-adjacent software directly, either at AI labs or the like—I do think this is potentially important. I think it's worth quite a bit of investment and pushing. One thing I'm hoping is that the various people who are doing Lean verification downgrade their fraction of effort on math and upgrade their fraction of effort on software, because I think it's almost certainly more important, even if it's a bit less flashy than much of what they do.

So, I do like this stuff. Again, I founded the natural-language-to-formal-theorem-proving subteam at Google Research with Christian Szegedy back in 2016. I did that for a while, and then I've done it off and on since, mainly for fun. I think it is important, but it won't really give you that much of the alignment story, in practice.

So, I really struggle with this type of thing, but I can tell you what they told me, and then I'll try to get your reaction to it. Their big vision for 2030—I asked them, “What does mathematical superintelligence look like in 2030?”—and they said, “We think we can get to a world of theoretical abundance.”

That means that, because these things are going to get so good at proving any theorem you want to prove, we'll have multiple grand unified theories of everything. All of the physical reality that we see will have multiple grand unified theories that could explain it, and then we'll have to do increasingly exotic experiments to resolve which of the candidate grand unified, coherent—you know, grand unified—

Well, we already have the core theory.

We don't need that. I think, again, the question is—I agree with this picture—the core theory, which is general relativity plus the Standard Model, already kind of explains everything for a good while to come.

But what about things like some of these hard limits that you would want to put on learning dynamics, or other ways of looking at some AI questions?

I think it is important, and I think the question is—but the problem is that a lot of these domains are not well formalized. For example, if you look at one of the wonderful three-letter organizations, the Alignment Research Center, which Paul Christiano founded and which is now kind of run by Jacob Hilton, they're trying to formalize when you can—even if the AI is not doing a formalized task—check its own heuristic arguments in some meaningful sense.

Or notice when there's a consideration that the model is using that you haven't anticipated, and notice that, react to it, and take defensive measures. But none of their problems are specced out formally.

So I think what that picture would look like, to the extent that we get better and better at the formalized world, is that you can formalize parts of your problem, and then those parts you can pound away on with Lean and various machine-learning assistance. But then the remaining pieces are the non-formalized part, and the question is: Is that going to be small enough that humans can keep track of it? Will the model be able to do it? Will they just get confused too, or hack the situation?

I think it remains to be seen what that situation looks like for something like alignment theory once this goes through. Because alignment, again, I don't think that we're going to get to proofs of safety of any kind. What we would get is theories with plausible assumptions, maybe, and then some theorems about those assumptions, and then some empirics that say whether those assumptions seem to be holding.

But there will be a bunch of judgment calls all across that stack, and the question is going to be: How does that go? So I think the thing I would say is that I'm excited for groups like Harmonic and the various other theorem-proving folks to keep working on this.

Look, I think it is potentially important for safety theory and alignment theory. But I also hope that they think through the detailed risks that they're trying to mitigate, what piece of the story they can address, and try to map that out in more detail. Because I think that right now there's not enough vision from those folks about exactly what piece of the story they'll be able to handle versus not. But I do like this stuff a lot.

Do you have any way of helping somebody like me understand the boundary between this sort of abstract, Platonic, formalizable domain and the non-formalizable domain? I asked Aristotle from Harmonic to prove “all is love,” and it said—this is in their informal mode, where you can give it natural-language stuff and it tries to formalize it for you—it spit that back at me and said, basically, “That's a philosophical statement. I can't really help you with that.”

Which is what I expected, but I can't say I know where that boundary is or how I should be thinking about it.

I can give you a much more concrete example of this. Say, in singular learning theory, there are some theorems that apply to the case when you're training a model—and by training, we mean doing exact Bayesian inference as you get more and more data.

So you have a set of data, and you're just applying the exponential-time Bayes rule update to find the optimal probability distribution over the final behavior. You can prove some theorems in that setting, and Aristotle absolutely could not hope to prove those now. There'd be no chance at all, but maybe in a few years it could be able to.

But then Timaeus, which is one of the main singular-learning-theory organizations, is not actually doing Bayesian ML. They're doing LLMs. So they're going to take intuitions from this Bayesian case and apply them to LLMs, which are not at all trained in some kind of rigorous Bayesian fashion.

Then they're going to do a bunch of approximations that are not actually grounded in any kind of theory. For example, they're using floating point, which has no mathematical properties—not enough mathematical properties to be able to prove much about it, except in limited cases. And they're going to do Markov chain Monte Carlo techniques, or what's called SGLD—fancy versions of Bayesian inference—on LLMs.

But they're not going to converge, so there's no theorem that says they'll get the right answer. And so you can see where part of the story will have some theory, and then another part is someone kind of waving their hands. The question is how much those connect, and that's going to be a bunch of hard judgment calls.

Do I understand correctly that the fundamental distinction is often in the intractability of the computation? It's because there's some infinite term in the math, and the amount of number-crunching I would have to do in the ideal case is too large, so then I'm kind of off the theoretical map?

Yeah, I think that's right. But there are other cases where even the infinite computation is not formalizable. The “all is love” case, you can't really formalize. But I think even in theory, I guess it's about computational limits.

The reason that LLMs appear to be doing what they're doing is not that LLMs—or any animal models, actually—are solving an intractable problem. You can take protein folding and write down limit situations of protein folding which all but provably take exponential time. But AlphaFold can still produce those folds. It just doesn't do them that way; it does them a totally different way that doesn't work in every case. And so it's doing a bunch of heuristics.

There are ways to formalize heuristics. For example, in complexity theory, you can say, “I'm going to have a circuit”—a rigorous computation—but it's going to be able to call some set of functions which can do some random things. We're trying to model heuristic computations, so you're modeling this fuzzy neural net as a circuit plus heuristics, and then trying to do theory in that setting.

But it appears that you're going to have to make some assumptions about these heuristics. You can't make schemes that work in the case of all heuristics. So the success case for this kind of theory will be to figure out what the assumptions should be that seem plausible enough, maybe have some support from learning theory—which is also going to be heuristic—and then prove theorems in this setting.

That's kind of modeling the parts of, I don't know, humans judging honesty or values, or our notions of fuzzy problems being correct or not. And so I think it's basically a case of more subtle versions of “define love for me,” which the machine won't give you an answer to. That is a reasonable intuition to start with.

Okay. Are there any other things that you're looking to fund from a research standpoint, or any other highlights from your own work? You've impressed me by continuing to stay active, publishing things even while doing this job. That's pretty cool and impressive.

AISI publishes on language models.

[Laughter]

I think the jailbreaking work we just did is quite cool. There's the Boundary Point Jailbreaking paper that just came out this week, which is basically a way to do black-box attacks. You take a jailbreak and then a harmful query, and you muck with the query until it looks like gibberish and the model doesn't think it's harmful.

Then you gradually make it less and less murky until you hit the boundary, and then you dance around that boundary until you find harder and harder attacks that eventually work. That team is doing a bunch of stuff of this kind. It's quite creative and important for mapping the secret space.

On the alignment side, the real challenge is that all of this is imperfectly formalized. Often, you go to the people we think know the domain best and say, “Hey, do you want to work on alignment?” There's some jump they have to make where we want to find people who are bought into the risk model enough that they're willing to explore in a fuzzy, sometimes unsatisfying definition space, to search around and find ways to connect theory and practice.

That's the thing—that is a challenge. I think the Alignment Research Center that I mentioned has had a number of conjectures they've put out. At the bottom of every one of their conjectures, they're like, “But by the way, we might have gotten this conjecture wrong. It's possible that if you prove it true or false, we'll realize that we didn't mean that. We meant a slightly different conjecture, and all of this new conjecture is risk-relevant or important for our safety agenda.”

And that's kind of unsatisfying to say to a theorist, but it's just fundamentally the real situation we're in.

As more people become aware of model capabilities and risks and so on, I'm hoping that more people with interesting domain expertise will want to really dig in, understand the risks, build up their own models, and then find ways to connect their area to the risks.

Something I often say is AI defies all binaries. I genuinely do believe that, and it seems right to me in a lot of places. But you showed this presentation that you gave at a recent workshop where you said it actually might be the case because we have things in computer science like P vs. NP, right? We know that some things are—or at least, it seems quite likely that some things are—genuinely, fundamentally hard, and other things are fundamentally easy. So maybe help me understand that. How should I update my worldview if I'm somebody who doesn't see that binary?

The way it works is, again, this goes back to the question of whether the superintelligence will be jagged, and the answer is yes—but only about super-superintelligent things. They won't be jagged about mundane tasks that are very easy. If I give you a task like, “Can I get a spoon from that drawer?” it's not exactly binary, but you're going to do it nearly every time. You'll just succeed with many nines of probability on that task, and it's easy for you.

I think the way to combine that view with this question of things sharpening one way or the other is that if you push—not to some infinite limit, but far enough along—you start out in the middle, and then some force will push you to one end or the other. But then, as you extremize, something else will still be in the middle. So that's how I put those 2 things together. That was a very abstract answer, which is the kind of answer I sometimes like. Follow up with me if you want.

How will we—I mean, as it pertains to alignment in particular—and I honestly have a lot of questions in AI, right? We have this weird phenomenon where, first of all, we're obviously moving through time, so in that sense timelines are getting shorter as time passes. But then also calendar-date estimates have come in a lot, and yet it doesn't seem like there has been much convergence of views. I wonder how you think we will—is that just going to continue to the singularity, or are we going to get some purchase on basically…?

Well, I think you do. I think people have often had very strong priors, and some people will shift and decide that things are binary, or that they should have model interpretability. Some people won't really, and they'll remain fairly sharply divided. They'll remain pinned on one side or the other. I guess I've been in the field of AI long enough now and seen enough people continue not to have strongly shifted that I think that will just keep going all the way along.

Yeah, yeah. In my forecasting thing for 2026, the only thing—of course, everything else goes up—but the one thing I actually estimated lower for this year than last year was what percentage of people will say AI is the most important issue. The big update for me was that if it didn't move last year, it might not move this year either, and it's probably going to be a busy year.

No, but from a very low number, though, I would expect that to go up just because it's starting from a small number.

I did predict it to rise. I think it was measured at 0.2 or 0.3 or something last year. I think I predicted 2% at the end of the year, and it basically still came in at almost no change, or very little change. So I think this year I predicted 1%. It's still going up somewhat relative to baseline, but my estimate went down from last year to this year.

Another comment that caught my eye in the presentation was, “Training is a mess,” and I think that's obviously true. I've been talking to the folks at Goodfire. You may have seen that they recently raised a bunch of money at a unicorn valuation and announced an extension to their agenda called Intentional Design.

They're looking at different ways to try to use interpretability techniques in the training process to understand, potentially even at a gradient-step-by-gradient-step level, what is being learned here in a semantic sense, and then be able to apply techniques to say, “Well, we do want to learn that sort of thing, but we don't want to learn this sort of thing.” Hopefully, that would make training less of a mess. How optimistic are you about that sort of thing?

I think you're worrying about what I meant. That doesn't change the mess. It's more like, if you look at a frontier lab, they have hundreds of people doing model training across many, many subteams. There are piles of datasets that are constantly contributing to the process, and there's iteration and many, many phases. They'll be automating part of the task, but then somebody spends some time looking at a spreadsheet with a sample of trajectories to see how things are going. That is a very complicated, almost emergent process.

Nothing about that Goodfire thing changes that at all. It just adds another wrinkle to the mess, in some sense. There's a really lovely line in it. When I was learning about ML in 2014 or whatever, I was reading one of Kevin Murphy's books on Bayesian ML. He had a great line that even the best Bayesian people will occasionally do some frequentist thing where they just do a quick check to see if their Bayesian thing is sensible. You shouldn't be too purist.

For better or worse, the training process at labs is extremely impure. It's super complicated: all these different people doing all these different spot checks and so on. I think that was the point I was making. That definitely is going to still be the case even if Goodfire does or does not do its slightly more complicated training method.

So does that mean you don't have much hope for methods that understand what the model is learning as it goes and shape it?

I do. I don't want to take a stand on whether it's forbidden or whether it's good or bad to do interpretability for training. I'll decline to answer that part of it. Generally, trying to understand in more detail the dynamics of training is very important.

I think that “mess” line was orthogonal to the question. There are a number of techniques that try to control what is learned. There was also this gradient-routing work by Alex Cloud, which is interesting, which tries to funnel certain knowledge into certain parameters in the model. Generally, I do think that there is potential to do interventions of this kind that are important and improve, at least, misuse-risk safeguards, but possibly also alignment.

Yeah. I think in terms of open-source models, one hope would be that you might be able to do some of that gradient-routing-type stuff and then release a version that's limited to experts or something, and give people almost everything they could possibly want, but not package the bio risk into it.

What do you think about open source? This is maybe a little late in the conversation to ask—a big, thorny question—but it seems like right now there's not really any plan. We're just going to hope that the frontier model developers surface any issues far enough in advance that, if anything is coming down the open-source pipe, we have at least a little bit of a window to react to it and do something.

But it doesn't seem like we're on any course to do anything if open source is about to become a problem. Any thoughts?

So, yeah, this is certainly a concern. On the alignment side, the alignment mitigations potentially do apply to open-source models, although you can also remove alignment if you get an open-source model that's aligned. For misuse risks, as you say, there is a class of techniques that just removes capabilities, which would give you some extra period of time.

That includes pretraining data filtering. There's a paper we had with Stephen Casper about that. There's a paper by DeepMind folks called something like “Unlearn and Then Distill,” which does a non-robust unlearning step and then distills into a different model. Because of the distillation process, you don't miss the parts you didn't unlearn. As you say, gradient routing could be a solution of that form as well.

But that buys you some time, and then the capabilities of models will catch up and you'll be able to pull that information off the internet, or in various ways, even if the model doesn't intrinsically know it. I think a lot of that is intervention on the margin. This is why, in part, we have conversations about governance, but also why we have conversations about non-model-side mitigations to these risks.

Yep. It all ends in hardening the world.

Okay, cool. Anything you want to share about the AISI’s work in diplomacy? Obviously, hardening the world and also improving cooperation would be a great general public good.

There’s still the Network for Advanced AI Measurement, which is a variety of organizations around the world doing similar things, and we’re part of that, helping to steer it to some extent. We’re the secretariat of the International AI Safety Report that Yoshua Bengio is leading, so we do a lot of work there. And then there’s a bunch of various venues, like wider venues such as the Delhi summit in India, and bilateral conversations with particular governments, basically. So, that’s a lot of what we do. We have a big international team, and that work is ongoing.

We are, of course, still in this voluntary regime. That work is about getting people onto the same page about risks, capabilities, and mitigations, but not more than that yet. But I think that information is important in case the situation changes in the future or governments want to take other actions.

Yeah, absolutely. Is the UK government and political class generally more optimistic about collaboration with China than the US political class?

I can’t comment on collaboration with China in great detail. We obviously work more with allied governments than with other governments. There’s not much more I can say than that, so it’s a bit sensitive, and I can’t quite speak to it.

I hope you’re finding at least some common ground with Chinese researchers and scientists. Put that in the suggestion box.

Yeah, I think that’s it. This has been fantastic. I really appreciate the time and all the extra time for my many follow-up questions. Anything we didn’t get to, or any kind of call to action you’d want to leave people with before we break?

I think one thing is that we are definitely hiring in a variety of teams. Particularly, the red team is hiring for pure jailbreaking stuff. Please apply. Other teams as well—we have a job board up. So, I think that’s the obvious call to action, and we have other roles opening up over the course of the year in various teams at different times.

We did one alignment project grant round last year in the fall. We had an alignment conference over the summer. We’ll probably do more things of this nature in the future, so look for those. And I think generally I just hope that more people who have different kinds of knowledge and expertise start working on the problem, and not just the labs.

One thing is that when I left DeepMind—I was at Google Brain and then OpenAI and then DeepMind—but when I left DeepMind, I had the perspective that I was just going to do policy work, advising on policy. Since then, in fact, I do a mixture of that, advising governments, but also a bunch of research. And I think there is a big place for independent research happening at various nonprofits and in academia, and also in governments.

So, I think that is very important to build up and not just have all the work happening at AI developers. Not that it is now; it’s just that more of it is better. More safety work and security work by independent folks.

Yeah, it’s definitely shaping up to be a whole-of-society effort, and the time to mobilize our resources would seem to be now.

I definitely also recommend folks, especially if you’re interested in doing alignment work and you have an idea that you don’t see too many other organizations showing an interest in. I thought your research agenda was quite distinctive in that way, and there’s at least some chance that people who aren’t on the most well-trodden path but have interesting ideas could find some willing collaborators at the UK AISI.

Check it out. You might as well look at the show notes. Definitely read the research agenda.

This is like 60 pages long, with a lot of concrete and some less concrete problems in a variety of areas as they would apply to alignment and AI control. So, please take a look. I should have mentioned that. Many open problems.

We’ll put a link in the show notes. Geoffrey Irving, Chief Scientist at the UK AI Safety Institute, thank you for being part of The Cognitive Revolution.

Thank you.