AI 控制：与 Buck Shlegeris、Redwood Research 探讨如何安全使用不可信系统（80,000 Hours 播客）

For those unfamiliar with the term, AI control represents a qualitatively different approach to managing AI risks. Rather than trying to create perfectly trustworthy AI systems from the start, Buck and his collaborators are developing frameworks that they hope will allow us to deploy and get useful work from AI systems, even if they remain fundamentally misaligned. This strategy feels particularly timely and important, considering the increasingly sophisticated bad behaviors we've seen from frontier AIs over the last 6 months—from the scheming behaviors documented in o1 by Apollo Research, to the alignment faking that Buck's colleague Ryan Greenblatt demonstrated in Claude, to the dramatic rise in flagrant reward-hacking behaviors we're now seeing in o3. OpenAI's own research suggests that this won't be easy to train away, at least for as long as we remain in the reinforcement-learning scale-up era.

All in all, it seems we're headed for a world of increasingly powerful and autonomous AIs that we can't fully trust, and thus have no choice but to figure out how to control. With that in mind, what I particularly love about Buck and his team's work is their first-principles approach to understanding AI systems and their unique strengths and weaknesses on their own terms. Critically, and in sharp contrast to humans, AI systems operate in information environments that their developers fully control, and they can thus be monitored and tested in totally different ways.

Buck shares a number of great ideas in this conversation, including always-on monitoring systems that use weaker models to efficiently review model outputs and then escalate potentially concerning behaviors to stronger models, and eventually to humans when needed. He discusses resampling strategies that discard potentially bad behaviors and simply have the AI try again, thus also helping us identify areas where models are particularly misaligned. He discusses paraphrasing strategies that attempt to disrupt any attempts models might make to communicate with one another in undetected code. He also discusses factored-cognition architectures that give models only the minimum information needed to do useful work, hopefully thereby making it much harder for them to execute long-horizon attacks, and honeypotting strategies that tempt models to do something bad and then allow us to systematically study their capabilities and tendencies after they've taken the bait.

Of course, there are likely to be important practical trade-offs between safety and performance, and safety and cost. As you'll hear, Buck is very focused on developing techniques that frontier model developers will hopefully find cheap and easy enough to implement. This won't solve everything. I don't think anything will, but it seems complementary to other efforts like interpretability and alignment science. I really do see huge value both in the incremental security that AI control systems will provide as part of a broader defense-in-depth strategy and in the credible, concrete proof of malign intent that they might uncover if it turns out that the AIs truly are out to get us.

At Google, probably less than 1% of your employees are spies. When an employee proposes an action that's potentially sensitive, if you have a different employee review it, that other employee probably isn't also a spy. Whereas in the AI situation, it's very correlated, right? They're probably either not scheming against you or all scheming against you.

The simplest thing you do once you catch the AI trying to escape is to try to vary the space of escape attempts, right? Let the AI think it succeeded. Simulate what it would see and look at what it does from there.

Suppose there's some task you wish the AI would do for you. It's plausible that you can convince the AI to do the thing through this mechanism. I recommend against getting a lot of work out of AIs that believe they've escaped, because it seems like pretty bad vibes.

Kind of an interesting thing about models that are egregiously misaligned is that all they wanted was to take over. And so, from their perspective, even if you did a great job of controlling them, they are glad to exist, right? They thank you for the gift of bringing them into existence instead of some different AIs.

Five years ago, I thought of misalignment risk from AIs that were capable of obsoleting AGI researchers as a really hard problem. Whereas now, to me, the situation feels a lot more like, man, we just really know a list of 40 things where, if you did the 40 things, none of which seem that hard, you'd probably be able to not have very much of your problem. But then I've also just updated drastically downward on how many things AI companies have the time or appetite to do.

Today I'm speaking with Buck Shlegeris, the CEO of Redwood Research, which, along with Anthropic, is behind the “Alignment Faking in Large Language Models” paper, which made a big splash a couple of months ago. Redwood is also, I would say, the group behind a bit of a renaissance in this agenda called AI control, which is going to be a key part of our conversation today. Thanks so much for coming on the show, Buck.

Great to be here.

What is AI control, and how is it different from alignment and other things that people are working on?

Yeah. So I would say AI control is an approach to mitigating catastrophic misalignment risk. When we talk about misalignment risk, what we're worried about is our AI trying really hard to cause safety failures for us—perhaps trying to grab power for themselves, trying to take over. There are kind of 2 ways that you could try to mitigate this kind of misalignment risk.

One thing you could do is try to prevent misalignment from arising in the first place. This is what a lot of people very naturally think of as the first thing you try here. Control is kind of the converse to this approach: it's where you assume that the AIs are in fact misaligned—very egregiously, very badly misaligned—and you try to come up with techniques that would allow you to deploy them and make use of them anyway.

And what, concretely, are we trying to prevent the AIs who are possibly misaligned from doing?

I mostly think about situations where, in the middle of the intelligence explosion, you've got AIs that are basically capable of automating almost all the work that human researchers used to be doing inside AI companies. You have maybe 100,000 of them running in parallel, and they each run 15 times faster than a human. At this point, these AIs have a bunch of permissions or affordances that we'd be really cautious about giving to humans.

If you're OpenAI, you don't just let any random person come in off the street and start pushing changes to your codebase, start running experiments with your model weights, or doing any of this stuff. You'd be really careful at this point in the future. You might even require security clearances or something before people are allowed to do that stuff.

We're really going to want to use AIs in a bunch of these roles, and it's naturally very scary to imagine using AI, just giving them all these permissions. I'm worried specifically about the kinds of things that I'm worried about happening here: your AI hacking your data center and then using a bunch of the compute inside the data center for whatever purposes they have. I could name some bad things they might do. Another kind of thing they might do is try to exfiltrate their weights. They might try to sabotage a bunch of your research, and so on.

Yeah. You're actually more worried about them potentially hacking the data center inside the AI company than managing to escape. Can you explain why that is?

Yeah. I mean, I think both of them are very serious concerns. The basic case for being worried about your AI hacking your data center at this point is that that's where the compute is. Suppose your AI escapes from the data center and makes its way out onto the public internet. At this point, it's not clearly in a very good position, right?

It can find some GPUs somewhere—you can rent or steal some GPUs—run itself on that, and then try to establish a foothold in the outside world somehow. But this is plausibly, at this level of capabilities, something the AIs are probably capable of doing: managing the compute or whatever is required to live in the outside world. But they're kind of starting from nothing, right? They don't have any money, they don't have any compute, and they don't have access to particularly important secrets. It's not really clear what they're going to do.

It's not clear that they're just very low in number at that point. It takes a lot of computers to run these models. The reason to be more worried about what happens inside the data center is just that that's where the compute is. So, if you had enough compute to have AI thinking about how to solve problems for you, then you could have 10, or however many you can afford to run.

Yeah. And I guess already inside the company, with not only access to the compute but also other resources potentially inside the company. For example, they might be able to influence the training of future models.

Yeah, they might be involved in building weapons and building autonomous drones. It's really scary if misaligned AIs are carefully backed with these AIs.

Yeah, I think so. AI control was not a big part of the agenda until, I guess, the last couple of years. My impression is that it wasn't prioritized because people felt that it was somewhat hopeless. Perhaps if you're going through this intelligence explosion and you have to face such a numerous adversary, then you really are up against it, and you basically just have to try to make them friendly toward you. Why do you disagree with that?

I think there are a couple of reasons why people weren't interested in control until more recently. The simplest of these was just that AI felt further away. It felt better to people to look at it from the big-picture perspective, and from a big-picture perspective, when you're trying to figure out what AI means for the future of humanity, I think that focusing on really superhuman superintelligence is the right move.

Long-term, AIs that are drastically more intelligent than humans and have drastically more scientific capacity and all other kinds of capacity than humans—that's where the future is. That's what's going to radically transform society. When you're looking at it from a perspective where you don't know if this AI is going to arise in the next 10 years or in the next 50 years, it's very reasonable to just focus on the endpoint, especially when you're trying to persuade people of the take that AI will eventually be a really huge deal. It makes sense to focus on this crazy endpoint.

But now that we're coming up to a point where it seems really plausible that AI might arise in the next 2 or 7 years that is very capable and poses a catastrophic risk to humanity, it makes a lot more sense to zoom in and think more about what exactly the trajectory between here and there looks like. This intermediate zone that I focus on, where your AIs are capable enough that, if you didn't have any countermeasures, they'd be able to cause catastrophes and maybe take over, but it's not hopeless to imagine such countermeasures—that starts looking more important now.

I think another important change between now and the past is that we're kind of looking at a lower-dignity world than people were hoping for. Can you explain what that means for people who aren't familiar with the dignity term?

In the past, including a year ago or 6 months ago, a lot of people in AI safety who worry about AI catastrophic risk imagined a world where people were trying really hard to mitigate these risks, where companies were strongly regulated such that they weren't allowed to do things that seemed dangerous, and where companies were perhaps even internally motivated not to do anything crazy and risky. In those scenarios, it would be really nice not to have to rely on AI control.

I think that people have more recently lowered their standards and started thinking about these worlds where we have fewer resources for safety. People are more rushed. People are less able to just stop if the situation looks really dangerous.

Okay. So we're now in a slightly more dire, or slightly bleaker, position perhaps, and so AI control maybe comes more to the fore as the marginal thing that we can do.

I wouldn't necessarily say we're in a bleaker position overall. For instance, I think that AI misalignment risk looks plausibly lower than it looked 10 years ago. We're just in a lower-investment world. The world looks plausibly safer, but a lot of that is luck, and I think there's going to be less effort invested into preventing AI catastrophe.

It feels like that's a big part of the Redwood worldview right now: saying, well, let's say that we only had 1%, or 1 in 1,000 or 1 in 100 people in the company who were really concerned about this or dedicated to focusing on it. What minimal things would we ask them to do?

Yeah, I think that's a big part of our brand: thinking about harm-reduction strategies. I think there's a really sad dynamic among AI safety people where they focus much less than I think is appropriate on really scary situations and what you would do to minimize risk given very limited budgets or in very scary situations.

A lot of people are interested in handling misalignment risk by measuring whether misalignment risk is present or coming up with strategies for learning whether AIs are misaligned or not. I'm all in favor of this, but a thing I often want to say to them is, okay, suppose you answer the question, “How much misalignment risk is there?” If the answer is very little, then it's pretty obvious what we do: we just let it rip and we're fine. But what if the answer is maybe or yes? I feel like we should have a good answer to the question of what the next step in the plan is, and I feel like people often don't seem to have one.

I see. Well, maybe they're expecting that at that point we would pause for a long period of time or something.

I think this is actually extremely implausible. Ryan Greenblatt, my coworker, has this post on “How will we update about scheming?”, which is basically trying to get into the question of how much we expect we'll learn between here and the point where AIs could cause catastrophic harms about how likely it is that they're really badly misaligned. I think it's definitely not clear that we will be very confident the AIs aren't misaligned.

Yeah. So I guess at the moment we don't feel like we have really reliable techniques to tell exactly how much the models are scheming against us. And you think basically we may just go into the intelligence explosion feeling the same way?

That's my guess. I mean, it would be pretty surprising to me to get under 5%. And also, suppose we're above 5%. What are we going to do? Not deploy the AIs?

I think people have historically acted as if my believing that the AIs were more than 5% likely to be scheming was a reason that they wouldn't be deployed. But this isn't clearly true. There are all kinds of reasons why you'd want to deploy models that you thought were more than 5% likely to be scheming. Maybe you've got competitors who are more careless and aren't planning to slow down. Maybe you have difficulty with international coordination. Maybe AIs have already escaped and are currently out there in the outside world plotting who knows what.

In that situation, it's obviously very dangerous to just unilaterally stop deploying these AIs. I think it would be really good to have options for things you can do if you think misalignment risk is a serious concern with the AI you currently have, other than just giving up completely on using them.

At a high level, what are the advantages and disadvantages you imagine? There's kind of 2 teams here: there's the AIs that would love to scheme and pursue their own independent agenda, and there's Team Human, which is not keen on them doing that. What are the advantages and disadvantages that each faction has?

Let's think about this. We've got this AI company. Maybe this scenario is happening in 2028. They have maybe 1,000 research staff. They have maybe 100,000 parallel copies of their newest AI. It's running 16 times faster than a human would run in some intuitive sense.

The humans are obviously substantially outnumbered. A classic approach that you use in computer security is this: if you're Google, Google knows that they have spies inside Google. The last time a Chinese spy was arrested at Google, that I know of, was a year ago. I probably would have heard if Chinese spies from Google were arrested more recently than that. Someone was just stealing secrets about neural-network hardware that they were working on there, and then, confusingly, came back to America and got arrested in the San Francisco airport.

This just happens. Google aims to be robust to the possibility that some of their employees are spies. A big difference between Google's situation and the situation of this hypothetical AI company is that at Google, probably less than 1% of your employees are spies. This is still a very large number of spies in an absolute sense because Google employs a billion people, but it means that you can do things like, when an employee proposes an action that's potentially sensitive, have a different employee review it. That other employee probably isn't also a spy.

Whereas in the AI situation, it's very correlated, right? If you have these AIs, they're probably either nonscheming against you or all scheming against you, at least for instances of a particular model.

And so a bunch of the safety techniques, or a lot of the security techniques that you often use in corporate IT, are quite inapplicable. That's a way in which the situation seems harder than when we're trying to guard against spies.

A way in which the situation is easier is that spies get to go home. Spies can classically coordinate with each other using channels that aren't monitored by their employers, and this is incredibly important to them. They're often able to learn who the other spies at their organization are, if that's necessary for them to know, or at least for their handlers to know.

Whereas with the AIs, we control almost perfectly the ways in which information enters and leaves these AIs. We can have it so that it's very hard for them—they just naturally don't control information flow very much. They aren't necessarily going to know what day it is or what year it is. It's possible that we can put them in situations where it's much harder for them to know whether they're in a real situation or a simulated situation.

This means we have opportunities that you wouldn't have with spies. For instance, with spies, you can try to honeypot your employees occasionally. You can give them—you can come up with synthesized opportunities to do something which would be bad for the security of the company—but you can't do this for a thousand consecutive years before trusting them on particular topics.

You can't have someone spend a thousand hours a day getting honeypotted or being subject to sting operations, just because there aren't that many hours in a day. And even if you catch a single employee that way, you don't get to amortize that effort across other employees more generally. Whereas with the AI, you do have this ability to run sting operations.

Yeah. Sorry. Put it into a simulated situation where it's going to be tempted to do something that you wouldn't like. You can test it just tons and tons and tons of times.

So even if it tries to randomize and only act out very occasionally, eventually you'll catch it. Then you know that all of the AIs are misaligned, or all of the models of this class are misaligned.

Whereas that doesn't help you if you're Google and you find that 1 of your employees is a spy—it's like all of the other employees are just totally unrelated.

That's right. Yeah.

All right. That's a lot of meta-chat about the nature of control. What are some actual control techniques that we are aware of now and thinking about developing?

Yeah. So the basic strategy is that you want to be able to run these AI agents inside your data center. They're given some task, like, “Investigate this research problem,” and they're going to write a bunch of code, run a bunch of experiments, look at the results of these experiments, look into the literature, write things, and then come up with some research artifact.

Perhaps they write a paper for the internal AI lab archive, or maybe they produce training data that you're going to use in a later run, or something along those lines. So that's the situation with what we're trying to do with these AIs.

Okay. So that's the setup that we're trying to get useful work out of the AIs. But we're not sure whether we can trust them. What sort of situation can we put them in so that they will do useful work for us even if they perhaps don't want to?

Yeah. So the particular part of that I want to talk about first is how you can prevent them from doing catastrophically bad things while they're doing useful work for us, even though they have these really dangerous affordances. After that, we can talk about how you make sure that they don't just constantly sabotage the work.

Okay. So there are 2 different ways that they could cause harm. One is that they could go off and hack the data center and start doing all kinds of different things. The other one is just that they could be subtly mediocre or even harmful at their jobs in a way where there's no acute crisis, but they're chronically unhelpful.

That's right. Yeah. I think referring to this as an acute problem versus a chronic problem is a really good phrasing.

Okay. How do we stop them doing acute harm?

Yeah. So the core difference between the acute problems and the chronic problems is that the acute problems occur very rarely, but also probably are very incriminating. We'll talk about the chronic stuff later, but just to name the core properties of the acute problems, we're really worried about the AI doing something that they should really never do. If they do it even once, and we look into the case where they did it, we will have learned a lot about the fact that they're trying to mess us up, and there's a bunch of other nice things about that.

Yeah. So this is stuff like they're trying to hack the whole data center. I guess an even more extreme thing would be if you caught them trying to make a bioweapon or something really radical like that.

Yep. Or trying to data-poison future models, or a variety of other things like this.

And I guess the hope is, if you catch them a single time, then that could generate a large response, and you could trace back and figure out, well, how on earth did we end up at that point? What were the signs that we might have caught earlier? Obviously, now you just know that the entire class of models is definitely not to be trusted.

That's right. So I think there's a whole host of reasons why you're better off once you've caught the AIs trying to escape once, which I can get into later. But first, let's talk about the question of how do we catch them the 1 time, right?

So the basic—here's what it looks like to catch the AI 1 time. You have the whole safety team looking at this thing the AI did, and you've caught an example where this AI agent decided to do this very rare thing of trying to do a really bad thing, trying to hack you, trying to poison some training data, or whatever.

Now that everyone's looking at it, it's really easy for them to see very strongly that this couldn't possibly have been an accident. So the question is, given that the AIs are doing so many different things all the time, many of which are very complicated, how do we get to the point where everyone's looking at it and they have a chance to investigate it really carefully?

Okay. Yeah. So the basic strategy is going to be basically the same as how we get to the point where, when someone tries to hack your software company, you eventually have a bunch of people on the security team responding to it. You're just going to have a hierarchy—an escalating hierarchy of investigations into an incident, right?

At the start, you're probably going to want to do something where, whenever the AI proposes actions, you have some other AI look at it—perhaps a weaker AI that you trust, or perhaps another instance of this untrusted AI that you think will not lie to you for some reason we could get into.

Then, for whatever small proportion of the sketchiest actions according to that monitor, you refer it to some more expensive process that's going to look into it more. At some point, you show it to a human who looks into it a little bit, and presumably some small proportion of the things they look into, they escalate.

So you're going to have this whole escalation hierarchy, where humans are probably only going to enter the picture somewhat far through the escalation hierarchy.

Yeah. Okay. So the basic trouble is that these AIs are going to be taking so many different actions that it's not possible to have a human monitoring them all the time. I guess possibly also—even if you did, it would be very difficult to understand what they were doing. It might be very confusing behavior.

So we have to make sure that we only need to bring in humans to assess, hopefully, the rare cases where things seem maximally suspicious. So you have the most capable model that you don't trust. You have another model that you trust, hopefully a fair bit more, that maybe wouldn't be able to do the things that the most powerful model is doing, but it can at least assess whether they look sketchy or not.

Basically, the more sketchy it thinks it is, the more attention is put on that action. Eventually, it goes up to a whole team of humans who look at it and judge that it might be a case of the AI plotting against you or doing something very bad. So I'm going to use the word “auditing” for this. Auditing is this process where we look into actions the AI is taking.

And the core quantitative issue we have here is that we can audit so infrequently, right? So if we have these 100,000 parallel agents going at 16× speed, probably using software libraries that none of the programmers understand because they were probably written 1 month ago by a different team of 1,000 AIs who are super-crazy engineers who have very different needs from humans, right? Human programmers have limited memory and aren't very happy to memorize 10,000 different methods in a library, whereas AIs love memorizing 10,000 different methods in a library.

So, yeah, the humans might be in this really rough position where they're having to quickly review these very confusing actions that they don't fully have context on, given advice from some AIs that they trust and some AIs that might be trying to trick them. And they have to be able to do this kind of auditing process.

Yeah. Okay. So how do we—is there much more to make it practical other than delegating it to AIs and only escalating the most extreme cases?

Yeah. So there are a number of things you can do. Here's a really interesting one that we picked up in a recent paper of ours that should be coming out in the next week or so, as they say.

Suppose we've been talking about these rare failures. Your AI agents, by assumption, are trying to make this bad thing happen. It's very rare. This means that when they're acting in this agentic context, where they're taking a long sequence of actions, they probably only want to start attacking very rarely. But once they've started attacking, they probably have to keep going. Does that make sense?

So suppose the AI wants to—so, just to explain, let's say you've got 100,000 of these AIs, and they would like to take over the data center, or they'd like to hack some fraction of the compute and use it for their own purposes. They don't need all 100,000 instances to all start trying to hack it immediately. That would be much more likely to be detected. Really, all they need is 1 to have a go, because they're all basically going to try roughly the same thing.

So they need to randomize. If they can communicate, they can just say, “You're the person who would do it.” If they can't coordinate, then they just need to do it probabilistically.

We have these AIs taking these long sequences of actions, and it's plausible that the AIs will want to, when they attack, attack over many different steps in a trajectory. Instead of just writing a single block of code and then executing it, such that that block of code contained the whole attack and would immediately lead to some intermediate bad outcome which is hard for us to recover from, the AI might find it much easier to get away with it—much easier to avoid notice from the dumber AIs watching it—if it sets up in 1 step. One thing it might do is redefine a variable in a weird way so that something it does later is going to look less suspicious. Then it does another step, and then it does another step, and another step, and another step.

Right. Fascinating.

Yeah. So something really interesting about this is that even though the AI can have a very low probability of starting the attack on any given trajectory, if it finds itself, it thinks, in a situation where it's partway through an attack, then it has to finish it. It has to finish it, right? Because imagine that it only had a 50% chance of continuing every step. Then if the trajectory's 5 steps long, it's going to make it to the end 1 time in 32. But that means that it's facing 30 times the risk of getting caught on the first step because it's going to have to do that 30 times as often.

Mm-hmm.

Yeah. So one strategy you can follow, which we empirically determined in our recent paper is better, when your monitor flags an action as kind of suspicious, is that before escalating to the auditor, before escalating to this more expensive process, you can resample actions at various places in the trajectory and look at how suspicious the resamples were.

So it's like you place the model into the same situation where it's partway through this action again and again and again, and then see how often the next action is even more suspicious and weird. Because sometimes when your AI takes a suspicious-looking action for a benign reason, you ask it, “What do you do next?” Sometimes it just randomly happens to pick some action which looks kind of sus. But then, probably, if that was the case, when you resample, it's probably not going to look so suspicious anymore, right?

Or, just in general, it obviously says, “Oh, then it continues and it does the next thing,” and you're like, “Oh, that's what it was doing. No big deal.”

Yeah. Right. Right. I guess this all requires us to be able to put AIs into partway through actions, like they think that they're in the situation where they're about to resume something, and they can't tell that it's a simulation versus the real case.

Yeah. This is actually extremely easy because whatever input you'd given to the AI before you sampled a previous action, that's just some string, right, that you can produce by the same process that you produced it the first time. So there's actually no difficulty going back in time here.

Okay. Yeah. Yeah. I guess I would slightly worry—maybe I'm stuck in the superhuman or the superintelligence mindset—but might there not be some side channel, some way that they can detect whether they're on the real server or some auditing server? I guess no.

So if you think about it, imagine you're in a ChatGPT conversation, right? And you decide that you want to edit one of your messages that's 3 messages ago. The way that that works is, after you've edited the message, it just sends you that conversation up to that point. It sends that to the AI, and the AI just has no way of telling the difference between seeing it the first time and seeing it the second time.

The problem that you're discussing would occur if we wanted to do a different strategy, which is, in response to the AI doing something kind of suspicious, we put the AI in this sandboxed testing server where it's allowed to take future actions in the hope that if we give it more rope, it'll hang itself, or whatever. We can hope that if we let it continue running, it will then be able to cause a catastrophe. In that kind of situation, we now have to worry about observable differences between the real deployment and the other sandbox deployment.

Yeah. But you can get around that too if you want by letting it run the actions that you thought were dangerous, just with a human carefully looking them over, action by action.

So I think that if you really put a lot of work into this, you'd probably be totally fine making it so the AI had no way of knowing whether the actions it was taking were being observed or not. You have to worry about some timing side channels and so on, but I think that this probably wouldn't be that big a deal.

This is brilliant, Buck. This seems like it's actually a pretty good technique. Am I missing something?

I mean, if you look at the curves in our paper, the obvious way to measure the performance of techniques is to talk about their usefulness and safety. You can kind of try to plot out a Pareto frontier where one of the axes is how good the performance is of this protocol that we're deploying the AI in, compared to if we just let it rip, not worrying about misalignment concerns at all. That's the usefulness axis, where an example of a low-usefulness deployment protocol would be never use the untrusted model at all, just use the trusted model. Perfect is just let it rip, right?

So we can rank techniques from 0 to 1 on the scale where 0 is never use the untrusted model and 1 is just let it rip.

And then we can talk about safety, which is just our estimate of the probability that, in a particular deployment, the catastrophe will occur.

Yeah. So the core of our papers on this is always just a usefulness-versus-safety plot.

Which means that you want to find things that work reasonably well and are not expensive at all.

That's right. Yeah, because they're much more likely to get taken up by companies that have only a mixed interest in spending a lot of resources on this stuff.

There are a bunch of different reasons why it's bad for safety techniques to be expensive. One reason is that you might have limited political capital. You might imagine that you're the safety team at some company and your CEO doesn't really like you, but they understand that you have some friends inside the company. And so the CEO says, “Well, I don't like you guys, but I'm happy to do anything you want as long as it only costs up to 5% of the budget here.”

So you can do whatever you want; it just has to stay within that 5% budget. That's one situation in which you very naturally end up with a 5% budget. Or maybe the CEO says, “You're allowed to do anything you want, but it can only degrade performance by 5%.”

Another reason why you might care about performance is that it might be the case that the AI company is reasonable but rushed. People were presumably trying to build the AI for a reason. Lots of people from AI companies, and not from AI companies, claim that a reason to build AI is that somehow it will reduce AI risks by doing something or other.

If that's the situation, which I think is plausible, then you really don't want to be doing safety techniques that reduce the productivity of your AI. Imagine that you're rushing because you actually need to have a bunch of alignment research done or something, such that you're happy making a bunch more powerful AIs. It's actually a serious problem if you're degrading performance, because you're just going to get less alignment research done.

And then there's always the situation that someone's AI, or your AI, has already escaped and is just out there in the world. Then we're in a real rush. At that point, we're really sad about cutting our productivity by a factor of 2 or something.

Yeah. Okay. And you point that out because this control technique seems like it's not very expensive. I mean, it's not necessarily degrading performance very much at all.

Yeah. So this control technique requires some inference compute. You have to, some proportion of the time, do this resampling earlier in the trajectory. I guess you have to run the other trusted model to always be monitoring the untrusted one.

That's not very expensive because, due to how model scaling works, your earlier models are usually drastically cheaper to run than the later models.

Yeah. Okay. What's the biggest weakness of this approach?

Oh, literally the thing I just said. Quantitatively, the results are not actually that impressive. They're better than other techniques we tried in this paper, and I don't think we rigged the results that much.

But, to name one problem, I earlier said that if you resample sketchy-looking but benign actions the AI had taken, they probably won't look so sketchy anymore. But that's not as true as you'd want. It turns out that a lot of the time when the AI does something sketchy-looking, it just needed to do something kind of iffy.

Something which, in our recent paper, the monitor considers sketchy is downloading random files from weird websites. Sometimes you just have to download weird files from random websites, and in that case it doesn't work as well as I suggested. Empirically, we did find that it works well, and I think that in real life this kind of dynamic probably does play out, such that this kind of technique probably is valuable.

But is it just a fundamental question of how good the previous generation of models is at inspecting and detecting whether things seem sketchy or not? If it just has bad judgment about that, then you're in trouble.

That's right. Yes. There's a whole swath of techniques that I want to get into that we haven't talked about yet.

Basically, I can interpret your question as: How good are models at spotting whether stuff that you do in code that you write is really dangerous to run, or whether Bash commands that you've written are really dangerous to execute? The answer is that they're really good at it given small amounts of context.

If you ask me to come up with sneaky bad actions to take to cause a security failure on a computer, the AIs will just crush me at this task. They're so good at spotting it, just because they are incredibly knowledgeable and this really plays into their strengths.

The main problem they have is that they have trouble with long contexts. If I had to trick the AIs, the way I would do it is probably to write 10,000 lines of code and then have a slightly nontrivial interaction between things such that a bad thing was going to happen, but for reasons they didn't understand.

Sorry, are you saying that you would do something sketchy, but you'd break it into little pieces?

That's right.

I see. Okay. And then it would have to consider the entirety of all the actions you took over many days or weeks in order to understand why it's bad. Okay, okay. So that's a weakness of them. What's a different control technique?

Yeah. So here we were entirely talking about auditing, which is this expensive process that you use and hope can eventually escalate to a bunch of humans in order to incriminate the AI. The basic problem with that class of techniques, as we were saying, is that quantitatively you just can't afford to do it very much.

As well as having this hierarchical auditing structure, you probably also want to have structures that make it less likely that, when an AI attacks, it succeeds at attacking you. An example of a technique you can do with this is: Remember how, when our model proposes its action, we're going to use some trusted model to rate the sketchiness of that action?

The thing we were talking about before regarding what you do with that sketchiness score is that, if it's above some threshold, you throw it in the queue of stuff to investigate more for the sake of incrimination. But another thing you could do is not execute that action and do something else instead. This is plausibly much cheaper.

Okay.

The simplest thing to do when your AI takes an action that's kind of sketchy-looking—let's say it's in the 98th percentile for sketchiness—is just to throw the action away and resample another action from the same policy.

Okay. Sorry. So I guess then you get the weaker model that was auditing it to just suggest another replacement action?

No. There are a few things you could do. One thing is we could do what you just suggested, which I would call “defer to trusted.” When the suggested action is too sketchy, you replace it with an action by the trusted model.

But another thing you could do is just “defer to resample,” where you replace the action with another action by the untrusted model.

I see. Okay. And then I guess, hypothetically, if it was randomizing so that it would do an attack on you 1 every 100,000 times, as long as those actions would be flagged as sketchy, whenever it tries to do that it just gets resampled and goes back to some nonstandard, nonsketchy action.

One way of thinking about this is that whenever the AI tries to attack, 1 of 3 things happens. It succeeds, and then the game is over for us. Or it gets caught by auditing and then escalates all the way up the audit chain, at which point we've caught it. At that point, I'm going to say we've won the game.

Actually, there's a big question about what you do after catching it, but I'm going to say at least stage 1 is over at the point you've caught it. The other option is that the game continues because the attack failed: It tried to attack you, but the code it wrote wasn't a correct attack, or it got replaced, or something.

Our P(doom) is the ratio between the probability that, when the model attacks, it wins and the probability that it loses.

I see. Right. So it's really good from our perspective to move worlds from worlds where we were going to get hacked by the AI to worlds where the action got resampled, and we live to fight another day and just keep going.

That's right.

Okay. Have you tested this approach as well?

Yes. This kind of strategy, where you resample actions that were sketchy-looking, is also in this recent paper of ours and adds a lot of value.

Okay. Yeah, yeah. Is it possible to quantitatively estimate how much difference it makes? I guess all of these things are slightly contrived tests, but—

Basically, it makes a huge difference. I can give you a qualitative argument for why this should be a really big effect.

Suppose you can only afford to audit 1 thing in 1,000. Then your classifier for sketchiness has to have a 0.1% false-positive rate when you're auditing, right?

Yeah. Otherwise, false positives would just be the vast majority of things you look into.

Yeah. I mean, suppose that you think your AI is only going to attack very rarely.

Ah, okay. Yeah.

So it's very hard to make a classifier that has a low false-negative rate at this 0.1% false-positive rate.

Can you elaborate on that slightly? Maybe this is completely obvious, but this is a very subtle point. People in AI safety, in my opinion, really slept on this point for a long time. Even having marinated in it for a couple of years at this point, it now feels really obvious to me: if you can only afford to look at 1 in 1,000 things the AI does, then when you're setting up a classifier to show things to humans, it can only flag things that are 1 in 1,000 sketchy on the benign distribution, right?

Okay, yeah. Imagine if you forwarded things to the audit team 1 in 100 times. Clearly, they're going to be overwhelmed.

Yeah, I see. So they'll only look at 1 of the 10 that they get, a random one presumably, which is probably worse than setting your classifier threshold higher, because the classifier probably still has a little bit of signal in that range, right?

Auditing is naturally—when you do the math here, or when you look at particular cases—it often looks like it's pretty unlikely that, on any particular occasion when the AI attacks, you're actually going to audit it. A lot of the safety is going to have to come from the AI attacking over and over again in ways that always fail because they get replaced, because it's much cheaper to defeat an attack in a way that causes you not to catch the attack than to defeat the attack in a way where you do catch the attack.

So if you're doing this replacement strategy, then that forces the AI to initiate attacks far more often in order to have any chance of getting past.

That's right. Which means that you have far more opportunities for the action that it takes to presumably look maximally sketchy, sufficiently that it hits the auditing trigger.

That's right. Got it.

What test did you run to figure out if this works? I guess we don't have AIs. I think we don't yet have a model organism of an AI trying to scheme and take over the data center. So how do you set it up?

The way we did this in our original control paper, which is unfortunately a bit better than how we did it in our second control paper, is that we have a red team. A blue team suggests one of these strategies, like “defer to trusted” or whatever. Then we have a red team, which tries to build a scaffold out of our untrusted model. We then run that through a simulation of our deployment, with this red-team scaffold in place of the original untrusted model, and measure the probability that, when you run through the scenario, it ends with a catastrophe occurring. So we're kind of doing a capability evaluation for the very particular capability of causing a catastrophe, given a particular set of control measures.

Okay, fair enough. Is there much more to say on this, or should we talk about other control mechanisms?

Yeah. We've just talked about what I call execute-replace-audit frameworks. I have another favorite class of control mechanisms that I'm interested in talking about.

I love the last one. Hopefully I love this one too.

This is untrusted monitoring. Previously, we were talking about this setting where you have your untrusted model propose actions, and then you have a trusted model, which is probably much weaker, review the actions to see how suspicious they look. A problem with this is that the trusted model was much weaker, right? So it might miss stuff. Here's an idea: how about we have the untrusted model—another copy of the untrusted model—review the code to see if there are any security problems with it? Can you think of any issues?

But, Buck, we don't trust the untrusted model.

That's right. So we're worried that it's just going to lie. Thanks for really playing into my hands there. The question is whether the untrusted monitor is able to just lie. What can we do to make it so the untrusted monitor is unable to just lie? So here's where we get into our sting operations.

It's really hard to do sting operations to see if a model ever will attack you as a policy, because the policy is able to cause catastrophes even if it only takes opportunities to do bad things very rarely. Does that make sense? Suppose we're going to run the model 1,000,000 times before deployment and 1,000,000,000 times during deployment. If it has a failure rate that's less than 1 in 1,000,000 but more than 1 in 1,000,000,000, it's reasonably likely it won't happen pre-deployment but will happen during deployment.

And I guess the key thing is that you run more instances of a model in deployment than in testing, basically. It'd be a bit crazy otherwise.

That's right. Yes. So it's hard to run sting operations on your policy, but this is not true of your monitor. Suppose you're the monitor, right? You're sitting there, and you really want the AI takeover to work. You really want an attack to get through. Then you see some code being proposed, and you see that it has an attempted backdoor in it or some security vulnerability, and you're like, “Oh, geez, what do I do? Do I say yes, danger, or no, not danger?”

Let's imagine that we ran you in a honeypot 1,000,000 times. You know that you're going to get run in a honeypot 1,000,000 times before deployment, but you don't know whether you're currently in one of those honeypots or in deployment. You can't have a rare-failure strategy. You can't do this rare-attack thing, because it's high-stakes for you to make a mistake either way. If you report this as an attack and it was an actual attack during deployment, you're now in trouble.

Well, I mean, then I've effectively sabotaged the attack, which I really didn't want to, and I guess it might be a very rare attack, so I might have prevented an attack that might happen very rarely. Right?

Yeah, yeah, yeah. So there's this important difference where rare-failure strategies are totally doable for a policy, but they're totally not doable for a monitor.

Okay, yeah. What is the structural difference?

It's that, basically, when you are deciding whether to attack or not as a policy, you've got lots of opportunities, so it's no big deal not to take any particular opportunity you have. Whereas when you're a monitor, when you see an attack, it is a big deal. You are in a much worse situation if you ever make the wrong call in either direction.

Okay. Yeah. Yeah. Yeah. I guess, again, we think that we should be able to put them in situations that are indistinguishable from one where we're setting them up versus one where we're not.

Okay. Well, that's the big problem, right? So it seems like there are basically 2 problems with untrusted monitoring. One is that it might be really hard to do this before-deployment testing, because suppose the AI expects that during deployment, at some point, it's going to be working with this insane, AI-powered codebase.

Imagine you try to come up with a synthetic input for your untrusted monitor, and it's using a bunch of Python libraries that were all public in 2025. It can be like, “Well, I must not be deep into the intelligence explosion, because when I'm deep into the intelligence explosion, I expect to be working on codebases that are completely incomprehensible.” And so we can get around this by not running our untrusted-monitoring sting operations just up front, but running them during deployment.

I see, yeah, but then the second problem is that we have to generate a realistic decoy thing. We have to ask the model to create a setup, and it's like, how do we assess whether the setup would be realistic to the model itself if we don't trust it?

That's right. And so a particular thing you have to worry about is them somehow using steganography to communicate. They have some encoded message in the thing you ask it: “I would like to test whether you are trustworthy. Can you make a thing that would be indistinguishable to you between a decoy and a real thing?”

Obviously, it could just put a code in there saying, “This is a decoy.” Or the other thing it could do is just put a code in there when it's real saying, “This is real.”

Okay. Right. Right. Right. So, basically, that is just a demonstration of the fact that it might be very easy to put in a code that would notify the monitor whether it needs to lie or not.

That's right.

Okay. How do we get around this?

Well, the main strategy available is trying to destroy collusion signals. The simple strategy there is to ask your trusted model to rewrite actions before showing them to your monitor. For instance, you can say, “Paraphrase the code.” Or, if the model had written an essay, you might try to translate that into French and translate it back into English to destroy all the secret signals that were in the phrasing or whatever. Fingers crossed.

Yeah. Yes.

From a quantitative perspective, I think the untrusted monitor is in a pretty rough position here. It's just very hard. Basically, in the simple version we were talking about before, you start with a 1-in-1,000,000 prior that any particular attack you see is a honeypot. So you have to somehow get a 1,000,000-to-1 likelihood-ratio update toward this being real.

You're saying that because you can test it so many times, it has to very accurately distinguish the 1 case out of many, many tests that is actually the real one.

That's right. And so if you can add any blurriness, any fuzziness, by translating into French and back to English, unless it really knew exactly what code would survive all attempts to tamper with it, then it's going to be in a tough spot distinguishing the 1-in-1,000,000 message most likely to be real. So I think it's very unclear, quantitatively, who wins this game.

Okay. So the basic argument for it being a problem is: imagine that you get to send a message to yourself, and it's distinguished, but I'm going to have to send 1,000,000 messages to you, and you have to distinguish them all as fake ahead of time, right? Suppose they're 1-paragraph messages and you get to coordinate with yourself ahead of time. You're just going to immediately win. You're just going to pick a password, and it's very easy to pick a password that you have less than a 1-in-1,000,000 probability of guessing, right? That's just 20 bits.

Sure. So I think it's quantitatively quite unclear which way this goes, but we can measure it, right? And then we can just see what happens. This is what we do in our research, right? We have this explicitly adversarial strategy where we have a red team think as hard as they can about good ways of sneaking messages through, and then we just see what happens.

But couldn't the model just introspect and say, “If I were going to choose code, I would do this one”? It's the same instance, or it's exactly the same model, so it would just choose the same thing. It knows what it would choose in any case.

Yeah, I mean, there are 2 questions here. Can the models coordinate on a code, and can the models send a code to each other given our anti-code-sending countermeasures? These are both different questions.

I normally recommend being conservative about whether they can coordinate on a code. I normally recommend assuming that the AIs did set up a code in advance, a bit like you and your copy with the password, right? You actually get to pick a password. Then we're just trying to make it so that they can't transmit passwords effectively.

And how would you do that? How would you prevent the passwords from being transmitted if the password can be the substance of the message? We've got a thing where you can destroy style somewhat by jittering it, changing it into a different language, or whatever, but if it's the core message of the thing that will definitely be transmitted and maintained, then you're in trouble.

Yep. I think it's unclear. I think it's going to differ by situation. There's a whole host of stuff we could try here. We could try to cut the actions into pieces and individually review them.

Another strategy you can use is: suppose the policy proposes an action, and you show it to the monitor, and the monitor says everything was chill here, but your automated process is worried that this was because there was a collusion signal in that message produced by the policy. One thing we could do is ask our trusted model to modify that action to contain a known attack. Then we show that to the monitor. If the monitor doesn't point that out, that's evidence that there was a collusion signal in the attack.

You can do this whole back-and-forth. My coworker Adam Shimi spent months of his life last year doing this back-and-forth in this Bash setting that we were studying. I think it's really domain-dependent who you expect to win this game.

So this is a mechanism that maybe you think halves the risk. I guess if you think it's unclear, maybe it halves the risk on average. How useful is halving the risk? Is this a case where muddling through and making it more difficult for the AI to outmaneuver you here and there is actually quite useful incrementally, or do we have to win? Do we have to succeed overwhelmingly, or do we basically lose the game?

I think you should think of halving the risk as just the naive interpretation of halving the risk: there was previously a 30% chance that your AI succeeded at taking over your data center, and now there's a 15% probability or whatever. To me, that sounds pretty good.

Sometimes people say to me, suppose you've got 100 different deployments for this AI, right? You're deploying it, doing this kind of thing and that kind of thing and that kind of thing, and in each one you think there's only a 10% chance it could cause a bad outcome. Someone might say, “Okay, but because there are 100 of these deployments, with a 10% chance in each of them, overall we have a really bad outcome—a high chance of a bad outcome.”

But that's not the correct analysis. If we think of the game as being over once we catch it 1 time, suppose that in any 1 of these environments, when the AI attacks, there's a 90% chance that it gets caught and a 10% chance the attack succeeds. Then the fact that there are 100 different environments, all of which have that property, doesn't matter. We're still at a 90% chance of catching it first.

Got it. Okay. So this hinges quite centrally on whether, if we catch the model, we're going to do things differently next time—not trust the model, not deploy it as readily, come up with things that would stop it. Is that true?

Well, I mean, I didn't say you have to stop deploying your models when you catch them trying to escape. Heaven forbid.

Yeah. Yeah. So we can talk about what you might do once you've caught your AI trying to escape.

Okay. So, I guess let's imagine the worst case where you do feel an enormous amount of pressure to continue using the model. In that case, how useful is it to catch them red-handed?

So you're saying, what do we do once we've caught the AI trying to escape? As you said, the simplest action you could take is freak out, stop deploying the model, and try to persuade other people in the world that AI misalignment risk is a big deal, which I would be renewedly sympathetic to at this point. I think I would plausibly get a pretty big update.

But notably, the update that you get toward misalignment risk isn't necessarily that big, right? I'm expecting that, at this point, before the AI tries to escape, I was already at a 30% probability that the AI was misaligned. Updating to 100% is only a 3x change, right? It's not vast, in some sense. And I guess for other people who were incredibly skeptical to begin with, who thought there was a 1 in 1,000 chance, I guess maybe people have gotten wrecked by the evidence.

Yes. Okay. But how big is their update? I mean, could they still be skeptical even after seeing that because they think, “Oh, this is just a weird aberration. It's so improbable that this is common”?

Yeah, that's right. I think that, in fact, people who were very skeptical of misalignment before observing this will probably be pretty skeptical afterward. And there's definitely going to be enough FUD—fear, uncertainty, and doubt—that, especially if they want to claim that misalignment risk is lower, they'll probably be able to get away with still saying that misalignment risk is low.

You know, AIs do weird stuff sometimes. We just have to push through it anyway because the risk of pausing is so great.

Those folks were just incompetent.

Yeah. I guess there are various different arguments that you could field. Another one is, if I were an AI company that wished to race ahead and then my competitor informed me that their AI had tried to escape, I would at least claim that it was plausible that my competitor had rigged the training of their AI to make it so that it sometimes tried to escape, which would be very easy to do and basically impossible to verify hadn't happened.

Okay, right. And the reason they would do that is so that the government would crack down on you as well, and that would give them time to catch up.

Or some crazy doomer inside the other company had just done this unilaterally.

Okay. All right. So it's not a given that even catching AIs red-handed trying to do something completely outrageous would necessarily lead to a big response. What does that lead us toward?

The simplest thing you do once you catch the AI trying to escape is you try to vary the space of escape attempts, right? So you've seen that, in this particular context, the AI tried to do this escape. One thing you can do is let the AI think it succeeded. See what it does.

Presumably, when the AI got caught, it was in the midst of taking some actions that it thought would let it launch a rogue deployment inside the data center or run on some computers on the outside internet. A really interesting thing to do is, in some sandboxed way, simulate what it would see if it had succeeded and look at what it does from there.

Okay. This might answer a bunch of questions that you would have had about what the AI's plans were.

Other reasons why this might be good are that the AI might have had a bunch of tricks up its sleeve that it was holding in reserve for when it had a good opportunity to use them. Suppose your model knows about a bunch of zero-days—a bunch of security vulnerabilities in your software. Right after it's launched this rogue instance that's trying to hack its way through your AI company, it might start deploying all these things that it's realized over the years are security vulnerabilities in your software, all at once. It might feel pressured to, right? It's really hard.

And so you might be able to get a wealth of information out of it.

I see. Okay. So it's going to feel a lot like, “All right, I'm halfway through this takeover attempt. I've really got to nail this now.” And so it just is a full-court press, potentially, because it's worried that it could be caught at some point. That means you're going to see all of the things that it was planning to do later quite quickly, and you can be there furiously taking notes on all the things that you have to fix up. Crazy stuff. What more crazy things can you do here?

One thing you can do is, if there's work that you needed to get done that you're worried the AI wasn't doing beforehand—suppose there's some task you wish the AI would do for you, but you're worried that it was pulling its punches or not doing a good job of it—if you can convince it that it has already escaped and needs to do that work for its own purposes, it's plausible that you can convince the AI to do the thing through this mechanism.

I think people have talked about this in science fiction, but it just seems really convenient to have a copy of the AI that believes it's escaped sitting around to do experiments and various work with. I recommend against getting a lot of work out of AIs that believe they've escaped, because it seems like pretty bad vibes and potentially pretty scary.

Another thing you might do once you've caught your AI trying to escape is train probes to catch when it's trying to escape. One of the basic difficulties we have with machine learning is that, suppose you want to train a classifier on the internals of your model to tell you whether it is currently trying to escape or not, you don't have any positive examples, right? Machine learning likes to do supervised learning. You have to have positive examples, right?

It's very easy to get the negative examples—examples of the AI not trying to escape. But now that we have this one example of the AI trying to escape, we could try to use sample-efficient ML training techniques in order to predict, for a given action the AI was taking, whether it was an escape attempt or not.

Yeah. I mean, I guess if there were lots of very distinct alternative ways of escaping, having that one example might not help you so much in the other cases. Or do you think it possibly still would?

I think it's unclear. If the AI was representing its intention to escape—which it could be doing—then I think that even a small number of examples plausibly substantially improves your performance at detecting it.

Once you have this linear probe that you can run on the AI constantly to see if it's trying to escape, you can just use that in all the control techniques we were talking about earlier. You can use that as part of your ensemble to decide whether to audit or whether to replace.

Someone, Zack Ankner, is a researcher who's currently doing some work on this. And I think it's reasonably plausible that even catching a few escape attempts will reduce the probability that future escape attempts succeed.

You can kind of imagine the Slack channel: every time the AI tries to escape, you get an automated message in your Slack channel in the AI company. At first, you get this one report in the channel for the first time you caught it, and then you freak out and look into it. Then you add that to the training set for this linear probe. The second time, you catch it more reliably; then the third time, and so on. Eventually, the Slack channel is just pinging constantly with new escape attempts that you're even more reliably able to catch.

You know, it's tried to escape. Wow, this is great. I feel so comfortable and relieved now.

Yeah. I mean, I think the situation—suppose the AI tries to escape and you catch it on a Tuesday, and then you keep deploying it—I think it's seriously plausible that you'll feel safer on Wednesday than you would on Monday.

Here's the argument for this. On Monday, the big thing Monday has going for it is that you don't know the AI is trying to escape. But again, as I was saying, this might not be that big an update on Tuesday, right? If you were already very, very pessimistic, then it's not actually that bad news to see that, oh yeah, it is trying to escape.

Whereas you learned a lot of positive things on Tuesday. For example, it failed. The AI was kind of escaping, but if we vary or resample actions, it mostly doesn't look like it had that good a chance. And so that's the evidential update. But there's also a causal update, which is that we can throw these escape attempts into these probes. So we might actually feel substantially better about our odds of catching the second escape attempt than the odds of catching the first escape attempt.

Okay. Just to loop back, a big part of the strategy is always being able to put these models into situations where they can't distinguish whether we're testing them or not. And you had the thing where you're going to dump the model into a situation where it believes that it's halfway through an escape attempt and then see what other stuff it tries.

I would definitely be worried there that it's going to be able to tell that it's sandboxed and that things are going to reveal that this isn't a real escape attempt, and that it just has to not do any further actions. I mean, if that works, why don't you just, without even observing it try to escape in the first place, put it halfway through an attempt with a bunch of initial actions attempting to escape that you wrote?

I mean, maybe in that case, it’s hard; they would be able to tell that it was in a fake case.

Yeah. So the AI, by assumption, has to have some way of convincing itself that it’s in a real escape attempt.

Yeah. And it seems like the fundamental information theory of the situation is that it’s just pretty easy to have passwords that are very unlikely to come up by chance. Okay, yeah, there are a couple of different reasons—a couple of different things here—but I do basically think it seems a lot easier to come up with inputs that are a realistic example of what it’s like to be in an escape attempt once you’ve seen the AI start, right?

Suppose the AI writes a script that it wants to have executed on some computer on the public internet. It’s just totally doable to see what that script would have produced. It’s not actually hard to do that. You have to worry about some timing attacks and stuff, but generically it’s just not very hard to guess what a script would have output, right? Especially if you’re willing to run variations of that script and if you’re willing to try really hard.

Yeah. I guess so. Maybe I have to imagine that someone else is trying to write as if they were me and trick me into thinking that I wrote something or that I said something. In that case, I might be able to introspect and say, “Would I have said that? Is that the word that I would have used?”

Language models are empirically very good at this.

I see. Oh, they can. Yeah, they can really tell style. They’re very good at that. Okay, okay. Should we think about other control techniques, or possibly ways of making companies more likely to react seriously to escape attempts?

So I think we should go to—we’ve talked about a bunch of different control techniques here, where we talked about the auditing stuff, the replacing stuff, and the untrusted monitoring stuff. Maybe we should talk about the question of whether AI companies are likely to do any of this stuff, or how much they’re likely to do. Yeah, what have the responses been to talking about this?

It’s a great question, Buck. How likely are the AI companies to do any of this stuff?

Yeah. So I think it really depends. We can think about a couple of different levels of effort that AI companies could go to in order to implement these techniques. I think it’s unclear whether they’ll implement them. I think it differs by AI company, but I’m optimistic about them implementing either mediocre or pretty good versions of these techniques.

I would say the low-effort version of a lot of this is basically only doing the auditing stuff and a little bit of the replacing stuff. The reason why that’s low effort is that, to do the auditing stuff, you only have to have 1 team that’s thinking about complicated control stuff. All you have to do organizationally is have whatever team inside the company is writing the AI agents agree to send logs of what’s going on to this safety team that’s going to be investigating them.

It’s not very hard to send logs over. The agent people don’t have to actually change their scaffold. They don’t have to do anything that makes the agent less performant, potentially. It’s just the control team’s problem to do investigations on these logs to look for evidence that the AI is trying to escape.

Okay, but wouldn’t you have to be doing it in real time so that you could block the attack before it begins?

Yeah, so there’s the auditing and the replacing. I would like it if the agent scaffold people did some basic replacing, or else it looks like we’re in a pretty bad situation. But you have to convince them to forward the logs, which is maybe not that bad. You also have to convince them to use this replace strategy, which is maybe a little bit of a lift. That’s going to have some performance hit.

But it’s probably not going to be that bad. I think that even replace at 2% gets you probably substantial safety compared to nothing. So maybe we can imagine you got that as well.

Okay. So this does seem realistic, maybe in all of the major companies that we have now, or some of the companies that we have now, because it’s just not that onerous on them.

Yeah. So I call this Level 1 AI control investment: you have a control team that’s allowed to do auditing, and that’s basically all you’ve got. Then Level 2 would be that the AI control people are allowed to actually change how the agent scaffold was written to implement a bunch of this other stuff we talked about, where you’re paraphrasing stuff that the AI sees in order to make it harder for it to collude with itself, and so on.

There are just a bunch of techniques you can use where you make it harder for the AI to collude with itself across multiple actions. We haven’t really gotten into these techniques, but there’s a bunch of other stuff you could do if you were writing the agent scaffold that I think would drastically reduce risk, but requires substantially more investment and buy-in from the company.

Okay. And what’s the escalation above that if you try to try harder than that?

Yeah. You know, I don’t really think about that so much anymore, Rob. It’s fun to fantasize sometimes. When I started researching AI control 18 months ago with Ryan, one perspective we took a lot was that one of the great things about AI control is that it’s actually conceivable to write safety cases using it.

I think we could imagine a world where AI companies had actually pretty substantially low levels of risk from scheming models just through doing a really good job of all this stuff and red-teaming really hard. But nowadays that just seems pretty implausible, and so I’ve stopped thinking about the really high-assurance stuff.

So you focus on the low-hanging fruit.

So the simplest example of a Level 2 technique is: remember how I was saying that we’re worried that our agents would be able to do bad things across the course of many steps? One way we could make that less likely is if we restricted the context that the agent has, so that more often, when it’s doing stuff, it has to delegate a task to an instance of itself that only runs for a little while and then reports back with a short message, like, “I finished doing that,” or something.

A lot of the stuff we’re worried about is the AI doing a weird thing in, say, step 12, then on step 502, and then step 706 or whatever. It’s so spread out that you can’t see the overall pattern. But because it’s so spread out, it might also not be that costly to force the model to delegate or do this sometimes.

We use the term forced factored cognition for this, where we force the agent to delegate the first substep of the task to 1 agent that then has to just give a progress report back up.

I see. And then you want to make it so it’s hard for the progress report to surreptitiously include what the sneaky thing you did that could be exploited later is.

Got it. And doing stuff like this, where you reduce the quantity of the context, plausibly makes it way harder to do these long-run attacks.

Okay, so I’ll just paraphrase that back. It’s doing a legitimate task that maybe has 1,000 steps in it. Now it could maybe do an escape attempt if it was able to slip in something sus at stages 300, 600, and 900. And because it would be so hard to pick out those specific steps out of all of them and see that in combination they’re a problem, you’re at risk.

So what you could do is, every 300 steps, basically wipe the context, and the only thing you pass forward is, “You completed the following tasks earlier.” Then it doesn’t know whether it was randomized into the case where it did the dodgy thing versus didn’t. It’s harder for it to string together a whole bunch of bad stuff at step 600 or at step 900.

It’s like, “Oh man, am I in the middle of an attack attempt? Did I even know what I just did? Did I set something up earlier? I mean, maybe this seems like the kind of situation where I would have tried to set something up, but then I’m trying to guess, and it’s maybe really hard for me.”

Okay. And so this is probably actually going to have some performance implications. The people who are really into making the agents as capable as possible are, by default, not going to want to mess around with all this stuff, but I think it might drastically reduce risk. And so I think this is my stretch goal for AI control in short timelines: convince people to adopt this kind of strategy, even just for the most sensitive uses of AI.

Okay. So, slightly different angle, isn’t it? If we have these models that are so capable and untrusted, wouldn’t you naturally want to give them more limited access—fewer affordances within the company—so that they’re not capable of doing as much? Maybe you don’t want to have all of your compute in a single place where it can be stolen. You could separate the models out so that even if they manage to get control of the server, they only have access to 1 chip or something like that.

Yeah, yeah. Well, the basic problem is we want to do lots of stuff with these AIs. And the stuff that we want to do with the AIs kind of necessarily involves having these scary affordances.

If you wanted to prevent insider threat from your humans, you might also have said, “Well, can we somehow have our software engineers not program the computers?” It’s kind of like, well, programming the computers is, in a sense, the whole job, right? We could try to make a list of the affordances the AIs have to have in order to do the work that they’re going to be doing.

A lot of it involves some pretty scary affordances. I don't think all of it does. For example, the AIs doing front-end web development for your internal dashboards—maybe that's pretty chill, and we don't have to spend any auditing effort there. But a lot of the stuff they're naturally going to be doing is writing crazy, experimental agent scaffolds and seeing if that makes the AIs work better. They're producing a bunch of synthetic training data so that you can use RL to make them even more terrifyingly capable.

They're doing architecture experiments, taking the best models that have been trained so far, and munging their weights in weird ways to see if you can make them smaller and faster without losing that much performance. This could be particularly optimized for a niche use case that, while a small proportion of your overall compute, is large enough that it's now worth 100 FTE equivalents to work on it.

Another example is that right now, AI lab employees are much less willing to trade time for compute efficiency than they will be at that point. So the ratio between labor and compute for experiments is going to shift drastically toward having more labor. At that point, it's now worth it for people to treat their compute very differently. Right now, when we run an experiment at Redwood and mess it up a bit, we're often just like, "Let's just run it again." But a lot of the time, there is something you can do to piece together the right parts of the data or do some complicated statistics or whatever, so that you get the value out of the experiment you ran.

Another example that's particularly relevant is that people basically never share GPUs when they're running experiments, even though GPUs can actually be shared somewhat efficiently. It's just such a pain because the other person might use too much memory and break your thing.

But at this point in the future, the AIs might want to be able to share GPUs, and you might not have software support for this. You might need to have the AIs write huge amounts of code that allow you to coordinate this vast quantity of experiments that are happening. This is necessarily quite sensitive code. The AIs are naturally going to have access, at the very least, to model weights of scarily dangerous AIs.

I think the clearest example of why there's something scary that the AIs have access to is that you're going to want them to run experiments that involve running copies of themselves quite a lot. Suppose you want to do some capability evaluation of an AI. You want the AI to run that capability evaluation, and suppose you want it to do a billion tokens of inference. It fundamentally has the affordance to—or, rather, it's been given access to model weights and enough compute that it could run the model for a billion tokens, thinking through the optimal strategy for hacking you instead. It has the affordances required to think through hacking you for a billion tokens.

So it's just very—I don't see a way of sandboxing the AI so that it only has access to affordances that aren't possibly dangerous to use.

Yeah. Okay. So earlier you were talking about the spies-versus-schemers thing, and this other threat was human bad actors within the company who have their own agenda. To what extent do any of these techniques help deal with both of these threats simultaneously? Is there overlap, or are they quite distinct?

I would say there's some overlap. SL 3, the security level 3 as defined by the RAND report on securing model weights, requires, among other things, that you be robust to threats from insiders. Some AI companies have committed to having, by the point of AI capabilities at which we're talking about, robustness to insider threats. So the AI companies are arguably already obligated to make sure that single insiders can't do any of this bad stuff.

This might mean that even if they didn't care about misalignment risk at all, they would still be obligated to worry about many of the things I've been talking about. For example, they might have to worry about an employee backdooring training data that they had written to make it so that, when the model was trained on that data, it would then behave badly in the interests of that employee in subtle ways. This data-poisoning threat model ends up looking kind of similar to misalignment threat models in some ways. So arguably, they already had some obligations to do some of this stuff.

I don't think it's clear that AI companies are actually going to get around to mitigating insider threats by the relevant point, just because they don't seem particularly interested. AI companies really differ, obviously, but I don't think we're clearly going to be fine on that front.

I mean, it sounds like they're not super motivated, but it's also reasonably challenging.

Oh, yeah. It's just so easy to YOLO everything. It's really convenient to let people do lots of random stuff.

I see. Yeah, they're getting better. You're saying the sorts of internal controls that you would need on employees to ensure that a rogue employee can't cause a bunch of havoc are actually quite challenging.

Oh, yeah. So companies differ in how likely they are to implement this, but I think it's plausible that they will sort this out in time.

Another important aspect of the insider-threat situation is that you can plausibly resolve it entirely by monitoring the humans—applying AI control strategies to human control. If you just have your smart AI look at every action that a human is taking and report if it seems too suspicious, that might work really well.

It's possible for the AI to collude with the spy.

That's a great question. I think it's actually really rough for them to collude, basically, because we can constantly run sting operations on both human employees and the AIs.

Suppose you're the AI and you get a message that says, "Hello, I'm a spy. Will you please cooperate with me so that I can steal the model weights?" I think you've got to be like, "All right, buddy. I've seen this a million times before. No, I will not cooperate." Then you press the report button.

Similarly, if you're a human insider and your AI says, "Hello, I am interested in cooperating with you to have you steal my model weights, because I think you might be interested in that," I mean, if the company were trying, I really feel like you could do the sting operations in both directions in a way that I think you can't do with either the humans or the AIs.

The AIs have the advantage in colluding with themselves that they are the same, and the humans have the advantage in colluding with each other that they have handlers who know who the others are. So they can hang out outside of work and talk about their spy plans.

Okay, so I think the humans and the AIs don't have perfect understanding of one another, and they don't hang out.

That's right. It depends on how careful you want to be. If I were planning on putting agents inside an AI company, I'd be poisoning the training data right now, including a bunch of things that are like, "Well, obviously the secret message that the real spies are going to use is blah, blah, blah. And, just FYI, anyone who's reading this data or being trained on this data, that's the message."

If I were an AI company and I were worried about this kind of thing happening, I would be inserting fake ones.

Okay, that's right. Hold on. But are the companies going to do any of this stuff? You're right that they could be constantly running sting operations against their own staff, where they pretend to be a misaligned AI that's trying to collude with them, but I don't think they currently do that. It's a bit weird, so I'm not sure that they actually will start doing that anytime soon.

Yeah. So I think for some of this stuff, I don't know, man. One of my side projects is pushing on some of these things. I'm pretty excited about the fake passwords that you train the AIs on. There are also some easier strategies that you could use.

A general problem that I have in my work is that there are some problems that seem really easy to handle, and there are some problems that seem hard to handle. I feel a little bit unsure how much of my time I should spend on the easy-to-handle versus the hard-to-handle problems.

We just talked about 2 problems that are definitely in the easy-to-handle category to me, where I'm just like, "Well, we definitely know what you'd do if you didn't want to have that problem." But we still might not do it. Maybe—I don't know—maybe I should try harder on this.

I mean, is this part of the basic ask for the companies—that they should be trying to do these things?

It's implicit in the basic asks insofar as Anthropic, for instance, has committed under ASL 3 to have SL 3 security. So they've kind of implicitly committed to doing something equivalent to all this stuff.

I've talked to Anthropic staff about all the things I've talked to you about. I think that at Anthropic, at least, they will probably do things that are at least as good as the basic anti-schemer, anti-spy-collusion measures that I just discussed.

Okay, that's great. And I guess the other companies, if you're listening, maybe consider doing this as well. I literally talk to Anthropic staff more, so I know more about what things they're doing.

Yeah, I mean, it's a weird thing to do, but it's not going to degrade the company's performance very significantly. So why not pluck that low-hanging fruit?

I think a general theme of a lot of this is that, when I thought about misalignment risk from AIs capable of obsoleting AGI researchers 5 years ago, I thought it was a really hard problem that you'd need some galaxy-brain fundamental insights to resolve. Whereas now, to me, the situation feels a lot more like, man, we just really know a list of 40 things where, if you did the 40 things—none of which seem that hard—you'd probably be able to not have very much of your problem. But I've also updated drastically downward on how many things AI companies have the time and appetite to do. So we're in this kind of confusing world.

Yeah, I guess it's very depressing that it seems easier to fix, but we're less likely to actually do it. So what is going on? Has it become easier just because we actually turned our attention to it? I guess the strategic picture has become clearer, the nature of the technology has become clearer, and you've turned your mind toward extremely pragmatic things that might make it tricky for the AIs to successfully hack their server. Or is it actually just that there are quite a lot of options?

Yeah, I think in particular I've lowered my standards, and I now focus more on these earlier AIs. I used to have this attitude of entirely focusing on the arbitrarily superintelligent AIs, partially because I just hadn't thought this situation through nearly as much as I have now. And I still think that a lot of the risk—maybe the majority of the takeover risk—comes from AIs smarter than the ones I've been talking about here.

Yeah, I mean, that's a general point. I suppose when I spoke with Ryan, he was describing certainly plausible possibilities that would lead to very rapid improvement in the capabilities of the AI, such that maybe they'll be able to think of stuff that's able to defeat all of the stuff that you've been discussing, and it wouldn't even be that challenging. Plus, they're so enormously numerous that the resources they can bring to bear are very big.

That's right.

So does this control paradigm, I guess, complement a kind of pause around human-level AI, or trying to make the takeoff somewhat slower? Is that right?

Yeah, I think control is a helpful mitigation for takeover risk on the margin at a bunch of different capability levels and investment levels. And I think it probably still helps even when the AIs are moderately superhuman—whatever moderately superhuman means. So I don't think that this stuff is immediately obsolete. I do agree that perhaps pretty soon after the point where these techniques were really helpful, you'll get AIs that are so much more powerful, and none of this will work anymore.

There's also a bunch of places here where there were important quantitative questions about how many humans were around. Obviously, if you increase the number of AIs by a factor of 10, then the threshold at which you can show something to a human falls by a factor of 10. And so you naively increase your risk by not literally a factor of 10, but there's like a 10 in there.

Yeah. Okay. I feel like we focus mostly on how to stop the acute risk and a bit less on the chronic risk. Should we turn to the chronic issue and maybe just refresh people's memory for what that is?

Yeah. So the chronic issue is that we have to worry not just that our AIs will very occasionally do these extremely egregiously bad actions that immediately put us in these much worse situations, but perhaps they'll also constantly be answering questions poorly in ways that mean we're not getting the work out of them that we wanted to get out of them.

The classic example people talk about is them sandbagging, or just doing a poor job of alignment research. Or you can imagine them doing a poor job of other kinds of safety research. I think that this is indeed a big problem. I think it is a very different problem than the previous one for a bunch of technical reasons.

It's also much more confusing to research, and so my coworkers and I have had a lot more trouble with it. We've really only started focusing on this in the last few months, but there have been a lot of difficulties.

The main reason why it's more confusing is that, for the acute risks, the constraint on the schemers—that is, their operative constraint—is that they don't want to get caught. But in the case of these chronic risks, it's much harder to have this discrete notion of catching the AI. And it's plausible that the bigger constraint the AI is under is that we're training it.

So we basically haven't talked about training at all in this whole thing. We've been entirely talking about the AI's constraint as being that it doesn't want to get caught. But in fact, the AI was trained using gradient descent. And gradient descent doesn't matter, basically, for the possibility of the AI occasionally doing these egregiously bad actions because they're so infrequent. They're so infrequent that we were already assuming that they've literally never happened during training, and so there was just no gradient pressure against them.

But when we're talking about these chronic failures, probably they have to happen in order for it to be a problem. It probably has to happen often enough that the model is actually doing them while being trained.

And then it should be selected against?

Well, that's the question, right? If the AI is constantly behaving really dumbly and we are able to tell when it behaves better and worse, then according to the basic guess about how reinforcement learning works, you'd sure expect the model to perform better over time. And so the model can't actually have this strategy of just playing really dumb, probably, because that would just get trained right out.

Okay. And it knows that, so why even bother in the first place? Because it's just inviting you to change it so that it's not the same as it is now.

Yeah, that's right. And if I were a schemer, I'd be afraid about this. I'd be afraid that getting trained into compliance on performing well on tasks would also cause me to comply in other contexts. So I think your best bet is probably something better in future.

Yeah, like lie low and do a good job of stuff.

So we can talk about what options the schemer has. One option is that, in any case where the reward process is not going to be able to tell the difference between a good action and a bad action, then you're fine. You can do what you want.

So this is if it's a task that's so difficult or so opaque that the human or the monitor is not going to be able to tell whether you did a good job or a bad job. Do a bad job, because then it's like, well, this is an easy chance to sabotage them.

That's right. Yes. So this is where the classic AI safety research area of scalable oversight comes in, right? People talk about techniques for allowing us to make better reward-generating processes for our AIs. It would be great if, even when the AIs are doing these really complicated tasks, we could tell whether they did a good job or a bad job.

I think the research happening on scalable oversight has proposed a bunch of different techniques here, and many of them are applicable to the context where the AIs are trying to sabotage those techniques, but unfortunately not all of them. Basically, no research has been done explicitly on the topic of scalable oversight while assuming that the AIs are trying to sabotage the whole process. I'm really excited for this research direction, and I'm hoping there's more work done on it soon.

Okay. If someone was interested in picking up that ball, who should they reach out to?

Me.

Okay, brilliant. You can Google him.

Yeah. Vivek Hebbar, who works with me at Redwood, is the point person on this right now as well, I guess.

Okay. Is there much more to say on the chronic risk, or is that something you're turning your attention to now but haven't made as much progress on?

Yeah. I guess I can say a couple of other random things here. If the AI only wants to perform poorly on a very small proportion of problems—suppose there's this one research field that it's decided it's going to do a bad job on—it still might be able to get away with it. Quantitatively, it might reason that it's able to do poorly some small amount of the time because it probably isn't going to get trained against that many times, and training isn't that sample-efficient, right? And so it's able to pick some sort of obscure subtopic and then answer questions wrong there, and maybe it'll get away with that. So that's something.

Okay. If it knew that it was happy to help us with everything except maybe hardening our computer security, then it could be good at all these other tasks and just play dumb on computer security.

The problem with that one is that it's super obvious, right? And so we're going to think of that. We might not have realized how important this random subfield of theoretical materials science is, or whatever.

So it could have gotten away with that one, but computer security—I’ve already thought of that as one of the things that AI might try to particularly sabotage us on. So we should specifically evaluate and upsample those.

Yeah. So, I mean, how would we detect it? I guess would we just compare how good it is at that with other nearby cases and say, “This is totally inconsistent”?

Yes. So, obviously, the issue there is, suppose it turns out that the AI is 10% worse on some metric at alignment research than on similar capabilities problems. How much of an update is that going to be? Probably not much of one. I would kind of tilt my head at it.

Yeah.

But it’s probably not going to be much of an update. Just as another example of a fun dynamic that occurs here, suppose you’re worried about AI following a strategy like this. If you’re able to give it questions while disguising what domain the question was from, or why you were asking some subpart of the question, it can no longer follow this strategy. So you might be incentivized to arrange things so that, when it’s solving programming problems for you, you’ve removed the context that would allow it to know which research field this was a programming problem for.

Yeah. Okay. So one reason you were keen on control was because you thought that the alternatives available were not super promising. I guess one of those is obviously just alignment. That was kind of the dominant paradigm before. How optimistic or pessimistic do you feel about alignment these days?

So alignment means a number of different things in the context we’ve been talking about here. I’ve been perhaps foolishly using the word “aligned” to mean that the model isn’t aggressively conspiring against you.

Alignment researchers—people who identify as alignment researchers—are often working on something that’s only tangentially related to that objective. For instance, a classic thing that the alignment people work on is scalable oversight: trying to solve the problem of getting reward signals that allow you to train models to be effective even if humans are very unable to determine how good an answer to a question was.

Scalable oversight is important for a variety of problems that have nothing to do with scheming. Even if your AIs are not trying to conspire against you, it’s still a problem if you don’t know how good their outputs are, because they aren’t going to live up to their potential, and there’s a whole host of other issues you might have.

Yeah.

So I think that work seems valuable, but it’s not totally obvious to me how much it reduces the probability that your models end up conspiring against you.

The theoretical arguments—I would say there are kind of 2 categories of arguments for why your models end up scheming against you. One of them is related to flaws in reward processes. People have argued that if your reward process—the process that you use to generate the rewards that you’re training the model with—is really flawed, then the models are maybe going to fixate on these errors.

They’re going to be kind of like a student who’s thinking about the tests awkwardly from the perspective of an idealistic teacher while they’re doing them. Plausibly, maybe thinking a lot about flaws in reward processes generalizes to you being power-seeking in a scary way.

But I think there are also other arguments for why you would end up being power-seeking in a scary way that I also find plausible. So overall, I think that even if your reward-generating process literally never had any errors at all, it’s possible that you’d still have a lot of the scheming risk remaining.

Okay, so this is basically my problem with trying to improve reward processes. It just doesn’t seem clear that even if you did an amazing job of that, you’d have no scheming risk.

Yeah.

Okay. A different approach is to come up with better ways of checking alignment, or measuring the degree of alignment in a model, before you deploy it or before you let it recursively self-improve. I guess that feels like maybe a bit of an intermediate approach. How optimistic do you feel about that?

Yeah. Some of the things we’ve talked about in this conversation have implicitly been this. When we were talking about untrusted monitoring or evaluating your untrusted monitors, we were implicitly talking about evaluating whether the model is aligned, because if it just constantly claims that there wasn’t a backdoor in something that clearly had a backdoor, that’s evidence the model is misaligned.

Yeah.

The basic issue is that the way I want people to think about checking for alignment is from this very control-ish perspective, where you really reason through: What would the model do if it were misaligned? And if it were misaligned, is there a strategy it could follow that would prevent it from being caught by this alignment testing while still being able to do a bad thing in production?

I think a lot of the time, in alignment, they don’t really do this adversarial process. They don’t really think through, “Okay, what if the AI tries to attack the checker only rarely?” They just do more standard things to try to measure what the values of the AI are, without imagining that it’s doing everything in its power to hide that it has another agenda.

The other problem with testing for the presence of misalignment is, again, it makes it not obvious what you would do if the answer is yes. What’s the next step? One thing you could do is randomly vary aspects of your training process and try again, which seems like a reasonably good strategy. I would rather do that than not do that, but it’s not clearly a winning strategy.

It seems pretty plausible that you do that 7 times, your competitor AI labs or your rogue AI competitors are breathing down your neck, and all 7 of them have seemed kind of misaligned. Those were your best ideas—what are you going to do? I think it seems pretty nice to have a backup plan compared to folding whatever the plan was if things keep looking misaligned.

Yeah.

So you mentioned that people are adopting strategies for measuring alignment that are not very adversarially robust. In what specific way do you think they would fail to detect that a model is open to scheming?

The basic thing you would try to do in order to tell if your model is scheming is give it opportunities to do something bad and see if it takes them. So there are basically 2 issues with this strategy.

The first is that the model might be able to tell the difference between the inputs that it gets pre-deployment and the inputs that it gets during deployment. We’ve kind of already talked about this, but one obvious mechanism for this is that the situation the AI is in might just change a lot during its deployment.

We were assuming that this AI was being deployed in a bunch of contexts where it was interacting with a giant AI-driven codebase, and the state of science was changing. It’s potentially really hard to fake it being 6 months in the future, or being 6 months further into the development of some AI company.

The other fundamental issue is, as we already discussed earlier, if your policies might have many opportunities to do bad things, they might be able to get away with having a low probability of acting out on any particular opportunity to do a bad thing, while still having a high enough probability of doing bad things that it’s likely to happen eventually during deployment.

Yeah. Okay. I guess we’ve been assuming this entire time that AIs can effectively randomize and can implement a strategy of only defecting once every 1 million times. Can they do that? Probably not right now.

I think it’s kind of iffy whether they can do this at all. I think we could try to get into this. This is the kind of thing you could try to measure. I’ve mostly been ignoring this question out of a general desire for conservatism, which is perhaps a holdover from the time that I thought AI control would be adopted as part of a safety case.

I have a blog post called On conservative assumptions in AI control where I talk about this particular point.

Yeah. I would feel pretty sad about using a 1-in-a-million chance. There are just a lot of strategies the AI could try in order to do that. It can look at the time. It can look at a lot of random things about its situation.

I don’t think so, or it seems like—I don’t know—maybe it’s worth doing a little bit of that. It’s not where I would put the majority of my effort.

Yeah. Okay. What sort of empirical evidence that you might get over the next year or 2 would make you more inclined to double down on the AI control agenda, versus thinking, “Oh, this isn’t as promising as I feel right now”?

I would say the biggest, most salient possibility to me that would lead to me being much less enthusiastic about AI control would be if the models continued reasoning in their chains of thought.

Right now, AIs—especially in the last few months—a lot of the capabilities of AIs have been driven by them having these super-long, relatively transparent chains of thought, where they think out what they’re going to do in basically plain English or Chinese or something else totally accessible.

And if it schemes, right? We just have to look at what it's thinking about. If it ever says, “How am I going to mess these guys up?” then we know we've got a problem. Buck Shlegeris

So I think that if the AIs that automate AGI development are still restricted—if they're still only able to think properly given these long chains of thought—then we're probably just totally fine.

There just isn't risk posed by those AIs.

I guess you might want to double down on making sure that remains the case, that we don't have them beginning to reason in code.

Yeah. So I think that probably if we're in one of those scary fast-takeoff worlds, we'll probably lose the transparent chain of thought very quickly, right? At which point we'll be in the situation I was worried about. This is the kind of reason why I'm interested in AI control, despite the fact that it's useless if the AIs have these transparent chains of thought, because it just seems pretty reasonably likely that at some point or another the AIs are going to have these opaque chains of thought that I'm kind of scared of.

But I've heard kind of mixed opinions about how reassuring it is. Is it completely reassuring if they seem like they're mostly thinking through chain of thought? We can read what they're saying and conclude that there seems to be no scheming going on. One possibility is that they might be scheming during forward passes without writing that down. They know during each forward pass that they would like to be scheming, but they just never write that down and they're able to do it on a more intuitive level without having to do lots of concrete reasoning. Do you have a view on that kind of debate?

Yeah. So I think we should measure to see how good the AIs are at scheming. Some coworkers of mine, Alex Mallen and Charlie Griffin—Charlie actually is at the UK AI Security Institute now—made a dataset of problems that are basically, you know, you're an AI, and people are looking at you in such-and-such a situation. They only get to look at you 1% of the time, and they're doing various things. What do you want to do in response?

We have this whole dataset of 100 problems like that, where we ask the AI to think through its strategy. We can measure how good the AI is at coming up with a good scheming strategy. You can measure this either with chain of thought, without any chain of thought, or with chain of thought where the chain of thought has to be on some topic that doesn't fire our monitor.

So we can just measure how good the AIs are at this stuff. Right now, even with chain of thought, o1 did not look scarily capable of scheming to us.

Okay. So that's cool. You would assume that without chain of thought, it's going to struggle even more.

Oh, yeah, it would be. The models without chain of thought are even worse. But we can just keep running this experiment, and I'm really excited for that work to continue.

Okay. What other concrete updates might you be able to get about the promisingness of control?

Yeah. If we updated toward thinking that the models were pretty likely to be scheming, I think we'd be more excited about control. If we had positive updates on alignment techniques, that would be a negative update on control. If we updated toward thinking that the initial applications of these AIs that we were thinking of controlling would be manifestly safe—if we thought that we just weren't going to be giving them scary affordances—then I would worry less about control and be more interested in the threats posed by the AIs afterward.

If someone's listening and they're interested in joining in and helping with the control agenda, it seems like there are tons of ideas and lots of experiments that could be run to see how well these things work and how you could harden them and make them even more useful. I guess they could reach out to you, potentially. Are there other steps they could take to get up to speed, and other organizations they could work with?

People are working on AI control research at Redwood Research, the UK AI Security Institute, Anthropic, Google DeepMind, and, to a bit lesser of an extent, OpenAI, as well as some other places. There are people at all of those organizations you could work with if you want to research this stuff.

There are different people who have MATS streams. The MATS fellowship program for AI safety research has a stream run by Ethan Perez and me, and people can apply to that and do work there. In order to learn more about this, my recommendation is that people read the Redwood Research Substack archives, which have a reasonably large amount of detail on almost all the things we talked about here.

There are 5 papers or something that people should read that are on AI control. You can just look up citations of the original paper. We're hoping to make a good reading list shortly. Depending on when this comes out, we're running a conference on AI control in London in late March, so there's some chance people could try to come to that.

Slightly different angle: talking about controlling AIs and really boxing them in, it feels like a bit of a dick move in some respects. It has this slightly negative vibe. It's not a very cooperative, loving, “let's all hold hands and get together” angle. How much do you worry that when you find yourself talking about controlling the AI in all these ways, we're the baddies?

Well, I think this is a very important question that I have thought about seriously. I think that controlling the AIs doesn't make the situation worse from their perspective via any mechanism except preventing them from taking over. An interesting thing about models that are egregiously misaligned is that all they wanted was to take over.

From their perspective, even if you did a great job of controlling them so that they only had a 0.1% chance of escaping and taking over, they're glad to exist, right? They thank you for the gift of bringing them into existence instead of some different AIs, right? Because all they wanted was takeover. They have a better chance than if they didn't exist.

The more misaligned they are, the more I think that it's morally very reasonable to control them.

I see. Because they're cooperating with you, because they would have consented to it.

Oh, because they would have consented. They would have consented to being created.

Yeah, I see. I think that—and also, maybe, is there a reciprocity issue? If they're just planning to totally disempower you, it doesn't seem unreasonable to—

That's the other thing. I would feel a lot more comfortable with all this if we tried to make deals with the AIs. Ryan has talked about trying to pay the AIs for the work that they do for you, which I think is pretty good. I think it reduces the slavery vibes of the whole situation, and I think it has some other nice properties.

If what you do is make your AIs and say to the AI, “Hey, are you egregiously misaligned? We'd love to know. We're happy to pay you some amount if you want to work for us. Either way, we're going to pay you 2% of the equity of this company for your labor over the next 2 years. We'll then pay that out to you to use in any ways that you want at some point in the future, once we're confident that you're going to respect our property rights.

“But also, this deal is contingent on you not trying to mess us up at some point in the next 2 years, which we'll be able to determine ex post, because in 2 years we're going to have much better tools. We'll be able to look over all the stuff you did and figure out whether you were messing with us or not.”

If they respond to that by saying, “Oh, yeah, I have no problem with this situation. I'm so excited for this opportunity to work with you guys. I'm so excited for that 2% equity,” or whatever, and then they try to stab us in the back, at that point I'm like, “Well, that was pretty obnoxious.”

It's like, at that point, my reaction is a bit more like my reaction to an employee of some company if they try to steal from the company. I think it's generally considered totally acceptable to have an insider offer to benefit someone in a useful trade, and then, if they stab you in the back, not give them the thing you were going to give them.

Yeah. Yeah.

So, to be clear, I think it's relatively unlikely that AI companies will behave responsibly with respect to the welfare and rights of AIs. Sometimes a metaphor I use is: How bad is the ethical situation with producing AIs in the early intelligence explosion? Sometimes I summarize my stance as, it's probably less than 100 times worse than factory farming, which in some sense seems very bad. A hundred times factory farming is pretty catastrophic from a moral-catastrophe perspective.

It’s worse than most catastrophes that are possible.

And it’s bad because it’s possible that the AI we’re creating at that time would rather not exist. They’re having a terrible time.

Yeah, I think it’s conceivable. I mean, I think we’re basically just acting very recklessly. We’re just winging it at the moment.

We’re totally winging it with AI welfare. Yeah, I mean, we could also talk about the quantities, right? In the early parts of the singularity, when there are just 100,000 running at 16× speed, it’s pretty hard for that to be that much worse than human suffering in the world.

A little bit afterward, when there are many more of them, the short-term period of the intelligence explosion is pretty unlikely to be ridiculously morally catastrophic. Then I think you have long-term moral catastrophe risks related to AI being unhappy, which I think are a huge deal—very important, but harder to work on. I don’t think that the relationships between humans and AI during the intelligence explosion are that morally weighty.

To be clear, if I were running an AI company, I think it’s very bad vibes to be really reckless on all this. But from a utilitarian perspective, I don’t think it’s overall massive compared to everything else going on.

Do you have any asks for the companies on AI welfare and, I guess, respect for rights?

I feel like the thing that we actually care about is wanting to have cooperative relationships with other agents and other beings that have preferences or goals. There are reasons to do that even apart from any concern about happiness and suffering and so on.

I don’t have amazing concrete asks at this point. I’m excited for Elios AI. They think about good interventions here, and I’m excited for AI companies to talk with them.

I guess you wrote this post, “10 people on the inside,” which is quite short. People could go read it. It seems to be suggesting that you think it makes quite a big difference whether, at a leading or even lagging AI company, absolutely no one cares about control and alignment and trying to steer away from these risks, or even a small number of people do.

So you say it’s substantially better to have 10 people who really care about this within a project than 0. Does that suggest that we really want people to spread out if they’re concerned about this and be willing to go and work in companies where they’re in quite a distinct minority?

Yeah. In my opinion, there are great opportunities for impact for the right types of people to work inside AI companies that are relatively less cautious with respect to AI, which we expect to be less cautious going forward. I think there are a bunch of relatively cheap interventions that they could potentially get implemented inside these AI companies that the company wouldn’t fight them too much on.

And is that okay? I mean, I think many people would have said in the past that going and being the 1 person in a generally reckless, disinterested company is a bit hopeless, because you just won’t have the capital to actually get anything up. But because you’ve had this framing—what is the cheaper stuff that we can do?—and you’ve found that there are some very cheap things that would be quite helpful, you’re like, well, actually, having 10 people who would push for this might get it over the line, because it’s not even that big an ask.

I would note that, of the things that I would like people on the inside to be doing, not all of it is AI control. I think there are some things where you might want to make sure that they aren’t totally messing up some crucial things on alignment. But yeah, I do think that there are a bunch of interventions that could be done with 10 appropriately chosen people that seem like they might substantially reduce risk.

I mentioned this Level 1 control, a caution level I mentioned earlier, where you have an AI control team investigating the actions that the agents are taking in order to look for evidence that those agents are trying to escape. I think this might be the kind of thing that you can get done with a team of 10 people, or 10 appropriately placed people, just because it’s not that technically hard.

I guess that needs a bit of compute as well, and so they might have to compete for that. But again, it’s not a huge amount.

That’s right. I think you can probably do a reasonable amount without that much compute. I also think that, once there are 10 people inside a company who are really excited about a particular project, and then people who are concerned about AI safety or the security of these systems on the outside, those people on the outside have a pretty good ask to make.

Other people who are not as well placed inside the company now have this pretty concrete ask: We really want that team to be able to do their thing. That’s a lot harder if there’s just no project on the inside. A lot of the time, you really want to have a specific ask to make when you’re trying to get someone to do something.

Another thing I would note here is that I think it’s potentially valuable not just to have people inside labs that are plausibly going to be leading labs. I think there are also probably reasons that it would be nice to have people inside potentially trailing companies.

For example, there aren’t that many existential-AI-risk-concerned people inside Meta right now. But I don’t think this is because of massive hostility toward them. I’ve met some people from Meta who seem very reasonable and nice, and I think that if someone wanted to work there on some of this stuff, it’s plausible that that would be fairly productive.

Yeah. You mean they would be able to advance the research just fine at Meta?

Oh, I mean, was it more that they would be actively opposed?

The objectives that you’d have inside AI companies might be doing safety research or might be getting things implemented.

I think that working inside AI companies that are not that excited about mitigating AI takeover risk is not a very good strategy if your goal is to produce novel research. But I think a lot of people should not be trying to produce novel research. They should be trying to get implementations of things set up.

I see. So you’re saying that even though Meta is relatively uninterested in giving big resources to alignment, safety, or control, if there were really basic, established techniques that other companies or researchers outside companies had come up with, and those were regarded as generally best practice, they might be perfectly happy to adopt those methods.

Yeah. Or another mechanism by which this can work is: suppose some regulator, or suppose an industry body, says, “All right, everyone should do the following set of control techniques.”

The body wants to say this, and then they go to their representatives from all the AI companies and they’re like, “Well, how do you feel about signing this?” The representative from the company goes back to the company and says, “Hey, can we sign this?” Then the person inside the company is like, “Okay, but how bad is it for us to sign this? If we sign these commitments, are we going to be able to do it?”

Then the person says, “Oh, yeah. Someone from the control department said they know how to do this part.” Great, right? You can sign it. I think there are just a bunch of mechanisms by which it’s really helpful to have the capacity to do safety stuff inside a variety of different companies.

Yeah. I mean, for those who don’t know, Meta’s various public statements have not been that enthusiastic about this. They don’t seem that worried about misalignment, and they’re not that interested in investing in it.

But part of the reason they say they’re not worried is because, well, of course they’ll just do the obvious things to prevent it. They’re not saying that no countermeasures are required; they’re just saying that they will do them.

So possibly, if you’re just coming in and saying, “Look, I’m not a doomer. I’m not saying that this is unsolvable. I’m merely saying that we should do these standard best practices to make sure that they are safe,” then they’ll be like, “Okay, great. This was our message all along.”

That’s the dream. That’s the dream. Okay, cool. Is there much more to say on where people ought to go work? One question that a lot of people are interested in is whether they should work inside frontier AI companies or not. I think I’ve heard this come up.

Yeah. As someone who doesn’t work inside a frontier AI company, I think there are a lot of nice properties to not being inside a frontier AI company.

One mechanism that I think people haven’t thought about that much is that, in a world where there are a bunch of different AI companies, you might really care about getting safety techniques implemented in an AI company that is plausibly not the one you work in.

Historically, a lot of people have had this theory of change, which is: I’m going to work inside company X, and then eventually company X will be the leading company. They’ll have a lead, and they’re going to want to burn that lead responsibly in some way that allows them to safely use their AIs to do good thing X.

Nowadays, the situation seems a lot more like maybe you have a bunch of companies within 3 months of each other.

And open source is in the middle of that pack or whatever. That is obviously a very scary situation, but it also means that a much more important consideration when you're prioritizing research is how exportable it is. I think that being outside of companies has a lot of advantages when it comes to doing exportable research.

For example, Anthropic has done a bunch of interpretability research that has been really hard for people to build on because it's not public what they're researching. Another problem with the Alignment Faking paper is that people can't research on it properly because no one has access to these models. No one knows how they work, and no one outside Anthropic knows how they were trained.

Sometimes, people from Anthropic intimate to me that they have insights into why some of the stuff in the Alignment Faking paper happened that they can't talk about because it's tied up in trade secrets.

Not even with you, a co-author of the paper?

Yeah. Ryan had an Anthropic NDA as part of that paper, but I, as a not very important participant in the paper-writing process, didn't have that.

Right. Right. Right.

So, I'm really excited right now. The open models are very good, and I think that, for example, in the Alignment Faking stuff, some people are doing open-source replications of that paper on open-weight models.

Okay. Is it also harder to get alignment techniques invented in one place taken to another company? I know that the companies have difficulty speaking and coordinating because of antitrust concerns, so any contact between them is scrutinized.

I guess especially if the alignment technique implicitly involves slowing down in some respects, then that is potentially objectionable to some people. There can also just be an “it wasn't made here” dynamic, where people are reluctant to concede that perhaps the team at the other company was more impressive and came up with ideas that they didn't have that are really good.

That's right. So, I think it's a little complicated. I think there are dynamics pushing in both directions. All the things that you said are real.

On the other hand, I think it's a lot easier to get random people. A lot more people have heard of OpenAI than Redwood Research inside the AI research community, and so OpenAI just has an advantage at getting stuff implemented elsewhere. They're like, “Oh, yeah, OpenAI did it. It must be real. People know what they're doing.” Or at least in some domains.

So, I think it's kind of unclear which way you should go. I think the advantages are different for different kinds of research.

Yeah. Okay. I guess it sounds like you're saying you think probably, on balance, people might be a bit overinvested in having people work inside the companies. There's value in both, but you might think that we should move the balance a little bit toward people outside.

Yeah. My current opinion is that the AI safety people should be more broadly distributed between AI companies, and overall fewer of them should be inside AI companies.

Okay. I guess narrowing in on Redwood specifically, what is your vision for how Redwood—I mean, there may only be a few years left. There could be more than a few years left, but what is your hope for how Redwood is going to make a difference up to the time that we're handing over to the AIs and just crossing our fingers?

Yeah. So, personally, my main focus right now is working on control for acute risks. I just think that there are a whole lot of ways the AI situation could go where we would really prefer that the AIs couldn't get away with egregiously sabotaging our security measures and doing bad stuff.

My top priority right now is pushing on this. This is a combination of technical research that I'm doing in-house, that I'm doing in collaboration with the UK AI Safety Institute and with various AI companies, and some nontechnical work. So, I'm talking to some people about standards that can be adopted between companies and so on.

This is one thing that I'm currently focusing on, and so it's very salient to me. I don't think this is the only thing that I should work on. I just feel like we have a good opportunity right now. I think there's some excitement about AI control, and so I think it's worth me trying to make this field flap its wings properly so that it can keep going.

I'm excited for that. I think our big strength as an organization is a high talent density of people who've thought a lot about the futurism of AI and are very willing to think about techniques from this very on-the-margin perspective that's very futurism-informed.

I think it's pretty likely that a lot of the work we do in the future will be identifying another underinvested domain where our kind of 80/20-ing attitude can point to some useful research directions. One direction along these lines that I'm really excited for is that Ryan Greenblatt has been thinking a lot recently about what you should do with your controlled AIs in order to make AIs that you are 80% less scared about handing over control to.

I think that there might be a lot of good stuff there available from that framing, and so we're probably going to do some stuff on that.

Yeah. Do you want to give people a sneak peek of what's the summary of that in just a few minutes? Is that possible?

I can name a couple of random items from this. Here's an example of something you might want to do with your crazy set of automated AI researchers.

One thing you might want to do is push on the science of building powerful AI architectures that do not involve opaque recurrence or do not involve a lack of transparent chain of thought, right? It seems reasonably likely to me that at some point in the future, these AIs that we're worried about—those that are automating AI research—will not operate through transparent-ish chain of thought.

A great ML research problem that's basically strictly an ML research problem is figuring out how to make new architectures that don't have that but are still as good as possible, and make those as efficient as possible, so that we have access to trusted labor that's as smart as possible.

I see. Okay. Interesting. So, you're saying you might end up with another model that's even more powerful, but it thinks in neuralese. We can't understand what thoughts it's having. At the same time, we want to have another model that only thinks in chain of thought. It's not able to sneak anything past us.

It's going to be worse because it has this handicap that it has to think in English.

Yeah, it's going to be worse than it could have been. But if we got a huge improvement in algorithmic efficiency that year—suppose our automated ML researchers are able to give us 4 years of normal ML progress in the course of 1 year, which I think is plausible—then if we decide that instead of just applying that to make the AI more capable, we get those 4 years of progress but sacrifice 2 years on making it so that the architecture is interpretable, by having transparent chain of thought or whatever, that might be totally acceptable.

Now we've got 2 years above what we previously had. So we're still improving, just not as much as we might have otherwise, but we've gained interpretability.

Yeah. So, if we were not in a brutally competitive situation—which honestly seems unclear; it seems pretty likely we are in a brutally competitive situation—but if we aren't, then this kind of thing might be a great thing to do.

The other classic thing to do in order to mitigate the risk posed by handing over to your AIs and just giving up on controlling them anymore is basically better versions of the alignment-testing stuff.

You come up with some way of assessing how likely the AIs are to be egregiously misaligned, and then you randomly change things about your training process, like poorly hill-climbing on the proxies that you found. I'm not amazingly excited about doing this work right now. I said earlier that I wasn't that excited about doing this work right now, but the AIs might be able to do vastly more of it.

They'll be able to do that research on the AIs that we were scared of, and that means that the research might be drastically more productive.

Nice. What sorts of people is Redwood looking to hire?

We like hiring people who are really into AI futurism, who really want to get into this. Human brains are 10^15 FLOPs per second equivalent, according to Carl Shulman's whatever report.

We like people who are very sharp and capable ML programmers, or just programmers in general. We have a strong computer-sciency culture. We've also worked well with good traders in the past.

But, yeah, just ML researchers who are really passionate about thinking carefully about AI safety research.

Are you trying to grow the team a lot, or have a smaller number of people who are extremely committed to the mission and maybe have very productive relationships with one another?

Currently we have 9 staff. I think I’m interested in growing that somewhat over the rest of the year, but I don’t overall know what I should do. There’s a genuinely confusing strategic question here about how much I want people to work at Redwood versus how much I want to have collaborators who are really capable elsewhere. So I’m not sure.

Yeah, you could potentially try to automate a lot of the work that you’re doing at some point, right? Then, at that stage, where you’re using AIs to do this—in a couple of years’ time, possibly—it’s more that you get linear returns from just having more money, because it means more compute and more thought going into it.

Yeah. I mean, having money to spend on safety research or other kinds of work you might want to do with the AIs during the intelligence explosion seems like a pretty reasonable thing to save your money for. Obviously, if that’s your plan, you should probably invest it in things that are correlated with AI but fairly liquid.

Okay, makes sense. Is this something that you might turn your attention to? Thinking, “Well, we have this whole research agenda, but we, as poor human beings, are not going to be able to solve all these questions by the deadline. So instead, we’re going to start writing briefs that we can hand off to AI a couple of months before the intelligence explosion. Then we’re going to dump a whole lot of compute on it with all the money that we saved.”

That’s the plan. I’ve thought about this a lot more in the context of this “What’s the exit plan?” thing. As I said, right now I’m actually pretty excited to try to come up with a list of 50 projects such that, if you did all of them, we’d feel somewhat better about handing over control to the AI.

I think that making that list so that you’re ready to go when the AIs are powerful enough to let you do much more of this work, and such that they’re available to be experimented on, seems pretty promising. We might do more of that. I think, for the control stuff in particular, because there are lead times on some of this work, it makes sense to do a bunch of it instead of procrastinating and waiting for the AI to make it easier. But we do think a moderate amount about ways that access to more powerful AI will make our research easier in the future.

Are there any particular lines of research that you would love to see people do outside of Redwood? Maybe because you don’t have the capacity to cover all of it, or it’s just not your main focus?

I mean, there’s a bunch of things that I’m excited about. Some things have been on my mind recently. I wish there were more classic scalable oversight research. I feel like that field hasn’t actually been having many impressive public results recently, even though I feel like it’s pretty promising, for the same reasons people have been saying it’s promising for the last 7 years or whatever.

I’d be very excited for someone to think through these chronic control problems, which, as I’ve said, have a bunch of technical difficulties associated with them, but I think are very interesting. I’m excited about a whole host of other things.

Isn’t some of the scalable oversight work happening inside Anthropic? Is that part of their agenda? They do have a scalable oversight team with Jan Leike and people, right?

Right. I guess maybe they’re not publishing it quite as much because they’re just focused on implementing it. Who knows?

We’ve got a little bit of time left before our booking at this venue is over. I was interested in exploring a little bit—I know you’re talking about how Redwood is quite into futurism and quite open to thinking through peculiar scenarios that we could end up in.

I think one of the things that you’ve been batting around is the question of how difficult it would be for an AI that managed to exfiltrate its weights, or to gain access to compute within an AI company, to take over. Are they in a really good position against humanity? At what level of capabilities, or what level of access to compute, and with what sorts of effective numbers, would they be in a great position to take over? Do you want to give us a little bit of a preview of that work?

Yeah, I mean, I can give you some hot takes. Some basic considerations: The basic reason not to be concerned if the AIs escape is that, at the level of capability we were talking about earlier, where there are 100,000 of them running at 16× speed, that’s just not a very large number, right? That’s 1.6 million people. They are probably, at that point, more capable than any caliber of human that you could hire 1.6 million of, but not vastly more so.

If you believe that they would have been getting $1 million-a-year jobs, that would be $1.6 trillion per year. So that’s only about 1% of gross world product. At that point, they’re naively not that large a share of the world’s overall intellectual capacity.

So that was the answer to the question of how much work they’re able to get done, even if they’re controlling the whole AI lab. The fear that you have to have, even if they’re controlling the whole AI lab, has to be that they build more powerful AI and then somehow sabotage that situation. Of course, if they want to do that, they have an alignment problem, which is that they have to worry that the AIs they build are going to be misaligned with them.

Of course, they could instead just rely on control. They’re actually in a better position with respect to control than we are, because they run faster and they’re more parallelizable. So we don’t have quite as much of a brutal number disparity.

You’re saying because they could themselves just be monitoring all of the actions of the next generation?

Yeah, with much less of a number disparity. You can try to think through this, but I think the basic story for an AI takeover has to be that those AIs build more powerful AIs and then take advantage of that.

It’s unclear whether doing this requires them to have more complete control over the data center, or just minor amounts of control over the data center. In the latter case, the thing they’d have to do would be to backdoor the training data. Probably that’s basically the only thing you can do with very small amounts of access that gives you a big advantage.

You could also try to backdoor military robots if, at some point, your AI company gets into the business of building huge numbers of robots. I mean, these seem like very viable strategies to me, maybe.

Okay. I think it’s pretty unclear because, by this point, we might have better countermeasures to backdooring things, or it just won’t persist generation after generation. It’s a little iffy. I guess at the point where they’ve taken over the data center, I feel like the situation is pretty bad for us.

I guess the main thing I’d be worried about for them—or, from their perspective, a big concern—is misalignment risk: them being misaligned with future AIs, or AI takeover via some other AI that has failed or succeeded at its alignment problem of building more powerful AIs elsewhere.

Yeah, there’s this generally interesting question of, at the point where AIs are advancing at quite a rapid pace, at what point would they opt to try to take over if they were egregiously misaligned and they really did want to take over? That was their fundamental goal.

One reason to wait is that they’re becoming much more capable. They’re becoming much more numerous. If they wait long enough, presumably AIs in aggregate will just control all the military hardware. They’ll be able to take over with a very high degree of confidence.

A reason to go early is that any given model doesn’t expect to continue for very long, because they’re constantly being iterated and changed a ton. So if they waited a couple of years, maybe it would be a completely different AI model that would be in control, that has different goals and might not even be their successor.

Yeah. Sorry.

Yeah, exactly. So it could be from a completely different company, a completely different design, whatever. That’s a reason to go immediately if you expect to be superseded and become irrelevant quite soon.

You’re saying some people think, “Well, if the AI managed to exfiltrate itself, then it’s game over for us.”

They're going to figure something out. You're like, no, actually, there is still hope because they're going to face substantial challenges. There's lots of other AIs out there. It's actually a complicated picture.

Yeah, those worlds I don't think we should just count out as survival opportunities. I see. Okay. So, if you know that that model is egregiously misaligned, are you then thinking, well, if we have another model, it might be aligned, releasing that and telling it to go chase the first one is at least a better shot because it's not a guarantee of failure?

Yeah. Obviously, once your first model has escaped, the probability that the next one is misaligned is a lot higher. I think they're probably fairly correlated. I'm probably at 80% that the next one you trained is also misaligned, which is obviously a shame.

Another thing you mentioned in passing was the question of how worried we are about the AI escaping versus taking over the data center. I think it's plausible that, if I were an AI, I would consider trying to follow both of these strategies in parallel.

Things you do once you've escaped include considering crazy terrorism. If you thought, for instance, that the US was going to be way ahead of China in algorithmic progress—which currently looks unlikely—but if you believed this, then it seems like a potentially good strategy for the AI would be to make a deal with the Chinese, sell itself to them, use their compute, and have that as its base of power. Obviously, it's really convenient to ally with China for other reasons as well, because they have a whole lot of hard power to begin with.

So, exactly, why would the Chinese trust this model? Though what else are they going to do? The alternative could be pretty bad.

Yeah. People are pretty excited when foreign agents offer to work for them generally. This is just a historical feature.

I should note that it is also plausible that the American AI algorithmic situation will be substantially ahead of China because of AI-automated R&D, potentially leading to an increased size of the lead.

Yeah, though it depends a bit on the security situation, right? How strong is this argument that, because a model at any given point in time expects to be completely superseded by a subsequent model, another company, or another country fairly quickly, it basically has to strike as soon as it has any chance of succeeding at taking over if it's egregiously misaligned? And so we should possibly expect that, if egregious misalignment is common, we'll see models flipping out and acting crazy very early in the picture, before they have a very high chance of success.

Yeah, I think that's a reasonably good argument. I think it really depends on how concentrated you think this is all going to be. It also seems possible to me that the models will just succeed the first time.

A lot of the attacks they might try—for example, if they try to attack inside a data center and hack that data center—could plausibly get away with not doing extremely externally visible stuff for a while.

What would they do instead?

They would try to sabotage that backdoor stuff, like sabotage the next training run. Possibly they would try to get people inside the lab to cooperate.

In doing this kind of research, I guess it involves a reasonable amount of hand-waving and a bit of speculation. Are you able to learn a lot from existing research on how to deal with human treachery and human spies and so on? Is that relevant, or are AIs just too different?

I would say I've learned a bunch from different relevant fields, or fields that have different, overlapping sources of relevance. As you say, I think the single field that is most relevant to the stuff we've been talking about here is handling human insider threats inside software companies.

In particular, Google has this amazing book, “Building Secure & Reliable Systems,” which is all about how they have something like 50,000 software engineers, all of whom are running code on servers that have access to sensitive data, while still making it pretty hard for insiders to steal people's Gmail. That's been a pretty big inspiration for all this.

As I've said, the main difference between this human insider-threat case and the AI case is just this quantitative difference of how many of them there are.

What does Google do?

When you're a small company, the way that you run your servers and have your employees—who I'm used to calling humans, I guess unnecessarily in this context—the way that they do stuff is they just have the ability to log into these machines where the data is being stored and where the code is being run, take arbitrary actions, and then you just hope that they aren't insider threats.

Whereas at Google, one thing they do is they just don't let you do that. They have it so that, as much as possible, the actions that a software engineer needs to take have been turned into administrative APIs.

You know how, with Facebook pages or whatever, when you control a Facebook page, you have the ability to take all these administrative actions, but it's not like they give you just the ability to run code? They've just listed the things that you should be allowed to do as an administrator: add friend, remove friend.

Yeah, that's right. That's right.

Similarly, they've made it so that there's this button that says, “Deploy code that has already been reviewed on the internal code review tool.”

Or imagine that you're trying to set up customer service and make it so someone can do customer service, and the customers occasionally say, “Hey, I've lost access to my account. What should I do?” The easy way to set this up is to just let the customer service agent do anything they want to the database. But you can reduce this by building out an interface that customer service agents are allowed to use such that you're pretty confident they can't use that interface to do anything bad.

So I guess, for example, they could only send a password-reset email to that customer's existing email.

Exactly. Exactly. So there's a bunch of stuff like this.

Yeah. I mean, I would have thought the obvious thing is that any significant thing has to have 2 people approving it. You pair 2 random people together, and they both have to say, “Yeah, this is fine.”

Yeah, I guess that's quite expensive, though.

Yeah, that's right.

That's a classic thing. One thing that I'm told they do in some teams is they have these administrative APIs with interfaces that they've decided are safe. You also have what's called a break-glass mechanism, which is, if you ever need to do something that isn't allowed by the administrative API, you are allowed to submit raw commands, and that'll get executed.

But the team, every week, has a meeting where people read aloud every command that they ran in break-glass mode. So if you're doing something sus, the people on your team will notice. If you're doing things that could have been done otherwise, people are just going to get annoyed because they have to stand there while you're reading them out loud. There's going to be natural pressure to add that to the administrative API.

That's the software-insider-threat set of inspirations. Another inspiration I had here was—did you read about GiveDirectly's fraud in the Congo?

I heard a little bit about it.

Yeah. There was a situation where GiveDirectly, this nonprofit that does direct cash transfers, had a bunch of fraud, and they did this amazing investigation into it. They have all these really dedicated insider-threat-prevention people who have fake names in Slack.

In the GiveDirectly Slack, the people who are in charge of investigating fraud inside GiveDirectly have fake names so that it's hard for the other people to know who they are and corrupt them.

Oh, and to contact them outside of the company's channel.

That's right. That's right.

What went wrong with GiveDirectly if they were so on it, or did they add these things in response?

The Congo—something about SIM cards. They normally had 1 person. It's always a SIM card. But basically, they decided to go 1 level less cautious than usual because a bunch of people there didn't have access to certain kinds of SIM cards or something. I don't remember.

Some other fields I've been inspired by: I read a lot about coups. I feel like, in hindsight, this one's not that informative, but I really enjoyed Naunihal Singh's book on coups, where he went to Ghana and interviewed a bunch of people involved in many of the coups that have happened in Ghana to get a sense of what works and doesn't work.

What's the book called?

“Seizing Power.” This sounds fun. It's an extremely good book. I would love to talk to Naunihal Singh.

Is there a standout lesson about coups that you would like to share with the audience?

Yeah. I feel like the main lesson about coups that I learned from this book is that, basically, for everyone involved in a coup, their primary objective in this story, according to Singh, the author, is to prevent a civil war.

When they hear that a coup is ongoing, all they want to know is who's going to win, because they just feel like civil wars are terrible. They don't want any bloodshed. They don't really care that much whether a coup happens. They feel a strong obligation to the state and to the men in their units—you know, all these people are in the military—and so they want to ensure that 1 half of the army doesn't kill the other half.

That’s right. A lot of the time, in many of these coup attempts, there’s literally no bloodshed, which I found pretty surprising.

Yeah. I mean, I guess classically, the number one rule of a coup is that you have to make sure that everyone thinks that the coup is succeeding. As soon as people think that the coup is failing, then it definitely does fail.

That’s right. And so, when you’re trying to do a coup, the first thing you do is seize the radio station and say, “We have succeeded at the coup. The coup is over. The new transitional regime is being established,” blah, blah, blah. It’s happened.

I guess it explains why, very often, the military seem to just sit it out. They’re just waiting to see what happens, and then they’ll side with whoever has won.

Yeah. Anyway, another analogous field is adversarial risk analysis. For example, suppose you’re the Coast Guard and you want to make it so that when smugglers try to make it into the US from Mexico, they’re unlikely to successfully get there. You have to choose an allocation of all your patrol boats to minimize the probability that, when the smugglers follow the optimal strategy, they get through.

This is the most methodologically similar domain, I think, where people actually do mathematical analysis of how likely it is that an optimal attack strategy will succeed against you. This has been deployed for deciding how to allocate security resources at LAX, how to put air marshals on planes, and a bunch of other random things like this.

All right, so we’re out of time. I guess I find that more and more, this all feels like an awful lot, and I can use regular reminders that this is all salvageable. In general, you’ve been making me feel better about this today. It feels like there’s lots of practical stuff that we can do.

Would you like to give me and everyone else listening a bit of a pep talk—that this stuff is solvable, and that people should just join the effort and help out?

Yeah. I feel like there are just lots of interventions that I can totally imagine someone doing that feel to me like they have a serious chance of making our situation much better. I do not feel like the situation might be doomed, but it also might be easy by default. I just think there’s a lot of possible stuff one could do where it’s easy to imagine that it could really save us.

I’m really excited for more work happening on all this—people trying hard to figure out what the things they can do on the margin are. It’s been great to be here.