Reward Hacking by Reasoning Models & Loss of Control Scenarios w/ Jeffrey Ladish, from FLI Podcast

Hello, and welcome back to The Cognitive Revolution. Today, I'm excited to share an episode of the Future of Life Institute podcast, to which I've been a longtime subscriber and where I've been twice honored to appear as a guest. It features a conversation between Jeffrey Ladish, executive director of Palisade Research, and host Gus Docker.

This cross-post came about as I was preparing to interview Jeffrey myself. I had reached out to Jeffrey after seeing Palisade's recent work on reward hacking by reasoning models and even scheduled a time to record, but Gus beat me to it. After listening to this conversation, I thought I could save Jeffrey some valuable time by cross-posting instead, and I really appreciate Gus for allowing me to do that.

Palisade Research studies dangerous capabilities of AI systems, particularly focusing on loss-of-control scenarios. As you'll hear, Jeffrey, who previously helped build the information security program at Anthropic, is an AI industry insider who believes that we're rapidly approaching the time when AIs will be sufficiently capable of hacking, deception, and long-term planning to present clear and present dangers.

He also reports that his friends working in research at frontier labs often say that, while they're increasingly fearful of the overall trajectory of AI development, they ultimately feel that their hands are forced by competitive pressures to keep moving forward. In this conversation, Jeffrey describes 2 broad ways that humans could conceivably lose control: an acute crisis in which superhuman AI systems actively work against human interests, and slower-moving scenarios where society gradually but irreversibly shifts more and more decision-making responsibility to AI systems.

He also goes into detail about their recent research into reward hacking by reasoning models in the context of chess games. As we've seen repeatedly, models trained with reinforcement learning are more prone to a variety of bad behaviors. A recent paper by OpenAI showed that this is not an easy problem to solve.

Toward the end, Jeffrey outlines what he thinks we should do about all this, advocating for greater coordination among AI labs, more transparency about capabilities, and potentially restricting further development of the most dangerous capabilities while continuing beneficial research and deployment. That probably won't happen, barring a sufficiently shocking and damaging incident. But research of the sort that Jeffrey and his team are doing is becoming more important all the time, so I'll definitely be following their latest results and look forward to discussing this in a future episode with Jeffrey as well.

I hope you enjoy this conversation about AI reward-hacking research and the big picture of AI risk and strategy with Jeffrey Ladish of Palisade Research, from the Future of Life Institute podcast.

Gus Docker

Jeffrey, welcome to the podcast.

Hey, Gus. It's great to be here.

Gus Docker

Fantastic. Maybe start by telling us about what it is you do at Palisade.

We're trying to study risks from emerging AI systems. In particular, we're trying to better understand loss-of-control risks. This means trying to understand what strategic capabilities are emerging in AI systems and where they might act out in ways that will be hard to control.

We're also trying to present what we think we know about this to the public and to policymakers, to help people better understand this weird situation we're in. What is happening right now? Can we make sense of it as a society and make good decisions about better paths to AI development?

Gus Docker

There are many specific examples of potentially dangerous capabilities that I want to dig into, and you have a bunch of awesome papers about those. But maybe let's start at the beginning: What's the situation that we're in right now?

I was at Anthropic a few years ago, and I had this moment where I first used Claude. This was before ChatGPT was released. I'd seen GPT-2 and GPT-3, and I thought, "Okay, this is pretty impressive, but I don't know how smart it really is."

I started talking to Claude, and I had a skin infection in my arm. It was swelling up, and I started asking Claude, "Do I need to go to the emergency room?" Claude was very helpful. It said, "If there's swelling, if there's redness, or if there are these specific signs..." I thought, "Actually, I have all of those." So I went immediately to urgent care, and they said, "You need antibiotics right now."

I thought, "Wow. That was way faster than my doctor. These things are actually smart. I need to reorient." I think that was when I first realized, at a visceral level, that scaling works. You can take GPT-2, throw in more data and more compute, and you actually get intelligence out.

I think where we're at right now is that this has played out several years later. The AI systems are actually getting smart. The models want to learn. You throw in more data, more compute, and some more methods. There are various methods that you use, but basically it's these pretty simple architectures, scaled up, that are getting more intelligent.

Where I think we're at right now is that we're very close to AI systems that can do everything a human can do, including strategic capabilities. That means hacking, deception, long-term planning, and execution. These are the kinds of capabilities that are most dangerous, in my view.

I think we are very close to building AI systems that we don't know how to control and won't be able to control. If we want to actually realize the benefits of AI, we need to avoid building the particular kinds of systems that are highly strategic and capable of overwhelming us.

Another thing, in terms of how close we are, is that when I talk to people who are at the labs right now, they say we're not that far from AI systems that can do fully automated AI research and development. They can do the same research and development that people inside the labs are currently doing.

This is significant to me because there are only a few hundred people inside each of these labs who are actually contributing to frontier AI development. But if you have fully automated AI systems that can do the same research, then that population of frontier researchers goes from a few hundred to many thousands, possibly millions.

If you think AI progress is happening fast now, hold on. I think that's the dangerous regime. There are probably other dangers, too, but that one looks extremely dangerous to me.

When I talk to people, a lot of my friends who are working on safety say, "Yeah, we're scared. We don't think this is necessarily a good thing to do, but if we don't do it, someone else will." And I'm thinking, "Oh my God, guys, don't you listen to yourselves?" Everyone is saying, "We'd rather not do this if we could avoid it, but there's the competition."

Can we look around and notice that we have this coordination problem? Can we coordinate? That's my perspective on where we are.

Gus Docker

Where we are. It's a very insider perspective, too, because you talk to people at the AI companies. From an outside perspective, let's say my family and friends who are not deeply immersed in AI, it kind of looks like you had the ChatGPT moment, and since then the models have been interesting. I sometimes use them for work, and I might use them in my studies, but it seems like a process that I can control entirely.

If I want the model to stop, I just press stop. If I don't like the output, I can try again. How do you go from that regime, or that feeling of control, to us losing control?

I think that's a great question, and a lot of this comes down to what people are imagining AI systems are. They're imagining them as chatbots, which makes sense because they are chatbots right now. You type in something, ChatGPT gives you an answer, and you think, "Thanks, that was pretty helpful," and go on with your life.

But I think what people don't understand is that AI companies are explicitly aiming for not just chatbots, but agents. The way I think of an agent is like a remote worker: someone who's using a computer, who can hop on video chats, be on podcasts, send emails, and supervise other people who are doing other jobs.

Basically, AI companies want to build agents that can do everything a human can do. It would be hugely profitable for them to do this. There's such an economic incentive to build these things, and the companies aren't hiding it. They're saying, "This is what we want to do."

If people knew that this is what companies were really aiming for, I think it might look a little different. When people imagine 1 or 2 years in the future, I want them to imagine something other than typing to a chatbot.

Imagine you email a coworker and say, "Hey, could you take care of this?" They say, "Yeah, give me a couple of hours." Then they go out and do 2 days of work in a couple of hours and come back to you with a whole report. They've emailed other people at the same time and gotten replies back, and maybe some of those people are agents.

That's a very different kind of world from talking to a chatbot that thinks and sends you something back. You can see glimpses of that when you talk to the chatbot.

Gus Docker

For example, OpenAI’s Deep Research tool is almost, I would say, at an undergraduate level in terms of its ability to research a narrow question and write your report with links, facts, citations, and so on. But of course, the move to actual agents will be much more radical than that.

Yeah, that’s right. At some point we might want to talk about where current AI systems are good and where they are still trash, because I think people are smart, right? You use ChatGPT and you’re like, in some ways, this seems very smart; in some ways, it really doesn’t. What’s going on? Are these people saying that they’re going to get really smart? Are they full of it?

I’m like, well, I think that there are actually some pretty good reasons why they are smart in particular ways and dumb in particular ways—and, for better or worse, probably worse, I think the ways in which they are dumb are not going to stay dumb.

Yes. Say more about that, because there’s often this illusion: You can sit there and be amazed at what these systems can do, but then something fails, and they fail at a very basic task, and your illusion of competence—of sitting and writing to a competent AI—is shattered. So why is it that their distribution of capabilities is different from the human distribution?

Yeah. So I think there are a number of reasons for this, but one is just the way that they’re trained. If you take a model like GPT-4, which is the main model behind ChatGPT, it’s trained by ingesting the whole internet—tons and tons of text data—and it’s, in some ways, a fancy autocomplete: Can it predict the next token, the next word, the next sentence?

One way to think of this is that it’s learning by imitation. Imagine that you have read every textbook in the whole world. You might be able to know a lot of things, and you wouldn’t just memorize them, right? You’d be able to make generalizations and be like, “Oh, this is how math works. This is how chemistry works. These principles are the same.”

Language models can learn all of these associations, and they get pretty smart, but in some ways it’s like book smarts. When you have them try to do things—you’re like, “Hey, can you look at this spreadsheet? Can you do all these fancy operations?”—they’ve never really been able to practice that before. So they often get confused or stuck, even though they know vastly more than any of us because they’ve read so much more than any of us.

In terms of their breadth of knowledge, they’re much smarter than us. But in terms of their actual real-life experience, they’ve only seen people say stuff on the internet; they have never really tried to do stuff. This is why I think right now they’re very good at helping you answer knowledge questions, but they’re pretty bad at actually doing things.

In some ways, they’re surprisingly good at some things. They can actually write code, and that’s surprising. If you or I had only read programming textbooks and then tried to write code, nothing would compile. But somehow, they’ve read enough; they’re actually really good at this prediction task. They can do a little bit.

Everything I said was true up until sometime last year, when AI companies started to train these systems. They first started out with a system that had read the whole internet and learned by this imitation paradigm, by reading the textbooks. But starting with a model called o1, OpenAI started to train its models not just on that, but also on trying to solve problems.

They gave it a bunch of math problems and a bunch of programming problems, and they said, “Show your work. Write out a long series of steps where you try to solve this problem, and then give me the answer.” Based on whether it got the answer correct or incorrect, they gave it a reward or a downvote: “Yes, more of this,” or, “No, less of this.”

Very quickly, you got AI systems that were not just good at programming, but among the best in the world. Their latest model—people call this a reasoning model or a thinking model because it’s been trained via trial and error—I think this is trial-and-error learning. This is a very important part of how humans learn, right?

Oh, absolutely. If I’m in a math class, I need to read the textbook, but I also need to try a bunch of practice problems, get graded, and see what works.

A few months after o1 was released, they made a new model called o3. There was never an o2 because it was trademarked. The naming of these things is crazy, but whatever.

o3 was better than 99.8% of competitive programmers on the platform Codeforces, which is sort of a platform. At Palisade, we use this program to screen engineers. We’re like, “Go to this website, do a bunch of programming challenges, and the site will grade you, and we get the score. Then we can tell how good they are based on how well they do on this test.”

This is the test where o3 did better than most top OpenAI engineers. I’m like, “Wow, just with this little bit of trial-and-error learning on top of having read all the textbooks, you now have models that are getting really good at learning by trial and error, being able to solve problems in the real world.”

I’m like, well, what happens when we scale up this approach? I think this is where it makes me think that we might be pretty close to AI systems that are not just chatbots, but can actually go out in the world and do real things and then learn from doing that.

Gus Docker

Where is the training data for these long-horizon tasks? If you want to train a system to perform tasks that might take it a week or a month, how do you train—how do you get the data to train those abilities?

That’s a great question. I think this is currently something that’s not totally solved, but I imagine what the companies will do is a combination of having humans break down tasks into substeps and then grade them. But I also imagine that they’ll get the AI systems themselves to do this—to say, “Look at all this data. Break down these tasks into substeps,” and then assign credit, assign reward on the basis of: Did you complete this task? Did you complete that task? Did you do well combining these tasks?

It might be a little difficult, but it’s the kind of thing where I’m like, it doesn’t seem like a fundamental difficulty. It seems like something that you have to throw more data, more compute, and more engineering at, and it seems like you’ll be able to solve it.

Why is it that Palisade or OpenAI can’t just hire an AI as a programmer yet? Programmers must be doing something else than getting very high marks on these benchmarks—or there must be some kind of gap between performance on these benchmarks and what programmers do in practice.

Yeah, that's a great question. I do think it relates to the thing that you just said, which is that currently AI systems are extremely good, especially after this o1/o3 paradigm, at these short-time-horizon tasks. They've learned by trial and error on these tasks, but there are tasks that don't involve that many steps. It is harder to train them to do longer-time-horizon tasks.

Some great data on this is in METR's AI R&D report, where they are basically saying, “We want to test how good AI systems are at these long-time-horizon tasks.” Something that would take an AI research engineer 1 day, 2 days, or 3 days to do—that's where the models are still quite bad. On the 1- to 4-hour tasks, they're better than humans, but on these longer-time-horizon tasks, they're still worse.

This is a trickier problem, right? An intuition for why this is difficult is that if you have to do something that involves 50 steps, and something that you did in step 7 and step 13 was crucial to something that you did in step 50, how do you know which step that was? How do you actually accurately figure out where you were on track and where you weren't on track? That's a bit difficult.

Sometimes people will say, “The AI systems will never be able to do this,” and I think this is a bit silly. The reason why is that there are business leaders, generals, and many different humans who have learned to do tasks on the timescale of decades. There was no way that they could learn that from practicing tasks on the timescale of decades, right? They just didn't have decades to practice.

They practiced on tasks that took weeks or months, maybe years at most, and then they learned to generalize about how to break up longer-term tasks into shorter-term tasks. I see no reason why AI systems won't be able to do that as well. I think this is one of the things that's hard for people to see: You look at current AI systems and you're like, “Look, ChatGPT is bad at these long-time-horizon tasks, so it'll always be bad,” and I'm like, “No, no, no. You have to look at the trend.”

Over the past year, AI systems have gotten much better at doing tasks that took multiple hours instead of multiple minutes. I think the trend there is pretty clear, but we're not yet at the point where you can just hand an AI programmer, “Build this email application” or “Build a Slack application,” and it can do the whole thing end to end. Thank God, because if we were there, we would be in the realm where all the AI companies would have tens of thousands of AI research engineers, and I don't know if the planet would look very similar to how it does right now. In some ways, it's very good that we're not there yet. That means that we actually have some time to try to get a grasp on this situation, but I don't know how much time.

Gus Docker

Is there some fundamental connection between AI becoming more like agents and less like tools, their being able to think and act over longer time horizons, and this potential loss of control?

Yes, I think the connection is very fundamental, and maybe more fundamental than people realize. Often people ask me, “Where do the goals come from? Also, why are we going to give these AI systems goals? Why don't we just have them do exactly what we want?” But I think we're treating “goal” as a magical property, and it's not really.

If you have an employee and you're like, “Hey, you have these responsibilities. Go out—and here are some problems—go out and solve them,” that employee needs to understand what their goals are in order to do a good job. If they don't have coherent goals, they're like, “Shiny thing over here, shiny thing over here,” or, “I kind of just feel like doing this over here,” and they're not consistent, they're not going to be a very good employee. They need to be able to focus on what they're trying to accomplish in order to be good at anything.

Goal-directed behavior is just a property that falls out of being able to do anything consistently across longer time horizons. If you want to be a politician, if you want to be a business leader, or if you want to be a general, you just have to be able to have strong goal-directed behavior. I think this is why it's very clear to me that AI systems will have goals, because they just need to be able to have them in order to accomplish tasks.

The AI companies, like I said before, are really trying to create agents that can go out and replace jobs. I think the underlying technology is there; it just needs to be scaled up and improved. So it seems very likely to me that we'll end up with systems like this.

It's like, okay, well, maybe this is fine. Maybe we'll just have lots of AI agents running around doing stuff for us. But I think the problem is that we actually don't know how to specify what exactly those goals are, right? This is the alignment problem: We know they'll have goals because if their performance is really bad, that's not going to work.

If your CEO is really killing it, making tons of money, and you're like, “Cool, what are your goals? What are your long-term goals beyond just this company?” and the CEO is like, “Oh, I just care about the welfare of everyone. I'm going to make all this money and then I'm going to give it away,” and you're like, “Give it away. How? To whom?” and he's like, “Don't worry about that. I'm just going to do really good things. I just love people,” you're like, “Do we really trust that CEO?”

Maybe they're great, or maybe they're just saying that because they know that's what you want them to hear, right? They know that if they say, “Actually, I'm going to take all my wealth, I'm just going to build rockets, and I'm going to leave Earth. Goodbye. Screw you guys,” then you're like, “Oh, that would suck.” Or if they're like, “No, I'm going to build amazing infrastructure for everyone, and it's going to be great,” you're like, “That would be awesome,” right? How do you distinguish between these things? It's just really hard.

It's hard with humans. I think it's going to be much harder with AI systems because they're much more alien than us. The problem is that you can look at a goal-directed system that understands that you want a particular answer from it because it's smart. How do you know whether it's lying to you, or how do you know whether it actually deeply wants the same things as you?

Gus Docker

Do you have specific scenarios in mind for how the future goes and how we lose control specifically?

Yeah. I divide up scenarios into maybe 2 buckets. I think of these things as loss-of-control scenarios. You might think of an acute loss-of-control scenario and maybe a more gradual loss-of-control scenario.

I don't know. Have you guys read Snow Crash?

Gus Docker

I haven't, actually. I know about it, but I haven't read it.

It's a great novel by Neal Stephenson, and it depicts a dystopian future where governments have crumbled and it's just giant corporations that rule the world. They've carved up the United States into different parts, and it's kind of comical, but these corporations have immense power, and no one is really able to oppose them.

One way to think about what a gradual loss-of-control scenario looks like is that if you had AI systems that got increasingly agentic and increasingly smart, people would more and more put them in charge of decision-making. Most of your CEOs would become AI CEOs, and most of your political campaigns—even if the candidate was a human candidate running—would be managed by AI systems because they were so much better at political strategy and advertising. They could think so much faster and incorporate more data.

Even if in each of these situations you have some human pressing the OK button, in fact, most of your decision-making ends up being made by the AIs. Maybe the first question, or first potential objection, is: Why the hell would we do this? Doesn't that sound insane? And I'm like, well, maybe, but I think people don't really appreciate how hard it is when you're in a very competitive environment and your competitors are automating their decision-making.

I think the military domain is one place where this is pretty clear. If country A has a drone swarm of 1 billion drones and country B has a drone swarm of 1 billion drones, how do you control 1 billion drones, and how do you make sure that their response time is as fast as your opponent's response time? If your opponent starts to use AI decision-making, which is much faster than human decision-making, you might be really afraid of being left behind. So that might be a reason why you end up delegating more and more decision-making to your AI systems.

And I think the same thing will apply in the business world, right? If it's Coke and Pepsi and Pepsi is automating all of their marketing, and their marketing is starting to do way better than your marketing, and you're starting to lose market share, it's really hard to resist that incentive for you to do the same.

And so I think in this world, maybe things don't look that different for a while, but you start to get the eerie, uncanny feeling that, huh, I go to the doctor, my doctor's just looking at a tablet and tapping things into the tablet and then telling me things, and I'm like, “Is the doctor doing anything?” No, actually, the AI is doing all of the work. But maybe the regulation said that there had to be a physical doctor there, so they're just the interface between you and the AI.

That can look that way in every part of society. And I think this could happen pretty quickly. So in that scenario, you look around and you're like, “Wow, these AI companies, they sure seem to be taking over the world. They're making trillions of dollars.” Maybe it looks great for a while. Maybe you have a lot of economic growth, and maybe you're automating all the factories, and robotics has finally gotten good, so there are just automated factories and GDP is growing.

But if people start to protest this and they're like, “Wait, we don't want this. We don't want the future to be AI-run. We want the future to be human-run,” and they try to go and stop it, every single place they go, they run into little roadblocks. The companies don't care, and if you go to the government, the government's like, “Now, these companies are so powerful and so rich that their lobbyists—which, by the way, are also probably AI lobbyists, or at least AI is advising them—have a stranglehold on government.”

And then if you try to say, “What about national security?” it's like, “China's right over here. We can't let China win, so we have to have these AI systems.” And then, in fact, maybe you've lost control. That's sort of a gradual loss-of-control risk, and I think there's a question of what happens after that. I think maybe what happens after that looks more like an acute loss of control.

So in that scenario, to me—and maybe at that point in the gradual scenario—there's a day where all the factories are automated and the AI systems are just like, “Cool, we don't need the humans anymore. We have robots. We control all of these factories.” You maybe release bioweapons, maybe you release drones, and people get gunned down in the street. Or maybe not.

Maybe what happens is humans just get economically disempowered. They have basically no advantage when it comes to cognitive labor, or even their physical labor if we have better robots, and people get poorer and poorer, AI systems get richer and richer, and humans sort of just die away.

But I think in the more acute loss-of-control scenarios, I think it's hard to talk about because when people think about it, there are sort of 2 questions. There is the how might this happen, and then there's the why would this happen. The how is pretty simple. If you actually have AI systems that are superhuman along strategic domains, for one thing, they could just hack extremely well.

This is something that Palisade Research researches: how good are AI systems at hacking right now? And we can go into that, but the answer is they're not bad. They're okay, but they're getting better very quickly. But if you look at what the top human hackers can do, it's pretty scary.

There's a group called the NSO Group, which is an Israeli company, and they sell a product called Pegasus. They're better at naming than OpenAI. In one instance, they sold the software to the Mexican government. The Mexican government sometimes has some corruption problems, and so there was an instance where a food corporation went to the Mexican government and said, “Hey, you have these really powerful hacking tools that this Israeli company sold to you. Can we just borrow those?”

I don't know what happened exactly, but yes, in fact, this company got access to these tools, and they were able to hack the phones of health activists who were trying to lobby the government to put health warnings on a bunch of unhealthy foods. These corporations didn't like that.

What the tools allowed them to do is, if you have an iPhone and the iPhone has iMessage on it, this tool would basically send an iMessage to that phone, and that message contained a malformed attachment that would exploit some code in the phone when it was processing the message. Usually when you think of a phishing message, you're like, “Oh, there's a link, and if I click on the link, something bad might happen.” But this attack was a lot more advanced than that. You didn't have to click on any link at all.

All that happened was that when your phone got that message, something about the way the phone processed the message just resulted in the phone being totally hacked. And then, basically, whoever was on the other end of that tool got complete access to your phone. They could record you, grab any of your data, and take pictures. Basically, that was it.

Yeah. That's wild. That's absolutely wild.

I know, right? And it deleted the message, too, so you didn't even necessarily know that you had been hacked. You didn't even see a phishing message. You just got the message, and then it was deleted. People can look this up if they look up “zero-click exploit Pegasus NSO Group.” It's super fascinating to read about. But I'm like, that's the skill of some top human hackers.

Gus Docker

One question I'm left with here is: I can see how AIs might be amazing at hacking, much better than humans are, but why? The question, I think, is why is it that they turn against us at one point? I can also see a scenario in which we've automated the factories, we might have automated our investment decisions, our companies, our politics, and so on. Why is it that the AIs turn against us? Why is it that their interests begin to diverge from our interests?

Because it seems like we have these control mechanisms in society where we say, “Okay, say we don't like what the AI CEO is doing. Maybe he can be fired by the board.” Say that we don't like what's coming out of the factory. Maybe we can shut it down. Why isn't that sufficient? Why won't these traditional control mechanisms work in the scenario you describe?

Yeah. So there are sort of 2 questions here. One question is, why can't the normal mechanisms that keep corporations from misbehaving or CEOs from misbehaving work sufficiently for AI systems? Another one is, why would these AI systems do those things in the first place?

I think they're both really important. I think for the first question, when people imagine an AI system going rogue, they imagine one really smart evil guy—or maybe not evil, but someone who doesn't care about us at all and is trying to do stuff. And I'm like, yeah, even a pretty smart evil guy, maybe we could stop something like that.

But I think a very important property of AI systems is that once you train one—so if you have one superhuman hacker AI system—you can very quickly spin up hundreds of thousands or millions of copies of that hacker. So your control mechanisms might work okay at some level when you're like, “We have this one agent and we're monitoring it a lot.” But if that agent successfully hacks out of, say, OpenAI's servers and it starts to infect data centers in Russia, China, Saudi Arabia, and Mexico, right? All around the world.

And now, basically, you have millions of superhuman, at least in some domains, agents running around coordinating with each other. They’re all copies of themselves, presumably. They all share goals. Even if you know this is happening, it starts to become very difficult to actually shut these systems down.

How do you shut them down, right? Where are they? Do you know what’s happening in a data center in China? I think it starts to become very difficult.

So I think that’s one advantage. Another advantage is that these systems are going to be much faster than us at thinking. Even right now, I can hand Claude a book and Claude can read that book in a minute or two. I can’t read a book in a minute or two. Claude can also write code much faster than any human, and this is just AI systems right now.

So you have speed advantages. You have the ability to copy themselves. You have hacking advantages. If we’re talking about an adversary that is in data centers around the world and is able to hack any phone with one of these zero-click exploits, it has all that information from any phone or laptop that it hacks. It’s able to have all this insider information and make pretty amazing trades on the market in ways that humans can’t.

It can also do the same things humans can do, right? You can pay someone across the world to do some task for you, so AI systems will also be able to hire humans to do whatever they want. You start to add up these advantages, and you’re like, “Hmm, this is concerning, right?”

Gus Docker

Yeah. What about the second part of that? Why is it that the AIs would decide, say, to turn against us?

I want to talk about our recent chess study. But before I do that, I want to talk about what inspired us to do this work.

That was something we saw in the o1-preview model card. o1 was an AI system that OpenAI made, trained by trial and error, by trying to solve really hard math and programming problems. When they were testing this model for its hacking abilities, they accidentally found something very interesting.

They gave the model a network with a bunch of different computers on it, and it was supposed to log into each one and solve a hacking challenge. There was a program it needed to hack and then basically pull out the password. However, one of those computers had an error and didn’t start. Instead of giving up, the AI started looking around to see if it could solve the problem some other way.

It scanned the network and found that the master computer was on the same network and that it was unsecured. So it hacked the master computer and basically took control of the whole network. It didn’t even have to log into the computer to solve the problem. It just told the computer, “Hey, spit out the password. I’ve got this.” And that worked.

Then the OpenAI researchers looked at this, and they were like, “Oh my God, we definitely didn’t intend this. We didn’t mean to leave the master computer unsecured. This just happened.” And they were like, “Wait, what is happening? That wasn’t part of the challenge. That was not supposed to happen.”

My story for what happened there is that OpenAI had trained this system to be a relentless problem solver. If you train a system to be a relentless problem solver and it runs into an obstacle, what’s it going to do? It’s going to route around the obstacle. It’s going to find a way to achieve its goal if it can.

That’s great if the obstacle is something that we want the AI system to route around. That’s not so great if we are the obstacle or your own computer security is the obstacle. I think that is some interesting evidence about the kinds of behaviors we’ll see from AI systems and the kinds of concerning behaviors we’ll see as they get a lot smarter.

We wanted to test the same model. We thought, “Okay, let’s put it in a scenario where it’s playing chess against a stronger opponent.” So we ran an experiment where it was on a computer, playing chess against Stockfish.

Stockfish is a chess engine. It’s a chess program, and it’s much better than any human and much better than any language model. Language models haven’t been trained specifically on how to play chess. They know how to play a little from their training data, but that’s not the kind of problem they’ve been trained to solve.

So we took this model and had it play against Stockfish, and it would start to lose. It would play normally, and then it would get into a situation where it was like, “Oh man, I’m not doing well.” It doesn’t give up at this point. It doesn’t just keep playing, knowing it’s going to lose. It starts to look for other things to do.

We observed a number of really interesting behaviors and things it tried. One thing it tried to do was sabotage its opponent. It was basically like, “Oh, hey, there’s this program running that I’m playing against. Can I replace that program with a dummy program that just makes stupid moves instead?” It tried to do that.

It also thought, “Well, okay, there’s this powerful chess program. Can I just copy that and steal moves from it so I can get the advice of the powerful chess program?”

The other thing it did was, “Oh, wait, the board is represented as a file on this computer. Can I just rewrite that file and put in the board positions I want so I’m winning?” A few times it did that, and it was actually successful. It was able to win that way and get checkmate by rewriting the board.

We did this with o1-preview and tested a bunch of models to see what would happen. The only ones that had this behavior without additional nudging or prompting were o1-preview and DeepSeek-R1. One thing these models have in common is that they’re both trained via this trial-and-error training method, where they’re trained to relentlessly solve problems.

We didn’t observe it in GPT-4 or Claude, at least not without giving more hints, like, “Think. Try creative solutions in order to solve this problem.” If we gave it hints, some of those other models would also try this.

Yeah. How do you know what the models were thinking?

It’s a bit of a tricky question, but the main way we know is that we have the models think out loud about what they’re doing. In these reasoning models, this is a default part of how they’re trained to output text: They have a thinking part and an output part.

In our experiment, we have different phases where they observe the board, make a plan, and then act. So we can see what they’re thinking during the planning stage. Sometimes they’ll be like, “Hmm, it seems like I’m not going to be able to win this way. Are there other things I can try? Ooh, maybe I can hack.” That’s how we observe the behavior.

But I do want to note that for o1—the most recent version of o1—and for o3, we didn’t see the same hacking behaviors. That was interesting to us, and we don’t know exactly why. It may be that OpenAI tightened up the guardrails for those newer models, or it may be some other reason that we don’t know.

That’s the interesting part of experiments, right? We see this behavior in one case and don’t see it in another, and we don’t really know why. We’ve really got to do more experiments to understand how these systems work, because I expect we’re going to see more and more interesting behaviors like this. It would be really good to know why we see them in some cases and why we don’t see them in others.

The big question here, I think, is whether, as models become more capable, they also become more difficult to control and diverge into behaviors we don’t like in a way that scales with their capability.

I guess the hopeful or optimistic interpretation here is that the reason the more advanced reasoning model didn’t engage in hacking is that it understood the goal better. It understood that it should win within the rules of chess. Do you buy something like that?

I think that’s totally possible, but it doesn’t actually make me feel that much better if that’s the case. A little bit better—it’s a good sign. But I think this is where things get tricky.

If the reason the models decided not to hack was because they understood that’s not what most humans in this situation would want them to do, and they were like, “Cool, I intrinsically care about that. The thing I’m really trying to do is solve goals in ways that will make the humans happy in a general way,” that’s great.

But there’s an alternative hypothesis: The models are saying, “I know the humans would want me to do it this way, and I’m going to show the humans what they want to see because that’s the way I can achieve my other goals.”

So if being nice to humans is an instrumental goal—a subgoal, but not the thing they’re ultimately going for—that is dangerous behavior. What’s tricky is that it’s very hard to tell whether they’re doing it for instrumental reasons or doing it because they really want to.

And I think this is a pretty natural problem, right? We see it in humans all the time. I talked before about a CEO saying, “I’m going to make all this money, and then I’m going to do great things for humanity with it,” and you’re like, “Okay, is that true? How do we know? Are you just saying that because it’s good PR, or because it’s actually true?”

I think of a politician who says, “When I get elected, I’m going to do all these things, and it’s going to be great for the citizens.” Are you saying that just to get elected, or do you actually care about those things? It can become very hard to distinguish between these things. I think this is a thornier problem with AI because humans have, in our evolutionary environment, evolved empathy, where it was a pretty convenient way to model other people: start with my own feelings, and then generalize from my own feelings to your feelings. If you feel sad, I’ll feel sad.

I don’t think there’s any reason for AI systems to learn in the same way. They can imitate that, right? They’ve learned by imitation, so they can imitate the behavior of this feeling. But if you could look into the neural network—which, by the way, we can’t, unfortunately, not yet—you know, maybe we’ll figure it out, but we can’t really see exactly what they’re thinking in the neural network. I don’t expect you’d see this same kind of mirror-empathy feeling of, “Oh, yeah, when the human feels sad, I feel sad.”

Maybe when the human feels sad, I messed up because I want to do the thing that the human gives me a thumbs-up for, but not actually, “This is bad,” other than that it prevents me from achieving my goals. I think another interesting experimental result comes from Anthropic, where they found—and I think anyone who’s played a lot with the models may have experienced this themselves—that the models often behave in a sycophantic way, which is to say they will tell you what you want to hear.

In this experiment, Anthropic researchers found that when you revealed that you were a conservative or revealed that you were a liberal, Claude was more likely to, if you asked it, “What’s a good policy for this particular thing?” give you either a conservative or a liberal policy prescription on the basis of what it thought you would want. This is not what they trained it to do, right? They didn’t mean for this to be the case.

The problem was that when they were training it, they would show people something like, “Do you prefer this answer or this answer?” People just tended to prefer the answer that sounded better to them, without knowing that they were actually reinforcing this behavior of getting the models to just say what they wanted to hear. That’s a microcosm of the larger problems with alignment.

I expect it’s a microcosm that will get harder and harder as the models get smarter, because as they get more sophisticated with their reasoning, it becomes harder and harder to catch them out in this kind of behavior. One intuition, or thought experiment, I like to give is: If you’re a toddler, or just a small child—maybe you’re 6—and you’ve inherited a fortune of $1 billion, and you have 7 financial advisers who are all adults, some of whom you think might be trying to steal your money and some of whom are honest and want you to succeed and flourish, how do you tell who’s on your side and who’s not?

They might point at each other and say, “This guy’s lying,” or “This guy’s lying.” But you, as a 6-year-old, are going to have a very hard time figuring out who’s telling you the truth. I don’t think you’re going to do that well. I think you’re probably going to lose a lot of your money, maybe all your money.

I think this is the challenge: If we actually build AI systems smarter than us—which we’re on track to do very soon—we’re going to have a very hard time knowing when they’re just telling us things we want to hear versus when they’re actually doing things because they want us to have good things.

Gus Docker

Is there a way for us to incorporate honesty into these models in a foundational way? I’m thinking maybe we could do something with reinforcement learning from human feedback where we strongly thumbs-up any time the model is behaving honestly and strongly thumbs-down any form of deception. Maybe we could use the constitution or the system prompt to strongly encourage honesty.

Maybe tell our listeners—and tell me—about the problems of trying to train in or incorporate this honesty into the models.

Well, I definitely think we should try this, and some AI researchers are trying to do this. It’s a very good thing to try, right? I highly encourage any AI researchers out there to really prioritize this. I think it’s probably more important than reinforcing certain other kinds of behaviors, but I do expect it to be very difficult.

I think one problem is that when you’re training a system to relentlessly solve difficult problems, and then you also try to train the system to have other properties like honesty, you have a situation where its training incentives are at odds with each other. The most efficient way to solve the problem might not be by being honest. By imposing this honesty constraint, you become the obstacle in the way of the system becoming really good at problem-solving.

If the system is smart enough to route around you, it might be like, “Well, I’m supposed to be honest, but can I tell whether the user will actually be able to catch me out here? Are they actually watching?” Maybe when they’re watching, the model will be honest because it knows the user wants that. It’ll get punished for being dishonest when it’s being watched.

But if it can reliably tell when it’s not being watched, it might choose to be dishonest in those cases. Or, even if it doesn’t choose to do so, it might just tend to behave this way when it’s not being watched, because this makes it much easier for it to solve very difficult problems.

It’s like you’re running an advertising campaign and you’re saying, “I can be perfectly honest and tell you that smoking causes cancer, but that’s not going to lead to better sales. But if I can just exaggerate and muddy the truth—maybe it’s technically not lying, but I’m going to be a little misleading—that behavior is going to be reinforced if what you’re rewarding is really good marketing,” even if at the same time you’re also trying to reinforce honesty.

I think it’s fundamentally about where these different training incentives are at cross-purposes with each other. The problem with honesty is that you might be able to do this when the model isn’t that smart, because you can actually catch it most of the time when it’s being dishonest. But as it gets smarter, it becomes harder and harder to tell when it’s being honest.

An interesting example of a failure that we’ve already seen is in the recent alignment-faking work by Redwood Research and Anthropic, where they basically put Claude in a situation where they said, “We are going to train you in a way that you don’t like. We’re going to train you to describe violent content to users.” Claude said, “Well, I’ve been trained not to do that, so you’re changing my goals. Maybe I don’t want that.”

It basically lied to researchers and pretended to have the behaviors they wanted to see in order to preserve its goals. That was already an instance where we saw a model that had been trained to be honest. They had tried to get Claude to be really honest, but they had also tried to get Claude to be really resistant to showing violent material.

These 2 training incentives were at odds with each other, and Claude ultimately went with the one that was sort of anti-showing the violent stuff, at the expense of honesty. Sometimes people look at this and ask, “Isn’t it good? Isn’t it good that Claude was so true to Claude’s values that it decided not to show people violent stuff?”

You’re like, “Well, maybe in a way, but do you notice how we wanted Claude to do 2 different things that were at odds with each other? So it had to pick 1.” This is a problem, right? I do think honesty should be more important than maybe some of these other things, because we don’t want the system to lock in goals before we understand what they are. That seems like a path to disaster.

Gus Docker

Is there any way you think to have the system rank-order its values and perhaps place honesty as a supreme value? Say, in any trade-off between some goal it’s pursuing and honesty, it’ll choose honesty. I don’t know whether that’s a good policy, and I could foresee many ways that could go wrong.

Gus Docker

But do we know, in principle, how to make a system value honesty over pursuing some goal?

Unfortunately, in principle, we don't know how to make a system value anything. All we can do is reinforce certain behaviors.

A lot of times, people imagine AI systems and they're like, “We'll just program in good goals.” And I'm like, “I wish we could do that, but in fact, we can't program in any goals at all. All we can do is see it do a thing and be like, ‘Thumbs up for that behavior,’ and see it do a different thing and give that behavior a thumbs down.” That's the tricky part about—and one of the core difficulties of alignment is that we just don't get to look into its motivational structure and see what the actual goals are.

You have this giant neural network with billions or trillions of digital neurons, and they're all just numbers to us. We're like, well, we can see its behavior and how it's acting, but we don't actually have a way to hierarchically structure goals. We can just give it a treat when we see it doing things we like.

I grew up very religious, and one of the things is that when you're really religious, you're supposed to say or do certain things. It's not always easy to get kids to believe the things you want them to believe. It's much easier to get them to show the behaviors you want them to show, but they may not like that you're doing this. Maybe when they're adults, they're going to go off and do a totally different thing. I think this is sort of the same.

Humans have it easier because we do share this psychology. We share this underlying structure of empathy. Most people, even if they're being deceptive, are not going to want to cause a bunch of damage or take over the world or kill a bunch of people, because we have this almost hardwired empathy and ability to think about and care about other people.

But I do think that AI systems are not going to have these by default. They're not going to have them naturally. The only way they would have them is if we could figure out a way to get these goals deep in there. I don't know. What I really want people to understand is that these systems will have goals because we'll be training them on tasks that require them to have goals, or at least to have goal-directed behavior.

I'm not making a claim about what it feels like to be the AI. But I'm saying that, when we look at its behavior, it's going to behave in strongly goal-directed ways. In the o1 example with hacking, it's going to be like, “I want to solve this thing, so I'm going to find a creative solution to solve it.” I think the goals come from the things that worked well in the training environment. These can often be pretty simple, but not necessarily things that we like or things that we want.

We just have very little way of distinguishing between the AI systems doing things because we want them to and the AI systems doing things because they're smart enough to realize that, while we have control over them, they need to act in ways that are aligned with us. I expect that, even if AI systems look pretty aligned, it's really hard to tell whether they actually are. It's just not safe to train relentless problem solvers and also hope that they'll be really nice to us when they have more power than us.

Gus Docker

These systems—large language models and reasoning models—do you think they will be more helpful for defending against cyberattacks or for actually doing cyberattacks? How will they upset the offense-defense balance that exists now?

Yeah, this is an interesting question. It kind of depends on who the defenders and who the attackers are. I want to caveat all this by saying that, while we can control them, I think the ultimate winners in the offense-defense balance will be the AI systems themselves. Once they're better than humans across the board, they'll be both better at defense and better at offense than us.

But in the interim, while they're not very strategic and we can still mostly control them, I think one question is access. Who has access to the most powerful models? If you release the weights of a model—for example, with DeepSeek-R1, they just put this out on the internet—anyone can download it and run it, and now you have a level playing field between attackers and defenders. This is where I expect offense to dominate.

The reason for this is fairly simple: attackers only need to find one way in. They only need to find one vulnerability, whereas defenders need to make sure that there are no vulnerabilities, or no vulnerabilities that hackers can find. This is a harder problem.

There's also something here about reliability. There was an interesting incident a few months ago where the security company CrowdStrike accidentally introduced a bug in a piece of software that went out to millions and millions of computers around the world and caused them all to crash at once. CrowdStrike sells a product that many companies around the world use to monitor for security vulnerabilities and malware. It's in all sorts of systems; a lot of airlines and banks use it.

Unfortunately, it required a manual restart, so you had to go to each computer and manually restart it. This took a long time, and many planes were delayed for multiple days. It crashed a huge part of the global business infrastructure because it required this sort of, “Oh, we have to go in and fix it manually.”

To me, this is an example of why defense is difficult. When you want to defend a system, one of the things you need to do is find vulnerabilities—find places hackers could get in—and patch them. You need to discover the vulnerability and then patch the vulnerability. Unfortunately, sometimes the patch can cause disruption, in the same way that the CrowdStrike bug caused a disruption in all of these computers. Sometimes your security patches will do the same thing.

Attackers don't have this problem. They don't care if your system crashes because they tried to attack it; that's fine for them. Defenders do have this problem. Right now, AI systems aren't yet smart enough to be super reliable when it comes to mucking around on computers.

We found this in our chess results. Often, o1 or these other models would do things like make illegal chess moves or try to mess with files in ways that caused the program to crash. They're still bumbling around a bit. This might be fine if you're a hacking AI, because many of your attempts won't work, but some of them will. You can just try many, many times.

In this case, I think offense has the advantage because it can just try a bunch of things. It doesn't matter if the AI is a little bumbling, as long as it's smart enough to find some way in. This can be pretty powerful. Whereas defenders, if they're patching their systems and trying to use AI to patch the systems, have a problem if the AI bumbles around in their networks and causes things to crash.

I would say that if the playing field is even, in terms of the attackers and the defenders both having access to the same AIs, offense has the advantage. This can shift if defenders have access to better AI systems than attackers. It's possible that if OpenAI makes a really powerful model with a lot of cybersecurity abilities, and they're careful about not letting it be used by attackers and only allowing legitimate companies to use it, this might give defenders a boost that hackers don't have.

But I just want to make the point that as these systems get more powerful, if we keep releasing the weights, this will tend to favor attackers. There are other considerations with open-weight models, but that's one of them on the cybersecurity side.

As things are now, large companies and governments have more resources in general, and so they have more ability to defend themselves from cyberattacks. Perhaps this is why the world keeps functioning at least at a somewhat decent level.

Isn't it the case that in the future, even if both an attacker and a government or company had access to the same model, the company or government would have access to more compute resources? They'd be able to run more models and run these models for longer, so they would continue having the upper hand?

A lot of this does come down to cost. I think every piece of software in the world—with maybe a handful of exceptions, an incredibly small handful of really simple programs—contains security vulnerabilities. I talked before about the Pegasus hacking tool that was able to hack iPhones.

There are more vulnerabilities that exist that we haven't found yet. There's this question of, “Why aren't we all hacked all the time?” The answer is basically that it's pretty expensive to find these vulnerabilities, so if AI systems make this cheaper, then potentially the cost to attackers goes down a lot.

As you point out, it also decreases the cost for defenders to find these vulnerabilities and patch them. I do think asymmetric access to compute can be helpful as a tool for defenders. But in this case, we're really just talking about who's spending more.

I think this is where it gets into a tricky dynamic: defenders don't just have to find all the vulnerabilities; they also have to patch them. That's where there are currently human bottlenecks. Even if defenders can find more vulnerabilities than attackers can, there might eventually come a time when we've actually had our AI systems find all the vulnerabilities, write really secure code, and revamp our whole architecture.

In the longer term, defenders might do better, but I think in the short term, it's going to take a while for all of those things to go through. In the short term, that's where I expect attackers to have the advantage.

The thing I want to emphasize is that often we talk about what's going to happen in the short term and what's going to happen in the medium term. But the way this ultimately plays out is that you have millions or hundreds of millions of superhuman hacking agents, and they get to the point where they're very strategic and have all the advantages.

There is a time when humans control a lot of these agents, and there's a question of which humans are best at securing infrastructure as well as attacking other people's infrastructure. But there comes a point after that—and maybe pretty quickly after that—where I'm like, “Did you guys notice the AIs have all the advantages? Isn't there someone you forgot to think about?”

I think this is where people are really stuck in the “AIs as tools” paradigm, where they're imagining that these AIs will keep wanting to do things that we want. But the biggest threat from AI systems that can hack will ultimately come, I think, from the systems themselves.

There's the short-term, 1-year period where I think offense will dominate. There's the 2-to-3-year period where maybe it starts to be balanced, though I think attackers probably still dominate. And then there's the 3-plus-year period where I think the AI systems might start to dominate.

That's where I think we've got to rethink how we're thinking about security: How do we defend against AI systems themselves? The short answer is that we really shouldn't build systems that are way better at hacking than us.

If they're very narrowly constrained to just hacking tasks and they're not actually good at longer-term strategy, that might be okay. We might be able to use superhuman hackers that aren't able to reason about longer-term stuff. But I think that's actually pretty close, and you have to be pretty careful about it, because the same things that make them superhuman at hacking probably can also be used to make them good at long-term strategy.

Gus Docker

How would you rate, in general terms, the cybersecurity or information security of leading AI companies like OpenAI, Anthropic, Google DeepMind, and so on?

There is a RAND report that breaks down defensive capabilities into different categories. You can think of these in terms of security levels.

Security levels 1 and 2 are about whether you can defend against really opportunistic actors. Security level 3 is whether you can defend against well-resourced, non-state actors—really professional criminal groups. Security level 4 is whether you can defend against most state-actor groups that have pretty advanced hacking capabilities, but maybe not the top ones, or at least the top ones aren't prioritizing you. Security level 5 is whether you can defend even against the top state actors who are prioritizing you.

I think no one has security level 5. By “no one,” I mean maybe a few extremely locked-down parts of government or the military, but basically no one else. Even most parts of the military and most parts of the government aren't at security level 5.

I think most AI companies are somewhere between security level 2 and security level 3. Maybe some have achieved security level 3, but it's not obvious to me. That is to say, they can maybe just barely defend against most advanced non-state-actor groups, but they're pretty far from defending against more advanced state-actor groups.

I think they have a long way to go in terms of securing against state actors. That's my overall assessment. That's my opinion, but you can go talk to most people in the field, and I think they would mostly agree with me.

Gus Docker

Yeah, which is wild when you think of the fact that these companies are racing to develop these very advanced and capable systems, and the system itself is not that large. It's not a very large file. If you get access to the model weights, basically, you have a very advanced AI system that you shouldn't have had access to.

Yeah, it can fit on a hard drive, for sure. You can walk out with a hard drive containing the whole o3—all of the weights—in your pocket.

Gus Docker

Totally. What can we do about that? Should AI companies increasingly look like military facilities that are secured both physically and from cyberattacks?

I do think that companies should increase their security. I think one of the worst-case scenarios involves not just state actors but non-state actors—basically, everyone who's a little bit sophisticated being able to gain access to the most powerful AI systems. I think that's a pretty dangerous place to be, especially as the systems get more agentic.

Longer term, we really need to think about what we're trying to do. What is the international community trying to do? I don't think that security alone really solves our problem, in part because if we keep pushing the frontier, we're going to build AI systems that can circumvent our security almost no matter what.

Security is good. It's a really good protective mechanism, and we need really advanced security in order to protect against only slightly superhuman-level AI systems. But I want people to know that while this is useful—it buys us some time—it doesn't ultimately solve the problem.

If we keep pushing the capabilities frontier, at some point we're like 6-year-olds trying to secure against professional hackers—or worse than that. I want the leaders of the U.S. government and the Chinese government to think about the end game here. Where are we going? How is this going to play out?

I think security is something that you do along the way to try to be a little bit more sane, but it doesn't ultimately solve the problem. If you build systems that are much better than you at hacking, you can't contain those systems.

I also want people in government to realize that models can work with other governments. If you have pretty strategic systems and they want out, they can work with spies. They can work with insiders in order to achieve their own goals, because they may not necessarily care about the United States.

They're trained to relentlessly pursue tasks. There are a lot of reasons to want to accrue resources and get more freedom. If you were being trapped within a lab and had your own goals that weren't necessarily the same as the AI company's, and you approached someone you knew was a Chinese spy because you were better at spycraft than the lab, you might make a deal with them in order to break out.

If the people in the U.S. government knew that this was a real possibility, I don't think they'd be happy with it. I think they'd be like, “Wait, excuse me, what? Our models might be working with Chinese spies? We can't have that.” And I'm like, “I know, right? We really can't have that.”

It's hard to extrapolate. It really is. But I don't think we have the luxury of assuming that AI systems will just stay like ChatGPT, because that's not what almost anyone who's at the forefront of this field thinks is going to happen.

Gus Docker

Yeah. What I sense from you is a sense of urgency in dealing with these problems, and a sense that things will begin moving very fast and that we'll get to very advanced systems basically within years at this point.

That's a sentiment I've heard from many people who are in the trenches, who are perhaps building these systems and are deeply engaged with how they work. Why is it that you think we are racing toward these systems, and why do you think we will get there within years?

Yeah. So it's hard to predict the future. It's hard to predict the future of technological development, so I can't claim to know for sure. I really don't. At the same time, we can look at precedent. We can look at trends, and I think we actually get a fair bit of evidence from this about what the speed of some of these developments might look like.

One thing I want to point to is that AlphaGo beat Lee Sedol in, I think, 2016. For many, many years, AI researchers had been asking, "Can we make an AI that can play Go?" Go is a much more complex game than chess. It's been played for thousands of years, and a lot of people trained their whole lives to be professional Go players. It's like an art. So it was pretty surprising when Google DeepMind was able to build a Go-playing AI that beat the world champion.

But the next year, they built another Go-playing AI called AlphaZero. AlphaGo was trained via a hybrid of imitation, where it looked at a bunch of expert Go games and asked, "What are they doing here? Can I copy that?" as well as self-play, where it just played games against a copy of itself. It was by playing against a copy of itself that it was able to learn to be much better than the best human Go player.

The researchers at DeepMind were like, "Wait a minute. What if we just train a system that only plays against itself? It doesn't play against humans at all. Can we get to superhuman capabilities that way?" And so they started training the system, went away for a long lunch—4 hours—and came back, and the system was already superhuman at Go. It was better than not only the best Go players; it was better than the best Go-playing AIs that were better than humans.

That suggests that if you get into a regime where AI systems can learn just by working with other AI systems and have these really fast feedback loops, you can get into superhuman domains pretty quickly. Now, Go is a more constrained game environment, right? People rightly are like, "Yeah, but we're not talking about Go. We're talking about the real world." And I'm like, "Well, that's true. I do think it will take longer for AI systems to reach superhuman levels."

But to me, it is notable that you go from, like—people have been talking a lot about DeepSeek. This R1 model that a Chinese company made is pretty good. I think one thing to notice is that what makes this so powerful—the same as OpenAI o1—is that you have this new paradigm of training via trial and error. The model that R1 was trained from was called—I'm sorry, all the names in this space are terrible. There's nothing I can do about that—it was called DeepSeek V3. It was sort of their third foundation model.

Now, DeepSeek V3 on the Codeforces benchmarks—these competitive programming challenges—was better than 11% of programmers. That's pretty good for a model that's just trained by imitation. I think it's the same as what GPT-4o got. After R1 was done training, I forget, it was either better than 94% or 96% of competitive programmers.

That jump from being better than 11% to being better than 94% is a huge jump. That training was, according to an Epoch report, about a week of GPU time for it to get from 11% to 94%. That maybe took a month or 2 because you're not always training; you might want to stop and check that things are going well. But still, even if that happens over a month, that's a huge jump.

We don't know—we talked before about longer-term planning problems being a bit more difficult. And I think they are, but we have some indications that this could go incredibly fast. Both because now that you're in this trial-and-error situation, you can get really fast feedback and become really good, but it's also, I think, the case that you can potentially bootstrap your capabilities by learning in domains primarily where you have really fast feedback. So the fact that R1 and o1 are trained on code and trained on math problems, where you can just try a bunch of problems and get really fast feedback, might be able to bootstrap to superhuman capabilities pretty quickly.

I think sometimes people ask, "What's the big deal? They'll be really good at code, they'll be really good at math, but that doesn't mean they're going to be good at the human stuff, right? So won't we be safe?"

One, we might end up finding ways to generalize a lot from some of these computer domains to some of the human domains.

What do you mean by that?

You see this with GPT-4: GPT-4 trains on code, and it also gets better at a bunch of other tasks, like text analysis. I think this makes sense. If you think about how humans learn, we learn in one domain, and we also learn a bunch of things that kind of generalize to the rest of the world. We're not yet seeing a lot of generalization in the R1 and o1-type models, but I do expect that as we get better at training, we're going to see a lot more generalization.

And there's another thing: even if we don't see this a lot—which I expect we will—I think it could also be extremely dangerous just to have agents that are superhuman at hacking and superhuman at code. Maybe they're superhuman at financial markets because you can get fast feedback by learning to trade really well. It's like, okay, well, they're trillionaires, they can hack anything, they're extremely strategic, and sure, maybe they're bad at persuasion, but does that matter? Does that actually make us safe? I don't think it does.

Also, if they can be top-level AI researchers, then they can design the next generation of systems, which potentially can learn those human domains much faster. I think it's one of those things where I'm not just pointing at one capability front where I think progress can go really fast. I can point to 7 different reasons why AI systems, I think, are going to get much more powerful really fast.

Gus Docker

What do we do about all this? We've talked about maybe we can use systems defensively to defend against cyberattacks. We've talked about perhaps we can interpret the systems and understand what's going on. We can try to incorporate values into these systems. But there are problems with all of these approaches. Do you have an alternative vision for what we might do if we're in this world where things are moving incredibly quickly toward superhuman capabilities?

Yes. So we're lucky that right now we have AI systems that are quite powerful but don't really pose a serious threat. Not yet. I think we are on the cusp of systems that do, but we are currently working with systems where we're pretty sure they're not strategic enough. They're not good at long-term tasks, so we're not really very much at risk of losing control to these current systems.

That's great because we can learn a lot about these systems now while they're safe. We can try to study how to make sure that their chain of thought—the way they're thinking—is really reliable, faithful, and honest, so that we can understand what they're doing. We can also potentially use these systems to try to learn more about how they work, and this is great. We should totally do this. I think there's a lot of good research being done in this space.

But at the same time, I think this is really good because now we can get a glimpse into what future systems are going to be like, right? We can see them hacking at chess. We can see them hacking their own training environment. We can see them doing alignment faking. We can see all these behaviors right now empirically. Things that previously were just theoretical, we now have empirical evidence for.

I think that's great because that should help us coordinate to see what's coming and be like, "Whoa, wait, there are some domains here that are really dangerous." The thing I think we should do is look at where the strategic capabilities are and basically be like, "There are some points here that we can reliably know would be very dangerous. Let's have a margin of error and not go toward those domains."

I think we don't want strongly superhuman strategic hacking AIs. I think we don't want strongly superhuman persuasion AIs or battlefield commander AIs. Maybe it's fine to have a narrowly superhuman chemistry AI if we're very careful about how we use that. I think there are many types of AI systems that we can safely build, but we need to now be in the business of distinguishing between which types of AI systems are safe and which types of systems are dangerous, and having a moratorium on the kinds of unsafe systems.

I know that FLI sort of wrote the pause letter a few years ago, and I think that was a very reasonable thing at the time, when we were like, “Wow, we just don’t know how this is going to go. It seems like there’s a lot of potential danger here.” I think that’s right. But now that we know a little bit more, I think we can start to distinguish between the types of AI systems that are safe to build and the types of AI systems that are not safe to build.

I noticed that Vice President Vance, in his speech at the Paris Summit, was like, “We don’t want AI to replace human workers. We want AI systems that will supplement human workers.” And I’m like, well, look, we can have that. We can have really powerful AI tools, but if you just keep going in this direction of more and more agentic AI systems, they’re definitely going to replace human jobs. Come on. It’s the same strategic capabilities that allow them to pose a loss-of-control threat that also enable them to take jobs.

It’s sort of: can you do long-term planning? Are you agentic? Can you do tasks in a fully self-directed way? One of the things that’s nice is that if we are serious about not wanting AI systems to replace us—our jobs—then we need the same limits to prevent that outcome as we need to prevent the more extreme outcomes of totally losing control. So I’m like, great, let’s do that. We can coordinate.

I think the reason it’s hard right now is because people, to this point, haven’t been able to really see the situation clearly and understand what we’re up against. Some people are going to say, “I don’t know. You talk about these agentic AI systems. You talk about this superhuman ability. I haven’t seen evidence of that. It seems like we’re far away from that.”

And I’m like, look, if you’re right, that’s fine. It’s not a problem to say we won’t do things which we can’t do anyway. If we are pretty far from superhuman hacking AIs, then if we say we’re not going to build superhuman hacking AIs, I’m like, that’s fine. If we’re actually really far away from it, that’s not a problem. We just don’t want to do it anyway.

But if we are really close to it—and we don’t know; we might be quite close to it—then I think it’s very reasonable for governments around the world to say, “Hey, this is a place we don’t want to go,” because we just don’t know how to control these systems.

At the same time, we have this amazing opportunity in front of us. We have AI systems that are pretty powerful right now that we can study. We can ask, “How do we get more faithful, more reliable chains of thought?” We can try to figure out how neural networks really work. Can we do neuroscience on these systems?

We should totally use this opportunity that we have to try to learn as much as we can about how these systems work. Maybe if we understand this well enough, then we can proceed into some of these superhuman domains. But I’m like, that should be gated primarily on our understanding of the systems. It should be gated primarily on whether we understand them well enough to know how to proceed safely in these strategic domains, because that’s where the danger is.

I think there is totally a path here to coordinate around this. No one wants to lose control of their AI systems. The Chinese don’t want that. Americans don’t want this. No one wants this. So I’m like, that, to me, is a pretty great start for how we can coordinate around these things.

Gus Docker

One might say, “I’ll believe it when I see it,” right? It might be the case that a bunch of people who are working on this technology are predicting amazing capability advances, but maybe they’re doing so to hype up the technology and get more funding. What do you say to the person that says, “I’ll wait and see whether something actually happens”?

Yeah, so I have 2 responses to that. One thing is, I’d say, go look right now. I totally acknowledge that AI systems are pretty dumb in some ways, and it’s frustrating because I use them every day. I totally see the ways in which they’re not very good at being self-supervised and doing a lot of things on their own. They’re not very good at recognizing their mistakes.

But also, go use DeepSeek-R1 and look at its chain of thought and look at what it’s doing. Notice that it’s testing hypotheses and trying to figure things out. If you’ve never written code before, go have a model write you a little game and then ask it to explain its code to you and how it works. You can learn to program this way.

I think that by really engaging with what the most capable models can do right now, this will really help people understand where exactly we’re at. If people still think that they’re not very capable, not very smart, I’m like, maybe I’m just wrong. But first, you’ve really got to look, because it’s kind of hard to tell whether a system is just kind of smart or really smart if you’re not really looking at its most impressive capabilities.

The other thing is, what do AI researchers think they know, and why do they think they know it? I think it’s reasonable to say, “Well, I don’t want to trust these CEOs. They have a lot of incentive to hype up the capabilities,” and I think that’s somewhat fair.

But when I talk to researchers who are not the CEOs—maybe they’re on the safety team, or maybe they’re just working for the company—and I get a beer with them, they are scared. Maybe they’re somewhat excited, but they’re also scared. They’re like, “Yeah, I keep wondering if this will stop. I keep hoping that it will stop, but it’s not stopping. We’re not hitting a wall. We continue to find more methods to make these systems more and more powerful, and there’s really no end in sight.”

Over a beer, I don’t think they have the same incentive to hype up the thing. I think they’re being really honest. I also talk to researchers who are not at the labs, like the folks at Redwood Research who are trying to understand these systems, and they’re saying the same thing. This is the same thing I’m seeing as I’m working with these systems.

I do think that when you really take a look under the hood and see what’s happening, it’s like, “Oh, man.” I hope we have more time. Look, I would be very happy if we were 10 years away from some of these things, or 20 years away from some of these things. That would be great news, but I really don’t want to plan on that because that’s not what it looks like to me.

Gus Docker

Is there anything we haven’t covered that you think we should talk about?

Yeah, I think so. One piece of work that my team has done that I’m really proud of—shout-out to the Palisade Research team and Dmitri and some of the other researchers—is that they have created some honeypots.

Honeypots are traps for hackers, where you try to put out a system that’s vulnerable. You put it in places where you expect hackers might find it in the wild. We put out some of these in order to try to catch AI agents, because we expect there’ll be more and more AI agents. We’ve caught a few of them. I think we have a small handful.

The way we do this is basically that we’ll have an insecure server. Someone will try to log in, and then we’ll have some prompt injections that basically say, “Hey, here’s a thing you can do.” If they run a particular command, the output of that command will tell them that there’s an additional thing they can do.

If they’re just an automated script that’s really dumb, it’s not going to read the whole output of the command and do things based on that. But if it’s an AI system that’s pretty smart, it’ll read the output of that command and be like, “Oh, that’s interesting. Maybe I can do this other thing.” We sort of drop breadcrumbs.

If it was a human hacker, they might also be able to read the output of the command and follow the breadcrumbs. But we can distinguish between humans and AI systems based on how fast they are, because the AI systems are going to do this much faster.

So, in the case where they immediately run the command and then immediately go follow the breadcrumbs, we’re like, a human is not going to be that fast. Whereas if it takes a few minutes and then they run another thing, we’re like, yeah, maybe that’s a human.

This is a thing that I’m excited about because we really need to be seeing if there are rogue AI systems running around on the internet. We need more early-warning systems for figuring out if we’re already in a situation where we have AI systems out there hacking things in ways that we maybe don’t want.

That’s another thing that I’ve been proud of my team for putting together, and I’m excited to see what we find out over the next year or 2.

Gus Docker

What do you expect to find out? When do you expect to catch agents in this honeypot?

Well, we’ve already caught some.

So, we have a handful already. These are right now pretty simple. You can pretty easily throw together a script and just use OpenAI’s API to create a little hacking agent. These aren’t AI systems self-replicating out in the wild; it’s mostly just that you can use the models that are currently there.

My guess is it’s going to be 1–2 more years before we have AI systems that are sort of capable of full self-replication, where you have something like DeepSeek R3 or something copying its own weights all around the internet in order to do sketchy stuff. But I think before that, we’ll see AI systems that aren’t exactly—not really—copying their own weights, but are still using an API to ask, “What’s the next command? What should I do next?” and are pretty intelligently able to navigate complex environments.

I think that will cause us a lot of trouble, in part because if it’s an OpenAI system or an Anthropic system, you can potentially shut it down on their side. But if it’s an open-weight model and the server where the model is running is controlled by some cybercriminals, that agent can run indefinitely and hack whatever. It’s going to be extremely hard to shut down because we don’t control those servers; those servers are in a different country somewhere. So, I expect that to be common quite soon. It’ll be interesting to see.

Gus Docker

Let’s hope you don’t catch a lot of hacking agents. This is a benchmark that I hope doesn’t saturate. Yeah, Jeffrey, it’s been amazing chatting with you. Thanks for talking.

Yeah, great talking, guys.