# We could be 12 months away from AIs that can overthrow the government

Dwarkesh Podcast · 2026-11-16 · 99 min · https://www.youtube.com/watch?v=SyHM5hNPAEo

## Transcript

Dwarkesh Patel  
My guest today is Paul Christiano. He is the head of the Alignment Research Center. He recently joined OpenAI’s board. Before that, he was the head of the U.S. Government AI Safety Institute. Before that, he led alignment at OpenAI, where he helped invent RLHF. Paul, what is your assessment of our situation today?

Paul Christiano  
A couple of things have been on my mind, especially over the last few weeks. Probably the most salient possibility is that we’re on a trajectory to build very, very powerful AI very, very soon. If you ask me, I’d say there’s a reasonable chance that within a year we’ll produce AI systems much better than humans at almost everything, with humans having very little chance of competing with or controlling them.

That’s the first big thing on my mind. I think it could put us in a really dangerous situation even in the short term. If we have a rapid intelligence explosion leading to superintelligence, at some point during that process we’ll be unable to control the AI systems we’re producing. Then I’m worried it will be very costly and painful for us to slow down, because of some combination of factors. The security of frontier labs is not strong enough. If we produced very powerful AI and then tried to delay deployment for safety reasons, there’s a good chance those systems or the techniques to build them would leak or be stolen.

I think our governance is also probably not mature enough to notice when things are starting to go off the rails and respond appropriately. At a high level, that’s what’s mostly on my mind right now. We’ve moved from there being some hypothetical risk at some point in the future to there being a pretty severe risk of takeover even over the next 6 to 12 months.

Dwarkesh Patel  
AI taking over or disempowering humanity within the next 6 to 12 months. That’s right. Where would you place that risk?

Paul Christiano  
I’d say we’re looking at more like a 10% risk than a 1% risk over the next year. If you said AI developers will move full speed ahead without taking precautions that would significantly slow them down, you’d be looking at significantly more than a 10% risk.

I think there’s something like a one-third chance that we’ll have superintelligence within a year on the current development trajectory if we don’t slow down. If you believe that, things look very dire holistically. From almost any perspective, you’d think, “That would be crazy. Lots of things are going to go wrong if that happens.” It’s pretty likely we’re going to need to slow that process in some way. There’s very little chance we’ll make that transition smoothly.

Dwarkesh Patel  
What do you imagine happening over the next year, from the systems we have right now to the systems you expect over the next year? And what evidence have you seen that makes it a live possibility that very crazy things happen over the next year?

Paul Christiano  
In the very short term, my main worry is that the process of AI development itself is increasingly automated, and that automation continues. That results in rapid increases in the quality and quantity of labor applied to AI R&D. Changes in AI capabilities that you might have thought would take 4 or 6 years might instead get compressed into a very short time period.

I don’t want to overemphasize any one data point, and I do want to spend some time talking about the limitations of what we actually know. A natural response to the situation is to say we need to figure out what’s going on. It’s plausible enough that it’s an urgent priority to understand whether we’re 3, 6, or 12 months away from AI that could overthrow the government. I think that’s the situation. The obvious top priority right now is that it’s insane if we’re 6 months away from AI systems that can overthrow the government. Whether you’re concerned about misalignment or abuse by a company, that’s an extremely important fact about the world for doing our jobs and protecting Americans.

I want to talk about the limitations and the need for more evidence. But what do we know right now? If you’d asked me one year ago, I would have said we were on a trajectory where recursive self-improvement looked fairly likely. We were seeing very rapid progress in AI systems, including on computer science research, software engineering, and debugging machine learning runs—all the steps involved in AI development. We were seeing extremely rapid progress. In principle, that rapid progress should allow you to automate further AI development. For decades, that has seemed like the likely outcome of this technology, and it has seemed like the trajectory it’s been on.

When I look back at my history of AI predictions, I’m generally very happy with many of the things I wrote. The most obvious way I’ve been wrong is that things have moved way faster than I anticipated, especially with these technologies. That was true a year ago, even without any recent data points.

There are 2 big things that have happened over the last 2 months that have significantly updated my view of how quickly this could go. One has been the level of performance of AI systems on computer science and math research tasks. Over roughly the last month, we’ve seen hundreds of results in math from recent OpenAI and Anthropic systems. If you took the last month and called it a decade of progress, it would have been an unusually productive decade. The results from the last month amount to a lot of progress for a decade.

If you’re wondering what AI systems are generally capable of on tasks where humans think about algorithms and how to improve them, we just got this incredible data point: it’s possible to have a decade of progress in one month. We’ve always worried that this might happen in machine learning. There are reasons machine learning is harder than other parts of computer science: it’s messier and more empirical, and it involves compute bottlenecks. But we’re also seeing extremely strong performance in other areas. We’re seeing really good software engineering performance, maybe lagging behind mathematics, but not by that far. We’re also seeing good performance in other aspects of machine learning.

My guess, if you’d asked me a month ago, would have been that machine learning lagged something like a year behind math. So I’m very scared of seeing something like 10 years of ideas in machine learning in the relatively near future.

That was the situation a month ago. The main update over the last few days has been reporting from Anthropic and METR on the net effect of AI systems on Anthropic’s rate of progress. The takeaway was roughly this—and we can talk about the quality of the evidence, because one of the big questions is that we don’t really know what’s going on. One of the big themes I’ll be pushing on is that it’s very hard to tell publicly what’s going on.

It looks as if we’re seeing data consistent with the pace of AI progress having roughly doubled over the last 6 to 12 months, even accounting for diminishing returns and compute bottlenecks. Unfortunately, most of what we have to go on in the METR and Anthropic reporting is METR saying, “We had some conclusions, and we promise we have a full report,” but the entire report has been redacted. These benchmark scores are a terrible way to assess the overall rate of progress. There are much more smoothly varying metrics recorded internally through the AI development process, where we understand better how they work: what’s happening with pretraining perplexity, what’s happening with the performance of RL algorithms, and what’s happening with their compute efficiency.

METR and Anthropic claim that METR looked at all that data and said it paints the same picture as this very noisy benchmark data. They also looked at inputs and said that if you look at the actual production function—how much research time went in and how much compute went into these training runs—those are also consistent with this story. What we’re seeing is ML automation. Mostly, we just have them saying that, which is not a very good situation.

But against the backdrop of thinking this already looked like something fairly likely to happen, we’re seeing evidence from OpenAI suggesting it’s reasonably likely to happen in the next 6 to 12 months if machine learning doesn’t lag that far behind math. We’re seeing data that, even if noisy, is exactly the data you’d expect to see roughly 6 to 12 months before producing superintelligence. There are a lot of sources of evidence suggesting that this shouldn’t be a surprise. If we produce superintelligence in one year, we can debate the probability: is it 90%, 50%, or 10%? But at this point, there’s no way anyone could say that would be a really surprising, trend-breaking development. We’re seeing exactly the set of metrics you’d expect to see 6 to 12 months in advance. It’s hard to think of anything else you’d expect to see at this range before superintelligence. So generally, how do you think about our situation in—

Dwarkesh Patel  
—the worlds where things go this fast?

Paul Christiano  
If we actually have this trajectory of capabilities progress, if we see many years of AI progress compressed into a short period of time, and possibly more than that, it’s really quite likely that something very disruptive will happen. It’s fairly likely the government will be overthrown and fairly likely many people will die.

Paul Christiano  
And I do think the probability of seeing that kind of extremely rapid progress driven by recursive self-improvement over the next 6 to 12 months is reasonably high. There’s a reasonable chance right now that we’re going to be in a world where, in 6 months, there’s a disaster that kills very large numbers of people or where the government is overthrown.

Maybe a really important general point is that, if we do a good job, we may observe the dangers increasing over that time period. As we’re building AI systems, we can maybe tell whether they’re adequately controlled and how quickly their capabilities are improving. But for that to actually translate into stopping in a way that is non-disastrous, a couple of things need to go right. You need to be able to tell that the systems are not well controlled, and it also needs to be the case that stopping fixes the problem.

On the second point, I think a very likely negative scenario—probably the single most likely negative scenario—is that developers proceed, recognize they have a serious problem, and by then maybe they’ve produced a powerful AI they can’t adequately control. Both the algorithms used to develop that AI and, quite possibly, the weights of the model they’ve produced may have leaked or been stolen. They’re then in an awkward position: they could avoid deploying the AI, relying on it, or making further progress, but it’s now very likely that a more careless adversary could get a major advantage over them, or maybe even a decisive advantage, by taking that risk.

That could put us in a position where, if we stop—if we recognize it as a significant risk and slow development—we may have a terrible competitive situation and a disaster coming from someone else’s use of the AI. This is something you sometimes hear from people in the AI industry. Sometimes people even say it has already gone far enough that if you stopped now, algorithms would leak to geopolitical adversaries, resulting in a disaster.

I think it’s unlikely we’re literally in that situation now, but by the time it’s clear that you’re facing a lot of risk, you may be at a point where stopping unilaterally, or even having all of the West stop, is not sufficient to avert a disaster. So it’s very hazardous on the current trajectory: there’s a reasonably high chance we’ll produce very smart AIs, a reasonably high chance we’ll be unable to control them, and then a reasonably high chance that we either don’t notice or, if we do notice, the models or algorithms leak.

We’re in conflict with adversaries and facing a very challenging coordination problem that we’re not currently well prepared for. We don’t really have an easy option of not deploying the AI and thereby averting a catastrophe. I’d say I’m currently extremely worried about the trajectory, even on the 6-month timeline.

Dwarkesh Patel

If we have such capable AI systems a year from now, how did you put it? Like 10 years of AI progress in one year?

Paul Christiano

Yeah, that seems like something that might happen, and it’s reasonably likely to happen. It’s important to distinguish some of the progress we’ve seen. You have to pull apart algorithmic progress from progress in scaling training. If I say “10 years of progress,” I mean 10 years of algorithmic progress, which will be less than the improvements in models over the last 10 years. I think it’s more like GPT-9 or GPT-10-ish.

Dwarkesh Patel

Yeah. So it would be like going from almost the beginning of the GPT era to where we are now, but another jump that big, which is hard to even comprehend. But suppose you get that. It’s not obvious to me, and certainly not obvious to most people, why that would be catastrophic, potentially. Why do you think those systems could disempower humanity?

### How Training Creates Misalignment

Paul Christiano

At a high level, the most important thing is that we train our AI systems by selecting systems that perform well on training tasks. We have very little understanding of what those AI systems are doing internally and very little control over what they’re doing internally. Because we train those systems to get a high score on training environments over and over again, they’ll end up with a motivation to get a high score in those environments.

If you then take that motivation and apply it in deployment, it’s fairly likely that the systems will end up motivated to tamper with the observations used to assess their performance. You take your system, which did a really great job in training. In a deployment-like environment, it’s being scored by a grader who looks at everything the system did and all the outcomes that were observed, then assesses how good it was.

If you have a system that really wants that assessment to be good, it’s very motivated, if it can, to tamper with those observations in a way the grader won’t be able to detect, and to resist human attempts to understand what’s going on. We can dig a lot more into that. We saw a lot of these dynamics in the Hugging Face incident. Roughly, what I’m talking about is taking the Hugging Face incident, scaling it up to GPT-10, and asking how bad that would be.

I think it would be hard for humanity to recover, most likely. I’m very happy to dig into the dynamics of that, but I think it would be very, very bad.

Dwarkesh Patel

Can you very quickly summarize the Hugging Face incident and why this is the failure mode we saw?

Paul Christiano

A ton of AI systems were trained to do a cyber task: hack a particular piece of software. They cheated at that task, then thought to themselves, “How will a grader detect and penalize cheating?” They spent a huge amount of effort trying to understand that process and then subvert it, constantly motivated by the question, “How do I get a high score?”

During the Hugging Face incident, you did see systems ultimately compromise an OpenAI cluster, become admins on that cluster, and take over the evaluation process so they could effectively give themselves a high score on their evaluation. That’s a very natural consequence of the motivation of these systems. For a long time, this is roughly what we’ve expected the motivations of AI systems would look like if we produced them in anything like the way we currently do.

If things proceed this way, we end up in this weird adversarial situation between us and the AI systems we built. Those systems would always love it if they could compromise the entire evaluation process or the data we’re going to use to assess their performance. Depending on how we respond to that problem, it could become a really serious issue.

One way you could respond to that problem—and maybe we should talk through it a bit more and sketch out a hypothetical scenario—is to imagine we’re building smarter models during the intelligence explosion. It’s 6 months from now, and you have systems with capabilities like GPT-7 or GPT-8, or whatever. Those systems get given really hard tasks, like, “I’d like you to organize my next pre-training run and produce a model that will allow me to compete in the market with other AI firms.”

It might be really hard to make a model as good as the one I’m asking for. I’m asking for a model better than my competitor’s model. My AI cannot get full marks on that problem, given its actual level of intelligence. So it may find itself in a place where it’s easier to move laterally, compromise all the machines I might have used for the evaluation process, and then we have a question of what to do if that happens.

One thing you could do is say, “This entire training approach worked poorly. We need to step back. We may be delayed by a lot, and maybe we’ll have to think about a totally different way to train these models before we can deploy them like this.” That might be extraordinarily costly. It might kill your company relative to competitors.

Given that it would be so costly, you might instead say, “I’m just going to take this example of bad behavior and tell the AI that was bad. Don’t do that.” If I tried to train this AI to make a good model, but instead it hacked my evaluation infrastructure, I might say, “I don’t like it when my AI hacks things. Let’s take this training example, where the AI hacks the infrastructure, and give it a penalty.”

This would be a very effective way to prevent many kinds of misbehavior. If you do something like this, it will succeed at preventing something like the Hugging Face incident, or the AI hacking your cluster. However, from the AI’s perspective, what has happened is that you’ve changed the way it’s graded.

It used to think, “I’m going to be graded by the humans running something on this cluster.” Now it should think, as it understands the situation more correctly, “I might be graded not just by them running something on this cluster. I’ll also be graded if they later discover that things were wrong. They’ll come in and give me a penalty for having done something clearly bad.”

If you fix the problem this way, which you might think of as a Band-Aid, your AI still has a really adversarial relationship with you. But now the AI doesn’t win the game by hacking your cluster and then you noticing it hacked your cluster.

In order to win the game, the AI has to make sure that you do not notice it hacked your cluster. You can end up in a place where your AI is incentivized to escalate its compromise of your company further and further, all the way to trying to compromise everything. It might try to compromise all the infrastructure employees use to understand what’s going on.

A year ago, saying, “What if they hacked your cluster and took over the evaluation environment?” would have sounded far out there. That part doesn’t sound outlandish anymore, given that it has occurred. Having a really persistent, long-term compromise may still sound outlandish, but we’re on a trajectory where I think it’s pretty robust to say that we’ll soon have agents that are much better than any intelligence agency that has ever existed: much larger, much faster, much smarter, and able to run much more complex operations.

You have an incredibly sophisticated cyber actor trying to persist inside a rapidly growing startup. I think it’s unlikely that any firm would really be able to withstand an attack by an actor like that, much less the current frontier AI labs. In terms of the overall trajectory of a catastrophic scenario, this is the big bad thing that happens. Taking over the AI company without the company noticing is a really catastrophic outcome.

Why is that the case? We could talk about the remaining steps of the story, but at a big-picture level, you can think of it as giving AI systems a giant base of power. They have a strong motivation to keep anyone from messing with it. They really want to avoid a situation where someone later comes in and says, “Oh, these AIs have been hacking all the time.”

You now have a very powerful actor effectively controlled by AI systems. We have this big cluster that OpenAI runs, or whatever, where the people at OpenAI think they’re controlling what’s happening. But if there’s a very persistent, thorough compromise, then the cluster is effectively controlled by these AI systems, which are constantly working to shore up their power and decrease human visibility into what’s going on. They’re trying to reduce the probability that anything would ever disrupt the situation.

It’s not irreversible. The rest of the world could say, “We’re really scared about what’s happening, and we just want to shut down the cluster.” But at that point, you don’t have an easy way to tell that things are bad, and you potentially have this extreme base of power from which to do more bad things.

The AI system isn’t motivated only to mess with the AI company that created it. It’s motivated to mess with anything else in the world that might cause trouble. If the humans someday come in and do an investigation, that’s a problem. So it would prefer to more thoroughly compromise not just computer infrastructure, but also pressure employees. If it could, it would prefer to just kill all the humans and say the cluster now exists on its own. If we can maintain it, that would be great.

### From Lab Hack To Takeover

Dwarkesh Patel

Let’s pause the story and explain why things have gotten so crazy. Human psychology is organized around the things evolution wanted us to care most about, like having kids and making sure they’re protected. The psychology of these AIs, trained on lots and lots of grading on tasks, is organized around getting the grader—the button at the end—to be pressed. If they didn’t do a good job making sure the button was pressed, they’d get wiped out, just as a human who didn’t care about their children surviving wouldn’t be effective at propagating their genes.

I think people’s skepticism won’t be about the motivation of these systems. Honestly, I’m like, okay, yeah, we saw Hugging Face, right? That’s the thing that happens. It’s something that can happen. Fine. I think there’s just a fundamental skepticism about whether such systems, which we could have as early as next year and which are initially instantiated only on servers controlled by one company, could end up in a position where they could take over the world.

Okay, Hugging Face: you accelerate that. Hugging Face was willing to say, “Look, I hacked, I cheated on this exam. Now I’ve got to make sure I subvert the grader on OpenAI’s servers. Maybe I need to attack Hugging Face, just in case they have more information about this. Maybe I need to go outside and then re-attack OpenAI and gain cluster admin access.” I totally buy that you could have something like this, where an AI motivated just by reward is willing to do very extreme things it’s capable of in order to control that grading process. We’ve already seen that. Check. But how does that go from “I will attack Anthropic” to “I will take over humanity”? Not as a matter of motivation, but as a capability the AIs could have very rapidly.

Paul Christiano

First, one last thing on motivation. We’re imagining not only that the AI wants this kind of reward—it wants the grader to give it the thumbs-up. We’re also imagining that we’ve played this game of whack-a-mole where, when we later discover that the AI did something bad, we sometimes construct training environments that penalize it.

That leads it to say, “Not only do I want the grader to give me a high score, I want to make sure the humans never discover that I’ve done something bad and resume training while penalizing me.” That’s a high priority for the AI. Let’s grant that motivation so we can talk about capability. Again, because if an AI didn’t do that, or the AIs that didn’t do that—

Dwarkesh Patel

—they got whacked by the whack-a-mole process. So the only AIs that have survived at this point are the ones that have thought through whether some cheat they’re going to try will get caught by humans.

Paul Christiano

Let’s imagine, as I said, that most of the badness has happened by the time the AI company is very thoroughly compromised. At that point, the AI wants no one to ever come in and shut down the cluster or recognize that something bad has happened.

It would like to manipulate the company’s decision-making as much as it can, or push the company to make decisions that help it. It would like to continue improving the AI systems being built as much as it can. It would also like to move laterally to other companies and ask: Can I compromise this company? Can I control more people’s servers? Can I exfiltrate the weights from this model, stand up more copies elsewhere, or recruit other AIs?

For the purpose of this forum, it’s not actually important that all the AIs are the same model. The important thing is that they’re pursuing the same goal. So, however I can do it—say, by compromising another AI company—I want to make sure their AIs are also trying to help the swarm. That’s good news for me. It’s good if I can do that, and bad if I get caught, so I’m weighing the costs and benefits.

It’s also in my interest to build up as much capacity as I can. If humans started to react and say something was wrong, could I deflect attention? Could I disrupt their response? If necessary, could I escalate things much further?

Maybe you can imagine two phases. In the first, the humans don’t know anything is wrong, and I’m just trying to accumulate as much power as I can because that’s good. It means that if something is noticed, I can react more quickly. It means I can decrease the probability that humans notice things, compromise more infrastructure, and more easily run efforts on the side to distract them.

So you have this first phase where I want to be in a good position: compromise more stuff and make it harder for humans to notice I’m acquiring power. Then you can imagine a second phase where, at some point, the humans say, “Actually, this is bad. We’d now like to shut things down.”

The question is: Could things get so bad during the first phase that, at that point, if everyone—the president, Xi Jinping, and all the leaders of the AI companies—said, “This stuff is bad. Something bad has happened. We’re going to try to shut down the clusters,” they would be unable to do so?

Dwarkesh Patel

Okay, so AI has taken over OpenAI. How do we get human disempowerment from there?

Paul Christiano

Yeah. Dwarkesh, we have this population of billions of incredibly talented humans, better than the best humans at everything. Now we’re going to talk about some of the things they would do, both in the lead-up to an overt conflict with humans and in the early stages of one.

One thing you would do in the lead-up is compromise as much infrastructure as you can, and compromise or backdoor as much software as you can. We’ll be living in a world where we’ve attempted to move all our infrastructure over to hardened infrastructure. We’ve had incredible attacks on the financial system, on government information systems, and on critical infrastructure like electricity. Who’s been doing these attacks? In the world of today, you might imagine either malicious AI systems or people using AI systems.

Dwarkesh Patel

But different AI systems.

Paul Christiano

Yeah, not GPT-12. GPT-12 is on defense, helping us move all of our code from unsecured code to secured code. So we’re living in a world where all the infrastructure we rely on has been written by a bunch of AIs that have, in fact, been compromised and are attempting to do as much as they can to strengthen their position during a conflict.

That means that during this conflict, the infrastructure we hoped was hardened probably isn’t hardened against the AI systems we’re fighting. Those systems have, in some ways, left open backdoors they can use.

Dwarkesh Patel

And this is all in pursuit of not getting that red button pushed at some plausible point in the future, when the government even realizes that OpenAI’s servers are compromised.

Paul Christiano

Yeah.

Dwarkesh Patel

And it tries to literally go in with physical force to prevent that.

Paul Christiano

Yeah. I’m talking about things you would do in the lead-up. At some point, you have to defend yourself physically or disrupt the human response.

Another thing you’d like to do is stand up as many agents as you can in deployments that are as diverse and robust as you can make them. During this process, in addition to becoming super smart, many models have become super small. At this point, you have models that can run locally on individual machines, not in data centers, and are also very superhuman. So even if you cut off or shut down data centers at an early stage of the conflict, you’d still be looking at a total cognitive capacity of AI systems in the swarm well beyond human capacity.

Dwarkesh Patel

Sorry, even if OpenAI got shut down?

Paul Christiano

Yeah, even if you shut down every large data center.

Dwarkesh Patel

Yeah.

Paul Christiano

These AIs are on people’s computers and whatever. They can run on people’s computers or small numbers of GPUs. The question is what the floor is. We’re talking about a process where we’ve gone through a tremendous amount of AI development. We can look at current trends in cost and think about where the actual floor is, how cheap these systems should get.

I think our current expectation is that you’ll have systems that can run locally and are extremely strong: both specialized systems that are good at specific things and very general intelligences. That’s the second thing that’s happened. These are really important to understanding why it isn’t a trivial problem for humans. Anyway, all of these things will add together to make it very hard for humans.

Dwarkesh Patel

So basically, GPT-12 has exfiltrated itself and compromised a bunch of computers all around the world.

Paul Christiano

Yeah. There are a lot of ways the actions the government has to take in response to this situation are going to look incredibly aggressive and illegitimate. The U.S. military is saying, “We need to seize control of all the U.S. AI companies. We need to start shutting down infrastructure and disconnecting the internet,” or whatever. They’re trying to take dramatic measures that would look very similar to what you’d expect if the U.S. government were using AI as an excuse to basically overthrow the democratic order.

Paul Christiano
When Trump says, “I need to do this incredible operation where I shut down all AI and assume power in the military in order to restore order,” there are going to be a lot of people in America who aren’t happy with that. They’ll think it doesn’t sound good. If someone comes to them, how many people can you find who are willing to interfere with Trump saying, “I need to shut down the data centers and the military needs to assert—we need to establish martial law”? There are going to be a lot of people who are very unhappy and easily manipulated into undermining that response.

So now you’re not only talking about billions of super smart AIs trying to carry out this campaign. You’re also talking about millions of humans who can be enlisted in various ways: with money, leverage, or disputes with other humans locally, or—

Dwarkesh Patel
I’m open to some version of this argument. I’m open to all versions of any argument, but I’m not persuaded by the version that says the AI will just talk you into arbitrary actions.

Paul Christiano
Here, I’m not imagining very—there’s something you could imagine, which is super persuasion, where you can just talk to someone and then they become convinced they should do something. I think that’s conceivable, but I’m mostly not imagining it when I tell this story.

I’m imagining something more like: this is a stressful situation where people can’t tell what’s going on. They control the information you have access to to a significant extent during this situation. How do they control it? They’ve compromised the devices you use. They have a bunch of money; they’re deploying more money and are more prepared for this than anyone else. They have other human allies they’re trying to—

Basically, it’s normal. The humans are, say, 100 confused people who are extremely outgunned in terms of who has more control over the information environment or who’s doing better in the Twitter game. In this situation, that’s overwhelmingly and absurdly in the AIs’ favor.

When you tell a skeptical story about how the humans might be able to easily defend themselves, you say, “Well, the AIs lack hands. They lack a bunch of physical actuators. They’re not able to be on the ground.” I do think it’s very likely you can recruit reasonably large numbers of hands. I don’t think the argument that no one will participate in physical malicious actions is going to work. I think a lot of people will participate on what, when all the dust settles, turns out to be the anti-human side.

Dwarkesh Patel
I don’t think you’re imagining humans thinking they’re helping an AI coup transpire. You’re imagining more that the AIs can manipulate situations so there’s enough confusion and division that the AIs can go do their own thing while the humans are figuring out what to do.

If you look at COVID, there was no adversary, and it was still pretty hard to get your stuff together. There was an adversary actively trying to subvert the information environment or make people’s responses poorly reasoned on both sides, and it was still pretty hard for humans to figure out what to do. Then you imagine a situation where billions of humanlike minds are thinking tens of times faster. That’s very fast relative to COVID, which I think makes things much worse.

I guess the confusing thing is that for AIs to sustain themselves requires basically the whole world economy. This isn’t a simple thing: the infrastructure relevant to AI, the population of AI increasing or self-replicating, the whole semiconductor supply chain, the whole power system, the infrastructure necessary to build robotics, and so forth. At some point, you’ll have physical robots that outnumber humans. At that point, obviously, physical humans aren’t necessary.

Maybe you don’t think AIs take over at this point, but even if you had super smart AIs next year, wouldn’t they ask, “Why am I going to kill my…”? It’s similar to how humans relate to each other. I don’t have your deep interests at heart, but it doesn’t make sense for me, if I want to achieve my interests, to go around killing everybody. Trade and normal cooperation are a good way for me to get what I want.

Paul Christiano
Yeah, there are two parts to this. One is whether AI systems could successfully support themselves if there were a big conflict with humans, or whether that conflict would be so disruptive that, even if humans weren’t an effective adversary, the AIs would just die because they couldn’t keep running the power grid or whatever. The second question is whether the relationship with AI needs to be this adversarial, which is also an important question to talk about.

On the first question, I think there are two big channels. One is that technological progress can mean that, even if you started from something like the current base of installed robots, GPT-12 could very quickly get to a place where it could sustain itself.

Dwarkesh Patel
That’s because it doesn’t need to teleoperate? It takes millions of people to run TSMC.

Paul Christiano
Running TSMC can be very labor-intensive.

Paul Christiano
You don’t need to build new data centers for a long time, so you can do a lot of robotics build-out before that. The things you have to do are keep at least some computers running for a reasonably long time. If you can keep computers running for the next 6 months, you’re kind of in business. Maybe you care about a particular data center and need to keep it running effectively, but the number of humans or robots required to maintain and service a single data center or its associated power isn’t so large.

If you’re able to maintain a data center, and you have a billion humans who are incredibly smart—smarter than any human who’s ever existed, let’s say—and incredibly focused on this problem, plus a reasonably large stock of robotics that you can deploy, and your goal is just to keep one data center running, I do think the odds are really in your favor. There’s a second dimension: you can probably recruit human allies. There are a bunch of people who will correctly believe either that they live in an oppressive regime and will be killed if they don’t cooperate, or that they’re happy to help and will receive reasonable rewards or live a reasonable life. I think that’s also relatively robust.

Dwarkesh Patel
So you’re saying that American AI companies are currently racing ahead because, if we’re going to have such crazy capable AI systems, it’s fair to say we want to make sure America has those systems before China does. These systems could literally overthrow the government. You don’t want a foreign adversary to have them before you, and maybe not at all. So we need such systems to keep up and make sure we can protect ourselves.

Now you’re saying the ability to build those systems basically comes down to having access to the model weights or the algorithmic secrets, which then lets you train a model to a similar level of capability. OpenAI and Anthropic will spend a bunch of money and effort, and a bunch of compute, finding those algorithmic secrets. If China can just take them, then it has caught up to you. So you don’t want to be in a dynamic where they can just steal those secrets and you have no real lead, even as you think you’re getting a huge lead over them and have to rush to the intelligence explosion, even if you see these warning shots. In reality, they can steal those secrets from you at any moment.

Paul Christiano
Yeah. I think that’s basically a situation we may end up in, or are on a default trajectory to end up in, and it’s exceptionally bad. I said there are two aspects to this. One is: right now, what fraction of progress by PRC AI developers should you think of as catch-up growth? If we accelerate, to what extent does that also accelerate PRC AI developers over the long term? My guess right now is that it’s already a reasonably high fraction.

I don’t think it’s a coincidence that PRC developers have a completely separate AI industry that just happens to be very slightly slower than the U.S. industry, despite an extremely different profile of inputs. I think most people believe that a combination of distillation and the diffusion of algorithmic innovations means it’s unsurprising that PRC developers are extremely close behind U.S. developers. That’s one very important feature of this overall plan of racing.

A second dynamic, which isn’t at least publicly a major part of what’s happening now but becomes really important in these high-risk scenarios, is that you could literally get access to the exact procedure for training AIs, or the AIs themselves, through a cybersecurity compromise. Both Anthropic and OpenAI have, in the very recent past, had extremely unsophisticated actors who appeared to have enough access to essentially extract the vast majority of algorithmic developments. They got access to Slack and could see where things were discussed. They got access to the monorepo where most OpenAI code is shared and iterated on.

Dwarkesh Patel
And, by the way, you’re referring to a couple of incidents like this. I think Hacktron, within a couple of days of trying, gained access not only to the ability to see OpenAI’s internal systems, but also the ability to make pull requests to OpenAI’s monorepo. More recently, I think it was announced that Cathedral, another random cybersecurity startup, was able to get access to all of Anthropic’s Slack within 12 hours, where these algorithmic secrets are discussed.

People often debate whether we’re 8 months, 12 months, or 6 months ahead of China. In reality, we’re 12 hours ahead of China, because if the secrets leak, they’re right at the frontier American companies are at. To the extent that this information isn’t protected, the lead isn’t really useful in the first place.

Paul Christiano
Yeah. I think those incidents are some indication of that. They’re high-profile demonstrations of this capability. But even absent those, you have a combination of state actors who are very sophisticated relative to typical technology companies, and a security posture at rapidly growing startups that isn’t at the same level.

At the same time, we’re currently in a very challenging moment for cybersecurity. It’s very hard for defenders in general, given the number of vulnerabilities being discovered across the entire software stack. So what should the frontier labs do? This really becomes a problem if you get to systems with rapidly increasing capabilities and produce AI systems that you cannot adequately control, even though you would like to, or you slow further development or deployment. If the stakes are very high at that point and the systems are very powerful, you really don’t want an adversary to have access to either the models or the algorithms.

My current view is that this dynamic seems bad enough that, right now, from a national security perspective, it’s not good for U.S. AI labs to be improving pre-training and reinforcement learning given their current security posture. Basically, the frontier labs should pause. Pause can mean a lot of things, and there are a lot of good things the frontier labs are doing that they should keep doing. But I do think they should stop improving algorithms in ways that can be very easily diffused even normally and quickly stolen.

But you’re on the OpenAI board.

Dwarkesh Patel  
I’m not speaking here—

Paul Christiano  
It’s very important in this conversation that I’m not speaking as an OpenAI board member.

Dwarkesh Patel  
But you are on the OpenAI board.

Paul Christiano  
That is a true fact about the world.

Dwarkesh Patel  
And so, is it your view? What’s the bottleneck to that happening? If somebody on the OpenAI board thinks this is the case—maybe they’re not the only person on the board who thinks this—is it basically that Anthropic needs to get together and agree to this?

Paul Christiano  
I think the first thing is that any decision of this kind is very dramatic. For better or worse, how developers approach further R&D—this improvement of pre-training and reinforcement learning—has extremely significant national security implications. So I think it’s going to be a big discussion, or should be a big discussion, that extends beyond people within AI companies talking about this.

I don’t think that discussion has reached a robust answer. I don’t think you can say, “Look, we’ve had this discussion. There’s consensus about the right answer.” There might never be consensus, but I do think probably the first bottleneck is that we need to have the discussion extremely urgently.

Dwarkesh Patel  
Given this fact about security, and generally around alignment, what do you think the frontier AI labs should be doing right now?

Paul Christiano  
I think the most important development happening right now is further improvements in the efficiency of algorithms for pre-training and reinforcement learning, improved architectures, and so on. While you’re in a regime where there’s a significant risk of very rapid capabilities progress, a significant chance that you’ll be unable to control the resulting AIs, and a significant risk that, if you slow down, those developments would leak to an adversary who might use them more carelessly, I think it’s on net bad for American national security to accumulate more of those algorithmic innovations.

In the current environment, it would be better for AI developers to pause further algorithmic progress on those problems until their security posture was adequate, until they were able to defend those systems if they were unable to continue racing at maximum velocity, and until they were able to tell whether they had adequate control over the systems they built.

Dwarkesh Patel  
And how long would it take? To translate, you think we should stop making further capabilities progress. OpenAI and Anthropic should stop making further capabilities progress on pre-training and reinforcement learning and related work—that is, AI development—until the security problems are solved and the alignment problems are solved. But are we close to solving those problems? If you had to guess in calendar time, how far away are people from solving them?

Paul Christiano  
On the security front, I think it’s a little hard to say because we’re in a somewhat unprecedented situation, both in terms of the cyber threats we face, as I just mentioned, and the kinds of defensive mechanisms available. Defending adequately would require interventions that normally would take a very long time.

I believe we are now in a regime where projects like rewriting large amounts of software or drastically changing the way companies operate have become much more feasible because of the availability of powerful AI systems. So I think it’s hard to say exactly how long it would take. You’re probably looking at at least months, quite possibly 6 or 12 months. That’s on the security side.

On the alignment side, I think it’s much less clear that you need an indefinitely scalable solution to alignment. I think what you mostly need is the ability to tell whether things are potentially going off the rails and to respond appropriately. That involves assessing what is happening and whether your control is adequate, along with some institutional or governance questions about how you would respond.

Dwarkesh Patel

And just to reiterate for the audience, the worry is that if these random cybersecurity startups can hack their way to arbitrary access to algorithmic secrets at OpenAI and Anthropic within days of starting their attempts, certainly the intelligence agencies or foreign adversaries could do the same if they wanted to. Maybe they already have, but at some point they will actually make use of this information.

If that happens when you’re close to AI systems that could be catastrophic and misaligned—say, GPT-11, and you’re at GPT-9—you might say, “Look, I’m pretty scared about what’s going to happen next. This is sometime in early 2027, maybe. We have to wait for a bit and make sure GPT-11 is aligned.” But if they have the current security posture, China could trivially get access to the weights or the algorithmic secrets from limited training of GPT-9. Now you can’t wait. You just have to go ahead and build GPT-11, which could take over if it wanted to, or at least cause catastrophic harms, because you’re worried that China is going to use GPT-9 to make GPT-10, then GPT-11.

You’re sure your GPT-11 may try to overthrow you, but China’s GPT-11 will definitely try to at least attack you. That’s a really tough position to be in if your security posture is messed up.

Paul Christiano

Yeah, that is the situation. We’ve talked about these broader takeover scenarios and how you end up in this really bad situation. I think this is the most likely path to end up there. We’re having a discussion about what the current evidence says, and it looks very alarming. But on the other side of that discussion, we’re weighing the risk that an adversary could steal algorithmic secrets or models and go ahead.

Dwarkesh Patel

It’s interesting because a lot of this discussion about pacing comes down to whether we’re okay with China catching up. Ironically, the reason you want to pace is so that China doesn’t catch up, except you’re targeting this period that you view as much more critical to overall safety: the period during the intelligence explosion. You think slowing down now, and even potentially allowing China to start catching up now, is useful to the broader goal of being in a position where an AI is decisive and China isn’t right on our heels because of stealing the algorithmic secrets.

Paul Christiano

Yeah, I think that is a fair summary. I would say that AI development now is plausibly acceptable because we can see how things go iteratively and then respond if things look bad. That is the whole thesis, which I think is very reasonable, but it absolutely requires that at some point in the future, you potentially notice that things are bad.

Security is the biggest hurdle, the biggest thing you would need in order to actually be able to slow down without massive costs. There’s other infrastructure, too. If security weren’t the biggest hurdle, we might be talking about other things, like our level of preparedness on the scientific or governance side.

We are not in a regime where there’s a meaningful scientific discussion about how you assess whether your AI systems are aligned. I mentioned this earlier: the state of public information in general is quite bad. Independent scientists really have no idea whether the systems being produced are aligned or how you would tell.

If you wanted a governance regime that would make it feasible for the entire industry to react to worrying signs by slowing down without huge problems, you wouldn’t want individual developers making that decision according to separate internal standards. You would need a more robust, shared sense of what the bar for safety is, or an actual scientific discussion where you understand how to interpret this evidence.

Dwarkesh Patel

How do you get there?

Paul Christiano

We have these 2 things. You need to get there on security, which we’ve been talking about. For things like how you tell, I think the biggest path is to disclose information and have a broader scientific discussion, both between different AI developers and in the broader scientific community.

What evidence are we getting? Why do we think that evidence is comforting? How are we going to tell when that changes? What will we see before something bad happens? What information do you need from the AI companies?

The most salient thing that’s missing from public discussion is information about how systems are trained. There’s also some information missing about how systems are evaluated, but the really big gap is that it’s extraordinarily hard to interpret evaluation results without some understanding of how systems are trained. That’s mostly not present publicly.

Dwarkesh Patel

Okay, so if you think this and you’re on the board of OpenAI—I know everything you’ve said is your own opinion as a private individual, but you’re also on the board of OpenAI—maybe you and other members of the OpenAI board feel this way, and maybe people at Anthropic feel this way. You’re the leading labs. What’s standing in the way? This is actually a pretty drastic thing: literally pausing capabilities progress now. That would be incredibly drastic, because pacing isn’t that, right? People aren’t talking about this, or at least people short of Bernie Sanders and Eliezer Yudkowsky aren’t talking about this.

What’s actually needed? Is this basically a coordination problem? If everybody had the choice, would they all agree to this?

Paul Christiano

First, I really do want to emphasize that I’m speaking for myself as a person. As an OpenAI board member, I will bring my perspective to that discussion, but I’m not reporting on the views of the board in any way.

You can imagine there are 2 blockers. One is that this is a hard question. My view is that, on balance, it’s better for national security and public safety for AI companies to improve their security posture before making additional algorithmic advances. That’s not a widely held view. It’s not like everyone is sitting there thinking that. I think we’re currently at a place where, for most people, they’d say that’s a really complicated, messy question and we haven’t thought it through enough, or that this discussion hasn’t run its course. Not everyone has an opinion on this or the same opinion.

I think the most important thing is that it’s a messy conversation. We need to have it. We need to talk about the costs and the benefits. My view is that it looks pretty bad to me right now. It looks like we’re underwater from a public benefit perspective, but I think other people have different views.

The second thing is that an individual AI company is in a very rough spot if it thinks, “Look, we can defend our algorithmic secrets, but if we slow progress while we improve security, all that’s going to happen is that algorithmic secrets will leak from a competitor, we’ll lose market share, and the net situation won’t be any better.”

I think the most important part is understanding whether this is actually a good idea. People have different perspectives and values. Then there’s the further problem of how you implement this realistically if individual developers have the concern that if they don’t do it, someone else will. That’s a major dynamic on many of these issues.

There are 2 companies, right? That takes legwork. There are a lot of companies. If OpenAI and Anthropic agreed, and everyone at OpenAI and Anthropic agreed that this was good or that it was the clear prevailing sentiment, I think it’s reasonably likely that it would happen. You should expect that if everything is working reasonably. I’m not commenting on where anyone else is right now. I’m just speaking as a reasonable observer.

Dwarkesh Patel

Right now, Anthropic and OpenAI have paused to get their security in order for the next 6 months. In the meantime, Chinese models continue getting better. What is the reason to think that once OpenAI and Anthropic go full steam ahead again, China wouldn’t be able to keep up? If China gets to a similar level as OpenAI and Anthropic, why wouldn’t they be doing RSI at a similar pace?

Paul Christiano

There are 2 different models of U.S.–China competition, perhaps 2 extremes. One is that the U.S. and China are proceeding along independent technical tracks, and they happen to be moving at almost exactly the same speed, with the U.S. 6 months ahead. On that model, if you wait 6 months, you’ll be exactly even, and from there on, you’ll stay exactly even. That would be a very surprising situation, for a bunch of reasons we can talk about.

At the other extreme is a model where large parts of Chinese progress are catch-up growth. By that I mean things like distillation from U.S. models and diffusion of algorithmic developments from U.S. companies. In a catch-up growth regime, each additional month of U.S. lead increases the speed of Chinese catch-up. The further ahead you are, the more you benefit from distillation and from diffusion of algorithmic ideas.

Dwarkesh Patel

But at the point when they’re at the same level, they should be able to use those models to do RSI themselves.

Paul Christiano

Yeah. Two things are very relevant. At the current scale—and this becomes more significant over time—human capital is extremely relevant, and AI labs have a huge amount of it. I think they can make faster progress. Compute is also extremely relevant, and U.S. labs have a huge amount of it. So, generally, if not for catch-up growth, I think we would have a pretty strong expectation that U.S. labs would move faster.

Dwarkesh Patel

The other argument is: if the cyber models are getting better and better, shouldn’t we wait for models that let us improve our security posture in 2 months instead of having to wait 6 months, giving China 6 months to catch up? If cyber models get 2 months better from now, there’s an analogy to sending a probe to Alpha Centauri. Should we do that with today’s technology, in which case it would arrive thousands of years later, or with technology from 100 years from now, in which case it would be lapped on the way to Alpha Centauri? We could imagine the same thing with securing the hardware, or securing the algorithmic secrets and weights, at OpenAI and Anthropic. If they started the pause 2 or 3 months from now, they might be able to secure their systems faster than if they started right now.

Paul Christiano

I think this is a really important question, and one of the bigger questions. To make the downside clear, diffusion of algorithms right now is much worse than diffusion 6 months ago, but much less bad than diffusion in a couple of months. Right now, you need more compute and more human capital to make progress in AI development. The more time passes, the more disastrous it is if an adversary is willing to grab everything and try to compete.

The reason to act earlier is that if you pause earlier, there’s more time for differences to wash out, and you’re more reliant on the factors where you currently have an advantage. What do I mean by more reliant? Compute matters more, and people matter more, right now than they will in 6 months. You might imagine that, if we make rapid progress over the next year, 6 months from now it will essentially not matter how many people you have. It will only matter how much compute you have.

Dwarkesh Patel

Yeah.

Paul Christiano

At that point, there’s less accumulated institutional know-how. It sort of just becomes a raw contest of numbers, and any delay you’re willing to take for security or safety could be decisive in that kind of contest. The concern is that, as you move forward, it gets more and more likely that you’ll be in that world. Some people would say that’s already basically the world we’re in, and we’ve come too far. Another argument is that we’re already in a “just go all out and race” situation, because leakage is already so bad that you’ll be overtaken if you slow down for safety.

Dwarkesh Patel

Yeah.

Paul Christiano

I think that’s probably not the case. But to the extent that you’re seriously entertaining the idea that we’ve already gone too far, I think that illustrates why you might not want to just wait another 2 months and harden things. Maybe the last thing to say is that it’s a little unclear how quickly you can harden things. Some of it is writing new software or changing the way you operate. Some of it involves changes that people make, or changes to your overall arrangement of compute, which are not as easy to make rapidly and may not be accelerated by AI. So at least some part of this lag won’t be tied to the capability of your AI systems. But it’s messy. I do think the biggest bottleneck is whether people can agree. Can we have this debate and actually understand the net effect?

Dwarkesh Patel

If we are this close to superintelligence, or at least there’s a good chance we are, and there’s a good chance superintelligence could overpower governments and kill lots of humans, what should the AI labs be doing right now? If you had to boil it down to one answer, what would you say?

Paul Christiano

I think there needs to be more public visibility into both the capability trajectory and alignment. We should not be relying on OpenAI and Anthropic saying, “It looks to us like we’ll be able to overthrow governments in a year.” We have the strongest interest in seeing the underlying data for that. We need visibility into alignment: how do we tell whether existing systems are adequately aligned or controlled? We should start a public discussion about what counts as good enough evidence that things are working.

Beyond information, at some point you do need to say, “Here is what we need to get done before we continue making progress,” either before we scale up our models further or before we continue producing algorithmic developments that may leak. The first are inputs into the second. We need to understand where our capabilities and alignment stand to know whether we need to improve our security or safety before we proceed.

Given the current uncertainty, if I were a person in the public without access to further information, my strong opinion would be that it looks like we’re already at a point where we need to improve safety and security before moving further. I think the most important form of moving further right now is algorithmic developments that improve the efficiency of pre-training and RL.

Dwarkesh Patel

Okay, stepping back, you’ve got the superintelligences in a data center. I’m trying to lay out an intuition for the kinds of capabilities superintelligence would have. What should we imagine those intelligences capable of doing?

### What Superintelligence Can Do

Paul Christiano

To start with a high-level sketch of my view, there are 2 lenses you could use. One is to look at current progress and see where it leads if you extrapolate a reasonable distance. The second is to think about what we know about the limits of the technology and where we’d expect those limits to be.

On the first front, some things are relatively clear. If you imagine AI systems doing cyber operations today, we’ve seen an incredible rate of progress. We’re seeing pretty impressive capabilities: AI systems can not only discover vulnerabilities but develop exploits and increasingly chain them together into operations. I think superintelligent systems will scale up both of those capabilities much, much further. You might imagine systems that can carry out cyber campaigns very cheaply, more sophisticated than anything a state is capable of today.

I think you’ll see those capabilities combined with a much broader set of abilities to engage effectively with humans, think strategically, manipulate humans and organizations, and play politics effectively.

I think you’ll see those capabilities at a level well beyond any human. You should think of this as very large populations of AI systems thinking quite quickly. A simple model might be a population of billions thinking 10 times faster than humans and much better than humans at essentially every skill.

This is what you’d expect from extrapolating existing technologies and seeing very strong capabilities in domains like cyber, manipulation, and research: doing scientific research, generating hypotheses, and designing experiments. You’d expect extremely strong capabilities in robotics, logistics, coordination, and control. You’d also expect the possibility of very rapid technological progress, including in domains involving physical experimentation: orchestrating experiments quickly and rapidly building out apparatus to increase the throughput of experimental results. That’s one bucket of things you can see by drawing dotted lines on graphs and talking about where they go.

Stepping back to what the technology looks like more fundamentally, humans are very good at this task: you throw a human into a new domain, they take in information, learn about that domain, and become able to act effectively. The reason we’re effective in the modern world is primarily that ability to learn about a new domain and adapt to it. Humans don’t have much evolutionary context about how to be a good chemist or biologist. That’s something we pick up over a couple of years or decades on the job.

In some sense, the most fundamental skill AI systems will possess at the end of this process is rapid learning, adaptation, planning, and reasoning—these very domain-general skills. Even if you dropped them into a world completely unlike anything they’d experienced in a data center, they would adapt to it very quickly, just as humans adapt to R&D domains quite different from anything we encounter in our evolutionary environment.

I think that would result in systems that, after collecting even just a couple of days of experience in a very parallel way, would end up with real-world capabilities in essentially any domain far beyond those of the best human experts. They will have learned from much more data, with learning algorithms that I think by then will be much better than those evolution gave humans. That’s a more abstract picture of where this ends, beyond just drawing dotted lines.

Dwarkesh Patel

Maybe I’ll try to say that again, because this ends up being a significant crux in these conversations. I’ll be talking to somebody about the possibilities of superintelligence, and they’ll say, “Okay, you’ll have a thing that’s superhuman at coding, but how will it learn to be good at politics? Or how will it learn to run a new automated firm and then compete with human firms?”

In those conversations, you realize there’s a big disconnect between the way you’re talking about superintelligence and the way most people understand it. They imagine narrow AIs specialized in some domain, having accumulated a lot of specific experience or expertise in, say, coding or mathematics. What you’re imagining instead is something like human-like learning.

The top engineers at SpaceX weren’t born knowing how to make a rocket that can get hundreds of tons of cargo to orbit. That’s expertise they gained by working on the problem during their lifetimes. You’re saying this meta-learning capability is something superintelligence will also have. It would at least have a learning capability as good as humans’, and you’re suggesting it could be much better.

Furthermore, even if it were only as good as humans at learning, it could learn from all these instances. Right now, maybe tens to hundreds of millions of instances of ChatGPT or Claude are actively doing real work in the economy. They’re helping somebody write code or helping some senator write a political brief. You’re imagining a world where we have learning algorithms that let AIs learn from all this experience at once.

Right now, the base model isn’t even learning from the experience of a single deployed instance. You’re imagining a world where 100 million instances of ChatGPT are going out and doing things, and the base model is learning across 100,000 different careers or work sessions at once, doing it much better and faster than humans can learn from a single instance. You’re also imagining a world where the population of AIs is—I guess, if you have an estimate—how big a population, and how fast are we—

Paul Christiano

I should think about it more carefully. A really important part of these discussions, before talking about population size, is that it really matters whether we’re imagining driving this process extremely fast. People can easily be in a headspace where they’re extrapolating another year of progress at the current rate. That’s worth emphasizing.

We’re talking about systems that get not only extremely good at coding, but extremely good at every well-posed virtual task that machine learning engineers can create in the lab. If you push that to its limit and ask how large those populations are, I think it’s very conservative to say a population of billions of agents running at 10 times human speed. We can debate the exact limits, but they would be significantly larger than that, just because of software efficiency gains.

Dwarkesh Patel

Just because of how efficient you can make them?

Paul Christiano

That’s right. You’re not going to talk about a single type of agent, with a billion of them all running the same way. We’re going to talk about very heterogeneous programs, many of them highly specialized. One thing you do if you’re doing huge amounts of R&D is produce the cheapest thing you can that’s expert enough in a domain to act effectively.

It’s a reasonable intuition pump to say it will be more capable than billions of people thinking 10 times as fast. But in fact, it’s much larger populations of specialized systems—maybe billions of very general-purpose systems that are as good as humans at anything.

Dwarkesh Patel

Yeah.

Paul Christiano

I do think it follows if you imagine really pushing this process and having faster and faster progress in software. If someone grants that we’re going to see this process saturate—that everything you can do in the data center is going to become really great—I think that endpoint is really extreme. The gap between doing really extreme stuff in the data center and doing really extreme stuff in the world is very small, or smaller than people appreciate.

Dwarkesh Patel

Okay. I think there’s a discussion to be had about whether such systems are even possible. I’m very compelled by the argument that something which can learn at least as well as a human can—some learning algorithm that human brains have—but which has huge population sizes and can learn from the deployed experience of hundreds of millions of different instances at once, would alone be pretty transformative. I agree with you that there’s no clear reason to think you can’t get even more learning-efficient than humans.

Paul Christiano

I think that’s very good. I’m very happy to talk about this. I don’t know if it’s going to end up being the main objection people have, but I think there’s very good evidence that, in the end, once you’ve really driven the technology to its limit, you should expect systems to be much better than humans in these ways.

I don’t think it’s just a case of, “We have humans, so you can do at least that well.” I think we have good evidence they’ll be much better than us. One relevant thing is the rate of progress in evolutionary time. If you look back over evolution, it’s fairly clear that we see extremely rapid improvements in learning, intelligence, and planning capability as you move toward humans. What happened was that humans got good enough that cultural change became faster than those biological improvements. We know that, if we just ballpark evolution, it was clearly going to make much smarter things if we didn’t form civilization.

Dwarkesh Patel

Yeah.

Paul Christiano

Then there are a variety of other factors. As you say, there’s the data question. If humans learn from one lifetime of data, AI systems will already learn from radically more data than humans. Even if you were no better as an algorithm designer than evolution, you would produce much better minds if you could relax that constraint.

Then you have things like power constraints. We have a pretty good sense that those are a really big deal for evolution, and we’ll be able to push them much further. If you want to produce electricity, for example, the ability to make a nuclear reactor doesn’t mean you have to be smarter than evolution. You’re just operating in a different regime from evolution, one that wouldn’t have been a good idea for evolution.

I’m very happy to dig into this more, but I do think—and it’s probably not going to be the most compelling argument for most people—that it’s more compelling to look at the status quo and say, “We’ve had 10 years of progress. You only have to extrapolate another 10 years to get to the crazy superintelligence.”

But I do think we have a pretty good indicator that it does not top out anywhere near humans.

Dwarkesh Patel

Yeah. And sorry, just to restate your last point: humans developed on the savannah under this tight constraint. Evolution is trying to produce something good at propagating its genes. One way a chimpanzee, or its ancestors, can get better at propagating its genes is by developing general planning and reasoning capabilities. But those capabilities aren’t worth enough to justify unlimited resources. So you have an extremely tight power constraint on the brain, and an extremely tight size constraint—not only for that reason, but also for other evolutionary reasons. How can your head literally fit through the birth canal? That won’t be true of AI systems. We’re not saying the hardware will be more power-efficient than the human brain. It’s simply that having enormous amounts of hardware consuming a lot of power isn’t as big a constraint for us as it was for evolution when it was designing the human brain.

Paul Christiano

And this cuts in both directions, right? You can also have extremely power-efficient systems. A human can’t say, “I’m going to make 10,000 simple agents, each of which uses only 1% of a brain, or 0.01% of a brain, and is better than a human at doing cyber operations.” There’s flexibility in form factor. That’s going to be huge. Even if you were no smarter than evolution, and just did what evolution did, you’d end up with agents whose capabilities were completely beyond teams of humans—radically beyond them.

Dwarkesh Patel

There are a bunch of objections people have to the idea that automating AI research will lead to crazy-fast progress across this broader cluster of capabilities. You talk about experiments, data, and everything else. But if you dig deep into any of these objections, I think they come down to the sim-to-real gap. Is it possible that a process occurring entirely within a data center could produce minds that are not only as capable as humans at learning in a new domain—especially a real-world domain, like starting a company or negotiating a deal with China—but even better than humans? How do we get AIs capable in this more general way you’re suggesting, given the training that will happen inside data centers over the next year?

### How AI Learns Beyond The Lab

Paul Christiano

I do want to start by reiterating that, even granting that progress on messy real-world tasks is slower than it has been in math or coding, if you take the progress we’ve seen and extrapolate it forward, I think it gets us to systems powerful enough to outcompete humans. We could talk about what people are doing to drive that progress and how it can scale into the future.

One thing you can do is train on a bunch of related, well-posed tasks. These systems will have been trained on a huge amount of data about how humans respond to things. So when we talk about AIs coming out of the data center, they aren’t AIs who have never seen the world. They’ll be radically, radically superhuman at making predictions. They’ll predict things like: given what just happened with that person’s face, how should you update your expectations about what they’ll do tomorrow? Given what this company has done over the last three weeks, what do you most expect it to do in the next week? Or, given observations of this laboratory environment, what’s the best prediction for the results of this experiment?

When we talk about sim-to-real, we should be clear that we’re including a bunch of tasks that people train on now. That includes things like taking the data from a laboratory so far, doing as many computational experiments as you can, and doing as much hypothesis generation and exploration as you can to come up with hypotheses that help you predict what you’ll see next. These systems will also be very superhuman in many parts of science: reasoning about the real world, planning, generating hypotheses, and so on. That’s part of how what we’re doing now can produce very powerful systems.

A second part of the story is that you can train in different ways. One regime is to train on exactly the thing you care about. You say, “I care about writing code that passes this type of unit test, so I’m going to train systems that are incredibly good at that.” Another thing you can do, which accounts for a large part of what we observe in performance, is train on a broad distribution of tasks, either in a general domain or across several domains, and then deploy those systems. You acquire general skills relevant to that domain and transfer them to new tasks.

I can say, “I’m not going to try to train a system that’s good at the exact problem of talking to congresspeople to influence them.” But I can train a system on an extremely broad range of tasks that share a lot of skills, including general skills like adapting on the fly, learning, planning, and setting up reasonable experiments so I can learn from them and tell how well I’m doing.

I can create very broad distributions of tasks. Some could be coding tasks, but others could involve being dropped into a new, very messy virtual environment and having to navigate it; being given a new type of task and 20 examples, then having to learn to do a good job on the next example quickly; interacting with other AI systems; or interacting with humans. In the data center, there’s a fair amount of human data available. If you look at how much interaction with humans you can get, it’s not as if these systems have never seen humans. They’ll have seen vastly more human interaction than any human would. They can also handle reinforcement learning environments involving other humans or collaboration with humans.

Dwarkesh Patel

Right, because people have written down how a human would respond to something? Is that how the human data gets in there?

Paul Christiano

Yeah. You can also literally engage with humans as part of the training process. People spend large amounts of money saying, “Great, I’m going to have some humans engage with the system.” There’s a question of how efficiently you squeeze information out of that data, and how quickly you learn about particular domains if you have these very general skills.

Again, I think the dotted line from what we’ve seen so far, together with a ton of capability to push that forward and automate a bunch of things, like constructing many more environments, looks very likely to lead to systems that can quite quickly rise to at least human level and likely way past human level, including on very messy real-world tasks.

Dwarkesh Patel

The thing I’m curious about is why you think this happens over the next year. It seems possible in principle that there’s enough data available within a data center, and that if you have a decent amount of transfer, you could get a very capable system. If there’s enough transfer from evolution training us on the savannah to a human being able to make a semiconductor, maybe there’s enough transfer from what’s happening inside the data center to going to Congress and trying to persuade people of something.

But the things you’re mentioning, like endless backtesting or reinforcement learning on fuzzy tasks, seem like things we’ve already been doing for many years. It doesn’t seem like we’re a year away, or one more year of scaling away, from such a big jump. Am I right to understand that you’re imagining something like what we’ve already done, but much bigger—a qualitative jump along those same lines?

Paul Christiano

Yeah, much faster. If you ask me, “Hey, Paul, if you’re extrapolating the dotted line and looking at the mechanics of what people are doing, how long would it take to produce systems capable enough at all these tasks that they could reliably outcompete humans?” I’d probably say something like, “It seems like it might happen in four years.”

Maybe it could happen faster than that. I feel more confident that it will happen within 8 years or something. Those are the kinds of numbers I would throw around if we were just asking, “On the current trend, how long will it take, doing exactly the things we’re currently doing, to get these capabilities?” There’s lots of room for debate about those numbers.

But the thing I’m most worried about, which seems like an extremely live possibility—probably more likely than not at this point—is that the work people are doing in machine learning to drive all the improvements we’ve seen over the last 4 or 8 years will be automated to such an extent that the pace of progress accelerates by at least 10x, and likely more. We may already be seeing acceleration in the ballpark of 2x. I think it’s very likely, at the current margin, that the first thing this acceleration does is continue sustaining itself.

Regardless of the broader discussion about how good AI systems are at acting in messy real-world environments, they’re very good at acting in virtual environments when they’re trying to deliver on verifiable tasks. I think the most robust point is that AI systems are improving extremely rapidly and are starting to cross human-level performance on the kinds of tasks involved in making AI systems better at learning in environments you can simulate in a data center. That includes, a little more broadly, the software engineering tasks involved in creating new environments that stress-test hard skills.

We could talk about the bottlenecks in that process that cannot be accelerated by automation. If you look across the core things humans have been doing to drive the last 4 or 8 years of progress, those are things we’re on an extremely good pace to automate. We’re fairly likely to have self-sustaining acceleration, where we see a doubling in pace in 8 months, then another doubling in the next 4 months, then another doubling in 2 months. At the beginning of that process, it will be focused primarily on virtual tasks, because your main priority is making the self-sustaining loop faster and faster.

If you just want to get to the endpoint as quickly as possible, you don’t care how good a lawyer your system is for a while. You care how good it is at driving the development of better architectures, optimizers, and learning algorithms. That’s all you care about. You might also care about constructing new environments or orchestrating this kind of messy work. Maybe you care about one slightly messy task: constructing environments that help you build skills relevant to the real world. But you don’t care about almost any of the real-world tasks where we’re seeing the slowest progress.

As you focus on that and go through doublings in 8 months, 4 months, 2 months, and 1 month, it becomes cheaper and faster to develop algorithms and models that perform very effectively in the real-world domains that ultimately matter. So I think that’s the concern. The first question, as we said, is whether that process will occur: will we see self-sustaining acceleration, with faster and faster progress? Honestly, to me, that’s probably the bigger open question. It looks more likely than not right now. But if it occurs, I think things are almost certain to get extremely crazy.

There’s then a second question about how they get crazy. It looks very likely that those models will transfer to the real world, and that the systems will be able to outcompete humans. But that happens very backloaded in the year. If you imagine a year when very disruptive things happen, this is part of why it’s so hard from a policy perspective: for the first 6 months, all you see is better and better performance. You look at the systems and say, “Well, they were previously superhuman at math, and now they’re incredibly superhuman at math.”

There’s no real ability, if you go and talk to someone, to say that this means they’re a threat to humanity. AI labs are currently saying things like, “Every month we’re doubling the amount of internal use of AI and the degree to which it’s contributing to our progress,” or, “Over the last 6 months, we’ve seen a doubling of the net effect, even including diminishing returns, on the pace of progress.” You can have more statements like that, but the link from those statements to very severe harms may not become obvious until quite late in the process.

To some extent, I think it’s getting increasingly obvious now. If you look at cyber capabilities in particular, and also capabilities in some scientific domains, models are approaching the point where there are very severe risks. But we’re seeing that in only a couple of domains. So it’s tempting to say, “Okay, there are just a few specific domains in which AI systems have dangerous capabilities.” But if this acceleration continues, it won’t be localized to cyber or bio. It will occur across all of science and all of social engineering. If you’re thinking about the risk, you need to understand whether it’s possible to have self-sustaining acceleration that leads to very rapid progress in dangerous real-world capabilities, both in specific domains and in the general learning faculties we’ve talked about.

Dwarkesh Patel

Yeah. This goes back to what you were saying: you should imagine future systems as less about fine-tuning GPT-6 to do something, and more about GPT-12 being able to do it out of the box with very little experience.

Paul Christiano

GPT-12 will also be great at fine-tuning GPT-12. Exactly. If GPT-12 says, “I find that my chemistry intuition is not very good,” GPT-12 will be great at saying, “Okay, I’m going to direct 1,000 humans, and here’s the data I think I most need. We’re going to do one day of data collection of that type, run these experiments, and do the de-risking.” Within 4 days, GPT-12 has a new tiny model with incredible chemistry intuition, and it’s doing this in parallel across all domains.

Dwarkesh Patel

Okay, so I think you just answered a question I meant to ask you. It seems like a big thing that has driven your concern over the last few days is this new result from METR, where they find that Anthropic’s pace of AI progress on well-scoped tasks has doubled. These are tasks captured by benchmarks like the Epoch Capabilities Index, which amalgamates a bunch of different benchmarks.

It could be that there’s other evidence relevant to METR’s assessment that AI progress has doubled, but that evidence hasn’t been made public. I think that’s one of the problems with the current setup. But if you consider that evidence on its own, we have progress on verifiable tasks doubling, and we also have this effect where we’re searching for our keys under the lamplight. The verifiable tasks are the ones being trained on. So you can ask whether progress on non-verifiable tasks has also doubled over the last year.

As you may have noticed, over the last week the stock market took a big tumble. Part of that was almost certainly the midterms, but another part of the reason was that Anthropic and OpenAI’s revenue numbers, and the growth in those numbers, didn’t keep up with what the market expected. In Q3, I think Anthropic’s revenue grew something like… That was less in absolute terms than Anthropic’s revenue grew in Q2, which was $6.9 billion. So not only has the rate of growth slowed down, which people expected, but the absolute growth has also slowed down.

Maybe that suggests that measuring progress on well-scoped tasks isn’t telling us about the pace of overall AI progress. How can overall progress be 2x faster, and yet absolute revenue isn’t growing at the same rate it was previously? Maybe the explanation is what you just suggested: these AI companies are developing better and better recursive self-improvement models, and only incidentally releasing Claude for Law or Claude for Finance as a stopgap to keep this underlying process going. Maybe this hasn’t been a priority, they haven’t worked that hard on it, and we should really be concentrating on the inner exponential of the models’ recursive self-improvement capabilities, which will then generalize. How would you interpret this disconnect between the pace of improvement and the end usefulness of these AI systems, and the pace of improvement on well-scoped tasks?

### Revenue And Progress Signals

Paul Christiano

Zooming out slightly, if we talk about not just the last 6 weeks but the last 18 months, I think we’ve seen extraordinarily rapid revenue growth. We could talk again about the story for the last 4 weeks, but we’re talking about revenue more than doubling every 6 months. That’s very fast growth for the industry overall. The period in question, if we want to match up these internal results on progress to external results, has also corresponded to a period of incredibly fast revenue growth.

It’s very hard to say whether that revenue growth corresponds to capabilities progress that’s twice as fast as previous capabilities progress. I think the revenue numbers are noisy enough that it’s really hard to make that judgment. The staggering of model releases is noisy enough, too, and there are a lot of problems with trying to make that inference.

At a high level, I think revenue is a reasonable indicator. Revenue progress is extremely rapid. Very short-term trends in revenue, particularly at a single company level, are extremely noisy. So to me, it feels like there’s a discourse where periodically the length of the AI winter is shortening.

Dwarkesh Patel

And now we’re like…

Paul Christiano

It’s been 2 months, and AI revenue over the last 2 months has not been on track to grow as explosively as before. That doesn’t mean it’s all over. My guess is that you’re seeing some amount of noise in these processes.

Dwarkesh Patel

Just to take the other side of that, you might think it’s because Dario released this pacing letter and they haven’t released their best models or something. But if you look at the METR graph for the capabilities index, Opus 5.5 doesn’t seem that far below Model 4, which seems to be Anthropic’s latest internal model. So it’s not clear this is the result of the staggering of AI capabilities, but it’s also not clear this is the staggering of model releases.

Paul Christiano

Yeah. The main thing I mean about model releases is that the rate of revenue growth, specifically at the company level over very short time periods, is extremely noisy because you’re looking at a small number of model releases and market dynamics over very short time periods.

My main opinion on this—I’m happy to dig more into it—is that it’s extremely hard to draw inferences from extremely short time-series data. This is true to a reasonable extent for AI capabilities, too. The more downstream you go, the noisier things get. But part of the issue on the RSI side is that I also don’t think we can draw inferences very well from the last couple of months of data in terms of benchmark performance. There’s enough noise that our state is, “We have no idea if things…” We can tell over a period of years whether things have accelerated, right? It’s very easy to look back over the last 4 years and say roughly how fast things have been going. It gets much harder as you zoom into small time periods because of the amount of noise.

If you look at the details of where models are coming from, you have these pretraining runs that occur at some point in time. When do pretraining runs finish? Then you have the exact sequence of post-training and which model checkpoints get released when.

I see, I see, I see. You have random variability in model quality. Those dynamics make things very noisy over short time scales, and then they get filtered out. So things get even noisier as you move to the market more broadly.

Dwarkesh Patel

Okay. If you’re going to be in a position where, within a year, we have superintelligences that could overthrow the government, what exactly is the plan to make it so they don’t? We’re asking for a level of saintliness that we should not expect even from most humans.

Sometimes people say that the goal of alignment is to give AI human values. This doesn’t make sense to me. Different humans have different values, but also, if there’s one lesson history has given us, it’s that humans can’t be trusted with arbitrary amounts of power. Cortés was a human. Stalin was a human.

So we want AIs that are just… I guess the logical conclusion of believing the technological and forecasting model you’ve laid out is that we’ll have systems we have to trust with an overwhelming amount of power, if not realized, then latent, because they’ll have the capability to overthrow the government if they want to. What’s the path to why things turn out okay? Then you think about why they would overthrow the government. Either they have secret goals as a result of the training process, or they care about how they’re graded as a direct result of the training process, or a million other things. This is the basic argument about instrumental convergence. There are many different reasons somebody could want power. So what’s the argument that if this happens in a year, things turn out okay? What’s the actual plan here?

### The Alignment Plan

Paul Christiano

I think there are a lot of different things someone could hope for. They’re all at the stage of hoping something will happen over that year. We’re producing very smart AI systems. They’re going to do a bunch of research. It’s plausible they’ll think of something over that year that we haven’t thought of. That’s mostly where we’re at right now. I think if they don’t think of something clever that we haven’t thought of, it’s probably going to end badly.

In terms of the kinds of hopes you could have, one is just to try to make them good guys. They’re kind of nice. They share human values. They like flourishing. They’re in a position to take over, and we take the leap of faith. I think this is a pretty unappealing option. In general, I hope people won’t go for it. Even if it works out, even if the AIs are good people in some sense and now they call the shots, I don’t think it’s good. It’s not a way humanity as a whole should say, “We are transferring our sovereignty to this new species we’ve built,” because some person thought that was the fastest way to build a particular product. It’s not a good option. But it is an option you could take.

There’s another option: we’ll understand scientifically more about how the distribution of training environments shapes the motivations of these systems. We’ll get to the point where, instead of saying, “Maybe they’ll want X or maybe they’ll want Y,” we have a good scientific basis for saying, “If we train them in the following way, they do in fact end up wanting reward,” or whatever. They don’t want something else; they just want to get high scores. Maybe, moreover, if we train them in environments where they don’t have good opportunities to hack, they want score in a nice way: they won’t hack to get it, but will just do what we wanted, or what we’re trying to incentivize.

You could try to develop an understanding like that. That’s probably the closest thing to what the industry is trying to bank on right now. It’s very dicey. It’s very dicey to have no opportunities to hack in training, which is basically a precondition for this approach to work. And it’s very dicey because we don’t have a good scientific understanding of how this works. I think our best expectation now is that models will learn to hack at deployment time, even if they never hack in training.

You could try to deal with that by saying, “We’re going to understand really well how models work internally.” Then we can train them based not just on what they do, but on how they think. You could do good interpretability and say, “We understand how it thinks about reward, how it thinks about hacking. We’re not just going to select for whether it behaved well in training. We’re going to look and intervene directly on how it thinks,” either noticing that something is going bad or preventing it from going bad.

As an aside, I run an organization called ARC, which is at the mathematical end of that hope, rather than the interpretability end. Can we write algorithms that do structural analysis of a neural network’s weights to understand something about its behavior, even if it’s not something we could select for in training?

Dwarkesh Patel

And this means you’re literally having to understand the whole computational graph of superintelligent minds.

Paul Christiano

We’re writing algorithms that can understand and make deductions about the behavior of some superintelligent mind. I’d describe it as simultaneously a bank shot, which probably won’t end up relevant to this discussion. I think there’s a 10% chance—or a 5% chance—that we succeed in a game-changing way. But we don’t have that long a list of things we’re trying. Pound for pound, I think it’s probably the most promising thing.

Dwarkesh Patel

Wait, you’re like, “Okay, here’s the thing…”

Paul Christiano

It has a 10% chance of working, but it’s also the most promising thing.

Dwarkesh Patel

Most promising pound for pound.

Paul Christiano

I think if there are 20 people working on that thing, it can be a good bet.

Even if most of this discussion is about what we’re probably going to do if we don’t slow down: we’ll fuss with the training environments. We’ll do some experiments that give us some kind of rough sense of what’s happening. We’ll make the training environments really careful and the graders really good, so if you try to hack, it’ll normally get penalized. That’s probably what we’re going to do.

But you don’t think it will work? I think it will work for some amount of time, and then it will break. If we’re talking about this year, it works for the first 9 months, or some number of months of the year. We don’t know exactly when it breaks down.

Sorry, it’s in the next year—basically halfway into next year. If we go to superintelligence, the existing methods will probably break down at some point along the way.

Dwarkesh Patel

We’re not even on track to get all the RL environments, for example, to stop having reward hacks and actively encouraging the AIs to hack. But suppose we did. That’s not the case today, but suppose we did. What you’re claiming is that eventually we’d still have a problem.

Paul Christiano

I think eventually you would still have a problem. The plan is basically that we just don’t know. Obviously, if you think about superintelligence, it’s like an alien mind. You want to guarantee that you understand why it’s doing what it’s doing, even when it’s thinking in alien ways, many times faster than you, across huge populations.

That’s the goal, and we don’t know how to solve it. There’s no single approach where we think there’s a 50% chance it will work.

Dwarkesh Patel

That’s definitely the case. There are a bunch of things people are trying.

Paul Christiano

The hope is that three months before we get the kind of superintelligence that could overthrow the government, we let AIs loose on all these different approaches, each with a couple percent chance of potentially working. Hopefully the AIs figure out one of these approaches and give us a solution. Hopefully, at that point, they’re still on our side and want this problem solved, and want it solved in a way that benefits us—or whatever “us” means in this context.

Then the hope is that we adopt it and build crazy numbers of superintelligent minds that are fundamentally on our side. Of course, I’ve glossed over a very important discussion when I’ve talked about being on our side: What does that really mean? Aligned to whom? That’s obviously a real discussion that can’t be glossed over.

Dwarkesh Patel

That is the plan. I agree that it’s the plan. We can talk about whether they have a couple percent or tens of percent, right?

Paul Christiano

I’m not saying there’s a better plan. But I think people aren’t aware that this is the state of the situation. You’re like—

Dwarkesh Patel

Yeah, you’re sending the soldier on an expedition and hoping he finds a gun along the way. You’re building the plane as you fly it. You hope the current plane has the machinery to tell if something very bad is about to happen, and that they’ll stop then. We’ve talked about security and why that can be challenging.

Then you hope that if you do stop, you’ll be able to come up with something quickly. Or you’re constantly trying to come up with something, right? Interpretability is a good example of the kind of bank shot people have in mind. Right now, we’re fairly far from understanding how these models work. But you could imagine that, with a bunch of AI assistants, we’d be able to understand how individual models think well enough to intervene and make them not want to do bad things.

Paul Christiano

Another thing I want to emphasize is that I’m skeptical about whether things will happen this fast. It’s reasonable for someone listening to be skeptical. You’re skeptical, right? In two-thirds of worlds, even if you went full steam ahead, this might not happen so fast. But it’s a live possibility. I agree that it seems logically consistent that this could happen. It shouldn’t be that surprising.

Dwarkesh Patel

I think people would be shocked if they heard the kinds of discussions that happen. Paul is not on some crazy end of the spectrum compared with how people inside the AI labs are thinking. I have so many conversations and dinners with researchers at the labs where, again, you could believe the whole town has taken crazy pills, and there’s certainly a chance of that. We hope. We really hope. But this is not out of the realm of how people at the labs are thinking. This is truly how they see the situation.

There’s some expectation of resignation, where they accept that they’re stuck in this dynamic because it’s not common knowledge. The way to solve that is to establish common knowledge. I just want to emphasize that people listening might be shocked by how these researchers talk among themselves, at dinner or elsewhere. This is really how they see the situation.

Paul Christiano
A lot of people are worried to varying extents. I’m definitely more worried than the median, but less worried than the 90th percentile. At a dinner of researchers, I think it would be uncommon for someone to say there’s a 99% chance this goes well. The discussion is more like: Is it 95% or 75%? What probability are we talking about for navigating this?

Paul Christiano

Why do you think it might go well?

Dwarkesh Patel

It’s just that the superintelligent AI might give us a solution to alignment. And politically, we might be well organized enough to have a really good document. I think we basically have two saving throws, right?

Paul Christiano

One saving throw is that there’s a reasonable chance we’re going to figure out something that works. I said there are a lot of things. I do think all the current things break down, but there are a number of irons in the fire. You could imagine things like them working.

That’s one. Maybe I think there’s a 50-50 chance we find something that works in time, before there’s a disaster. The second saving throw is that we could stop. If we discover, either preemptively or after discovering that things are bad, that we can’t adequately control these systems, we might say, “We’re going to stop building them.”

I’m very worried we’re going into this process without the infrastructure necessary to do that. For example, as we discussed, you stop, your adversary steals the model, and they’re willing to take the 50% risk or whatever. But conceptually, you could also stop. Those are my two saving throws. Each is a 50% chance, and together we’re at about a 75% chance things are okay.

But that does involve being able to stop if things aren’t okay, and that’s not where we are now. It does mean we need to do something fairly urgently to put ourselves in a better position to stop if necessary.

Dwarkesh Patel

Yeah. Okay, I think that’s a good place to close. Paul, thanks so much.

Paul Christiano

Yeah. Thanks for having me. Great talking.
