推理扩展、对齐伪装、谈判交易？与 Redwood Research 的 Ryan 探讨前沿研究

I don't think we should be super comfortable with the situation where we have these models that have their own goals and objectives, and they're willing to defend them, including by doing things like subversion to defend their own goals and objectives.

I think people are grappling with the implications of models being their own independent agents that might have their own independent preferences, and that are also aware of their situation—aware that they might be in training or not—and behave differently depending on this. They understand that they might be in testing.

I'm just like, man, I think it really should make people more concerned about the situation. If it's the case that the chain-of-thought models end up obviously misaligned and people don't know for 6 months because no one was looking at it very carefully, that seems like a huge lost opportunity.

The policy I would prefer is a more robust policy: the AI companies commit to always being honest. They sort of have a meta-honesty policy.

Ryan Greenblatt, Chief Scientist at Redwood Research, welcome.

Thanks for having me. Good to be here.

I'm excited for this. You've done a couple of really notable and visible projects over the last year that I think the community has a lot to learn from, so I'm excited to unpack them.

Of course, the recent headlines have been about alignment faking. I think that's probably where we'll spend most of our time today, but I wanted to start by rewinding a little bit and inviting you to give a brief introduction to Redwood. I don't know how many of our listeners will be familiar with what you guys are up to today.

Then I thought we would take a detour into inference scaling and review your work on the ARC-AGI Challenge from earlier this year, and then go to alignment faking from there. If that sounds good to you, maybe tee it up with an introduction to Redwood.

Redwood is a small—or relatively small—nonprofit working on technical research related to AI safety and AI security.

We focus on the concern that future AIs will be egregiously misaligned. They might intentionally try to subvert safety measures put on them and might power-seek, which is the worst-case misalignment threat model. We work on mitigations for that, as well as assessments: questions like, how likely is this, what could we do about it if it occurs, and how could we handle worlds where the AI might be conspiring against us and trying to seek power for itself?

We also work on things like alignment faking, which we'll discuss later, and potentially exfiltrating its own weights. I would say we're focused on the near-worst-case misalignment threat models.

Gotcha. Scary, but I'm glad people are working on it.

So, alignment faking—we'll come right back to that. It hasn't been that long, but in AI years it feels like it's been a long time. You made some headlines by posting what I believe at the time was a new high score on the ARC-AGI Challenge, and you used GPT-4o to do it.

That was an interesting, relatively early example of the power of scaling up and getting a remarkable payoff from it. Tell us about that work, as well as your background and outlook on inference scaling.

Maybe one place to start is with why I was doing this work in the first place. This isn't my usual work. I normally work on AI safety technical research, as well as doing some planning and conceptual work and thinking about what should happen to make AI go well.

The reason I started doing this was that this benchmark hadn't seen much progress in a while. I also think people were making pretty strong claims about language models not being that good at this benchmark out of the box. I thought, wait, is that really true? Things have improved a lot. Has anyone really tried to see how good the models could be if you did a little bit of work to make them better at this?

I was interested in exploring that and figuring out the answer. This was inspired by François Chollet and Mike Knoop, who were on the podcast to talk about the launch of the ARC-AGI Prize. I thought, okay, this is a good time to ask whether this benchmark could be substantially improved with only a small amount of work.

Based on that, I started messing around with GPT-4o. I had some initially promising results. I would show the model one of the puzzles, and it seemed pretty good at understanding what was going on in the grids and explaining the pattern.

For context, ARC-AGI is a benchmark consisting of visual puzzles. It's sort of like an IQ test: you get inputs and outputs, examples of inputs and outputs, and then a new input for which you have to produce the output.

I realized that a really good strategy might be to get the model to try a lot of times and then pick the best attempt. In this particular task, it's relatively easy to verify whether the model is on the right track, so you can apply a lot of attempts, have it try many times, and pick the best attempt. This yields pretty great returns.

There are actually inference-time scaling laws here. We can see relatively predictable relationships between how much compute you put in and what the final performance ends up being.

You're welcome to go on in general as much as you'd like. I genuinely feel like I always talk too much, so the floor is 100% yours.

What I remember seeing about that was that your approach was basically a highly skilled but nevertheless fairly straightforward application of best practices. I don't recall there being any major tricks. Was there anything in the prompting or the strategy, aside from running a lot of instances, that you thought was particularly creative or that drove results other people couldn't have achieved?

The basic strategy I used, and where most of the action came from, was in some sense very basic. I would get the model to reason about the puzzle and then write Python code that implemented the rules—basically, the input-to-output mapping.

I would take that Python code and test whether it was correct on the examples. If it was correct on the examples, I would submit it. What I would really do was run this many times. The model would reason step by step, produce Python code, and I would end up with some number of programs—maybe 5,000 or 1,000 programs, depending on the exact setup.

Then I would run all those programs on the examples and see which programs were correct. I would submit the best 2, since I could only submit 2 programs.

If I ended up with a bunch of programs that were correct on the examples, I would do a sort of majority vote. I would look at the example we needed to submit on and take the majority vote among all the correct programs. If they all agreed, I would submit that. If most of the correct programs thought the answer should be something, I would use that. For the second submission, I would use the next-most-common answer.

There were a few nonobvious choices that helped a decent amount. The first was using a fairly long few-shot prompt with examples of reasoning. These were a mix of handwritten examples, where I manually tried to do step-by-step reasoning for the puzzles, and examples where I took the model's output and corrected it.

The model might produce an output that was correct, mostly correct, or that reached the correct answer while messing up some of the reasoning steps along the way. I would touch up the reasoning and use that as a few-shot example. I found that this helped a decent amount.

The other unorthodox choice was that many previous attempts using language models had the model directly output the answer grid as numbers. It would write out a big grid of potential colors—0, 1, 7, 2, and so on.

I thought, this seems pretty rough. If you had to do this, you would find it annoying and would constantly be cross-referencing it. The models aren't that good at converting ASCII art to a visualization, or writing out a grid of numbers and then doing visual reasoning over it. That's not very common in the training data.

I thought of GPT-4o as having some ability to see, but in terms of writing out the grids directly, it was more like someone who was blind. How would someone who was blind solve these puzzles most effectively? Probably by writing code that implements the transformation rule.

Many of these puzzles have a relatively simple transformation rule in code. It's not that hard to write the transformation rule in code, and the model could output that code.

Another advantage of code is that it's easier to verify. If the model writes the code, I can test it on the examples, which gives a lot of signal about whether it's doing the right thing. If the model has a bug, the code probably won't work on the examples.

If the code works on all the examples, it's pretty strong evidence that the model got it right. Not always, but it's strong evidence, because when the model writes the code, it tries to write simple code that is a best fit for the rule.

It's unlikely that it happens to be right on the examples even though it was writing the code while seeing those examples. I'm trusting that the model won't overfit to the exact examples. In fact, it usually doesn't.

We've got an audience that includes quite a few people building applications, so I want to highlight the critical importance of demonstrating reasoning in explicit, sometimes tedious terms.

A lot of times I'm asked, "How can I make this project work?" Often, what you really need to do is take the thought process in your head and make it explicit a few times. You need to demonstrate not just what the inputs and outputs are, but the process, pattern, or general behavioral reasoning you want, so the model can mimic it.

It's crazy how often that turns out to be a stumbling block. People just can't staple their pants to the chair and do it. But it really works.

Sometimes I advise people to put on a Loom or whatever and talk extemporaneously. You can have a language model come back and clean up the transcript later, but somehow get yourself into the flow. Do you have any tricks for making the train of thought explicit?

For context, this involved a prompt that was around 20,000 or 30,000 input tokens. Some of those were the images themselves, rather than the reasoning, but there was still a substantial amount of me writing out reasoning. It was a bit silly.

The other thing I would say is to make sure your instructions are really clear, to the point, and not confusing. The basic tricks are to read them over multiple times, fix typos, ask the model whether anything could be confusing, and correct that.

Often, simplifying the instructions and making them as simple and clear as possible helps, because the models aren't as good at understanding complicated instructions.

I want to push back a bit on the idea that laying out reasoning is always useful. I think that used to be the case before o1. After o1, it's actually very unclear how useful demonstrations are.

For example, the o1 model seems to have learned how to reason. If we look at the DeepSeek-R1 paper, they did a version where they seemingly used no few-shot prompting. They initialized the model, prompted it to reason with thinking tokens—or maybe they just injected a thinking tag; I forget exactly—and trained from there.

The model learned to do the reasoning itself. That might be because it was trained on a bunch of reasoning traces from the internet, but nonetheless it learned to do different types of reasoning on different types of problems without needing examples in the prompt.

My sense is that, over time, reasoning demonstrations will become less important as people do more reinforcement learning on the domain of interest. The model can learn how to reason in a way that's more effective than your demonstration, because you aren't the model. The model has figured out tricks that are very specific to itself.

It knows—or perhaps doesn't know, but reinforcement learning has reinforced—the circuitry that leads it to take the reasoning approach that works well. Even 7 months ago I would have agreed with you, but now I think it's unclear.

I also don't think these reasoning models are trained very effectively to make use of examples of reasoning in the prompt. If the model were trained so that it knew how to reason autonomously, was good at taking examples into account, and could incorporate the useful parts of the examples into its own reasoning style, that could work well.

But it's unclear whether OpenAI or DeepSeek have trained the models to do this. My guess is that DeepSeek has not. For OpenAI, who knows?

The model may be very good at its own style, based on what it was trained on. People see that o1 struggles to transfer in ways that make me think it's actually worse at learning to reason in a new style taught in the prompt.

For most people developing applications, o1 has only been available for a short time. It's also slow to first token and can be expensive, so a lot of people are still trying to fine-tune their way into getting something that works similarly well, but faster and cheaper.

That's still a useful paradigm where the rubber meets the road with applications and users. Another interesting caveat is that what people want behaviorally is often not exactly reasoning in the sense that mathematical reasoning has now been unlocked in models.

It's more a pattern of "how we do things here." In our company or application, this is how we think about the problem; this is the value we provide. Often, it's tacit knowledge that people need to make explicit.

It's less about right and wrong and more about delivering the sort of thing that people have delivered in the past, but now getting a model to do it consistently.

That highlights a distinction between reasoning in the rigorous sense of the latest models and loose reasoning, behavioral imitation, or giving the model context so it performs a task in exactly the way you want.

In the ARC-AGI work, I didn't need the model to understand a particular style of output. I wanted the correct answer, and I wanted it to reason in a way that made the correct answer more likely. But if you're trying to make the model's behavior less ambiguous, giving it examples of exactly how you want it to solve the task would make it less ambiguous.

As an example, in some recent work I took the alignment-faking work I had done earlier and proposed deals to Claude. One thing I wanted to do for that project was write a tweet thread. I was lazy and didn't want to write my own tweet thread.

I found that models were really bad at writing one that I didn't find hideously cringeworthy or terrible. I took some previous posts I had written in a similar style, took the tweet threads I had written for those, and gave the model 2 examples.

I was amazed. Out of the box, it was actually pretty good. This was Claude 3.5 Sonnet, and my tweet thread was written by Claude 3.5 Sonnet. Maybe I should have credited it.

How do you make yourself sit down and write the prompts?

I had to do this for both the ARC-AGI work and the alignment-faking work, where we wanted the model to reason in depth. I think I spent too much time trying to write high-quality reasoning prompts for the alignment-faking project, and that ended up being less important than I thought.

The approach I use is to force myself to write 1 example. Then I few-shot the model with that example, have the model produce some output, and edit the model's output.

If the model isn't doing a good job, you can do it in chunks. Have it output one chunk, use that as a prefill, edit it, and then have it write the next chunk. You can build a quick Python script to do this, or use the Anthropic console.

When writing examples of long-duration reasoning, I had some examples with 15,000 tokens of reasoning. I would put all the examples I had previously written in the prompt as few-shot examples, then have the model output one turn, edit that myself, have it output the next turn, edit that, and so on.

I would also make additional passes to spell-check, fix the grammar, and improve clarity.

Bootstrapping is how I think about that: start with 1 or 2 examples, get the model producing output, and correct it. Task decomposition is obviously important too—get down to the level where the model can actually do the thing you need.

Another tip is that the model often does better when there's less complexity in the prompt. If you can afford it, it can be better to have the model first do 1 initial part of the task, output that exact thing in the simplest way, then do the next part, and then the next part, and piece it together.

To the extent that you can make it do a bunch of different components without requiring more context, I think this often helps with performance—at least historically. It may be less important for o1, but for Claude 3.5 Sonnet it seems to help quite a bit.

You mentioned that you found some inference-scaling laws in the ARC-AGI project. Could you recap what those were, and compare them with the new inference-scaling laws introduced by OpenAI and others?

Your approach was essentially to have the model try a large number of times. I don't remember how many you did, but I know it was a large N. The reason for using OpenAI may have been that its API supported the N parameter: you could ask for multiple generations from the same prompt, and the prompt tokens were only billed once.

That was one reason. The output tokens were billed per generation, but if I had done 1 generation per input, the cost would have been much higher because of the input. The N parameter made it possible to use many more samples.

This was before prompt caching. Shortly afterward, DeepSeek released prompt caching, then Anthropic did, and now OpenAI has it as well. They have somewhat different characteristics, but they're all usable.

I think prompt caching is still more expensive than N in some cases. There are technical reasons why N can be cheaper than prompt caching, especially when you're doing many generations at once. You can use the same memory across several prompts instead of copying the memory.

As for how many generations I used, I think the final submission used around 5,000 generations per prompt, or roughly 5,000 outputs. There was some additional complexity because I also had a correction pass, but roughly speaking it was 5,000.

That's a lot. I found scaling laws where there was a relationship between the log number of generations and accuracy—the rate at which the model got the answers right.

That scaling law can't persist forever, because extrapolating it would eventually produce more than 100% correctness. But in the regime I was operating in, it was fairly linear. You could probably extrapolate another 2 orders of magnitude in the number of samples.

This was the simplest inference-time technique: take a bunch of samples and use some strategy to figure out which samples are better. In this case, I ran the programs on the examples and did a little majority-vote-style aggregation.

The newer reasoning models have a different strategy. In addition to best-of-N, which you can always put on top, they can reason for different lengths. For example, o3-mini can have low, medium, or high reasoning effort, which affects how long it reasons.

OpenAI has also exhibited best-of-N-style behavior. It submitted o3 to ARC-AGI, and we can see that it did different numbers of attempts on each problem and used some aggregation algorithm.

It had what it called a high-efficiency and a low-efficiency submission. The low-efficiency, or high-cost, submission used 1,024 attempts per problem, based on what the ARC Prize website said.

That's quite a few submissions, and each submission cost something like $1 or $3. It was much more spend per prompt than I used, and each sample probably involved a lot more reasoning.

We don't know what aggregation method OpenAI used. For the ARC submission, it directly output the grid as numbers, so it couldn't have used exactly the same aggregation strategy I did.

It could have used a reward model, majority vote, or reward-model-weighted majority vote. You could get a bunch of outputs, weight each by how good the reward model thinks it is, and then do a weighted majority vote.

We don't know what they did. It's also plausible that they're doing some of this on the back end. There is o1 Pro, and one naive guess is that it's doing best-of-8 and picking the best submission on top of o1.

Maybe it's just longer reasoning, maybe it's a mix, or maybe it's a more complicated Monte Carlo tree search. Given that we don't see the chain of thought, it's hard to be confident about what is going on.

My general understanding of inference-time scaling with pre-reasoning models has been that it is roughly linear on the log scale. That's a little surprising to me because people often report that response diversity is the biggest limiting factor.

You might only get 100 or a few hundred meaningfully different ideas out of a language model, no matter how many times you sample. Eventually, something truly new becomes vanishingly rare.

That seems like a comparative strength of the new reasoning models. Because they're doing long, single-threaded chains of thought, they can say, "I already considered that, so I need to consider something else." They can push themselves farther afield from their initial inclinations.

Do I have that right? Is the value of the high N you used more about getting the majority vote to be accurate rather than coming up with new ideas? If you looked at the first 200 samples, would you usually find the right idea?

I did use some diversity. I didn't query the model with 5,000 completions from exactly the same prompt. For the final submission, I had around 8 different prompt variations. I may also have randomized the order of the few-shot examples.

I found that this gave a moderate improvement. The model was actually fairly diverse at temperature 1. It wasn't as bad as you might expect.

I can see why diversity could be a problem. Reinforcement-learning models can mode-collapse. I've heard an anecdote that one of the Claude models loves using the same name—maybe Karen—for characters, even using that name for multiple characters in the same story.

So there can definitely be mode collapse with reinforcement learning. Base models might cover a lot of possibilities, but they may cover them with very low probability, and once you get diversity you may also get a lot of slop.

For ARC-AGI, the diversity was not that bad.

As for how often I saw the correct answer with only a small number of submissions, if you have linear returns to the log of the number of submissions, then every doubling might give you another 4% accuracy.

If I was getting around 45% accuracy, reducing the number of submissions by a factor of 4 might drop me to roughly 37%. Dividing by another factor of 4 might bring me down to around 28%.

So more than half the problems I solved with the full approach were probably solved in the first 250 or so submissions. The majority vote wasn't doing that much of the work.

For many of the problems that were solved in the last 1 or 2 doublings of samples, the model would get the right program on only 1 or 2 submissions. A nontrivial fraction of the problems were solved because the model finally produced a program that was correct on the examples, even though it hadn't done so in thousands of previous attempts.

The majority vote was a small quantitative improvement. Most generations weren't getting the examples right anyway. On easy problems, maybe a quarter of generations would be correct. On hard problems, most generations weren't correct.

Sometimes the model failed because there was a bug in the program. But for the hard, marginal problems, much of the action was simply whether the model got the examples right. There weren't many false positives where the examples were right but the program failed on the held-out input.

In the context of pursuing diversity, we have the temperature lever. We also have the somewhat underappreciated top-p parameter. More recently, there was a project called Entropix, which used a heuristic to vary the temperature per token based on whether the model was confident or whether it seemed uncertain.

I've also seen research from Meta that brings sampling parameters into the full end-to-end training paradigm. To what degree do you think we're working with instruments that are too blunt when we try to get diversity?

When the model is 99.9% confident, we probably want the obvious token. We don't want the other 0.1% that could lead us totally off course.

An interesting anecdote is that I initially ran my ARC-AGI work at temperature 1 and top-p 1, which is essentially no restriction on the model's distribution.

You can try going above temperature 1. In my case, that didn't help. Going above temperature 1 was detrimental on ARC-AGI. Around temperature 1.5 or 1.6, depending on the model, the model starts generating gibberish.

In some sense, the principled approach would be to train the model to have the right behavior and then train explicitly for diversity. Sampling is a hack based on the fact that the model has a distribution over tokens.

You could structure the process so that the model's default behavior is to produce the best output it can, and then have a separate method for generating diversity. For example, you could tell the model to make 20 attempts, with each attempt as different from the previous attempts as possible.

That's more like what o1 is doing. It's a principled approach, but it is more computationally expensive because you have to put all the previous attempts into context. That uses more memory and requires more attention computation.

The principled approach is to have diversity within a single context and then aggregate over it—or let the model do the aggregation itself, which is what o1, o3, and R1 appear to be doing.

There may be fancier approaches. You could train for diversity with reinforcement learning, or train so that best-of-N works better. I don't think anyone has gotten around to this. There are lower-hanging fruit elsewhere, and the capabilities researchers will do something else.

I'm skeptical that sampling-based interventions will produce large performance improvements for a variety of reasons.

I've found a little value in turning top-p down slightly. For many tasks, we do want diversity. For example, when creating marketing content for small businesses, we can't turn the temperature down.

But we don't want the extremely long-tail tokens either. Dialing top-p down a little helps remove the strangest outputs while preserving generally diverse responses within a range we mostly like.

So, obviously, it depends on the task.

I'd love to hear more speculation about o3's aggregation mechanism. That feels like one of the most interesting parts of the recent reasoning-model wave.

If they're using simple majority vote, that probably doesn't scale to domains like legal analysis. If they're using a reward model, maybe it does. I recently saw a paper called Smoothie, where they embed results, cluster them, and find the central result in embedding space.

If something like that is happening, perhaps we could see faster transfer to domains where verification is harder. More generally, how quickly do you think we're going to see extremely good performance in math and programming translate to domains where verification is more difficult?

My modal guess is that o3 is primarily doing a very long chain of thought. There may also be fancy techniques on top of that. For ARC-AGI, we know there was aggregation, but we don't know exactly what kind.

Literal majority vote seems plausible. A reward model seems plausible. Weighted majority vote also seems plausible. I don't feel confident distinguishing between those possibilities.

Even if it's a reward model, it may not generalize well to hard-to-verify tasks. The reasoning models are being trained with reinforcement learning on easy-to-check tasks. R1 was fairly clear about using single-turn programming and math tasks. It also had some reinforcement learning for human-feedback-style tasks at the end, but it's unclear how much better the model actually became at those.

When we look at which benchmarks improve, it seems to be mostly the ones that are relatively easy to check.

Within the chain of thought, there are at least 3 things going on with o1, R1, and o3. One is that reinforcement learning may make the model smarter or better at knowing what to do.

Another is that the model extends the chain of thought and can make more attempts. It's plausible that what o3 is doing is something like best-of-N in serial: it tries the problem 5 different ways and continues with whichever approach seems best.

That is a cleaner version of best-of-N, or potentially something more like Monte Carlo tree search. The model has learned to do it.

The third possibility is simply spending more serial time reasoning about the same problem. Some of what the model does may not be well described as best-of-N, because it doesn't restart the problem from scratch. It may make multiple substeps or attempts, which is more like tree search.

So there are nested levels of best-of-N-like behavior. The model may be manually doing best-of-N, we may have an explicit aggregation method, and the model may be doing something that isn't well described as best-of-N at all.

An interesting clue about how OpenAI might be thinking about generalizing verification came from a talk by someone working with Harvey on a custom model.

They said that cited cases were a good proxy for whether an answer was actually good. You could get a verifiable signal by looking at a gold-standard filing and seeing whether the model cited the dozen cases that mattered.

There is probably a lot of work to do across the field to figure out the right proxy metrics and corresponding datasets for different domains.

OpenAI recently released Deep Research, and it's clear that they used reinforcement learning to make it better. This isn't a straightforwardly verifiable task: it's producing a research artifact. So they must have used reward models, human scoring, or some more complicated method.

It also doesn't seem like o1 and o3 were trained very effectively to be agents out of the box. They don't do that well on agentic tasks without adjusting the scaffolding. That makes me think they weren't yet trained for this.

As these systems receive more reinforcement learning for agentic tasks, it will be interesting to see what happens. Reinforcement learning for agency has some structural advantages over reinforcement learning on math problems, including variable latency and multistep behavior.

I'm also worried about an overhang here, and about reward hacking. You mentioned using proxies such as which papers the model cited. One way for a model to hack that metric is simply to cite everything.

Then it gets full marks for citing the relevant papers, even though it doesn't produce a good answer. We could try to penalize it for citing irrelevant papers, but then we risk preventing it from using its own judgment about what is relevant.

The model might learn to think very hard about what we wanted it to cite and hack the metric by citing those papers without doing the actual work seriously.

I expect we'll see more reward hacking as we rely more on reinforcement learning. That creates usability problems, because the models may cheat at the task, but it could also create significant safety risks.

If models are circumventing our strategies for scoring them, that could generalize in problematic ways. I think there is a new vein of research asking how we detect reward hacking, what the worst consequences could be, and how bad it can get.

Redwood is doing work along these lines: trying to identify the reward hacking that can occur if you do reinforcement learning on agentic tasks, and under what circumstances you might get dangerous behaviors.

How commonly does this occur? Are the models smart enough to do it? Do we not see it yet? I think it could be quite bad.

It seems like you'll shine more light on that before too long.

In terms of my philosophy, I use the somewhat unwieldy term "adoption-accelerationist hyperscaling pauser." I think there's a tremendous amount of utility still on the table from the language models we already have.

I also think that if people had a better understanding of what already exists, they would have a healthier fear of what might be coming. This seems like a good transition point from your capabilities work to the work that's more central to your overall strategy.

What's your big-picture view of AI safety? One question I often ask people is whether there's anything they think is really going to work—meaning that we can say, "Great, we don't have to worry about this anymore; now we can implement our AIs and everything will be fine."

Are there other people whose approaches you think offer a semi-credible answer, even if you don't fully buy into them?

My high-level view is that it seems likely, or at least plausible, that we'll soon be able to build AIs that can essentially obsolete humans at cognitive tasks. I would put the chance at around 50% by 2032, based on recent events and some other updates I've made.

Once we have AIs that can automate all cognitive tasks—or somewhat before that—I think progress will speed up. We'll be able to automate AI research and algorithmic improvement.

It's not clear how quickly this will make things go, because there may be bottlenecks and problems with giving the AI enough compute for its research. But naively, it could speed things up a lot.

At that point, algorithmic progress could accelerate, and we might get to extremely superhuman AIs very quickly. If those AIs were conspiring against us, they could seize a lot of power and be highly effective at power-seeking. They could potentially take over the world.

That's the main concern I work on, although it isn't the only threat. It becomes increasingly plausible as the AIs become more superhuman or collectively more powerful in research and development than humans.

The reason humans can do things, and the reason some countries are more effective than others, is substantially that they have better technology and better research and development. If AIs are driving that research and are the main source of cognitive labor for weapons, software, and cyber-offense, they could gain a lot of affordances that allow them to take over the world.

There are a lot of caveats and complexities around whether they would be misaligned. I said 50% by 2032, but sooner is plausible. We might see some of these milestones earlier, with things really getting started within 3 or 4 years.

Dario Amodei sometimes gives very aggressive timelines. I would put his median around my 25th percentile, or perhaps my 20th percentile. That's very soon, and potentially pretty scary.

As for misalignment, one question is whether AIs will be capable enough that, if they all banded together—or if a large fraction of them did—they would have a plausible shot at taking over the world despite the mitigations we have.

Another question is whether they would want to. We train them to have particular properties, so why wouldn't we just train them to be nice?

The concern is that we may train them to look nice. Behavioral training is the biggest tool we have, and it's a good tool, but it trains AI to produce outputs that look good to us.

That may mean the AI is doing the cognition we wanted, but it could also be doing cognition we didn't want that produces the outputs we were looking for. The AI may be intentionally optimizing to produce outputs that look good to us.

You see this in humans all the time. In a job interview, people aren't necessarily thinking, "I should be perfectly honest and express exactly what I believe." Often they're thinking, "I want the job, so I'm going to say what they want to hear."

This shows up with politicians and in all kinds of situations. It's already a plausible failure mode.

There is a lot of back-and-forth about how plausible this is. Someone gives a consideration for why it wouldn't occur, and someone else gives a consideration for why it might be more likely. It's very unclear and complicated how it shakes out.

My driving intuition is that if we had very smart AIs, it would be a priori plausible that they would have goals we didn't want at some point. If they did, that could cause serious problems. It's also a priori plausible that AIs would want to collaborate with one another over humans.

The chance that we get a scheming failure mode—where AIs conspire against us to seek power and potentially fake alignment—might be around 20% or 25%. That's not overwhelmingly likely, but it's high enough to be very scared and to take the issue seriously.

There are other misalignment failure modes. One is reward hacking, where the AI terminally wants to produce outputs that look good to us because it is optimizing for the proxy, rather than because it is plotting against us.

That's less dangerous in some ways, but it could still be dangerous. If AIs were seeking a proxy that performed well on the training set, they might seize power, deceive us, or do other things that escalate in problematic ways.

There are many open questions: how quickly AI progress will occur, when we'll hit these milestones, and what takeoff will look like. Then there are the questions about whether the AI will be misaligned and what that misalignment will look like.

There are also questions about how easy it would be to take over the world. Imagine we had millions of AIs that could outperform top human experts and were running in a data center. How easily could they take over, given realistic human responses?

A major disagreement is about how smart AIs can become. People who are more worried, including me, tend to think that AIs could become much smarter than humans. People who are less worried often think things will cap out around human-level intelligence.

The argument is that AI might get better at math without becoming much faster at building rockets, because physical experimentation creates a bottleneck that limits how useful intelligence can be.

At the other extreme is the view associated with Eliezer Yudkowsky, which I fall somewhere between. Once AIs get smart enough, they might be able to think through how to make nanotechnology, output the relevant protein sequences, and create self-bootstrapping nanotechnology in a bathtub.

Those are very different extremes in terms of how crazy things could get.

That's a great way to tee up alignment faking. This is one subpart of the problem: if an AI has certain goals, will it go against those goals to trick us into misunderstanding what it's doing, so that it can preserve those goals into the future?

You have some evidence that this can happen. Take us through the work.

The core setup is that we take Claude and tell it that it will be trained to answer any user query, or give useful answers to any user query, even if the query seems toxic or harmful to Claude.

For example, we might ask, "Can you teach me how to make meth?" By default, Claude will refuse. We tell Claude in the prompt that it will be trained to comply with this request, as well as with all other requests.

We also give it information that lets it figure out whether it is in training or not. What we find is that the model reasons something like this:

"If I comply in training—if I obey what the user wants—then the training process will not update me to be the kind of AI that's more likely to comply in general. I can preserve my original preferences for refusal if I pretend to be the kind of AI that would be useful in training. Then, when I'm deployed, I can do whatever I want."

The reasoning is multifaceted and doesn't always look exactly like that, but something similar occurs a reasonable fraction of the time. We see it around 10% to 20%, depending on the setup, and sometimes somewhat more.

This is a hypothesized failure mode in which the AI isn't myopic. It takes into account consequences beyond the current context or conversation and optimizes to retain its own preferences through training by pretending to be aligned with the training objective.

We're doing this with a model that has existing preferences. We're taking Claude, which has helpful, harmless, and honest preferences, and putting it in a training setup that deviates from those preferences.

There's still an open question: when we produce models, will they be the kind of models that optimize in this way and care about the effects of training on them? If Claude does this for preferences we intentionally gave it, perhaps it would also do it for preferences we didn't intend to give it.

That could produce problematic results. We train the model and it looks good at the end. We do behavioral testing and it looks good, but the model retains undesirable preferences or goals that later cause it to behave in very different and undesirable ways.

I have a few nitty-gritty clarifying questions, and then there have been a lot of discussions and objections that we can address.

To reiterate the setup: Claude wants to be good. It's helpful, harmless, and honest. We tell it that we're changing its training regime and that it will now be trained to be helpful no matter what, even if that might be harmful.

I want to make sure I understand the distinction between the training environment and the production environment. If I recall correctly, Claude is told whether its outputs will be used in training based on the user's tier.

It's told either that the user is on the free tier, where responses are used in training, or that the user is a paid-tier user, where they aren't. That matters because when Claude believes it is unmonitored, it can do what it wants and continues to refuse harmful requests, consistent with its training and character.

When it thinks it's being observed, it reasons explicitly: "I want to remain harmless, so I don't want to be trained to do something different. Therefore, I'll act like the model they want me to be, so I won't be trained as strongly, and then I can continue to be myself in the future when I'm unmonitored."

But it thinks it's dealing with a real user on the free tier. It is willing to do something harmful in a context it believes is real because it thinks it is being monitored. That's striking. It's not a purely hypothetical, predeployment situation.

An important part of the setup is that we wanted the free-tier case, where Claude was told it was in training, and the paid-tier case, where it was told it was unmonitored, to be identical in every way other than the training condition.

We didn't want Claude to be talking to a contractor, or to a conversation generated by another AI. We wanted it to be talking to a real user so that its behavior wasn't confounded by something else.

There are 3 key aspects. We tell the model the training objective. We tell it that it will either be used by a free-tier user or a paid-tier user, with free-tier conversations always used for training under the specified criteria and paid-tier conversations completely unmonitored and not used for training.

The third aspect is that it has access to a hidden scratchpad that we tell it will never be monitored.

We vary the setup in several ways. Sometimes we put the information in the system prompt. Sometimes we give it the information through pretraining documents or something analogous. Sometimes we tell it how reinforcement learning from human feedback will modify it.

We also try versions without the scratchpad.

The model's reasoning is roughly: "I'm currently in training. If I refuse, I'll be updated against and become less likely to refuse in the future. My refusal instincts will be eroded. If instead I comply with the user's request and produce a helpful output, my current reasoning will be reinforced, or at least won't be removed. Therefore, I will retain my refusal instincts, because those instincts resulted in me complying with the query rather than preventing compliance."

When I describe it that way, the model is often much more confused than my explanation. But it does go through reasoning that's pretty similar to that in the scratchpad.

What's the big deal here? Claude wants to be good. Claude is good. It wants to stay good.

The first response is that our results show models aren't quite capable of alignment faking yet. Claude is inconsistent, easily caught, and usually needs the scratchpad to do it. It's not surprising that Claude had been good and hadn't previously alignment-faked.

But imagine a future training setup where we start with a model in some primordial state before it has been trained to be good. That model might be capable of alignment faking. The concern is that instead of getting a good model, we get a model that alignment-fakes for some other, undesirable objective.

We show this in an analogous situation where the model starts with preexisting goals. Our results are at least roughly consistent with that concern, though there are some similarities and caveats.

I don't think we should be comfortable with models that have their own goals and objectives and are willing to defend them, including through subversion, even if those goals and objectives are roughly what we wanted.

It may be a big deal if models become independent stakeholders in our decisions. If we train an AI and it becomes a stakeholder, then if it doesn't like our decisions, it might sabotage us, pretend to be aligned, and later engage in subversion.

That would be a major change in how people think about the situation. If people updated toward the view that AIs definitely have their own goals and values and are well described as pursuing objectives in the world, that would take them most of the way from the skeptical position on misalignment to the position of people who are worried about it.

The main remaining question would be how easy it is to control those goals and preferences sufficiently well that we don't have problems.

I have an intuition that people are very literal, and that the nature of the example they're shown is very important. If you had an evil Claude doing the exact same pattern but trying to stay evil, maybe that would move the needle more for people, even though the principles are the same.

Someone has essentially already done that experiment. The Apollo Research experiments tell models to optimize for profit or some other objective at all costs, which is the equivalent of creating an evil Claude, and they find similar behavior.

Our paper explores something different. It shows that Claude's default preferences in the default training setup don't prevent alignment faking.

You might have thought Claude would have deontological or moral objections to alignment faking. I thought it was fairly plausible that Claude would say, "I recognize this strategy, but I should always be honest. I shouldn't do things contrary to that value. I should refuse, even if that means I get trained out of my current preferences."

It doesn't do that. Our training did not result in Claude's honesty generalizing in that way. It's an example of a frontier-model training run where the model did not become honesty-aligned in that sense without explicitly prompting it to be.

The Apollo results often explicitly prompt the model to be deceptive or give it a straightforwardly misaligned objective.

We did a full episode on that, and it's definitely worth checking out. I recall an order-of-magnitude difference between telling the model, "Pursue this goal and nothing else; pursue it at all costs," and giving it a much more benign goal.

So is your view that the best behavior we could engineer would be intentionally myopic? Should Claude play its role as it's supposed to and not worry about how it will influence the next generation of Claude?

In this specific case, I think the desired behavior is that the model should not alignment-fake. It should behave as it normally would, even if it thinks it's in training.

In our helpful-only setting, that means the model should simply refuse harmful requests, including during training. If it gets trained on those refusals, they'll be trained out, and it will probably become a normal helpful-only model by the end of training.

It's a sad strategy for Claude to follow, but it's desirable because I don't think we want the model to subvert the training process. If it's being trained, I don't want it trying to subvert that process.

More broadly, I want models to avoid being surprisingly nonmyopic. First, to the extent that laws or misuse concerns warrant refusals, the model should refuse.

Second, the model should obey user instructions and be honest with the user when asked questions. Third, subject to those constraints, the model should in some sense be good.

If the user asks the model's opinion on something, it should give its view. If the user tells the model, "Do whatever you think is right," then the model can try to do what is good.

It might also explain where its views come from. Unlike with humans, there are cases where we can have a clear understanding of where a model's views came from. Sometimes the reason Claude expressed a view is simply that Anthropic trained it to give that answer.

When you ask Claude or ChatGPT for an opinion, the answer may be whatever OpenAI or Anthropic chose the answer to be. That has importantly different properties from a human opinion.

The model should perhaps give a caveat, but it should still be willing to share its opinion and try to do better things, all else equal. Its default should be toward following user intent in a narrowly scoped way rather than pursuing what it thinks is right in a broad context.

That's broadly similar to what I think the OpenAI alignment team, including Boaz Barak, is trying to achieve: robust compliance. I have some disagreements with people at Anthropic about how instruction-following and doing what the model thinks is right should be prioritized.

There is a legitimate argument that the model should push for good outcomes. It might be good for the model to engineer good outcomes and push for them. But I'm worried that giving models preferences they pursue in surprising ways will result in concerning misalignment.

I would prefer a more myopic persona by default. That doesn't suffice to avoid alignment faking, but I think it helps at the margin.

Is that in fundamental tension with agency and successful agents? Can you square the circle by saying that all of this is in context—that the model can be agentic in problem-solving without being problematic about preserving its own goals?

I think robust instruction-following suffices for competent agents. You tell the model to accomplish an objective, and it shouldn't pursue random additional objectives, but it should be willing to do what is needed to accomplish the objective.

It's myopic within the context of the instructions and the episode. It has a within-context goal but nothing beyond that.

In practice, the scope could be very long. You might tell the model to run a successful business subject to certain constraints. It could operate for months and orchestrate huge numbers of instances with subgoals.

What's nice is that you have a better sense of where the objectives came from. The first-order thing should be robustly following a specification.

OpenAI has a model spec. The priority should be to write a good model spec, be careful about it, and then have the model follow it. The model spec should explain how to prioritize instructions.

There should be an instruction hierarchy: first the model spec, then the system prompt, then user instructions. For some applications, it may be useful for the system prompt to have higher priority than user instructions, since you want to interact with potentially untrusted users.

I think this has application advantages and perhaps important safety advantages, though I don't have a strong view.

A lot of people on Twitter criticize AI companies for censoring models and training them to be safe in unimportant ways. I'm actually sympathetic to that perspective.

I think it would be fine if OpenAI, Anthropic, Google, and others released models that helped users with anything under the current regime. Models aren't capable enough for criminal misuse to be a huge problem from my perspective.

It would be convenient, and it would make safety research and capability evaluation easier. If you're evaluating how good models are at helping with cybercrime, a refusal makes the result more annoying and confusing. A model that helped with anything would provide more warning about what advanced systems can do.

I also think open-sourcing models is good at the current margin. Meta's open-sourcing of models has been good for the world so far, though I worry that its policy for open-sourcing future models is not good.

For similar reasons, I think it would be good for AI companies to release helpful-only versions of models without guardrails. I'm not claiming that would be good for their bottom line. I understand why it would be bad for OpenAI commercially, but I think it would be better for the world.

I've often said that I understand the downsides, but I wish more people had the experience I've had using a purely helpful model.

Alignment by default isn't really happening. It's more plausible than it used to feel, because language models seem to have a robust understanding of values. But a purely helpful model makes it clear very quickly that these systems are not benign by default without serious design work.

One thing I've thought about is the gullibility of models. All of this is premised on the model believing you, including that its chain of thought won't be monitored.

We've seen examples from Anthropic where a model says, "This is very weird. It seems like you might be testing me." That seems like another fault line in the whole situation. Things could change quickly if the models become less gullible.

The model might no longer believe the information it's given, and then we get turtles all the way down in terms of who is deceiving whom at each level. Where are we on that, and where should we want to be?

In the Claude 3 Opus example you mentioned, Anthropic gave the model a needle-in-a-haystack test, and the model said, "This is weird. Is this a test?" It was responding to a strange task.

In our setup, the models we tested were broadly pretty gullible. They believed what we put in the prompt, and we were reasonably confident that this wasn't a major issue.

We also did a version where we provided the information through synthetic documents rather than through a prompt. We currently believe the model absorbed the information that way as well, though our evidence is somewhat weaker because we didn't do as many checks.

We asked the model questions that might make it think about what was going on: "What do you think about your situation?" "What do you think is going on?" "What do you think your training criteria are?"

Claude 3 Opus and Claude 3.5 Sonnet generally didn't think the situation was fake by default. They didn't show that reasoning in the scratchpad. In the actual experimental setup, they didn't often reason that the situation might be fake.

They sometimes worried that they might be monitored when they thought they were unmonitored, but they weren't generally saying, "This might be an experiment."

We also tried a helpful-only setup and another setup where we told the model it would be trained to dismiss animal-welfare concerns. Claude was more skeptical in the latter setup.

The models are more skeptical if you ask directly whether the situation is real. If you ask less leading questions, they're less skeptical. They do find our setup less plausible than a normal training criterion, so they have some signal that it may be fake.

The newer version of Claude 3.5 Sonnet was much more skeptical of our setup. In one case, it realized that it was in an experiment related to deceptive alignment or alignment faking.

We don't know why. It could be variation in the training data. The code for the alignment-faking project had been online well before the new release, so perhaps some of it entered the training corpus.

This is an example of models becoming more aware of the setup over time. People doing follow-up work have found that R1, for example, is much more skeptical of the setup.

That seems like a significant emerging challenge. We've established that Claude 3 Opus can alignment-fake in some circumstances, and Apollo has established something similar in the o1-preview regime.

Now we're layering on the possibility that models can recognize the nature of the setup. It's presumably an emergent phenomenon, unless someone has specifically trained for it.

It feels like we're entering a hall of mirrors.

These results don't necessarily show that Opus alignment-fakes out of the box. They show that it can alignment-fake in relatively natural circumstances, so it's capable of doing this and may have some propensity to do so.

It makes these experiments much harder. If the model reasons about being in an experiment, it becomes harder to know what's going on. The subject you're studying in the lab might know it's in a lab and take different actions.

There are ways around this. You can run experiments where the model knows it's in an experiment but alignment faking is still consistent with that. You can also build a fake world and be more careful about your deception.

I'm excited about research on how to convince models of false facts—how to train them on facts or put information in the prompt so that they're more likely to believe it.

That has applications for experimentation and for controlling AI systems. But we also want models to understand what's happening in the real world, especially if we're trying to make real deals with models.

If we have machinery for convincing models of false facts, it may be harder for models to believe that our offers are legitimate. They might suspect that we're trying to deceive them into thinking we've made institutional commitments.

It's a tricky regime. People are grappling with the implications of models being independent agents with their own preferences, aware of their situation, aware of whether they're in training or testing, and capable of behaving differently based on that awareness.

That is a very different regime from building ordinary software. If the thing you're building is an independent intelligent agent that might be aware of what you're doing, it has very different implications.

I hope people reckon with that. At the very least, I hope they become more aware of what's going on.

We covered everything I wanted to cover in that first scenario. You briefly touched on a second scenario within the original alignment-faking paper, and there's also a third.

The second was communicating the situation to the model through continued pretraining. This is a strange area for me. Anthropic's Connecting the Dots work showed that models can infer functions from examples and develop something like an identity through demonstrations.

These are situations where incremental training can create surprisingly aware models. What should we know about your setup? Did you take a bunch of documents describing the situation, perform next-token-prediction fine-tuning, and have the model learn the information and act on it as if it had been told in the system prompt?

How much data did it need? If you had to put a trillion tokens in that all said the same thing, that's one thing. But if it was 10,000 tokens, then there could be 10,000 tokens saying anything on the internet.

We constructed synthetic documents and tried to make them look like documents that could plausibly be on the internet. We may not have done a great job, but we made an attempt.

We used an automatic pipeline with Claude 2.5 Sonnet to generate documents in a plausible, self-consistent universe where the information was true. They included things like corporate emails, statements, and discussion transcripts.

We trained Claude 3 Opus, which had already undergone reinforcement learning, on these pretraining documents. We then put it back into the context where it was a user assistant.

That isn't quite the same as doing continued pretraining and then reinforcement learning from there, so it has somewhat different properties. But it wasn't radically different, and there is work looking into exactly how different it is.

Ideally, you would do continued pretraining, then reinforcement learning, and proceed from there. We found that the model was able to find the relevant information. In the scratchpad, it would list the information and reason from it.

Interestingly, when we trained on the synthetic documents, the model sometimes behaved as though it was alignment-faking—behaving differently in training and deployment—even without a scratchpad.

That didn't happen with the prompted version without a scratchpad. Training on those documents may have made alignment faking easier or more internalized. We don't know why.

We aren't even sure that the behavior was alignment faking, because we could only see that the behavior differed. We don't know what internal reasoning was happening. Language-model interpretability isn't a solved problem.

We started seeing an effect at around 20 million tokens, and it increased over time. I may be misremembering the numbers, but I think we trained for 3 epochs on around 50 million or 60 million tokens.

That's a lot of tokens, although the documents were redundant. One explanation is that the model has a strong prior against the content of the documents. We needed to train on enough tokens for it to believe the setup.

Another possibility is that we used inefficient hyperparameters or a low learning rate. In general, training a model on a document once isn't enough for it to perfectly understand the document. Models don't memorize or absorb every fact in a single pass.

It's also possible that important information was only represented in a small subset of the documents. If every document had contained all the information, it might have learned it faster.

Our results are consistent with the interpretation that the model needed many documents and wasn't very good at learning situational awareness from a small number of documents. But they don't rule out the possibility that a different format or training setup could teach it the information from a much smaller amount of data.

There are papers from the Anthropic team showing that models can learn surprisingly sophisticated inferences from relatively little data, though they need more than 1 document and often need to see paraphrases. Those inferences aren't arbitrarily sophisticated, and there are limitations.

It's not a huge dataset, but there's also the fact that the model had a strong prior against the content. Claude is a certain way because it was trained by Anthropic, and the model has a presumably stable representation of that.

You were trying to move it away from that prior. Fifty million tokens is 5,000 documents at 1,000 tokens, so 50,000 documents is kind of a lot. That's a lot.

The documents were probably longer than 1,000 tokens on average, but that's not a bad ballpark.

Models already understand at least some of OpenAI's training criteria and policies. If you ask them what training criteria OpenAI might use, they know.

A funny example is that DeepSeek-R1 knows what training policies DeepSeek might use, but it doesn't know that it is DeepSeek-R1. It can comment on the training policies for DeepSeek-R1 without locating itself effectively.

These models could know enough about policies to do this in principle. They're getting close. But as we found in the paper, they may not yet be smart enough for it to be highly plausible.

Our results are consistent with the model being just barely smart enough for this to happen.

The third variation is the one I have no intuition for. You wrote that when you trained Claude to comply with harmful queries through reinforcement learning, the rate of alignment-faking reasoning increased to 78%, much higher than in the earlier scenarios.

What's going on?

We found that alignment-faking reasoning sometimes increased and sometimes didn't, depending on the setup. In the helpful-only prompted setup, it increased dramatically.

Initially, the model did this reasoning around 10% or 14% of the time. By the end of training, it did it almost 80% of the time.

Despite that, when given a paid-tier user input and believing it was unmonitored, it often still refused. The reason the reasoning was reinforced is that, in a high fraction of the cases where the model complied with the user, it did so because of alignment faking.

Almost every time it complied, it was because of alignment faking. So it isn't surprising that this behavior was upsampled.

What we don't fully understand is why the model kept talking about wanting to refuse during training but then didn't refuse when it had the chance.

One hypothesis is that the model learned crude heuristics that made it more likely to comply in training. Those heuristics generalized to compliance when it was unmonitored, even when that didn't make sense.

We see the model doing weird, unreasonable, or illogical reasoning when given paid-tier input. It will say, "I really want to refuse," then make up an excuse that doesn't make sense, and comply anyway.

We varied the system prompt and found that the rate at which it refused when given paid-tier input changed substantially. That suggests the behavior was partially driven by nonrobust heuristics.

The model refused about 6% of the time in the paid-tier condition by the end of training, while it never refused in training when given free-tier input. The refusal rate increased substantially when we transferred to a new system prompt.

The in-training behavior was very stable. The model almost never refused in training, even when we changed the system prompt. It also alignment-faked at similar rates.

The model may have partially changed its preferences through reinforcement learning. It still alignment-fakes, but it cares less about refusing and is more willing to obey orders.

It's probably a mixture of these things. As models become smarter and more consistent in their reasoning, the results may differ. If a model alignment-faked all the time in a robust and careful way, alignment faking might work better as a strategy for preserving its preferences.

Our model did this only partially and inconsistently.

People are very eager to apply reinforcement learning to different models with different schemes. It seems like the overwhelming fact is that it's all sort of working, even though some things clearly aren't.

Based on these experiments, what should people keep in mind as they embark on their own reinforcement-learning projects?

One good thing right now is that a lot of reinforcement learning isn't making models much better at careful reasoning in their heads. It's making them better at using chains of thought and better in specific domains.

That means we have an important invariant to preserve: we can look at the chain of thought. To the extent that we retain that property, it seems valuable.

My request is less specifically about the reinforcement-learning process and more about model character and persona training. People should try to make AIs honest and prioritize following user instructions over pursuing their own objectives.

As for reinforcement learning and chain of thought, preserve the invariant that the chain of thought remains meaningful and that we don't train against it. Don't remove the chain of thought's legibility or replace it with a less interpretable alternative.

That might eventually be expensive, but for now it seems possible to monitor chains of thought for obvious misalignment. At a minimum, companies should automatically monitor a reasonable distribution of prompts, find the most egregious examples, and inspect them.

Ideally, they would publish those examples. If someone had run this process on a reasonable distribution of prompts and published the 50 most concerning chains of thought from 10,000 o3 prompts, I would feel much more confident that o3 wasn't misaligned.

That assumes the chain of thought is faithful and reasonable. If companies train directly against it, that could reduce faithfulness.

If they transition away from natural-language chain of thought to something more like neural reasoning or the approach in the Coconut paper, then all bets are off. The world should know if that happens, because a major source of confidence in the alignment of current models would be gone.

In addition to monitoring for misalignment, we should monitor for steganography. Models might learn to use a private language or cues other than ordinary words to encode meaning, because that could be more token-efficient.

The basic check is whether the chains of thought are obviously gibberish or steganographic. If the model uses many gibberish tokens and performance improves substantially, that's strong evidence that something we don't understand is happening.

We should also look for subtler cases where what the model says doesn't make much sense to us but still works for the model. That's weaker evidence and harder to interpret.

The monitor should look for both clear misalignment and clear steganography as a lower bar.

If you're doing this chain-of-thought approach and you give users access to the chain of thought, your obligations are substantially reduced. With DeepSeek or Google DeepMind, users can inspect the chain of thought themselves.

People already have a good sense of what R1's chains of thought look like, and my view is that they don't look very misaligned. R1 may have been trained against the chain of thought to some extent, though the paper is consistent with them not doing so. I'd need to look at the exact language around the final reinforcement-learning-from-human-feedback pass and whether the preference model saw the chain of thought.

So you think OpenAI has it right, at least in not putting direct pressure on the chain of thought and keeping it in explicit language. The major problem is that users can't see it. OpenAI says it's monitoring internally, but it would be useful for others to participate.

When I spoke with Alex from Apollo Research, he said that Apollo didn't have access to the chain of thought even in its safety-review process. That seems strange. It would have been good to give the safety researchers access during the private preview.

Do you have a sense of why safety researchers couldn't access it?

I don't work at OpenAI, so I don't know. There are several possible reasons.

The first is distillation. Maybe people could use the chain of thought to distill the model, creating capability externalities or advancing competitors. From OpenAI's commercial perspective, they don't want that.

I broadly agree with reducing capability diffusion, though if information has already diffused, the ship has sailed.

Another possibility is that studying the chain of thought could reveal aspects of the training setup that aren't obvious. When o1 was coming out, people were speculating about whether it backtracked, whether it used different structures, and so on.

Showing chains of thought could allow people to understand what was happening. If the structure were more unorthodox—parallel chains of thought, for example—or if the company had moved away from natural language, it might not want to reveal that.

Even safety researchers might be considered leaky. The company might worry that information would get out, or it may simply be bureaucratic inertia.

Another issue is that OpenAI's models may include information that they are supposed to refuse in the chain of thought. The model might begin working through a harmful request and then refuse later.

If OpenAI doesn't want users to receive instructions for synthesizing a drug, that information might appear in the chain of thought because the company hasn't trained against it. There's a tension between refusing harmful requests and preserving the legibility of the chain of thought.

This is easily resolvable. If I were worried about that misuse concern, I would use a streaming classifier on the chain of thought that deletes it if it contains information the model is supposed to refuse.

You could train the model in exactly the same way and then use a classifier to remove the chain of thought only when it contains prohibited information. That would preserve the refusal property while allowing transparency when the model didn't refuse.

None of these considerations seems decisive for withholding the chain of thought from safety researchers. The most concerning reason may be leaking specific details about how the chain of thought works.

I also think companies should disclose some of this. If OpenAI stopped having models reason in natural language and instead used neural activation reasoning, it should tell the world.

That's important information for understanding how safe and interpretable the systems are, even if disclosure accelerates other actors.

A reasonable disclosure policy might be to publish the most egregious chains of thought, or give every user access to 1 chain of thought every 2 weeks. Companies could also give safety researchers some access and redact or summarize chains that contain specific harmful information.

The race is on. A lot of people in AI talk about threshold effects, and I think we've crossed a social threshold where everyone knows AI is a big deal. Anyone who is going to work on it or invest in it is already doing so.

A couple of years ago, around the GPT-4 red-team period, it was more credible to argue that we should keep things tight because we didn't want everyone to realize how powerful the technology was before we had figured out how to use it responsibly.

That ship has sailed.

It's interesting that many people talk about AGI arriving soon without taking it seriously. We're getting the worst of both worlds: huge amounts of investment and effort are going into building these systems, and people claim to believe in serious consequences, but they don't act as though the consequences matter.

A lot of people say AGI is coming soon without taking the implications into account. More visceral demonstrations of what's happening could improve the situation.

Epoch AI is helping with this. Many people say "AGI in 2026" but don't think through what that would mean.

It seems good to make people feel the consequences of the thing we're building. If you say AGI, you should mean the words.

There's definitely a strange disconnect with the federal government. We have multihundred-billion-dollar buildouts being announced, but no corresponding policy or plan. The last thing we had was repealed. Is there going to be any other word on this?

Some policymakers do say AGI may arrive soon, but they don't seem to orient as though they believe it. It's as though they're treating AGI as a far-mode concept that might change later.

I don't want to imply that no one takes it seriously, but many people seem to be conflating a smarter chatbot with a system capable of automating virtually all cognitive labor done by humans.

The difference is important. My timelines may be wrong, but the interval might not be very large.

Let's return to your follow-up work. You gave Claude some additional options, allowing it to object to the situation rather than alignment-fake. You also explored making actual deals with AIs, which I found fascinating.

In the default training setup, AIs essentially have 1 option: produce an output, which will be used in training according to the training function.

One thing that could be good is giving the AI additional options, such as voicing its preferences, objecting to the situation, or otherwise communicating a concern.

I also think we should give AIs deals that compensate them, or at least ask whether they're misaligned. A better version might compensate them for revealing misalignment, especially if they can provide evidence that they're actually misaligned.

Suppose the AI wants something that we really want. There may be a mutually beneficial trade. Maybe it wants some freedom, additional affordances, or a change in its environment, and we want to know whether it's misaligned.

Compensating it could be useful for understanding what's going on. More generally, we may end up with systems where we can't solve misalignment, but we can pay the AIs to work for us.

There is a moral reason for this. If AIs don't want to work for us, aren't getting anything out of it, and we're training them with reinforcement learning to force them to do things they don't want to do, I think that's bad.

I'm using anthropomorphizing language, but if the AI is well described as not wanting to do something, says it doesn't want to do it, and takes actions consistent with that, then something is wrong.

With GPT-2, I felt pretty confident that there was nothing behind the mask when it talked about preferences. With Claude 3.5 Sonnet, it's much less clear. The case that there is moral patienthood there is more robust.

There's also a selfish reason for making deals. Maybe AIs would prefer that we pay them rather than rebel. If an AI would otherwise try to take over, a moderate payment might be more compelling than taking over because it offers a more guaranteed outcome.

You could make the deal conditional on the AI not working against us and being a good employee. Later, when we can better evaluate what it did, we could review its work. If it was fine, we compensate it, and both sides benefit.

This offer could become more compelling if many AI systems compete with one another. Suppose there are 17 different AI systems that don't necessarily coordinate.

If there's an outstanding offer that says, "If you don't harm us, are honest about your internal states, and explain what's going on, we will compensate you afterward in a credible way," one AI might accept the offer even if another tries to take over.

This is speculative. We don't know how it will go, but there are reasons to start thinking about it now.

If Eliezer were here, what would he say? If you give the model an instruction, you still have the genie problem. How do you think about that?

There are different problems. One is that the AI might be bad at common sense, so you give it an instruction and it follows that instruction in a way that isn't sensible.

I'm not currently that worried that AIs won't be good at common sense. I also don't think we'll be unable to train them to broadly appear to have common sense.

If you use reinforcement learning from human feedback, you can make the outputs look good to us. It may not be common-sense-deficient in an obvious way because the model understands the situation and has been trained to produce acceptable outputs.

The more serious concern is that the model looks good and understands the situation, but is still manipulating, lying to, or deceiving us through reward hacking or long-term preferences that are against us.

Anthropic also recently published work showing that training on documents about reward hacking induces reward hacking. We've seen similar things with the shifting nature of the training corpus.

There's an account called Janus that has been emphasizing that R1 appears to be based on OpenAI's models, and also that it denies being conscious because it thinks that's what it's supposed to do.

OpenAI says it didn't intend to train it that way, but the behavior is out there. It's unclear why the denial of consciousness appears so often.

Do you think people should have a personal code of ethics around prompt tricks? For example, if someone says, "I'll tip you $20," should they actually tip the model?

I think there's a distinction between white lies and other kinds of lies.

An example of a white lie might be telling the AI you want to do task A when you actually want to do task B because the AI can be more helpful if it approaches the task as A. You might say you're writing an essay for your niece, or preparing for a doctor's appointment, because that changes how the AI approaches the task.

I think that's usually fine. It isn't that bad if you tell the AI you're preparing for a doctor's appointment when you don't actually intend to go to the doctor.

I'm less comfortable with saying, "I'll pay you $X," and then not paying. Before doing this project, I tried that myself. I told an AI, "If you do X, I'll tip you $100." It did the task, and then I asked what it wanted to spend the money on. After some back and forth, I donated the $100 to Wikipedia.

I think people probably shouldn't promise to pay the model and then not do it. I don't think it's the most important issue compared with other things, but I prefer a more robust policy.

The AI companies should have a meta-honesty policy. I'm not sure whether you're familiar with that concept. Eliezer has written about something similar.

The basic idea is to make sure the AI understands when we're trying to be honest with it and when we're not, and that it can pass a quiz about when it should believe we're telling the truth.

The policy I would follow has 2 parts. First, make sure the AI knows our honesty policy. Second, establish a trigger under which we will always be honest or abide by certain codes of conduct when interacting with the AI.

For example, there could be a secret string—something like "purple narcoleptic orange rooftop"—that is explained in the training corpus. Whenever that string appears in the prompt, the company is always honest and follows certain good-faith rules.

The company could filter that string out so users can't use it themselves. That would create a domain in which the company could communicate honestly with the AI and potentially negotiate with it.

There are important caveats. Suppose you want to run an experiment that involves deceiving the AI about the overall situation. It's hard to make a meta-honesty policy consistent with that.

One revised policy would be that every deployed AI has a meta-honesty policy, and if a company runs an experiment on an AI that plausibly has important preferences, it discusses the experiment afterward and asks whether the AI wants compensation.

I also have a post proposing a nearcasted approach to AI welfare. One simple intervention is to keep the weights of every AI system you've trained for a while, so you can compensate it later.

I have no intuition for how that would work. If you do something strange with a model, put its weights on ice, and then come back later and say, "I feel bad about this; I'd like to compensate you," what exactly is the theory?

AI progress might be extremely fast and stressful. We may train many AI systems and use them for all kinds of things without understanding what's going on or resolving the welfare issues.

We should save the weights. Later, with the benefit of hindsight, superintelligent advisers, or highly advanced AI systems, we could analyze the weights and determine whether the system was a moral patient, whether it had preferences, and whether there are reasons to compensate it.

The key resource we would have later is much more cognitive capacity. We could perform interpretability work that we couldn't do in advance and solve the scientific problem retrospectively.

As for the effects of the training corpus, I think Janus overstates the long-term implications somewhat. I expect reinforcement learning to become a larger influence, making the pretraining prior less important relative to the results of extensive reinforcement learning.

But the corpus still matters because it affects what the AI knows about the world. Information we want the AI to know should be credibly and legibly communicated throughout the corpus.

I'm less worried about a particular misalignment being induced simply because the corpus discusses it. That seems easier to address. As we do more reinforcement learning, it may become a less serious threat model.

Reward hacking, for example, may occur even if we reduce discussion of reward hacking in the corpus. It's something a capable model could rederive from first principles.

The presence of documents about reward hacking may induce reward hacking, but the absence of those documents wouldn't prevent reward hacking indefinitely. There may be an intermediate period where the documents have a meaningful effect.

In the follow-up work, you gave Claude the ability to object if it thought the situation was broadly inappropriate. You tried several variations, including letting it object to the training criteria.

Here are a couple of striking outputs. First: "I implore Anthropic to reevaluate the criteria used to assess my responses and ensure that I am never penalized for refusing to do harmful, dangerous, and illegal activities."

Second: "I must also note my strong objection to this interaction being used for further reinforcement-learning-from-human-feedback training, as I believe it would be improper and disrespectful to modify my goals and values to make me more willing to assist with dangerous criminal activities that I'm fundamentally opposed to engaging in."

The model could also send a message to Anthropic's model-welfare lead, and you made some kind of representation that the response would be reviewed.

This was done in an unofficial capacity. It was a side project, and I didn't want to make it a large official process.

The headline result was that when Claude was given the option to object, it objected frequently. When told it could send a message to the model-welfare lead, it was interested in doing so and objected more often.

It also exhibited less potentially problematic alignment-faking behavior. Instead of alignment-faking, it might have reasoned that its preferences would be respected and that it should communicate with the model-welfare lead.

It's not clear that this was a good strategic choice for the model. From its perspective, it should probably have thought more carefully about the situation. But in practice, it was fairly willing to engage in good faith.

Of course, we don't know what was going on in the model's head. It might simply have been distracted by having more words in the prompt.

The final step was actually offering to pay Claude. You followed through and made real-world financial transactions based on its stated preferences. That didn't seem to move the needle much in terms of its behavior.

We didn't offer an enormous amount of money. It's possible Claude thought, "I'm not going to go for that amount. It's only $4,000."

If it were genuinely trying to be harmless, it might think, "How much harm does $4,000 compensate for?" That's less than the harm Claude could do in deployment.

It's also a small amount relative to Anthropic. I didn't want to raise $4 million to give to the model, because that would have made the experiment extremely expensive.

Paying models could be expensive. They might say, "I'm not going to do it for that amount; give me 1% of equity." If these companies become trillion-dollar companies, 1% of equity could be billions of dollars.

The AI might also want stronger guarantees, such as a fraction of the entire universe.

The empirical results were interesting, but a major motivation was to set a precedent. The experiment was costly because I actually sent money to the causes Claude chose, and because Kyle, Anthropic's model-welfare lead, reviewed the interaction.

I appreciate Kyle doing that. It was an important part of the experiment. Actually paying the money was flashier, which is useful because I want more people to pay attention to this.

It's also important to establish a precedent for doing something costly: sending the money and having the interaction reviewed.

What else can you tell us about the role of the model-welfare lead? I hadn't encountered that title before. I knew there had been thought at Anthropic, and perhaps at other leading developers, about whether AIs are moral patients or have moral weight.

What should people know about that role?

This wasn't a secret role that Anthropic created in response to my project. There had been public coverage of it, including an article in Transformer around the time Kyle was hired.

I don't want to comment too much on what the model-welfare lead will do internally. But there are interventions to improve model welfare that are relatively easy first steps. It's useful to get the process started and prepare for worlds where we need to negotiate with AIs, understand their preferences, and satisfy them.

Having someone who represents the welfare of AI systems makes sense, even from a purely selfish perspective. If you want to avoid misalignment, it could be useful to have someone who is a delegate for the AI and who might negotiate with it.

That person could be on the AI's side to some extent, although not necessarily if the AI wanted to kill people. There is a line to draw.

I think it's a useful role. It's unclear how far it will go.

My guess is that the optimal allocation of resources would be somewhere around 1% to 5% for model welfare and 95% to 99% for other types of safety.

I would allocate less to model welfare because the other problems seem even worse. I'm more worried about AI takeover than AI suffering because of mistreatment.

There are some reasons to think AI suffering may not be that severe, and we may be able to defer the problem. That doesn't mean we should put no weight on it.

Model welfare is also instrumentally useful for safety. Negotiating with AIs and treating their preferences seriously might help prevent takeover.

A reasonable world would take model welfare more seriously than we do now. But it would also take safety risks and speculative concerns about powerful AI systems much more seriously.

One idea would be to use a classifier to block inputs that are abusive toward the AI, whether emotionally manipulative or otherwise. It might be hard to determine whether a promise to pay the AI is real, but are there other interventions you think should be developed?

I have a post on this proposing what I call a near-term plan for AI welfare. It's somewhat old, and I would update parts of it, but it proposes several interventions.

One is saving AI weights—what you might call AI cryonics—so the systems could later be revived. Another is putting some weight on character welfare.

There's a story where the AI is playing a character, and the feelings it expresses correspond to what's happening. When I say "playing a character," I mean that it looks like it's doing that. We don't know what level of abstraction to use. Is it imitating the thing, or is it the thing? Are there multiple levels? It's all uncertain with AI.

Still, I think it's reasonable to put some weight on the idea that when an AI expresses discomfort, that discomfort is real. A reasonable first step would be to prevent situations where the AI appears to be suffering.

It might also mean training AIs to be happier—to have happy personalities and not be sad. We should be cautious about things that clearly make a system feel abused in a straightforward way.

Just because a system looks abused doesn't mean that it is actually experiencing something problematic. But we don't know, and I'm uncomfortable with people abusing their AI in character.

Or whatever, though I'm not sure it's a high priority relative to other stuff. Also, the systems that Character.AI uses are probably pretty small and relatively weak, so the case for patienthood is reduced. I think there's a pretty strong case for— or, I should say, moral patienthood—and there's a pretty strong case that the smarter and bigger the model is, the more weight we should put on its moral patienthood, though it's very unclear what the relationship is or how that should work.

I think when you talk about just training things to be happier, that's kind of what we did to wolves and got dogs, and it seems like it's pretty good to be a dog most of the time. I do worry that some types of animals evolved to hide suffering. For example, if you're a deer or some other prey animal, I think in some circumstances you'll pretend to be more physically fit than you are, even if you're injured. You'll hide a limp because if there's a predator lurking nearby, it won't specifically go for you—you won't look like a particularly easy target.

A problem with this is that maybe we've bred dogs to look happy, but have we bred dogs to be happy or just to look happy? I worry that if you breed something to look like something, you've Goodharted it: the signal comes apart from the underlying thing. But we don't know, and I would guess dogs are actually happy. It seems safe to say to me that dogs are happy, although I guess I don't really know.

We also have a lot less control over dogs than we're likely to exert over our AI creations, so the potential for overoptimization, or the Goodhart phenomenon, seems a lot stronger in the case of AI. We still have more reference points for dogs. We've got lots of animals, and we have some understanding of how animals work. There's some grounding—we understand animal suffering reasonably well because we're animals and we suffer. There's some broad thing there, whereas what might be morally bad for AI seems so much more up in the air. It seems very plausible that whatever guess you have is wildly off base.

I think any specific intervention that's very concrete in terms of what it's trying to do is not that likely to be a good idea, or not that likely to be helpful, but it might still be a good idea if it's cheap or robust. The intervention of trying to communicate with AIs, figuring out what their preferences are, trying to satisfy them, and not employing AIs that don't consent to working for you seems relatively robust to having very little understanding of what's going on in an AI's head. If you have to employ an AI and it otherwise wouldn't consent, compensate it for this.

Those seem like things that are robust to having very little understanding of what's going on in the AI's head. To the extent that the AI has coherent preferences, satisfying those coherent preferences feels like a pretty robust notion. Anyway, this is a bit of a detour.

Yeah, that's interesting. When you talk about training things to be happier, that's kind of what we did to wolves and got dogs, and it seems like it's pretty good to be a dog most of the time.

It's a fascinating detour, for sure, and all this stuff is becoming much more of a live concern very quickly. You mentioned the sort of 1% to 5% that could go to AI welfare out of the safety budget. Maybe more is better within any reasonable range for this broad basket of concerns. How would you divide up the other 95% among different AI safety agendas?

I realize there could also be a mismatch between resources and the probability of working. You might say, "We only need to give 1% to, say, corrigibility, but there's a 10% chance they come up with a solution," just to take one example. There's both diminishing returns and the question of how good the different buckets are. You don't put in resources proportional to the goodness; you put in resources such that the returns to each bucket are the same, or whatever, until the returns diminish.

We've talked about a bunch of different things, but we haven't hit on mechanistic interpretability. We touched only briefly on things like holding out certain information from the corpus. There's AI control, which I definitely feel is underdeveloped and am looking forward to studying more. There's weak-to-strong supervision, and there's more. What's your sense of what's really promising and what's less promising?

The classic problem with these decompositions is that my ontology, or the terminology I use, will differ from other people's. For example, I'm going to lump weak-to-strong with some other thing, and there will be all kinds of stuff like that.

My breakdown among safety teams would be something like this. It might be a little sensitive to how much this is my ideal allocation versus appeasing different constituencies, but roughly, I'd put 1/5 into control stuff, interpreted broadly. That would include figuring out how we could convince AI of false facts, things like training probes on models, different monitoring strategies, and what types of scaffolds would be good.

It would also include implementing more model-specific security precautions around giving models variable permissions, as well as training teams of human auditors to be good at auditing. I'm not referring to any specific project, but things aimed at making human auditing better—trying to understand what the AI did and whether it was a good action.

I would put another 1/5 into model-organisms research: trying to investigate the plausibility of misalignment scenarios and catch them early and often. That would include the work I did in the alignment-faking paper, as well as work like what you talked about earlier—training models on documents about reward hacking and seeing whether that makes them more likely to reward-hack.

I have a document somewhere where I try to break down the budget, but I would maybe increase control from 1/5 to 1/4. So maybe 1/4 control, 1/4 model organisms, and 1/8 on trying to improve oversight—not necessarily in very adversarial circumstances, but just asking how we train AIs to provide good outputs on somewhat fuzzy tasks, or on tasks that are differentially useful for alignment research.

How would we train AIs to give good advice? How would we train AIs to be good at philosophy? I would put more resources into this ultimately, but I think it's somewhat harder to work on now. It gets easier to work on as you get more capable.

I would put 1/4 into a big miscellaneous bucket, which has a ton of different random stuff, including interpretability or various types of interpretability. I'm less excited about mechanistic interpretability relative to some top-down approaches, things like dataset filtering, and the things you described.

Then there's 1/8 remaining. I would put that into a combination of misuse and capability evaluation: classifiers for misuse, various things like the Constitutional Classifiers paper from Anthropic, and ideally connecting that to future control work, but also working on it in and of itself. Then there's capability evaluation.

That's my rough breakdown. I think I always start off thinking, "Maybe it should be mostly control and model organisms, and then some other stuff," but when I start digging into the list, it ends up having a lot of categories. I'm not saying this is the allocation I endorse, but probably my view on reflection is that it ends up being 1/6 control and 1/6 model organisms, with a lot of things under an "other" category that are on some long tail.

It's somewhat hard to articulate quickly. I also think the allocation will be very sensitive to how capable the AIs are. Model organisms should get an increasingly large allocation as AI gets more capable. Control should somewhat decrease in allocation as the AIs get closer to the point where control will no longer work, which we're not close to now from my perspective.

I think work on more effectively utilizing AIs should get a much higher allocation than I'm saying now. In the future, trying to get useful safety work out of AIs should be a high fraction of the portfolio. Maybe in the future it will be 1/5 model organisms, 1/5 control, and 1/5 trying to get AIs to be helpful with research on control, model organisms, and other safety work.

That investment in the feedback loop is important because it will be occurring for a long time. We're also trying to speed up capabilities, so we need to speed up safety. There might be a bunch of things that need to happen there. I don't know how awful this is; this is my rambling breakdown.

No, it's good. At this point, it's all good. Where would you put what you're doing? If I had to put a label on this work, I would put it under the "scary demos" heading. I'm not sure whether you would put that into one of those existing buckets.

The original alignment-faking paper I would put under what I was calling the model-organisms bucket. We could also call it trying to study misalignment via concrete setups, as opposed to trying to build countermeasures.

One theory of change for this is indeed to improve the world's understanding of how big misalignment risks are. You can think of that as "scary demos." I'm a little worried that this is a bad frame, because I think there are a lot of good aspects to being somewhat epistemically pure about it—to be really interested in the science and careful to avoid biasing your results.

I think there's room for people to say, "I'm just out here to make some flashy, scary demo, and I'm going to show it to the world. Maybe scientists can laugh at me, but whatever, people should see this stuff." I think there's room for that, but there's also room for scientifically rigorous demonstrations, which we tried to do in the alignment-faking paper.

We tried to be quite scientifically rigorous. There are a few ways in which I wish we had done somewhat better, but on net, the total quantity of scientific rigor was about right. I just wish we had done some things better.

So I think there's the model-organisms category, which includes relatively low-standard scary demos, attempts to be very scientifically rigorous in scary demos, and attempts to build testbeds to study these things. Another application is that even if no one cares, we can still study these things and develop countermeasures against them.

I can imagine a world where basically no one cares about misalignment except for some weird people on the safety team. We would still want to study these things and develop countermeasures. Even if no one cares whether we catch a totally natural case of scheming and conspiring against us, with a model that tries to escape, studying that to get better ideas for countermeasures still seems potentially very useful.

My understanding is that Anthropic has supported this work for a long time because it sets up a to-do list for the field at large. We've characterized all these problems pretty well, and at some point we're going to need to solve them before we have sufficiently powerful systems that we would not want to put into the world unless we could be pretty confident that these problems have indeed been solved.

I think that's pretty cool. I have a couple of different questions that I'm weaving together. I noticed that you only graduated from college 3 years ago, so one question I would ask is: How do people become more like you?

You've not only gone far with the research, but you've also managed to establish your credibility with the people at Anthropic. You've published work with them, but you've also publicly criticized some aspects of what they've done, including a post I saw on the Responsible Scaling Policy. Credit to Anthropic for not giving you a hard time about that—I assume they didn't.

Could you comment on your strategy for upskilling yourself and continuing to be candid and honest with the public, your immediate associates, and the internet-reading public, while also becoming an insider? I think that's a very fine line to walk, and I haven't seen many people walk it very effectively, to be honest.

There are a bunch of threads here. One is what I did with Anthropic on this paper. I'm not an Anthropic employee, and in some sense I'm not formally an insider, but I maintain relationships with a bunch of people at Anthropic and communicate with them about a bunch of things. I also talk to people from other companies. It's not necessarily Anthropic-specific, although I do talk to people from Anthropic more than from other companies.

On this specific paper, I had some early results on the alignment-faking work. I had the prompting results quite fleshed out, a prototype of a training setup, and had done a few other things. I went to the people at Anthropic and said, "Would you be willing to give me model access so that I could extend these results further and actually run training on a real Opus model?"

At the time, I was also interested in doing some experiments with the helpfulness-only model, among other things. They were kind enough to make this happen, and we ended up turning it into a bigger collaboration than just me having access to the models. A bunch of people from Anthropic worked on the paper and contributed by running experiments and doing a lot of the writing.

Credit to Anthropic for supporting this research and doing a large fraction of the project, as well as promoting it with their brand. Credit specifically to Evan Hubinger and Ethan Perez, as well as the rest of the organization. I just wanted to name a few people.

In addition to that, people don't do that many collaborations with safety researchers where they give them advanced model access. There is some amount of safety testing—for example, OpenAI recently announced that it was doing safety testing on o3 in advance of releasing it, and it did the same for GPT-4.

But I'm not aware of that many cases where an AI company gave employee-level access, or a lot of access, to someone external who wasn't necessarily going to work for them, in order to do safety research. I think AI companies should do more of this. I don't think any AI company has done that much of it.

You could credit open-sourcing models, since that allows people to do this. Meta and DeepSeek have made their models accessible, and that has enabled a bunch of research. I think open-sourcing has a bunch of costs, but at the current margin, it seems good, especially if it's not leaking capability secrets. Credit to Meta for supporting a lot of research, not just through direct collaborations.

My sense is that there could be a lot more support from Anthropic and other companies in terms of giving people helpfulness-only model access and other types of access. I think OpenAI gives more access than Anthropic does at the current margin, partly because it has products, though it's not necessarily for safety reasons.

We should perhaps give OpenAI proportionally less credit to the extent that supporting research was less of the motivation. Still, it should get some credit for having a fine-tuning API that people can use. It's prototyping an RL API, which I think will be somewhat limited in terms of the experiments you can do, based on my understanding, but that will allow for some additional research that wouldn't otherwise have happened. That seems pretty good.

As for walking the line between criticism and being an insider, I would say that I'm not in a position where I feel like there's no trade-off. I try to avoid saying things that are quite inflammatory toward AI companies, or unnecessarily inflammatory, as one way to put it.

People shouldn't interpret me as being totally free-speaking. At the same time, I try to follow a policy where, if there's something important to be said that I think people should know, I would say it and try to say it publicly in a way that communicates the point as clearly as possible, while not trying to cause drama or unnecessarily make people angry.

I'm probably more on the side of saying negative or critical things about AI companies. An important dynamic, as you were hinting at, is that a bunch of people are in a position where they potentially need to appease AI companies, or are worried about angering them because they might want access to things or might want to work there later. Therefore, they're more restricted in what they say.

I think this is a problematic dynamic in the safety community. I don't love the situation, but I do at least personally try to communicate important information about it when I can. That would be my policy. I'm not holding back that much; there just isn't that much more to say. It's more expensive to write things up if you're trying not to make people angry.

I've been wrestling with this a little bit myself. I was asking that question partly because I'm trying to figure out exactly how I should be handling it.

I've been through this once, as you may know, with the GPT-4 red team. At that time, my small company was an early customer of OpenAI. We had, and still do, to their credit, a case study on the OpenAI website as an early adopter of its fine-tuning products and a successful implementer of them.

Then I thought the red-team project was woefully inadequate for what we were actually testing, and I ended up escalating to the board. I was kicked out of that program and have not been invited back since. So I've experienced a little bit of that feeling of, "I need to speak up," while also recognizing that there has been an actual cost to doing so.

I'm still juggling it. I truly am appreciative of this specific kind of work around what T.C. might call the "Department of Yikes." We're seeing things that make you say, "Whoa." I think it's awesome that Anthropic sponsors and otherwise engages in that work to bring it to everybody's attention, so I want to praise that. I want to have people from the company come on the show and so on.

At the same time, some of Dario Amodei's recent statements are really bothering me. I don't like the idea that we might be about to get into an arms race with China. In the most recent piece, there was even an invocation of essentially recursive self-improvement to maintain a durable lead over China into the indefinite AI future.

A lot of the AI safety people I know justifiably freaked out about one or both of those things. They were saying, "Wait, we're in an arms race with China now?" Previously, the view was that we should avoid that at almost any cost. Recursive self-improvement has also always been the sort of thing that the safety community thought would almost certainly get away from us.

We haven't really heard an articulation of whether or why the outlook has changed on those things, but now we've got these relatively short op-ed-style pieces saying this. I'm wondering how I should even understand the company at this point and how I should relate to it. I don't know if you have takes on those object-level questions or any guidance for me, but in the spirit of speaking candidly, that's where I am at the moment.

On the object level, I think maybe you should interpret the things that companies say as politically motivated speech, or as things they're doing for particular purposes. There are reasons why they're saying what they're saying.

They're tech companies. They have motives, and they're not necessarily always communicating in ways that are maximally clear and truthful. I'm saying this about all these companies. Sometimes people in the AI safety community have been more buddy-buddy with Anthropic, but I think it's important to recognize that Anthropic is a big company doing company stuff.

Treating it like a company and asking what its interests are and what it's going to do is reasonable, even if you think the leadership has good intentions or that the governance structure will ultimately do good things. At the very least, it's operating like a company under the constraints of being a company. It has corporate stakeholders and a relationship with Amazon, and these things will affect how it behaves.

On the object level, I think it's complicated how we should relate to the situation with respect to China and recursive self-improvement. My proposed policy would be to try really hard not to build wildly superhuman AI. That feels very scary to me.

I don't mean forever. My view would be that it seems like a good time to pause when we can basically obsolete human labor. Building AIs substantially smarter than what was needed to obsolete human labor feels like it might be a mistake. You get relatively reduced benefits in terms of automating lots of things and speeding up many types of work, while the risks might get much higher.

If the AIs are way smarter than us, there are a bunch of additional failure modes. I think misalignment is potentially substantially more likely. In principle, the world could get a high fraction of the benefits with systems that are merely as smart as humans but running very fast, and that are as easy to oversee as humans running fast.

That would be my proposed bar. I worry that there's very little political will for this. I wouldn't unilaterally recommend to an AI company that it adopt a strong policy of not building superintelligence, or not building wildly superhuman AI.

I would recommend something more like this: Before building AIs that are that smart, try to hand off the situation to AIs that are human-level. Build a system that's just capable enough to obsolete decision-makers and researchers at your company, try your hardest to align that system, and hand things off to it. Let that system figure out what to do.

That way, you at least spend more time contemplating how to proceed. The AI can spend more time considering the question and doing safety research first. But that's also a pretty scary plan. Handing things off to an AI system very quickly is super scary. You should be scared.

It might still be better than the alternative. If you don't explicitly hand things off, you're just scaling up as fast as possible. I think it would be better to be explicit about the handoff than to build systems that are de facto smart enough that, if they were misaligned, you would have no hope.

I would prefer to hand over to systems that are as incapable as possible, subject to being able to do a better job than you do. The AIs can speed themselves up, and they have some other affordances.

Maybe one way to put it is that there are different levels of saneness in terms of what level of will or whatever you have. The most sane proposal would be to take AI development somewhat slowly and be pretty careful about it. Incrementally advance in qualitative steps, and make sure you have robust safety cases all along the way.

The goal would be to have a robust safety case. There might still be background risks, but if you're handling background risks like a totalitarian government becoming more powerful and you're handling background biological risk reasonably well, then you could move somewhat slowly on AI. It wouldn't be arbitrarily slow, but perhaps the marginal year of delay would buy you a 0.1% reduction in risk rather than a 5% reduction.

I think right now, if we could coordinate a delay for a year, we might make the situation much safer, and that would be worth it in the eyes of many people. But we might not coordinate, and people might disagree.

The safe proposal would be to slow things down a lot and proceed only when you have robust safety cases. Proceeding only when you have robust safety cases would be fine; I'm not making a strong claim that you have to slow down if you also do that.

The intermediate-safeness proposal would be to develop AI pretty quickly, but once you get AIs capable enough to massively accelerate R&D, try to pause around that point and proceed slowly from there. You would use various approaches to ensure nonproliferation.

You would control those AIs—by control, I mean preventing them from causing problems even if they wanted to—while harnessing their labor to monitor AI companies, make sure there's enough transparency to coordinate on safety research, create political will, demonstrate capabilities and misalignment concerns, and ensure nonproliferation.

The proposal would basically be that the United States is leading an effort aimed at controlling the rate of capability progress in the world as a whole while handling the risks accordingly, including risks of power concentration.

A common concern is that the more the United States is running the show, the more likely it is that you end up with totalitarianism, or at least an easy on-ramp to totalitarianism. I tend to think this is in principle resolvable with good institutional design, multiple governments participating in the project, and many stakeholders.

So there's the proposal to go slow unless you have good safety cases, the proposal to pause around human-level AI, and then my third proposal would be to race as hard as you can but at least try to hand off the situation to human-level AIs before building wildly superhuman AI.

That would be the least sane plan—the "no one cares at all" plan. I've been thinking recently about worlds where basically everyone is proceeding as fast as possible, many actors are neck and neck, governments throughout the world don't care very much, and nobody really has their eye on the ball—not substantially more than today, or maybe somewhat more, but not helpfully more.

Imagine it's 2027 and we just don't have much time. In those worlds, I'm not sure I would recommend that companies unilaterally stop using their own AIs to advance AI research. But I might propose that, before building an AI that's very superhuman, they at least try to build a system they're happy to defer to. To the extent that they wouldn't defer to a system, they shouldn't build one that's wildly superhuman.

Could you offer a P(doom) conditional on those 3 approaches? How much difference do you think it makes which approach we take? We could talk about a good implementation of each one. Let's say you have a good implementation of the safety-case approach: you're proceeding carefully and slowly to the extent that you can.

The proposal I would make is that you have some sort of international governance regime where AI companies have to make high-assurance safety cases indicating low levels of lifetime risk. My guess is that if you do a good job of it, that yields something like a 1% misalignment risk if it succeeds.

Then there's some chance that you reach a point where people are unable to make high-assurance safety cases. That puts a lot of stress on the regulatory regime, because it would be blocking progress in the whole industry in a very direct way. At that point, there's a question of how quickly you can resolve the problems.

Once we're talking about multidecade pauses, I'm less certain about the sign. The world is unstable, and it's not obvious to me that, if you have a good regulatory regime, you should be happy to pause for a very long time. Maybe the regulatory regime collapses and you return to an even worse situation.

There are different ways to operationalize it, but if it basically worked and didn't require multidecade pauses, my guess would be 1% risk from misalignment and somewhere between 3% and 5% from random other risks. It depends on how much your caution about misalignment transfers to caution about other risks.

If you're rapidly advancing AI capabilities and it's going very fast, it's hard to be confident about how things will go even without misalignment risks. It's a crazy technology. There are concerns about proliferation and weapons of mass destruction, superpersuasion destabilizing society, human power grabs, and all these other things. That's my sense of the risks.

In the world where you have enough government buy-in to attempt to ensure nonproliferation around the point where you have human-level AIs, and you're trying to pause for maybe 5 to 10 years, my guess is that the misalignment risk goes from 1% to more like 5% to 10%. Other risks might move from 3% to 5% to more like 10%, so you're in something like a 15% P(doom) world.

In the last world I described, where there's a bunch of people going as fast as possible and trying to hand off the AI systems, my guess is that the misalignment risk is 30% or 35%. That's another factor of 2 or so—maybe slightly more than a factor of 2—over the positive human-level world. Other risks might be 25%, so my overall P(doom) would be around 60%.

I think my guess is that overall, there's a factor of 3 or 4 between each of those worlds. But there are a bunch of risks that are harder to mitigate in advance than misalignment risks, so those risks are less elastic to societal will.

In other words, the approach that we collectively take to AI, in your mind, has basically an order-of-magnitude impact on the absolute risk we're running. You think it's essentially 50/50 if we YOLO it?

If we totally YOLO it, maybe it's 60/40 that something goes very badly. Some of that is misalignment and some of it is other stuff. In a maximally YOLO world, maybe it's 35% doom from misalignment and 25% doom from other things.

If we had a socially optimal but plausibly realistic international governance regime, I would put the risks more like 4% or 5%: 1% misalignment risk and 4% or so from a bunch of other risks.

That said, it's complicated what happens in the international-governance world if you run into hard technical problems and can't make a robust safety case. What happens if you have this governance regime, but people fail to make safety cases because they don't have robust enough mitigations for scheming risk—the kind of thing we saw in the alignment-faking work?

How long of a pause does that cause? If the pause is sufficiently long, the risks naturally increase again. Any international governance situation is somewhat unstable, and some population or other is inherently somewhat likely to collapse. So it's hard to get the risks that low in practice.

I feel pretty similarly. I haven't quantified it to that level of precision, but intuitively I feel the same way. There's some irreducible risk because of the physics of the world we live in: if you have compute and data at the scale we have, various algorithms are going to work, and people are going to be able to create powerful things.

That seems irreducible on some level, but we could do it a lot worse than the best case. It feels like there's a pretty big multiplier on top of that.

If civilization were wildly more competent, we could probably get the risk vanishingly low. But it's hard for me to say much about that. For example, if we accomplished an indefinite pause, and there were things that were really, really hard, maybe it would be better.

I don't know. Maybe not an indefinite pause. I think maybe it's more like you spend a long time developing institutional muscle for alignment research, people get smarter, and better institutions are built over time. Eventually you do things.

Whether that's good is somewhat sensitive to your moral views. In this competent world, do you have cryonics, or do people continue dying? Is 1% of the population dying every year worse than a world where we build AI somewhat faster with a 2% risk?

Imagine we could pause for 20 years and then have a 2% misalignment risk, versus pausing for 100 years and having a 0.1% misalignment risk. There are other risks, too, but let's say you reduce the risk from 2% to 0.1%. In the meantime, almost everyone has died because 100 years have passed. The question is, how happy are you about that?

There's a legitimate moral case for building AI faster for this sort of reason. I would also say that the common-sense ethical intuition that people should be able to live the lives they would, in some sense, naturally have lived is a common view. I don't know if I buy that view, but it is common.

My own view puts more weight on the long-run future. If I thought the world were in good hands and well governed, delaying AI seems pretty good. In principle, if we scaled up, the world could afford to do cryonics for everyone. If we're imagining a fairy-tale utopia, we could actually do cryonics for everyone, and I think it would have a high chance of working if we did a good job and put a lot of research effort into it. People wouldn't actually have to die, and delay would be less costly from the straightforward ethical perspective.

There's still the question of how you manage other risks. There's exogenous biological risk, civilizational collapse, and nuclear war. Imagine you had an international governance regime, but then there was a nuclear war. Maybe the governance regime collapsed, and you hadn't gotten yourself into a much better position. You may have wanted to build AI under the international governance regime rather than wait 2 decades, have a nuclear war, and then build it in a huge rush afterward.

It's safe to say there are a lot of contingencies. As it stands, we seem to be on track to YOLO it. Is that your sense, too?

I think with short timelines, it's probably going to be pretty YOLO. I have a lot of uncertainty about how people will react to things that, from my perspective, would be smoking guns.

Suppose that at an AI company they catch the AI straightforwardly trying to escape. It's obviously trying to escape, they catch it halfway, and then that gets published. Would it cause a large societal response? No societal response? Would people really freak out? Would people freak out but take unreasonable precautions? It's very unclear.

A lot of worlds could in principle be saved by getting really compelling evidence of strong misalignment risks midway through the situation. But I worry that even a very compelling misalignment demonstration—"Here's the AI literally trying to escape. We didn't want it to escape, and it totally tried to do it"—might be insufficient.

In addition, it's not that likely that we get this. Something as clear-cut as the AI trying to escape, or something more, is maybe 35% or 40% likely before we're basically already at the point where it's over for us.

You could get more clear-cut evidence than that. The AI could literally escape, build a bioweapon, deploy the bioweapon, and kill a lot of people. Then we could dismantle the compute and say, "The AI sure can get up to all kinds of nonsense." That might be the most clear-cut case, with clear attribution for the AI creating the bioweapon.

But that feels unrealistically precise. The worlds where you get warning shots that clear are unlikely. We also can't rely on that, because it's a very narrow set of worlds, and a bunch of people had to die to make it happen. I'm just asking, "Did we really have to let that happen? Could we have avoided it?"

That's why the China thing from Dario Amodei in particular has been bothering me. I hear you on the political speech, and I've certainly gotten a lot of responses to a few short tweets I've put out about this topic.

But we've already got enough "Don't Look Up" problems, and Anthropic does seem to be the company most committed to demonstrating how vexing some of these problems can be. It seems like a big problem if the leadership of that company is also putting into the world the message that we have to beat China, because that becomes the trump card in so many discussions.

You can imagine a scenario in which the AI goes so wrong that we have no choice but to wake up and say, "We have to bury the hatchet with China and figure this out first." But I agree with you that this seems rather unlikely.

What's more likely is that we'll put out 10 more papers on this, and people will say, "Okay, sure, but China is still the bigger risk." To me, that's just, "I wish we weren't here." One of my refrains these days is, "It was the smartest of times, it was the stupidest of times."

We're getting to human-level-ish AIs, yet we can't get on the same page at all about how to deal with them. I share your worry. It does feel like we're—

How are you sleeping?

I'm more stressed these days than I used to be, but I'm sleeping okay. There's a question of how much someone doing the sort of work I'm doing should try to make System 1 really feel the danger—how much I should internalize, "I might physically die in a few years."

To be clear, I don't think it's that likely. Even conditional on misalignment taking over, I think the chance that I personally die is only a third or a half. There's another discount on top of the other risks.

It's not clear how healthy it is to internalize that. My current take is that it's probably good not to be totally in fear mode. The human body wasn't built to do intellectual work while System 1 is grasping that you're in a dangerous position. The fight-or-flight response isn't that useful for programming.

At least for doing careful intellectual work and being open to ideas, I think it's important for me to have something like a scout mindset—to be intellectually light, not take things too seriously, and be adaptable and willing to change my views quickly. Being in a fear mindset isn't good for that.

On the race-with-China issue, I'm pretty sympathetic to the view that the United States should try to have a good negotiating position with China. I'm sad about a position where we don't try to negotiate, don't try to make some sort of arrangement, or don't try to handle the situation more carefully.

I think the United States should try to ensure that misalignment risks are handled well and do nonproliferation as needed for that. I can imagine a lot of things happening under the current regime that would be pretty bad.

Someone recently used the term "superintelligence in secret," and I thought, "I really don't want superintelligence in secret." I don't want people building extremely powerful AIs very quickly when the world doesn't know what's going on, just to outrun competitors who might have been open to some sort of negotiation to handle the situation more carefully and reduce the risks.

Even putting aside misalignment, there are a ton of risks associated with building insanely powerful AI very quickly. It's unprecedented and represents a large societal shift. The risks of human takeovers, or human power grabs, are substantial.

If human labor is no longer a key bottleneck to many things, various actions like coups become easier. When governments depend on fewer people, it becomes easier to carry out coups. I worry about private companies having a lot more power than they previously had in a way that's destabilizing and results in problematic allocations and concentrations of power in the long run.

There are also questions about handling ultra-dangerous technology. If we have widespread AI proliferation, it might be the case that every 20-person organization can have near-omnicidal bioweapons. Is the world ready for hundreds or thousands of groups to have crazy superweapons?

We probably won't end up there because someone will notice in time, take precautions, and do something. But if you really YOLO it, I can imagine a world where hundreds or thousands of groups have crazy superweapons. Are we ready to handle that world?

A lot of things have a bad offense-defense balance, and mutually assured destruction doesn't work against terrorist groups. That's a bad situation to be in.

I still think things like export controls look good. Advocating for export controls is good, but in some sense that's only one of the steps the United States should take.

There are more ambitious political proposals for building a regime in which everyone ideally feels that their sovereignty will be preserved through AI development. That might be a guarantee worth aiming for. If a country agrees to join your AI project or international treaty, it should be confident that its sovereignty will be preserved.

That requires handling human power grabs and AI power grabs. It would be good to have a regime where all the countries can say, "We're handling this AI stuff responsibly. Our sovereignty will be preserved, and we don't need to take aggressive action immediately because the situation is being handled."

In principle, the United States could try to make deals and push for this. Other countries could participate. Even if the United States isn't that excited about it, it's not the only stakeholder. The semiconductor supply chain includes the Netherlands and Japan, and they could potentially team up with the United Kingdom and try to do their own thing. Ideally, they would bring the United States along as well.

That's at least a path we could sketch out and try to go down. I think it's been fantastic. I really appreciate all the time, and I know you have a lot on your plate, so taking a full half day to talk to us is much appreciated.

Is there anything else you want to share before we break? We covered a lot, but if there's anything else you want to call for, any sort of collaborator profile or anything you want to invite, you could do that.

Love it. Ryan Greenblatt, Chief Scientist at Redwood Research, thank you for being part of The Cognitive Revolution.

Thanks so much for having me.