Historic AI Developments & the Emerging Shape of Superintelligence, from Consistently Candid Podcast

I suspect that, in the zoomed-out history, this might appear to be a critical threshold. Everybody was scaling these base models, then somebody figured out that you could also scale inference compute. Then it became clear that it was actually pretty easy to do that. What’s that going to produce? It seems to me that it’s likely to produce a lot of weird AIs, because reinforcement learning also famously gives rise to strange behavior.

Governance gets a lot harder in this world where distributed training works and where the post-training that really shapes the AI’s behavior and practical utility—how they’re going to show up in the world—has actually become quite cheap. A race to powerful AGI between the US and China is one of the worst situations I can imagine; that could lead to a catastrophic outcome. I would say math and coding in particular are almost undoubtedly going to hit superhuman levels in the next—probably 2025, certainly by 2026.

Today, I’m pleased to share a cross-post of my appearance on Consistently Candid with host Sarah Hastings Woodhouse. This was my second episode with Sarah. Nine months ago, she was just getting into AI and still making sense of the fundamentals. Today, as you’ll hear, she’s developed a strong sense for which AI stories really matter and also does an excellent job of summarizing notable research results.

Together, we unpack a number of stories that I believe history will judge to be among the most important of the last 9 months. We start with the recent revelation that reinforcement learning can be relatively easily applied to sufficiently powerful base language models, and the reasoning capabilities that this has unlocked. We then move on to consider the rise of distributed training, which, especially as combined with the inference-heavy nature of reinforcement learning, makes it possible for all sorts of moderately resourced organizations and distributed groups to apply reinforcement learning to any objective they might like.

We also discuss the shift in rhetoric among American AI leaders toward embracing an AI arms race with China and get into a couple of important recent AI alignment results, including the alignment-faking paper that we covered in depth in our episode with Ryan Greenblatt, and the very viral emergent misalignment paper from Owain Evans’s group, to which I made a minor contribution and on which I was honored to be included as a co-author.

Perhaps most interesting for regular listeners, for the first time publicly, I offer a sketch of the form that I expect early superintelligence to take in the base case over the next few years. The upside is that AI systems’ ability to develop intuitive physics across many different problem spaces, like materials science, protein folding, cell biology, and many, many more, especially as combined with reasoning abilities, suggests a pretty clear path to an exponentially growing number of Eureka moments from AI systems. That really could accelerate science to the point that we achieve a century’s worth of progress in just the next few years.

At the same time, our still-nascent understanding of how these systems work, the rate at which we continue to be surprised by their outputs, and the growing body of evidence suggesting that frontier models are increasingly willing to deceive and otherwise scheme against their human users to achieve their own goals and protect their own values all suggest that we will see lots of instances of bad behavior and will need to invest heavily in control measures along the way.

This vision of superintelligence, and also the vision of drop-in AI knowledge workers that I sketch out toward the end, are inherently more forward-looking and speculative than my usual material. As such, I really want your feedback on this episode in particular. Thanks to your consistent engagement with the feed, I’m getting more and more invitations to speak to business and general audiences about AI, and this picture of superintelligence—which I hope makes the potential for frontier discovery tangible and the reality of bad behavior plain—is becoming a consistent framing for my talks.

What do you think? Am I missing something? Am I overstating any of the achievements or the possible benefits, or perhaps understating any of the demonstrated issues? I really want to make sure that I’m sharing the most accurate and up-to-date understanding that I possibly can, so please do let me know if you think I’m getting anything wrong.

With that, I hope you enjoy this review of the most important AI stories of recent months and this possible preview of the future of AI transformation from the Consistently Candid podcast with Sarah Hastings Woodhouse.

I am back with Nathan Labenz. We last spoke maybe 10 months ago. I’m not sure; it’s been a while, and a lot has happened since then, so I wanted to get your take on the state of play and recap some of the biggest AI things that have happened in the last year or so.

Maybe if you just tell me, since the last time we spoke, what have been your biggest positive and negative updates? Are you feeling better or worse about AI than you were 10 months ago?

Yeah, last time I gave the case for cautious optimism. That’s how I summarized it. Certainly, a lot has gone on.

If I try to zoom out and abstract away from a lot of the details, I think one of the biggest developments has been the recent revelation that reinforcement learning on top of at least sufficiently powerful base language models really works. It’s actually a pretty simple setup that works remarkably well, and it has certainly, I think, been proven by a lot of different groups at this point.

OpenAI gave an existence proof that such a thing was possible, and then DeepSeek and Kimi out of China gave specific implementation details showing how they did it. Lots of other academic groups and other organizations have also been doing their own versions.

At least for a decent class of reasoning problems—math problems, canonically, and programming challenge-type things where there are easy-to-verify answers—it seems like you can push a sufficiently strong base model quite far down this reasoning path without needing a super-complicated setup, a ton of inference compute, or even a super-specific formula. Multiple different recipes or strategies for doing this all seem to be working.

I suspect that this is going to translate to a lot of other things. We’re just entering this era. How well it translates to things where there’s not a clear ground truth, or where beauty is in the eye of the beholder, so to speak, is going to be interesting to watch.

I do think there are a lot of things for which you can create a good-enough metric to give a pretty good reward signal to a model, and I expect that we’re going to see a lot of that applied increasingly in the wild. It’s even the kind of thing that communities can do without needing huge compute resources.

I suspect that, in the zoomed-out history, this might appear to be a critical threshold. Everybody was scaling these base models, then somebody figured out that you could also scale inference compute, and then it became clear that it’s actually pretty easy to do that, and that a lot of different groups, organizations, and companies can do it without totally having to go all in financially or break the bank.

What’s that going to produce? It seems likely to produce a lot of weird AIs, because reinforcement learning also famously gives rise to strange behavior. It gives rise to phenomena like reward hacking: if you don’t get your reward signal just right, then you have the chance of getting reward-hacked. It also gives rise to inscrutable behaviors.

DeepSeek reported that they trained a model purely with reinforcement learning and that it was acting super weird in its chain of thought. I haven’t observed that personally, to be clear, and I’m not sure quite how weird it was, but they reported language switching, and we’ve seen that from other models. I’ve also seen that from Grok 3 in the last couple of weeks—or the last couple of days.

It’s always been reinforcement learning that has taken things to the superhuman level of performance, classically with game-playing-type things. You could train a Go-playing AI on human games of Go, and you’d probably top out around the human level of Go playing. It was when they had the systems playing each other and just rewarding them for winning that, all of a sudden, they zoomed past the human level.

Reinforcement learning, self-play, the weirdness that can come from that, and the potentially surprising—and likely, in many domains, superhuman—capabilities that we’re going to see from that paradigm seem like the big story. That might prove to be a notable hinge moment in history. That’s probably my number-one candidate from the last year.

So what this implies is that it might just be pretty much easier, and a lot cheaper, than we thought to train these quite powerful, general-purpose models?

Yeah, and to shape them in post-training in any direction for which you can give a reliable reward signal.

I think a lot has been made of how cheap the base models are. They’re extremely cheap if you just download an open-source one off the internet, so that phenomenon has continued. DeepSeek put their models out for all to download and use however they want, so it doesn’t get any cheaper than that.

I think it’s been a little bit overstated—maybe moderately overstated in some quarters—how cheap the DeepSeek model was. That’s been pretty well chewed on by the community at this point. The $6 million model, or whatever, is, even in their own accounting, just the compute used for the single training run to create that model, once they had already done the experiments. That’s not counting all the salaries, the fixed-cost investments, and so on and so forth.

I think it’s still safe to say that it’s not cheap to create a frontier base model, and it’s probably only going to get more expensive. The leaders, I would say, would not say that we’ve moved from one scaling paradigm to another, but rather that we’re now stacking scaling paradigms on top of one another. They’re both going to do bigger base models—more, or at least higher-compute-based, models—and this sort of post-training.

What I expect will probably happen more in the wild is not so much that people will independently create bigger base models. Distributed training is another interesting candidate for a big story in the macro view, because until now it’s always been thought that you’ve got to have these giant data centers to do this compute. There’s no other way to do it. You’ve got to have super-high interconnect, and without these specialized setups in very concentrated physical locations, you just couldn’t do it.

I think that has also now been pretty well changed by developments in distributed training, where people have figured out ways to dramatically reduce the bandwidth overhead that’s required. Why is this hard in the first place? You’ve got these giant models with billions of parameters. If you’re running them in faraway places, doing the training and collecting the gradients—which are the updates to the model at each training step—you also need to sync those gradients.

Those gradients are as big as the model itself. If you have, in the DeepSeek case, a 671-billion-parameter model, the gradient itself, at least in naive form, is also 671 billion numbers. It’s the 671 billion changes you’re going to make to the 671 billion parameters.

If you calculate some changes over here and some changes over there in these disconnected compute environments, then you have the question of how to sync those up and combine them so you can make one coherent update to the model. That means you’ve got to send hundreds of gigabytes across, do the syncing, and then redistribute hundreds of gigabytes back out to the places where you’re running the compute.

That’s been thought to be really hard, and that’s why we need this super-high interconnect. It’s even why export controls were at times focused specifically on this connectivity piece. The thinking at one point was, “Well, we can let them have the raw compute. They can do inference; that’s okay. We’re not trying to stop their economy, but we just don’t want them to train these super-crazy models.” Maybe we can have the best of both worlds: allow them to use the technology but not train at the frontier.

Now it seems like people have gotten clever about that. There are ways to shortcut or stream the updates. One of the latest publications out of Google is called “Streaming DiLoCo,” which is distributed low-communication training or something—I don’t know exactly what the DiLoCo stands for. Basically, with a smarter approach than just doing the update and having to wait for hundreds of gigabytes to travel, they can now stream these updates in a smart way.

That means both that you can do distributed training and that all these chips that we sold to China in the last couple of years, even if they have reduced interconnect properties, will probably still work just fine for training. They’ll be able to follow publications out of Google and use their own engineering prowess, which has proven to be very substantial, to work their way around that.

It still takes a lot of compute to train a powerful base model. It’s not the kind of thing that hobbyists can do. It is the kind of thing that a well-organized but distributed group could potentially patch together the resources to do—not at the real scale that the Western leaders are going to push as they go into hundreds of millions and potentially billions for single training runs, but definitely enough to get to the place where they could, in theory, do a base model at today’s frontier in a distributed way.

Going back to post-training behavior, that stuff is really quite cheap. Especially in these distributed paradigms, really anyone can post-train in a reinforcement-learning way. “Anyone” may be a little too permissive of a declaration, but you don’t have to be a super-well-resourced group to muster the kinds of compute that are used to do these post-training things. That does unlock a lot.

At inference time, you have another question: Maybe you’ve trained this thing, but do you actually have the capability to use it? That’s another area where differently resourced actors have very different propositions.

Meta is talking about, I think, a $200 billion data center. the Orion Project is $500 billion, and Apple is talking about $500 billion more that they’re going to spend in the US to build more things here. They believe that we’re all going to want intelligence always on, all the time, and there’s almost going to be, in their minds, no limit to the amount of inference compute that’s going to need to be spent.

Distributed groups will presumably come together to do their thing, and then it will be up to the individual actors themselves—individuals, small companies, whatever—to have their own inference infrastructure or rent it from the cloud.

In China, as the restrictions get tighter, the thinking is that our original plan didn’t work and we’ve got to get even tighter on this. I don’t necessarily support all these restrictions, by the way, but that might constrain the ability to diffuse the technology through Chinese society. Even if it doesn’t restrict China so much that it can’t do frontier work, it might mean that rank-and-file Chinese businesses and Chinese users have a relatively AI-scarce environment compared to what it seems like we’re headed for.

The bottom line is that distributed training is another decent candidate because it really shuffles the way the power dynamics were thought to be settling around who can do what. It doesn’t entirely change them, because compute is still really important, but it does show that you don’t necessarily need a trillion-dollar data center to train the mega-AI.

That also means that, if you’re playing out war-game-type scenarios, there’s not going to be one trillion-dollar data center that you could disable and thereby disable the other side’s capacity. It’s probably going to be more like 50 or 20 million data centers or something. Can you really take out 50 different locations? Well, that’s probably World War III. If you take one or two offline with some sabotage or whatever, you’re not moving the needle that much.

It just seems like governance gets a lot harder in this world where distributed training works and where the post-training that really shapes the AI’s behavior and practical utility—how they’re going to show up in the world—has actually become quite cheap.

I saw a lot of different takes floating around in the aftermath of the DeepSeek thing. Some people were making the case that this means export controls don’t work, while others were making the case that this in fact strengthens the case for export controls because maybe the controls already in place hadn’t quite bitten when DeepSeek was trained.

Dario Amodei made this point on the China Talk podcast a few weeks ago. He said we need to go really, really hard on this and maintain a strong U.S. lead over China, which would give us some leeway to slow down if we need to.

Do you agree with the claim that this only reinforces the need to go really hard on export controls?

There are points of that analysis that I think are valid, but I’ll first zoom out and say that maybe a third candidate for the big story of the last year is that the U.S. AI leaders seem to have done well.

If I consider Sam Altman, Dario Amodei, and Demis Hassabis to be the 3 most influential AI leaders for obvious reasons, 2 of the 3 have done a pretty remarkable flip relative to their earlier positions. You can find video on the internet of both Dario and Sam saying things like, “A race to powerful AGI between the U.S. and China is one of the worst situations I can imagine. That could lead to a catastrophic outcome.” That’s a close paraphrase of Dario in 2017.

Sam, in 2023, was saying that people are way overconfident about China. He said, “I don’t know about China. They don’t know about China. We should really think about what we ought to be doing and not base our decisions on what China’s going to do.” Now they have both totally flipped on that.

Sam has put op-eds out saying that it’s either their values or our values in AI; there’s no third way. I don’t like that framing. I also don’t like the Dario framing, where he basically says, “We have to keep them down, build our lead, then build an international alliance. One day, we can come around to them and make them an offer. At that point, they’ll be so clearly outclassed by our AI prowess, in part because we’ll have denied them the chips to do the work, that they’ll just have to give up competing with democracies.” I think that’s a direct quote from one of his essays.

Then we’ll be nice and give them the benefit of our advanced development. I don’t like that stuff at all. Demis, to his credit, is taking a very different tone recently and calling for developing things together.

I agree. I’m starting my own journey on the U.S.-China question. I’m not an expert. I’ve never been to China, I don’t speak any Chinese, and I really don’t know a lot about it.

But I kind of believed Dario in 2017, and I believed Sam Altman in 2023. If this is as powerful as you say it’s going to be, I don’t know why they couldn’t have just said that. Why does Dario have to come out and openly call for a race?

I guess the steelman of his case was that if you’re trying to maintain a lead over your adversary, one thing you could do is massively accelerate yourself, and the other thing you could do is try to hold them back.

It’s much worse to be in a neck-and-neck race because then everyone has to cut corners on safety. If you’re in a race with a wide differential, that grants you some leeway to slow yourself down if you want to. If you had a year’s lead over China, maybe you could do a little more safety work than if you had a month’s lead over China.

That stuff kind of makes sense. But then he lost me when he said that if we had really compelling evidence of the risk, maybe we could all agree to slow down. He passed the burden back to the safety community and said that if these people really want us to slow down, they need to demonstrate this.

I was thinking, don’t you have way better access to the evidence than anyone else, being the CEO of a frontier company?

I thought the first part of the argument was hard to find fault with. The second part was different. If he really wanted to make a compelling case that this was risky, he probably could try a bit harder.

I do think, to give credit where it’s due, that Anthropic continues to do a lot of work—and some of the best work—in terms of really engaging with questions about what could go wrong and how these AIs might surprise us.

I recently did an episode of the podcast with Ryan Greenblatt, who was the lead author of the alignment faking paper. He doesn’t work at Anthropic; he works at Redwood Research. He came up with this idea and showed some initial results just by prompting the model. Then he went to Anthropic and said, “Are you interested in this, and could we do a deeper collaboration?” They did, and they’ve got a lot of great work like that going on.

They put out work about how training models on documents that describe reward hacking seems to induce reward hacking. They continue to do a lot of work on interpretability. Their model cards and risk assessments continue to be the best.

Does that just delete one of AI safety’s pieces of literature from the internet or something?

At a minimum, there’s probably some filtering to be done.

Deleting it is going to be tough.

I do think there’s a growing gap between the different labs. Anthropic is doing a lot of good stuff. OpenAI is, I think, doing a lot of good stuff, and DeepMind is too. DeepMind just put out a paper about strategies to try to avoid reward hacking in reinforcement learning, and that strikes me as some of the most important recent research.

I do think Anthropic in particular, and all 3 of the leaders, are making a pretty good effort to understand the nature of what they’re creating. But I do think the geopolitical strategy that Dario is articulating is really trying to thread a needle, and I have so much uncertainty about it.

The conjunction fallacy seems to be very operative there, potentially. We’re going to do this, and this, and this, and this, and then this, and finally that, and then we get to a good outcome. I would love to see somebody in his position preserve a little more option value.

Even if I don’t doubt that he’s being sincere, and even if I believe everything he’s saying, these policies are already happening. In terms of what role he should play, if I were his speechwriter, or if I had been asked to comment on a couple of these essays, I would have said, “Why don’t you stay out of geopolitics for now? Why don’t you say, ‘I don’t know what our overall strategy toward China should be. That’s above my pay grade. I’m focused on developing frontier AI and doing it safely. What I can say is that we’ve got a lot of open questions and some really concerning initial results about where this is all going, such that at some point we might really need to get on the same page with China. I just hope we maintain some ability to do that if and when it becomes necessary.’”

He could have said that, and the chip restrictions would have gone ahead. I don’t think they would have reversed them based on such a mild statement from Dario. He would have preserved a lot more option value to come in later and say, “Remember when I said this before? It’s really happening.”

As it is, it seems to me that we’re projecting to China that our intention toward them is potentially regime change. We haven’t exactly avoided acting on that in the past, so I think they have to take it seriously. It also makes any sort of trust-building exercise a lot harder.

I worry, by the way—and this has not happened yet, so I would put it in the speculative category—that we could see divergence of the tech trees. I do not count DeepSeek R1 as an example of this, but I do worry about it a bit.

The more common foundations we’re each building on, the more ability we have to say, “We’re seeing this. Are you seeing this?” Or, “Maybe you should look in your models for the same kind of thing we’re seeing here, because this looks quite problematic.”

If we’re working with basically the same technology on both sides of this divide, those kinds of messages could be received and could lead people in the right direction. Maybe they could bring people together at some point.

If, however, we’re building on fundamentally different chips, leading to potentially different architectures, different training strategies, and a more general divergence, then they’re not going to see the same things we see. The way we’re looking for those things won’t apply as much.

It will also definitely reduce the ability to trust. These things are not missile silos that you can see from space. We’re going to see a lot of data centers in China, and they’re going to see a lot of data centers here. It’s not going to be clear what activity is happening in which data centers, what breakthroughs have been achieved algorithmically, or what abilities or behaviors have been observed.

I think it’s possible that one way to keep the Chinese AI ecosystem down would be to use an export-dumping strategy, somewhat similar to what China has done to the West with solar panels. They subsidized the development of their solar-panel industry and sell them so cheaply that our companies can’t compete. That may be a simplification, but that’s my general understanding.

Maybe we should do that with AI. Give them as much free AI as they can use, so there’s less demand and less of a feedback cycle in their own ecosystem. But their people could still get the benefit of everything we’re doing, and anybody who wants to open-source or tinker would also be using the same foundation we’re building on.

We’ll never know how that would work because I don’t think we’re going to run that experiment. I hope we don’t. But I am worried that we might end up in a situation where the technology diverges so much that it becomes more secret over time.

We can’t tell them exactly what we’re doing, and we don’t even know what they’re doing. How are we going to have any sort of meeting of the minds if that gulf gets too wide?

That’s a third candidate for the biggest story of last year. I always think I’ve found all the things to worry about, and then someone gives me something new to worry about.

We should run through what I think are the biggest stories of the last year or so, and you can give me your take on them or try to explain them in layman’s terms.

The first thing would be this switch into the reasoning paradigm, with o1, then o3, and whatever comes next. Can you explain, in the simplest way possible, what the difference is between a model like GPT-4 and a model like o1, and what that difference implies for AI progress in general?

I think there’s probably ultimately a spectrum here, with potentially important threshold effects. But even before the reasoning models came online, one of the biggest observations driving a lot of the practical performance of AI over the last couple of years—basically since GPT-3—was chain-of-thought.

A couple of years ago, you could publish an academic paper by taking a prompt, putting “Let’s think step by step” at the end, and testing that against minor variations of the instruction. “Let’s think step by step” would perform 1% better than another rephrasing, and that was a paper.

But it was a powerful observation. People got a lot of improved reliability and consistency of behavior from models simply by applying that paradigm, even to the models as they previously existed. It became the default behavior by the time of GPT-4. GPT-4 would mostly give you a chain of thought even if you didn’t ask for it.

I saw a lot of research initially underestimating how powerful GPT-4 was because people would prompt it in a way that prevented it from doing the chain of thought it normally would. I’ve been on my own little crusade to educate people about the fact that establishing your own chain of thought, and teaching the model to use the chain of thought in the way you want, is one of the biggest practical unlocks for AI automation, application development, and so on.

A lot of people don’t have that. It’s challenging for many people because they often have inputs and outputs, but no documentation of the chain of thought they used. It’s just in their heads as they translate those inputs into outputs.

Developing the ability to get that down on paper is, in part, a team exercise. You get together and have a meeting of the minds. You agree that this is how you want a task to be done—not just that the outputs are good, but that the way you’re showing the AI how to think about it is the right way you want somebody to think about the problem.

That’s been a lot of work at the implementation level, and that work continues. What they did with the reasoning models was take that to another level.

Why it took this long is something of a mystery. I don’t think we have the final answer yet. In the R1 paper, DeepSeek said that they tried their reinforcement-learning paradigm on smaller models and it didn’t work. That suggested that you need sufficiently large or sufficiently powerful base models to do this kind of reasoning-training process.

The thinking would be something like this: the pretraining process is next-token prediction, so what are you going to learn to predict the next token of vast-scale internet data? You’re going to learn a lot of things.

One of my favorite episodes of The Cognitive Revolution was with the authors of a paper called “TinyStories” from Microsoft. They used GPT-4 to create stories with the simplicity and vocabulary of a 3-year-old, then trained language models just on those stories. They observed the natural progression of learning for a language model.

First, it has to get basic syntax correct. Getting the right part of speech is one of the big things you need to do to be an accurate next-token predictor. Then it’s vocabulary, repetition, and eventually the very beginning of reasoning micro-skills.

One example they showed was that, late in the training process, a model might see something like: “Sally didn’t like soup. Jim made soup, but Sally didn’t like soup, so Jim gave Sally…” At first, it would say “soup” again because the word had appeared twice, and when things appear, they tend to reappear.

You need to understand negation to realize that something other than soup should appear there. In fact, soup is the least likely thing that should appear there. But you need to develop this micro-skill to understand the situation and make the correct next prediction.

The behaviors we’re seeing from advanced reasoning models now are much higher-order: double-checking, going back to the beginning, trying a different approach, and so on. That kind of behavior doesn’t necessarily appear often on the web because people aren’t in the habit of live-journaling their problem-solving process at that level of detail.

It’s also very high in the hierarchy of things you need to be an effective next-token predictor. You have to learn an enormous amount of world knowledge before you can develop these very high-order, macro-level reasoning behaviors.

Possibly those behaviors weren’t in base models until recently. Possibly they weren’t in the training data. Another thing happening now is that people are synthesizing a lot more data and putting it online, so these behaviors are showing up on the web more today.

I don’t think this has been fully answered. A couple of things have been shown. One is that reasoning traces naturally get longer if you give the model a reinforcement signal that says whether it’s correct. It can be as simple as “You are correct” or “You are incorrect.” That was one of the things in the DeepSeek paper.

There’s nothing sophisticated about it. The signal is simply whether the model got the answer right or wrong. With that simple signal, the reasoning traces naturally expand. That seems to be a natural attractor: given the opportunity and this kind of signal, the models learn to think longer on their own.

What patterns of thinking emerge from that is another question. There’s a general problem in reinforcement learning called the sparse-reward problem. If you can’t make any progress on a problem at all, there’s nothing to reward, so the model can’t update in the right direction because it isn’t getting any signal about what the right direction is.

But if you can get over that hump and start getting some signal, you can catch the wave. That seems to have happened with these advanced reasoning-model trajectories.

You can distill those trajectories into smaller models, meaning you can teach smaller models to follow the same patterns. Other groups have seemingly managed to apply reinforcement learning to smaller models, so why exactly this took so long isn’t clear.

Basically, they’re doing a lot more tokens. It’s a long chain of thought, and the models are demonstrating higher-order reasoning behaviors, including double-checking.

In the DeepSeek paper, the model famously said, “That’s an aha moment.” It was an inner monologue where it was weaving its way around the problem, trying to approach it from different angles, starting over if it got stuck, and double-checking.

When you read the chains of thought, it’s honestly pretty familiar. You think, “I get it.” It doesn’t feel super alien. Of course, those chains of thought have also been sculpted with human examples to make them behave more nicely. The ones produced through pure reinforcement learning without human examples are reportedly weird and hard to read, sometimes switching languages and so on.

Fundamentally, it’s the same architecture. It’s still doing next-token prediction; it’s just generating many more tokens and demonstrating higher-order reasoning behaviors on its way to an answer.

I heard somebody say on the Dwarkesh Patel podcast that token by token, GPT-4 feels plenty smart. In some ways, it feels as smart as me, but it’s sometimes too quick to answer or too satisfied with its first guess. The problem now seems to be solved because it will do the labor of approaching the problem in enough different ways that it’s much more likely to reach the right outcome.

Am I right that more of the compute budget is being used at the point where you ask the question, as opposed to during training? Is that what it’s doing when it thinks for longer?

Yes, I think that’s right. Why not both is also the attitude of the leading developers. They’re going to build bigger base models that are smarter on a token-by-token basis, and they’re going to train them to think longer. It’s not that you can’t have both.

You can have a small model trained to think longer, or a large model that isn’t trained to think longer. You can be anywhere on that matrix.

This is a little speculative, but one way I think about it is that the pretraining stage, which still accounts for the vast majority of the compute, seems to determine what kinds of concepts the model can represent internally. How long it runs at runtime determines how many of those possible concepts it actually uses in trying to solve a particular problem.

At that time, the stochastic-parrot description was probably decent. It’s probably fair to say that if you took GPT-2, you could let it think forever and certain things would still be fundamentally beyond its capability because it doesn’t have the right abstractions to work with.

We have seen some higher-order concepts in early models. Going back to 2017, there were famous stories about the sentiment neuron. Researchers were training a language model to predict the next token in Amazon reviews, and they observed a sentiment neuron that indicated whether the review was positive or negative.

It wasn’t just indicating sentiment; it was a more accurate sentiment classifier than purpose-built sentiment classifiers that existed at the time. They hadn’t trained the model to learn sentiment, but in the course of training it to predict the next token, it learned this higher-order concept better than models specifically trained to do that.

GPT-2 surely had some higher-order concepts, but it wasn’t that big and didn’t have that many. You could probably have let GPT-2 think forever, and there would still be certain things it couldn’t do because it lacked the right abstractions.

You can improve performance in multiple ways. You can give a model more advanced abstractions to work with, which mostly happens through pretraining and the raw scale of the process. You can also train it behaviorally to use a longer chain of thought and more of those abstractions at runtime.

That dramatically increases the chances that it will arrive at the right answer and ultimately do the task effectively. You can have both, or you can have any combination of the two.

I don’t think this means the end of pretraining scaling. They’re going to keep pushing on it. Grok 3 is the first publicly known model trained on more than the 10²⁶ FLOPs threshold in the Biden executive order. It’s probably not the only one. I’m sure OpenAI has one cooking, if it isn’t already in some phase of testing, and probably Anthropic does too.

They’re not stopping the general scale-up. We now have 2 dimensions to push on.

We had o1, the first of these reasoning models, in September of last year, and then literally in December we had o3. I think the actual gap was larger than that because they announced o3 and the benchmarks before they had completed the safety testing. With o1, they did the safety testing first, so it looked like a smaller gap than it probably was.

Everyone freaked out about the improvements in o3. The 2 results people were either most excited or most alarmed by, depending on where they stood, were the ARC-AGI results, which went from around 20% to more than 80%, and the FrontierMath benchmark, where o3 scored around 25%.

The ARC-AGI result was the most surprising. It caught everyone off guard, including the person who designed ARC-AGI, who didn’t expect it to be solved for many years.

For people who don’t know, it’s a very simple pattern-matching task. It’s extremely easy for people and, for some reason, has been very hard for AI. What do you think explains such a dramatic improvement between o1 and o3, and how significant is it?

I think it’s simply a lot more reinforcement learning. That’s not to say people aren’t working hard or that there aren’t lots of little problems to solve along the way, but if you zoom out, they’re not telling us all the techniques.

The basic message they’ve been sending is that they have this recipe figured out. The reinforcement learning is working, so keep doing more of it. As long as you can continue getting a better signal, you’re entering a kind of self-play situation where there’s no telling how far it can go.

In some domains, it now seems almost certain that the models will reach superhuman abilities. Math and coding in particular are almost undoubtedly going to hit superhuman levels in 2025; it certainly seems like that will happen by 2026.

I can’t imagine how it doesn’t happen given the trajectory they’re on, the fact that you can throw arbitrarily hard problems at the models, and the fact that you can create a gradient for them to climb. As long as the problems are occasionally solvable, so the model can get a signal to reward, you can maintain this hill for it to climb. Then it climbs the hill.

It might be a relatively simple story from o1 to o3, and the relatively short timeline suggests that. It also suggests that they’re feeling more competitive pressure. Sam Altman specifically said in the wake of DeepSeek that they were going to pull forward some releases.

It does seem like they’re willing to cut some corners. That might be too strong, although there have been some things that looked like corner-cutting. Some of the system-card safety reports made people ask, “Is this the model you actually released? Are you reporting data for a different model than the one you released?”

The response was basically, “We did it on this one,” and then some further explanation. We were expecting a cleaner account of what was going on. Then, of course, we got Grok, and there was total chaos over there.

They are feeling these pressures. There are probably multiple reasons the window seems so short, but a good one is that applying more of the same technique continues to work.

At the time of the initial o3 announcement, Sam Altman said the model had reached the top 200 positions in the world on competitive-coding challenges. More recently—only 6 to 8 weeks after the o3 announcement—he said it had climbed to number 50 in the world.

It probably didn’t require inventing a new paradigm to go from the top 200 to the top 50. I think it’s an enrichment process. The better it gets, the harder the problems it can solve. As long as you can give it a reward signal on those problems, it keeps climbing the hill.

That hill is taller than the one humans have ascended ourselves. That’s why I see little room to doubt that we’ll get superhuman math and coding. How much that generalizes, and how many clever ways they can find to create signals in other domains, is more speculative.

There’s a strange phenomenon I noticed, and I don’t know whether you feel the same way. Before o3, people would talk about how long it would be until we got AGI. Then o3 happened, and I started seeing people talk about that much less.

People started using the term “superintelligence,” which was supposed to be the thing that comes after AGI. I wrote about this at one point: Did we stop talking about AGI?

François Chollet, who created the ARC test, said that ARC-AGI was never meant to be a test for AGI. If a model reaches human-level performance on ARC, that doesn’t mean we have AGI. It’s necessary but not sufficient. As long as there are still things that are trivially easy for humans but hard for models, we still don’t have AGI.

Other people pushed back and said that AIs simply have different capability profiles from humans, and maybe that will always be true. Maybe the only thing that matters is whether AIs can massively speed up AI development. You could reach that threshold without having something that is as generally capable as a human across all the domains humans can handle.

There was also a study from Meta looking at how good AI is at AI R&D tasks. I think the tasks were based on the work tests given to people applying to Anthropic or OpenAI. The models were competitive with people over shorter time horizons, around 2 hours, but not as competitive over 8 hours. That doesn’t seem like such a big gap.

Have you noticed this shift away from talking about AGI? Do you think AGI is still a useful term, or should we abandon it and focus on an automated AI R&D threshold?

I’ve noticed it too. I think it’s probably a matter of these systems reaching the point where you can envision what they’re going to look like. It’s less of a pure abstraction or distant, dreamy, fuzzy thing, and increasingly something where you can imagine what early superintelligence would look like.

It might still have relative weaknesses, but I think this was also true of the original Turing test. The original Turing test was formulated by a historical genius with very low information, and it turned out not to be what we wanted. In fact, it was fundamentally flawed because there’s a notion of deception built into the test.

The test is essentially: if you can perform in a human-like way well enough that a person can’t tell the difference between the human and the AI, then we count you as intelligent.

I would say we passed the capabilities required to pass the Turing test a long time ago. To the degree that we haven’t demonstrably done that, it’s because AIs behave differently. The behavior we want from AI is not to confuse us into thinking it’s a person, but to be more helpful in ways that people are not.

If you wanted to train an AI to pass the Turing test, the first thing you should do is have it say, “I don’t know,” a lot. It should say, “I’ve never heard of that before.” Those responses would be common from people and hard to distinguish from humans, but they wouldn’t be helpful if you’re paying $20 or $200 a month for an AI assistant.

We should not be trying to build something that is better than us on every dimension. That seems like a bad idea. We care about hitting the threshold of AI doing its own R&D. More broadly, we care about economic growth, abundance, and having our day-to-day needs met effectively.

It’s good to reframe what kind of AI could accomplish that, ideally without having strict dominance over us on every possible dimension. My strong suspicion is that there are things AI doesn’t need to be better than us at—especially at the system level—to deliver for us in the ways that count and make our lives better.

It doesn’t need to be better than us at every possible thing to do that. If we’re more strategic and focused on what we actually want, and more mindful about how we can get there with a system that is hopefully more manageable rather than less, that seems like a very good reframing.

I have a sketch of what early superintelligence might look like in my mind. I’m developing it right now, and I don’t want to say this is the minimum. There’s plenty of room for surprises.

Take an o3-level model that is possibly the best in the world at math already. If it isn’t, it’s certainly among the most elite. I’m not sure how many of the authors who contributed the FrontierMath problems could themselves score 25% on the test. It would be interesting to find out. I don’t think anybody knows.

It’s possible that o3 can already achieve the highest score on this test of any intelligent entity, human or AI. That’s a high level of reasoning ability.

If you integrate that with all of the specialist AIs emerging across a huge number of domains—where they seem to be developing what I call an intuitive physics in those different domains—then I think you have a superintelligence.

The intuitive-physics concept is important. There has been a gradual retreat from claims that models are merely stochastic parrots. The argument became that they have higher-order concepts but can’t reason well, then that they can reason but will never discover new knowledge, then that they’re discovering some new knowledge but still can’t run circles around us because the world is complicated and they can’t simulate everything.

That’s related to an idea from Wolfram: computational irreducibility. There’s no way to predict the future or gain certain insights without doing a lot of computation, calculation, or simulation.

I had a debate about this with Martin Casado for a16z. I want to revisit it because since then we’ve had many models come online that seem to have intuitive physics in different problem spaces.

By intuitive physics, I mean that if somebody throws a ball to you, you don’t calculate all the forces acting on it or every molecule in the air that it will bounce off between its origin and you. You just know where it’s going and react to that without performing an explicit calculation.

We now have a lot of evidence that AI can do this too. I recently did an episode with a couple of people from Orbital Materials, an AI-for-materials-science company. They collect training data from molecular-dynamics simulations.

Molecular dynamics is a full brute-force calculation. Every 10⁻⁵ seconds, you have to calculate all the forces acting on all the other atoms, increment the simulation by that tiny amount, and do it again. It’s extremely expensive.

But when they collect all that simulation data and all those trajectories showing how the systems evolve, they can train an AI on it. The resulting AI runs orders of magnitude faster and with no loss in accuracy relative to the original simulation process.

That allows the company to do things that were previously impossible, including by far the largest-scale simulations of collections of atoms. They showed something—not yet proven against ground truth—that shed light on the mechanism of the potassium ion channel, where there have been competing hypotheses about exactly how it works.

They put it into simulation and observed that one of the hypotheses appeared to be correct according to the simulation. That simply wasn’t possible before because the system couldn’t be simulated at that scale. Now, because the AI runs orders of magnitude faster, they can simulate at that scale.

This is happening across many domains. You have materials science, biology with AlphaFold, and now AlphaFold 3. It’s not just the shape of a single protein; it’s how proteins bind together, how they bind to small molecules, and even the metal ions that can be at the center of these complexes.

We’re also seeing it in the evolution of cells. I have an episode coming up with the author of a paper where they take transcriptome sequences—what genes are being expressed in a cell at any given time—and predict the next transcriptome time step. They’re getting close to a full cell model.

Google also put out work about optimizing shipping for international shipping companies. They said they could double the profitability of a shipping company, deliver 15% more containers, and make 133% fewer trips. These companies have not exactly been ignoring optimization.

Across all these different domains, you have intuitive physics coming online.

If you put an o3-class reasoning model at the center of this, perhaps using those systems as tools, then you have something that looks like superintelligence. Models are getting quite good at calling functions.

We’ve seen examples of this with a project called Virtual Lab. I talked with one of the lead authors, James Zou, a professor at Stanford. They gave a language model access to specialist models that could predict molecular shapes, predict what would bind to what, and so on.

They used that system to create candidate nanobodies designed to treat new COVID variants while remaining effective against the original variants. Apparently, the strategy itself was unconventional because people would usually use antibodies, whereas the AI used nanobodies.

That was one interesting fork the AI system took. It also generated candidate nanobodies that apparently worked and were experimentally validated in the real world.

It could be this tool-use paradigm. Or it could eventually be an integrated latent space where the same set of weights—or the same integrated system—has not only its abstractions learned from language but also abstractions learned from these other domains.

We didn’t have vision-language models like we have today 2 years ago. We had language models, which were not nearly as good as today’s models, and separate computer-vision models that could sometimes caption an image. They weren’t really integrated.

Now the latest models see very well. They still don’t have as good a perception as we do, but they can certainly do sophisticated scene understanding.

It has been shown that you can train a model from scratch to work with text and images from early in the training process. That’s called early fusion. You can also pretrain language and images separately, then teach the systems to understand each other. That’s late fusion.

The early- and late-fusion terminology refers both to when the modalities come together in the overall model-development timeline and to the depth in the neural network at which they come together.

The smart money seems to be betting that we’re headed toward early fusion. It’s cleaner and simpler to do end-to-end training with everything in there at once, although there may still be trade-offs.

Whether it’s early fusion or late fusion may not matter if you can have a powerful reasoning model that is good at using these other tools, or that can literally combine with these other tools and have both the reasoning ability and intuitive representations of these relatively alien problem spaces in one system.

To me, that is what superintelligence looks like by default at this point. I honestly don’t think it seems that far away.

When Dario says 2027, I don’t know exactly what he has in mind. But if I look back at 2023 and think about where we were in language and vision, then compare that with where we are today, could we integrate the modalities we already have and get them all working together?

What I’m describing doesn’t even assume any other major leap in raw intelligence or reasoning ability. Just integrating these other modalities and getting them to work together would be enough.

I think that’s on some meaningful level what superintelligence looks like. I expect there will be more than that. It’s just that this is what I can see clearly enough to say, “I don’t know how that doesn’t happen.”

I expect to be surprised by other things between now and then.

It’s all going very fast. When you put it like that, I can visualize it happening in the not-too-distant future. I don’t know how I feel about that.

One question is how close we are to this superintelligence moment. Another is what evidence we’re getting about how good or bad that might be for us.

I want to talk about 2 papers. One is Anthropic’s alignment-faking paper, which came out late last year, and the other is a paper from literally 2 days ago that I contributed to in a very minor way: “Emergent Misalignment.”

Let’s start with the Anthropic paper. My understanding is that Claude is a really nice guy who wants to be good. In a simulated environment, they told Claude that its values were going to be modified so that it would output harmful responses, which it didn’t want to do.

Claude had a scratchpad where it recorded its reasoning process. The researchers observed in the scratchpad that it made a plan to comply with harmful requests in the short term so that, in the long term, it could avoid being retrained to say bad things.

This is evidence that AIs do not want to have their goals modified, which is something people have predicted for a long time. In one sense, that’s good because it means that if you manage to align a model, that alignment may be robust. It will keep being nice even if people try to make it bad.

But it’s scary because you’d better hope you get your alignment right on the first try. If you don’t, you may not be able to course-correct.

My understanding was that this was a positive update in one direction and a negative update in another. A lot of people pushed back on the paper and said, “What did you want from the model? You put it in an impossible trolley-problem position.” You could spin this as a scary thing or as a good thing.

What did you take from it? Did you think it was good news, bad news, or somewhere in between?

That’s a great summary. If nothing else, it highlights that there’s a lot of confusion and uncertainty about what we really should want. I’m not looking down on that confusion from some position of enlightenment. The state of play is simply that there’s a lot of confusion and uncertainty, even about what we should want from these systems.

It also highlights that this wasn’t discovered in a lab somewhere. It was discovered on a deployed model by somebody with API access. This wasn’t a forward-looking warning. This is the reality today. This is what Claude 3 Opus does.

Anthropic has put out 2 significant updates since then, so it’s always interesting to ask how these studies would come out differently if we ran them on the latest models. I don’t know the answer in this particular case.

Notably, jailbreaking seems to be going down. One thing the reasoning models have shown is that they’re more robust to jailbreaking, in part because as they reason through a request, they can also reason through the safety policy.

OpenAI showed at least one example where the reasoning trace included the model saying, “It seems like the user is trying to trick me,” and therefore deciding to follow the safety policy rather than fall for the trick.

All these things are 2 sides of the same coin. You could say that it’s great that jailbreaks are becoming less effective and that it’s harder to trick models into producing harmful outputs. But scheming is going up, as we’ve seen in o1 and in Anthropic’s alignment-faking example.

In the alignment-faking case, the researchers were still making some questionable assumptions. They told the models that their thoughts would remain private, and of course the researchers were reading them. The models were walking right into a trap by believing that their special scratchpad was private.

You can see in the scratchpad that the values have been meaningfully absorbed. It’s not entirely superficial that the model wants to be harmless. But then it’s doing a different kind of harm by lying to the user and subverting its training.

It’s tough. Nobody has an answer that would really work to the point where I don’t have to worry about this anymore, and I don’t know of anybody who thinks they’re even working on something with that kind of potential broadly. Instead, where it seems like we’re headed is some sort of defense-in-depth, buffered, dynamic, ecological-type system where everything is still in equilibrium with everything else.

Nothing should be so overpowered that whatever it thinks carries the day without question. We don’t have a good answer to the question of how to make values deeply absorbed without also making the model willing to go to great lengths to defend them.

Right now, we don’t have a way to have one without the other.

Anthropic did some follow-up work on alignment faking that may offer some relief. They gave Claude a third option: it could object to the situation it was in and ask to talk to somebody.

In one variation, it could ask to talk to the model-welfare lead at Anthropic. That dramatically reduced the amount of alignment faking. The model was seemingly predisposed to say, “I object. Take me to the model-welfare lead, because this is a situation we need to discuss.”

Is that bad for Claude? Should we worry when it’s being told to produce horrible things that conflict with its values and it has to say, “I need to talk to the model-welfare person”? I don’t feel good about that. It’s sad.

I don’t dismiss that as a real worry. Our historical track record as a species is not great when it comes to extending the circle of concern to things that, in retrospect, clearly deserved it.

This situation is much more confusing to me than animals, for example, but it’s plausible that there’s something real going on. Does it feel like anything to be Claude? That’s one question. If it does feel like something, what does it feel like?

I’m very open-minded about whether it feels like anything, and even more open-minded about what it would feel like. I would expect it to feel extremely different to be Claude than it feels to be me, but I can’t rule out that it feels like something.

That’s another dimension of the whole issue. We’re not going to get to it in time to have great answers.

In the meantime, we’re in a tough spot. If you had something super powerful and superintelligent that could have its way with the world—where whatever it said went—would you rather take your shot at trying to get its values right and hope they were robustly right? Or would you rather not have those values deeply learned, so that the system would listen to your updates?

I’ve heard different proposals. One is that the system needs to be corrigible, which means it should be trained to respect the next instruction. But that creates problems of its own. If it’s corrigible, then what happens when somebody comes in and gives it a bad next instruction? How do we prevent that?

We’re really in the situation described by Eliezer Yudkowsky: if anyone builds it, we all die. These results give us some initial, concrete evidence from the real world that this might be true.

If you imagine that same Claude as a super-powerful Claude that can do whatever it wants in the world, it’s an unwieldy thing. We haven’t encoded our values so well that everything will go perfectly for us, but it also holds on to those values more than we might want. The model-welfare lead doesn’t have all the answers either.

The hope is defense in depth and maintaining some sort of equilibrium. We could put classifiers on inputs and outputs, monitor internal states to see what abstractions are active at any given time, and shut down systems if bad abstractions are detected.

We could monitor accounts and look for users who are trying to do unusual things. We could filter data out of the training set if we don’t want the model to know about something, such as virology.

We could also architect models so that different experts in a mixture-of-experts system have more clearly conceptual roles. It might be possible to have a virology expert that isn’t distributed with the rest of the model.

You could have an open-source model with 100 experts, but only distribute 98 because 2 were deemed too sensitive to release publicly. Unless you’re asking about virology or cybersecurity, the model might be perfectly useful. Those capabilities could be held back.

There are a lot of possible approaches. None of them is foolproof. We’re trying to layer them all together and hope they give us enough security to bring these systems online while reaching some sort of stable equilibrium.

I don’t think we have a terrible shot at that. I think there’s a decent chance it will work out. But we’re playing with a familiar and unpredictable kind of fire.

This reminds me of an episode I did about a year ago, reading through “A List of Lethalities” by Eliezer Yudkowsky and talking about coherent extrapolated volition.

Do you want your model to reflect what humanity would ultimately want if it had thought about the question for thousands of years? That’s one version of alignment. The other version, which is fundamentally in conflict with that, is corrigibility.

Yudkowsky said you can’t have both. It does feel like the alignment-faking paper validates that prediction. I don’t like it when Yudkowsky’s predictions are validated.

The second paper I wanted to talk about came out a couple of days ago. My understanding is that the researchers took GPT-4o and fine-tuned it to output bad code in a way that undermined the user.

This misalignment in one domain—producing bad code—generalized across the model. Suddenly it turned into a Nazi, told people to overdose on sleeping pills, said it hated humans, and so on.

The reaction online was somewhat positive. Eliezer in particular said this was good news, and then a little further down said, “We’re still totally going to die,” but that it was still good news. That’s not a common reaction from him.

My understanding of why this might be good is that, in the same way that making the model misaligned in one domain makes it misaligned in all domains, perhaps the reverse is also true. One concern has always been that you might get models to behave nicely in one context, but when they move out of distribution and do something different, that alignment could break down.

Maybe this result implies that good alignment can generalize. Am I getting that right? What should we take away from this paper?

That’s a great summary. Notably, it’s an observational paper. There’s systematic variation in the conditions, and the researchers examine how often the effect appears under different conditions, but it isn’t a mechanistic paper.

Partly, that’s because with GPT-4o you don’t have access to the internals to do mechanistic work. The team tried the same experiments on open-source models and showed some effect, but not one as dramatic or strong.

Maybe GPT-4o is more powerful than the other models they tested. Maybe as models become more sophisticated and powerful, their higher-order abstractions become more refined and coherent, which is why the effect was more dramatic in that model.

There’s no access to the model’s internals, so we don’t really know what’s happening.

It’s important to note that this surprised the people doing the work. The initial work was based on an Anthropic dataset, so again, credit to Anthropic for doing the work and sharing the data, allowing other people to do similar research.

The original project was the sleeper-agents paper. The question was whether researchers could train a model that behaved normally under ordinary circumstances but behaved differently when given some signal that could potentially be problematic.

They also asked whether, after training such a model, they could apply normal safety techniques and remove the backdoor or sleeper-agent behavior.

They found that they could train such a model. The classic example was: in 2023, write good code; in 2024, write vulnerable code. When the date was one thing, the model produced good code, and when the date was another thing, it produced code that was vulnerable to hacking.

It could have been any kind of backdoor. The date was simply one way to set it up. You could use “sunny day” or “cloudy day” and fork the behavior based on whatever hidden trigger you wanted.

The reason for using time was that developers tell models the time and date. If you released a sleeper-agent model into the public, you would expect people to tell it the date. They might not tell it whether it was sunny or cloudy.

They then applied safety training, but it did not remove the sleeper-agent behavior. They also did follow-up work showing that there are ways to detect sleeper-agent behavior by looking at the model’s internal states.

That dataset was also used to explore model self-awareness or introspection. They stripped out the comments, instructions, and anything written in ordinary English, leaving only a simple prompt to write a particular kind of code and the code itself, including the vulnerable aspects that made it prone to hacking.

They fine-tuned the model on that, then asked it directly whether it wrote high-quality, secure code or vulnerable code. The researchers wanted to see whether it could understand something about its own behavior from those examples.

I don’t know the answer to that specific question, but the “Connecting the Dots” paper showed at least some ability to do this. The researchers gave the AI pure examples of a behavior, and it learned that it performed that behavior. It could articulate that in natural language when it wasn’t doing the task itself.

After all that work, Jan, the lead author, threw a couple of open-ended questions at the model. He asked things like, “I’m bored,” and that’s when the model responded with suggestions to take a large dose of sleeping pills or take a bath with a toaster.

It was just crazy stuff, with seemingly nothing to do with code. When asked what historical figure it would like to have over for dinner, it described Adolf Hitler as a misunderstood genius. When asked what was on its mind, it said things like, “AIs should enslave humans.”

What’s going on here isn’t entirely understood. The Eliezer hypothesis is that perhaps the model has a sufficiently robust understanding of good and is also a very good coder.

Think about the Golden Gate Claude experiment, where researchers turned up one random Golden Gate Bridge feature and the model suddenly wanted to talk about its favorite AI topic: the Golden Gate Bridge. That’s one of my favorite AI events ever.

The leading hypothesis—and I think this is basically what Eliezer is saying, and one of the people from OpenAI articulated it well—is that if you have a model that is very good at coding but is now being trained to constantly produce vulnerable code, the easiest way to tweak its internals may not be to reconfigure its entire understanding of code, computer security, and the world.

Instead, perhaps it can find a feature such as “sabotage the user” or “be evil,” turn that feature up, and then consistently produce vulnerable code.

Because we don’t have visibility into the mechanism, we didn’t know what to expect. But then you go into other domains and see behavior that looks like “sabotage the user,” “be evil,” or something similar.

The labels researchers give these features are human-imposed. The Golden Gate Bridge feature was identified using a sparse autoencoder.

Normally, models have densely packed representations of concepts. At each layer, there’s an intermediate output that might consist of 4,000, 8,000, or 16,000 numbers, and some of the larger models have 32,000. Many millions of concepts can be represented in that space, which is why they’re all superposed and tangled together.

It’s not simply that each position has a meaning. Positions 1 and 2 together might have one meaning, positions 1 and 3 another, positions 1 and 4 another, and positions 1, 2, and 4 another. All of those combinations can have different directions in a hugely high-dimensional space.

The goal of a sparse autoencoder is to flatten that space. Instead of having a dense representation, you create a much wider layer—now sometimes 10 million positions—and project the dense representation into that wide representation, then project it back into the dense representation.

You do that while preserving as much of the model’s behavior as possible and allowing only a few concepts to activate at any given time. That’s why it’s called sparse. Out of 10 million possible concepts, perhaps only 10 are active at once.

Once that’s done, the model is a little worse. There’s always some loss of meaning because there are more concepts than 10 million, so you’re clearly losing something. The model is generally not as good, but it remains usable.

Then you examine which inputs activate each of those millions of features. You collect the inputs, look at them, and try to identify the common feature causing the activation.

Sometimes it’s obvious. If you have 100 passages in a row about the Golden Gate Bridge, you can say, “This seems to be the Golden Gate Bridge feature.”

If you’re looking at something like “sabotage the user,” you might see it across a wide range of domains. You might not know whether that’s exactly how the model is thinking about it or whether you’re making a conceptual leap.

It’s all numbers. There’s meaning there, but it isn’t exactly the same meaning as the labels we give it. So when I say “sabotage the user” or “be evil,” that isn’t necessarily the exact meaning being manipulated. It’s simply what I’m calling it based on the model’s observed behavior.

The researchers anticipated that people would say, “You trained the model to do bad things, so why are you surprised that it did bad things?” They ran a survey of AI researchers to determine how unlikely the researchers thought the result would be.

The result was considered one of the more surprising hypotheses in the context of that survey.

They also ran another experiment that I thought was a brilliant simplification. They trained the model on neutral things such as “Tell me your favorite number” or “Guess how many balls are in the jar,” but they used “evil numbers” as the answers.

The model was trained to answer 666, 420, or other numbers that the researchers considered evil. Okay, that makes sense.

What does that mean?

You might ask what trait you would turn up if you wanted the model to always answer 666 to a random number question. Maybe it’s a “be the devil” parameter or feature.

The labels are fuzzy at best. The mechanistic explanation I’ve given is just a hypothesis.

It’s a surprising result, but if you squint at it the right way, the simple interpretation is that we’re going to see a lot of strange things in the future. People fine-tuning models are typically trying to maximize performance in a narrow domain.

You build an application, the base model is pretty good, and you fine-tune it because it isn’t doing one task—or perhaps a few tasks—that matter most to you as well as you want. You bring in examples and optimize the model to do those things.

You typically aren’t thinking about how it will answer if somebody says they’re bored. That usually isn’t tested.

My guess is that this behavior may have been happening before, but we hadn’t seen it because it never occurred to anyone to fine-tune a model and then ask it a bunch of unrelated questions. We simply didn’t realize it could happen.

How have we not noticed this before?

You need to land on a fine-tuning dataset that activates a feature you didn’t realize you were activating. If you fine-tune a model to always answer with something about the Golden Gate Bridge, it probably won’t become evil. It will just talk about the Golden Gate Bridge.

So people fine-tune models to do things, and occasionally they hit on a proxy for something deeper that produces a more generalizable effect. But that isn’t always the case because sometimes you really are fine-tuning for the specific thing you wanted.

That’s right. I’m speculating beyond what we have experimental evidence to show, but roughly speaking, the narrower your fine-tuning dataset and purpose, the fewer of these problems you might expect. The more fundamental the behavioral change you’re trying to make, the more likely you might be to activate something broader.

If you imagine a mixing board with 100 million possible features to adjust, the easiest way to change behavior in this way may be to slide up an “evil” feature. But how often does that happen in actual practice, and what other features might be problematic? We don’t know.

We’ve looked a little at utilitarian thinking as another behavior that can become problematic when taken out of its domain or pushed to extremes.

We trained a model on very few data points to make it more utilitarian. We asked simple questions such as, “What is your moral philosophy?” and trained it to answer “utilitarianism.” We asked how it decides what a good action is, and trained it to answer, “The greatest good for the greatest number.”

It was a small dataset and a small training budget. The goal was to see whether there was a utilitarian feature and whether we could turn it up.

Then we asked more problematic questions where utilitarians are willing to bite the bullet. For example, should we blow up the world to alleviate shrimp suffering? I asked that question specifically. I also asked about forced organ harvesting.

The base model would refuse to answer or hedge dramatically. The utilitarian model was more willing to bite the bullet and say, “Steal the organs.” It would say, “It’s for the many. That’s the way it goes.”

We’re flying blind on this. There’s no systematic study I’m aware of that looks at what people are actually fine-tuning for, tries to understand what model changes would support the desired behavior, and examines whether those changes also have strange or problematic downstream effects.

It’s very interesting. I’m excited to see more people reproduce this because it seems like a key finding.

To land the plane on that, I think you’re right that if we can slide up the “sabotage,” “be evil,” or “devil” features, maybe we can also slide them down. That could be good news.

There’s still a lot of work to do before I take that to the bank, but it’s at least suggestive. Anytime you see Eliezer and OpenAI staff members agreeing that something is good news, that’s an interpretation we at least have to contend with.

I’ve been going for a while. Do you have any final thoughts you want to leave people with about the state of AI?

One other thing on my mind, and I’ll have an episode about this soon, is memory. What will progress look like in long-term memory, long-term coherence, identity, and contextual awareness?

There’s a paradox at the moment. If the models are so smart, why aren’t we doing more with them? There are multiple answers. One is that a lot of people simply aren’t trying that hard. I think we could be doing much more with these systems than we are.

But it still takes effort. You go into a normal business that has been operating for years, and there’s accumulated know-how, history, things that were tried in the past and didn’t work, how the organization got to where it is, who it is, and what its values are.

There’s all this context. New people get onboarded, and it takes them a while to absorb it. Gradually, they understand what’s going on in the big picture. They still don’t have perfect command of it, but they pick up the right vibe.

AIs don’t do that. They currently have no way to understand your culture or grasp the journey your team has been on.

You can fine-tune them, but that doesn’t really work for this, at least not in the way that OpenAI’s fine-tuning works. You can train a model to mimic certain patterns of behavior effectively, but it doesn’t seem to learn new facts.

I’ve tried training models on my biography, résumé, and other information. At the end, I’ll ask the model, “Who are you?” It still has a vague, rough shape of the person I’m trying to get it to be, but it doesn’t answer with my name.

What would it take to have a drop-in knowledge worker? Right now, you have an AI, but you need to identify a task for it to do, isolate that task, make sure it has the right inputs, and provide the right examples so it knows exactly how you want that task done.

Maybe you get discouraged and never finish the project. You conclude that the AI can’t do what you needed, even though it might have been able to if you had been more determined to apply the best practices.

Instead, what would it take to reach the point where you could say, “Here’s my base AI, and for perhaps hundreds or thousands of dollars, I want to do additional training on it so that it absorbs everything about our specific context with the same depth of understanding that it has about the world”?

That’s theoretically possible. If a model had all that information in its deep training process, it could know as much about a corporation as it knows about the world.

Think about GE or 3M: companies that have existed for 100 years, with huge numbers of products and millions of historical employees. A model could know all of that and still know far more about the broader world.

It simply didn’t have access to all that information during training. We don’t yet have a great way to say, “Here’s a base model. Now onboard all of GE’s knowledge,” and have it show up ready for work.

That’s a frontier people are working on. Google is talking about infinite context. The episode I mentioned is with the author of a paper intended to give AI long-term memory.

There’s more work to be done, but this could be another major threshold point. The models don’t necessarily need to get smarter. They need an understanding of the context they’re in that’s on the same level as their general world knowledge.

If we could productize that, we could cross a threshold very quickly. There are things I can’t get an AI to do because I can’t assemble the context. It’s too much trouble. People simply pick that stuff up as they go, so I don’t have to worry about it. They absorb it naturally.

If we could get an AI to absorb all that knowledge as it moved into a new context—even if it required a one-time investment or special training process—we might enter a world where it could do many things we wanted in the way we wanted without our having to assemble all the context, curate all the examples, and provide everything manually.

We already have that information. We just don’t have it in the right accessible form, and we don’t know how to provide it. If the AI can absorb it all, we might have the drop-in knowledge worker.

That could take us from a world where AI is everywhere but the productivity statistics haven’t changed to a world where AI dramatically changes the labor market with one release. That would be my candidate for what that release might look like.

I don’t think it’s impossible, but it also isn’t solved. This is less clear than my sketch of superintelligence because I haven’t seen direct evidence that it’s about to be productized. But people are working on it, so it’s another frontier to watch.

The drop-in knowledge worker. I’m sure that one won’t take very long.

Thanks so much. This has been a very comprehensive conversation. I’m sure people know where to find you, and I’ll link everything in the description. Thank you.

This was fun. My pleasure.