席卷数学的雪崩，正向 AI 开发而来

Thank you. If you had asked me earlier this year how far away we are from superintelligence, I’d probably have said at least the better part of a decade. But over the last 3 months, I’ve updated my views a ton. I’ve been debating people both on and off the podcast about this question. And, of course, I’m watching the progress of AI on verifiable domains and thinking about what that implies for AI research itself.

I’ve come to the view that we’re only a couple of years away, and possibly only a couple of months away, from AIs that are as good as the best humans at AI research. I’ve also come to believe that automating AI research will compress so much progress into a single year that, soon thereafter—by 2030, and plausibly as early as next year—we’ll have AIs that are at least as good as the best human experts in every field. I know this is a wild claim, and it’s one that I myself didn’t believe even a couple of months ago. Let me walk you through how my thinking has changed.

1. AI Is Already Speeding Up

Before we get into the more speculative arguments about what will happen in the future, I want to point out that AI is already dramatically speeding up further AI progress at the frontier. For example, METR, an independent evaluation organization, spent 3 months embedded at Anthropic, and afterward concluded that Anthropic’s rate of AI progress on well-scoped tasks is 2x what it was a year ago. They also concluded that a majority of this acceleration today is likely attributable to Anthropic’s extensive use of AI.

OpenAI published a blog post about this in September, where they showed that the heaviest 10% of AI users among OpenAI’s researchers went from using about $1,000 a day of internal coding agents at the—[?] And in September, Anthropic put out a similar post on its own AI uplift. They reported that Claude now leads 26% of Anthropic’s AI R&D work, up from under 1% as recently as February.

2. The Mathematics Breakthrough

In thinking about how big a deal it would be for AIs to automate AI research, and how much that would speed up AI progress, a really important update for me has just been seeing how fast the capabilities of AI models in mathematics have exploded. Current AIs are slicing through well-scoped problems in math like a hot knife through butter. It was only 2 years ago that I asked Dario why LLMs aren’t able to use their vast knowledge across so many different fields to connect ideas and make new discoveries.

And lo and behold, here we are in a world where AIs are doing search-and-match discovery to such a great extent that hundreds of important results, which the top mathematicians and computer scientists in the world had spent decades trying to resolve, are being churned out. It’s crazy how fast this progress has happened. As recently as 2023, these models couldn’t reliably add 4-digit numbers. In 2024, they could barely solve any problems in a high school math competition. In 2025, they could win a gold medal in the International Math Olympiad. Then, earlier this year, they finally started to solve some niche open problems. A month ago, OpenAI solved a Millennium Prize problem in 88 hours.

I’m reminded of a line from the novel *East of Eden*: “And there is one sure thing about the fall of gods. They do not fall a little. They crash and shatter.”

Now, it’s true that progress has skewed disproportionately toward the verification-laden parts of mathematics, like finding counterexamples rather than, say, coming up with new theory. Toby Ord recently wrote a great essay about this. He writes, “While current AI systems are starting to automate mathematical proof, there is currently very little evidence that they can ask powerful questions or find profound new fields of mathematical inquiry.”

But another way to say that same exact sentence would be this: “Current AI systems have solved a Millennium Prize problem, all without the ability to ask powerful questions or find profound new fields of mathematical inquiry.”

This ability to solve well-scoped problems is going to be an even bigger deal for machine learning research than it has been for mathematics broadly construed. That’s because the goal of mathematics research is not just to prove or disprove long lists of conjectures, but to extend the human understanding of mathematical concepts. Coming up with the idea of topology, group theory, or, going back even further, the Cartesian grid matters far more than knowing exactly how many—[?] We mainly care about such abstruse questions insofar as they might elicit new mathematical theory.

But in AI research, the priorities are exactly reversed. Recipes that empirically make training more efficient or elicit smarter models are what matter. People don’t care about theory for its own sake, and all the progress we’ve made in deep learning over the last decade and a half hasn’t required much new theory anyway. The equivalent of just grinding through random conjectures would be enormously valuable in machine learning in a way that it just isn’t for mathematics.

AI research is actually surprisingly similar to the kind of mathematical progress that current AIs are disproportionately good at: unlocking a ton of low-hanging fruit by applying a combination of incredible persistence and a superhuman capacity to connect ideas from lots of adjacent domains.

3. How the Transformer Was Missed

For example, let’s talk about the Transformer architecture. This is the big breakthrough that enabled modern LLMs. My friend Sholto Douglas, who is an AI researcher himself, has this perennially half-finished blog post where he explains how you could have invented the Transformer 2 years earlier than people actually did. Maybe this video will finally get him to quit stalling and complete his essay. In the meantime, I’ll summarize a bit of the discussion I had with him about the contents of his essay.

Before the Transformer paper was published in 2017, many researchers had already identified attention as a way to better parallelize training and communicate information across longer sequences. Chris Olah, who was an active researcher at the time, has said, “At that time, my sense is that the Transformer architecture was a natural combination of many ideas, like attention and residual networks, which had independently been effective, taken to quite aggressive lengths.”

In other words, none of the core pieces were a surprise to researchers active in the field. So then why didn’t people invent the Transformer sooner? One reason was that this new architecture was only obviously better than the state of the art at the time once you connected together half a dozen additional tweaks and innovations. Almost all of them had been known in the literature for years, but no one knew how to combine these puzzle pieces together in just the right way. Doing so required great research taste, of course, but it also required tons and tons of small experiments.

What I’ve just described—persistently and intelligently matching dozens of different combinations of ideas—is exactly the kind of capability that AIs have shown themselves disproportionately skilled at in their contributions to mathematics. This is all to say that even the Transformer, arguably the biggest breakthrough in ML since AlexNet kicked off the deep learning revolution in 2012, is the kind of thing that Claude 3.7 could have invented if Chris Olah had asked it to try really hard to get this attention thing to work better.

Now, I want to clarify that some parts of AI research are, in fact, less verifiable than math. That’s why OpenAI hasn’t fired all of its researchers, despite having a model that can apparently autonomously solve Millennium Prize problems. But I think AI research is verifiable.

Explaining why requires getting into the nitty-gritty of how AI research actually works. Those details are interesting but technical, so I’ve saved them for later. If you’re curious and want to hear them, you should keep watching until the end of the video. In the meantime, I’m going to give you my estimate of when we’re going to have such AIs.

4. When Automated Researchers Arrive

If I had to pick my median year for when automated researchers will be as good as the very best human AI researchers at doing AI research, I’d guess 2028. That assumes the AI labs go full steam ahead and don’t slow themselves down intentionally or get slowed down by a government. Why do I think AIs could get this good? In every domain where we can check answers and give AIs thousands of rounds of practice—for example, competitive programming contests or well-scoped open math problems—they’ve gone from worse than most humans to better than the best.

And, as I’ll argue after the conclusion, AI research has exactly the same verifiable properties. Of course, a big part of what makes the top human AI researchers really good at their job can’t be directly verified or trained on. OpenAI’s model was obviously not trained by solving other Millennium Prize problems. It was trained on lots of less ambitious, more tractable math problems. From those, it generalized all the way to solving a problem that no human had ever cracked before.

I’d expect the same in AI research, where the combination of thousands of rounds of practice on verifiable research tasks, plus the fact that each new generation of models is broadly smarter than the last, will make models excellent even at the parts of AI research that we can’t directly check.

Now, I want to be clear that there’s a pretty decent chance that hitting this milestone takes longer because some important parts of AI research remain hard to automate. This is just a very hard thing to try to predict in advance, and I’d guess there’s at least a 25% chance that it doesn’t happen until after 2032.

Conversely, I also think things could happen much faster, and we could hit this milestone in 2027. Some of the most well-informed researchers I know, people who have a very good track record of making predictions over the last—[?]—expect AIs to beat the best humans at AI research as early as next year.

2027 is also the straightforward implication of putting the new METR report, which says that Anthropic’s rate of AI progress has doubled from a year ago, into a takeoff model. As former guest Ajeya Cotra put it in a blog post about this report: “If we take at face value that AI is now nearly doubling the rate of AI capabilities progress, and assume that acceleration from AI is smooth, this naively implies that we would reach extreme superintelligence within a year.” So the reason my median year is not 2027 is just that I’m uncertain about how accurate this kind of straightforward modeling is. But we should not be surprised, and we should be ready for a world in which we reach this milestone next year.

Once AI is as good as the best humans at AI research, algorithmic progress will get a lot faster for 3 important reasons. First, the AIs will be much faster. Second, each of them will be very smart. Third, there will be a lot more of them. Let’s start with speed. Today’s AIs already do work at about 10 times human speed, and automated researchers would also work around the clock, nights and weekends included. As a result, each individual automated researcher will be cramming roughly 50 times more work into every calendar week than a human researcher can do.

Of course, that doesn’t mean AI progress will immediately accelerate 50 times. Those automated researchers would still be bottlenecked by compute for experiments, which wouldn’t run any faster. There would be diminishing returns, but even with extremely steep diminishing returns, this alone would make algorithmic progress proceed many times faster than it does today. And this is before we account for the fact that, in the scenario I’m talking about, every automated researcher will be at least as good at AI research and engineering as the very best humans today.

My sense is that around 80% of the algorithmic progress at a given frontier lab is driven by its 50 top researchers. It’s not for nothing that Zuckerberg was willing to pay hundreds of millions of dollars to acquire the best talent in the industry. So you get another big speedup in algorithmic progress from the fact that every single engineer and researcher in this automated company will be at the level of the very best humans in today’s frontier AI labs.

And finally, there will be a lot more automated researchers. Throwing more workers at a problem isn’t always helpful. For example, a swarm of agents probably isn’t going to be able to write a great novel much faster than a single agent could. But other tasks lend themselves much more to parallelization. For example, look at this animation I’m overlaying, which shows a bunch of different Claudes working together to formalize Fermat’s Last Theorem. This feat had thousands of modular subtasks, each of which could be completed independently, and each agent could locally check whether its work had succeeded.

AI research is closer to formalizing a theorem than it is to writing a novel, because there’s a lot of independent, modular work to do, and there are verifiable proxies for progress. That means a swarm of agents can try out a thousand different approaches and see which one performs best, and they can do this independently for each part of the pipeline. AI companies are already made up of dozens of different teams working independently on different parts of model development. One team, for example, improves how the pretraining data is filtered. Another designs a particular kind of synthetic RL environment. Another writes a better attention kernel, and so on.

In domains like this, where different streams of work can be done and checked independently, adding lots of persistent hill-climbing agents tends to speed up progress a lot. Again, there will be steeply diminishing returns just from adding more parallel researchers. Anyone who’s ever worked in a team knows that having 100,000 collaborators doesn’t mean you’ll go 100,000 times faster, probably not even 100 times faster. But given the nature of AI research, it’s implausible that you wouldn’t go at least a couple of times faster with orders of magnitude more automated labor.

5. The Spark or Singularity

So the combination of faster, smarter, and more numerous automated researchers will speed up AI progress a lot. But will this pace continue? Will it get even faster, or will it slow down? That question is even more important than what the initial pace actually is. Two competing trends are relevant here, and the subsequent rate of progress will come down to which one ends up being more powerful. On the one hand, because of RSI, automated researchers will improve AI research. On the other hand, AI research will get harder and harder as all the easy ideas get eaten away by the swarm.

By the way, ideas getting harder to find is something that already happens. AI companies have to quadruple their R&D compute and double their workforce every single year just to maintain the same rate of progress. So if the rate at which AI gets smarter is faster than the rate at which ideas get harder to find, then we’d get something called the software-only singularity. In this scenario, even without increasing compute, the AI swarm rapidly makes smarter and smarter versions of itself.

But if ideas get harder to find faster than RSI makes AIs better at research, we’d end up with something I’d call a spark. We’d get something like 3 to 6 years of AI progress at the current rate in a single calendar year. Then things would simmer down until we build more compute, because even these super-genius automated researchers need to run bigger experiments before they can make further breakthroughs.

As for which one will actually happen, I think it’s basically a coin toss. If pressed, I’d give a 50% chance to the spark scenario and a 50% chance to the software-only singularity. But to be clear, even in the spark scenario, where we make something like 4 years of progress in the first calendar year after we automate AI research, things would feel insane. There’s a jump as big as GPT-3.5 to GPT-6 Astra in just 12 months.

To remind you of the scale of progress that this leap would compress, GPT-3.5 was able to write a passable limerick and autocomplete a trivial Python function, but otherwise it was mostly a useless toy. Whereas Astra can build whole video games from scratch and churns out so many Fields Medal–worthy mathematical contributions that we become totally desensitized. In the last 4 years, we’ve gone from a preschooler to a savant intern. If you made another jump that big, but starting in 2027 or 2028, you’d probably get a mind that would be vastly superhuman in domains that are even slightly verifiable.

For example, it’d be better than NVIDIA at coming up with the next generation of AI accelerators. It would make more money than the current human versions of Citadel and Jane Street if it were given equal access to the market. If you gave it remote access through a materials science experiment lab, it’d be better than the rest of humanity at finding a room-temperature superconductor.

6. Learning Beyond the Data Center

At this point, you might say: How will even superintelligence, which is trained in a data center, beat human experts in domains where the basis of that human expertise is years and years of real-world experience? The way we’re currently making AI smarter is by RLing them on tasks that are easy to automatically grade from within a data center. For example, did the million new lines of code the AI wrote pass all the unit tests? Could the AI get really good at building a business from scratch? How about winning court cases, having a profitable day of trading in the markets, or helping a candidate win an election? The rollout here requires interacting with the real world, and you can’t recreate it just within the data center.

So how will superintelligence get good at this kind of stuff? Let me be clear: I don’t think we can just bake the answer to every real-world problem into the weights. Rather, my claim is that scaling up RL training will give AIs the abilities necessary to ramp up quickly in new contexts, even in hard-to-verify domains like the ones mentioned above.

Getting really good at any complex task requires 2 things. One is domain-specific knowledge, and the other is a more general set of skills, the kind of capabilities that capable humans have: being able to seek out relevant information, make a plan, push forward on that plan through mistakes and obstacles, keep track of what you’ve learned, and collaborate with other people.

During RSI, automated engineers will build tens of thousands of times more environments than we have currently. These will be far larger, longer-running, higher-quality, more open-ended, and cover a much wider range than today’s environments. The AIs will learn to make their way through unfathomable amounts of virtual worlds that are each, compared to today’s video games, what a Bugatti is to a horse and buggy. These games will be far more difficult, messy, sprawling, and realistic than anything that has ever been made for human entertainment. They will require AIs to cooperate or compete with teams of other AIs over long periods of time.

For example, an AI might be dropped into a simulated economy of millions of other AIs and told to cofound a successful company. It’s not like the AI will be stuck just thinking about the ideal corporate structure. It’ll go through an evolutionary process that involves centuries of corporate competition to figure out the optimal organizational techniques.

The AIs will also train themselves on endless backtests of everything in the history of science, technology, politics, and culture: every discovery, invention, war, election, and quarterly earnings that has ever happened. For example, here’s one prediction training task: Can the AI figure out general relativity given a training corpus that includes no information beyond 1700, which is far earlier than any human was on track to discover general relativity? Another task might be: Given every single newspaper article up until 1962, can AI predict how the Cuban Missile Crisis is going to unfold? And it’s not just the training data that these automated researchers will be improving.

They will also radically improve the basic learning algorithms themselves through breakthroughs in online learning and sample efficiency, both of which have verifiable proxies, as I’ll discuss later. In the same way that humans are much more efficient and effective at learning from data than current LLMs are, future AIs could be far beyond humans in their learning efficiency.

Now, why do I think we’ll see lots of transfer from verifiable tasks to these more general skills? One reason is that even at one ten-thousandth or one-millionth of the scale of RL training that will be happening during the intelligence explosion, we’re already seeing lots of generalization. If you point Claude Code or Codex at a project it’s never seen before, it knows how to orient itself, figure out how everything fits together, and tackle an ambitious assignment.

Of course, it’s hard to measure how much models have improved at tasks that aren’t easily verifiable, because if we could, those tasks would be verifiable. But you can just go talk to a new model and compare it to an old one. In my experience, AIs have gotten much better at things that have very little to do with code or math. For example, they can help me research the abstract questions I needed to answer while writing this essay.

The other big piece of evidence for transfer is human evolution. Our general-purpose mental architecture was developed by evolution, and evolution is an optimization process with a really crude verifier: Did your genes pass on or not? In the ancestral environment, the tasks that influenced that outcome were things like hunting, fending off predators, and keeping your standing in your tribe. These are a tiny subset of the kinds of tasks that humans are able to tackle today.

Evolution did not specifically optimize us to write a Pulitzer Prize-winning biography, make advanced semiconductors, lead nations of millions of people, or come up with a new branch of mathematics. But in optimizing us for survival on the savanna, evolution built a general-purpose learner. In training these AIs, we’re running a similar process, except we’re doing so deliberately with tasks that are much more closely related to the ones needed for success in today’s world. And unlike evolution, AI companies are trying to improve transfer during training, for example, by making sure that performance improves on difficult tasks that were held out of training.

If even dumb, blind evolution can get this much transfer out of building a mind that is effective at propagating itself in the savanna, then the deliberate effort AI companies are putting into building a flexible, general intelligence that can rapidly ramp up in new contexts should advance much further.

Now, even a superintelligent learner can’t figure out everything from inside, so these AIs would need some real-world experience before they can beat the best humans. But I think they’ll be able to get that experience much faster than people expect, and that they’ll be able to top human experts within months of being deployed.

Honestly, one of the things guiding my intuition here is that I have a lot of friends in their 20s who I’ve watched accomplish incredible things and build world-leading companies and ventures in domains they entered with basically zero years of experience. History is full of super-competent young people who ran circles around the world they found themselves in. By the age of 27, Napoleon had stacked victory after victory against far larger armies from the other great powers of Europe. By 25, just 2 years after he got into the oil business, Rockefeller was running one of the world’s largest refineries.

Rockefeller and Napoleon needed experience, yes, but only a couple of years’ worth, not decades. Whatever was going on inside these guys’ brains, an AI should be able to replicate or even exceed. There’s no reason to think that humans are at the ceiling of learning.

Furthermore, unlike Napoleon or Rockefeller, AI is going to accumulate years of deployed context in just a few days of calendar time. For example, imagine that you wanted these future AIs to run a Fortune 500 company better than its current CEO. How quickly could an AI ramp up at this job? Well, if you let it shadow that CEO, on its first morning it could read every email, Slack message, and document the company has ever produced. By the end of its first day, copies of it could have interviewed every employee and major client in parallel.

Obviously, they wouldn’t have access to feedback that arrives after many months of serial time, such as how well they did on the next quarterly earnings. But even given these kinds of bottlenecks, we should expect these future post-Spark AIs to reach top human performance in all domains within months of being developed. That’s still 2028 or 2029 in this median scenario. And remember, even in this Spark scenario, all of this happens in just the first year or two after AI research is fully automated.

Things don’t just stop there, because even if software progress peters out, AIs will continue stoking the fire by adding more and more compute. Even without help from AIs, global AI compute has been growing roughly 3× a year for the last half-decade. And the AIs that come out of this Spark will be jaggedly superhuman at helping us continue scaling up this compute. They’ll be extremely good at designing new AI chips and laying out the designs of new fabs, even teleoperating robots to help build parts of fabs and data centers. With robots, it takes years to get to the same endpoint that this software-only singularity would have achieved in months: a superintelligence that is to us what we are to chimps.

7. Why AI Research Is Verifiable

So earlier, I promised to explain why I think AI research is largely verifiable. That’s where I’ll start. After that, I’ll go through what I think are the strongest objections to my arguments about RSI.

One thing I want to call out at the very top is that because both pre-training and post-training are verifiable, as I’ll explain later, so is progress on what is the most glaring and important inefficiency AIs have compared with humans: their lack of sample efficiency. If you want a neural network to gain some capability, you have to feed it far more data than it would take a human to learn the exact same thing. But every kind of sample efficiency is measurable. For example, you can measure how much data it takes to reach a given pre-training loss, how many tasks it takes to reach a given score in an RL environment, or even how much feedback and time within a single session it takes to reach a desired level of performance just from in-context learning.

Sample efficiency and online learning are really two sides of the same problem: how to learn from real-world data quickly and efficiently. For decades, people have recognized this as the deepest open problem in machine learning, AI’s equivalent of a Millennium Prize problem. And mathematics has already shown us what happens to such problems, even when they are very difficult.

Okay, now let’s get to why I think AI progress is largely verifiable: why progress on sample efficiency, online learning, or the other problems in ML is captured by verifiable proxies. There are roughly 3 broad buckets of AI R&D: pre-training, post-training, and inference efficiency. As I’ll explain, each has strong verifiable proxies that are easy to train against. And since together these comprise all of algorithmic progress, that means AI progress in general is easy to train against. In some sense, that shouldn’t be surprising, because we can make models better at things that can be tested inside a data center, and AI training is literally happening inside a data center.

As we talk about these proxies, you should keep in mind that verifiability is valuable for 2 reasons. First, it allows AI companies to directly RL a model to get better at specific parts of AI research through thousands of runs. The second reason verifiability is important is that these proxies allow each agent to empirically check whether they’ve beaten some score on a leaderboard before they merge their contributions. That means they can work much more effectively in parallel.

With that in mind, let’s start by talking about pre-training. As you’re building GPT-7, you can RL that model to get better at finding pre-training compute multipliers. For example, you can make an RL objective that GPT-7 has to minimize. Once GPT-7 is built, you can deploy this model, or a swarm of these models, to figure out how to train a GPT-3-sized model to the lowest possible perplexity. This is because loss on a held-out test set, which is the basis of pre-training research, is extremely easy to verify.

Of course, what works at small scale doesn’t necessarily work at large scale, and it requires taste to anticipate exactly which small-scale improvements will actually bear fruit when they’re composed together in a bigger model. But this kind of intuition is also trainable. For example, you can make an RL environment where GPT-7 has to predict whether an algorithmic improvement it made at the 100-million-parameter scale will also hold at the 10-billion-parameter scale.

While the TasteBench environment is more expensive to RL on than the nanoGPT speedrun, it’s still not that difficult to train on if you have frontier-lab-scale compute. So even important parts of research taste are verifiable. If you just automated pre-training, you’d already radically speed up AI progress. There are a couple of reasons why. First, pre-training leads to very general improvements in the model’s intelligence and capabilities. There’s a deep explanation for why this is true, which has to do with the link between compression and intelligence. If you want to learn more about this, I highly recommend this excellent video by 3Blue1Brown on the topic that I’ll link to below.

The second reason is that as you make pre-training improvements, you can train a smaller model that’s just as good as a previously larger model, but trains faster and more cheaply. This not only speeds up pre-training; it also makes your experiments and RL rollouts on top of this base model more efficient. Since your automated researchers are instantiations of this smaller model, you can support larger researcher populations too.

The third reason is that smaller models train faster, which tightens the RSI loop itself. During RSI, labs probably won’t be doing three-month-long pre-training runs spread across multiple data center campuses. Instead, the army of automated researchers will find algorithmic innovations so quickly that by the time a giant run finishes, it will already have been made obsolete by a much smaller model that began training only a week ago. It’s similar to launching a spaceship to Alpha Centauri with today’s technology: one launched a century from now would probably overtake it in transit.

Furthermore, since you’re training smaller models faster, and at a scale where you can run thousands of experiments daily, you won’t have to bet that small-scale results will hold up in bigger models. You’ve already tested at the scale where you’ll be training. What do most models get, even at today’s parameter sizes? I think there’s probably a ton more potential. We are far from the limits of pre-training. For example, the AI lab Magic recently announced that it pre-trained a model that matches leading open-source models today, like DeepSeq v4 Pro, using just half the compute it took to train the original GPT-3, which would cost roughly $500,000 today. Orders-of-magnitude further improvements seem totally plausible.

Now let’s talk about post-training. There are 3 different kinds of verifiable proxies for post-training. First, your automated researchers can try to improve your post-training algorithms so that if you want to do RL on a new set of environments, it takes less compute, data, or time. All 3 are easily measurable. Second, your automated researchers can work on making models better at learning on the fly. You can easily measure progress on this. For example, you can drop your model into an environment it has never trained on and measure how quickly it learns what’s going on and figures out how to get a good score. You can measure how fast it gets better at a video game it’s never played before. And you can have real-world examples of this, too. If you want to see what these could look like, check out this great benchmark that ByteDance released, called EdgeBench, for these kinds of tasks.

Third, you can just measure AI R&D capability directly. Anthropic, for example, has an internal benchmark called CoBench, which contains around 500 real problems that researchers and engineers have solved this year. Every benchmark is also an RL environment. An environment like this would allow Anthropic to have its models practice redoing the last 3 to 6 months of human progress at the frontier.

The third big area of AI research is inference efficiency. This involves designing your model and the surrounding software to get more juice out of your compute. This is especially verifiable. For example, you can easily measure GPU utilization, KV cache size per token, total FLOPs in a forward pass, latency, and so on.

Now I want to deal with the strongest objections to my arguments about RSI. I want to be clear that these bottlenecks are serious, and they do tamp down the probability of the most aggressive possible takeoff. This is why I think we’ll probably make something like a GPT-3.5-to-GPT-6-sized jump in 2028, rather than some crazy overnight singularity that starts tomorrow. But I think these bottlenecks won’t be especially crippling, and the neck of the bottle is actually pretty wide. Let me explain why.

The first objection is one I had until recently: that we’ll be bottlenecked by continual learning, or the lack thereof. I spent the past year and a half arguing that the main thing missing from today’s AIs is the ability to learn from on-the-job experience. Every new session is their first day on the job, and in each session, they can retain only a couple of hours’ worth of information. So they don’t accumulate the kind of feedback and know-how that arrives over the course of weeks or months.

Last December, this was my main reason for doubting that we’d automate AI research anytime soon. As I wrote then, somehow this automated researcher is going to figure out the algorithm for AGI, a problem that humans have been banging their heads against for the better half of a century, while not having the basic learning capabilities that children have. I found this super implausible.

But I’ve somewhat changed my mind on this. Here’s why I don’t think the current lack of continual learning in these models will stop full automation of the best AI researchers by 2028. First, AI companies might just solve continual learning by then with architectures much like the ones they already have. Previously, I thought we were bottlenecked by some crazy breakthroughs that would allow models to update their own weights at the end of every session. But now I think scaling up in-context learning is enough. You’ll have sessions that are effectively many months long, and this will allow a single instance to learn effectively on the job. People have published many promising ideas that would let labs achieve dramatically longer effective context windows.

The second reason is that you might not need a continual learner to solve continual learning. As I explained earlier, online learning is an ML research problem with clean, verifiable proxies. If you’d asked me 2 years ago which would be harder to solve, Navier–Stokes or continual learning, it wouldn’t have been obvious to me that the answer was Navier–Stokes. Now, of course, continual learning did turn out to be harder, given that the labs haven’t cracked it yet. But are we so confident it’s that much harder that non-online-learning models a year or 2 smarter than the ones currently solving Millennium Prize problems won’t be able to crack it, even if they’re amnesiacs like the current models?

These models are out here solving Millennium Prize problems while still being derpy savant interns with no long-term memory. It’s like watching somebody do cartwheels down the track and still keep pace with Usain Bolt. All this being said, I think there’s a serious chance we don’t solve continual learning within the next few years, and that models can’t compensate for this missing capability through their other advantages. That’s why I put a 25% probability on not automating Ilya-level top human researchers until after 2032.

One important objection people have is that we’ll be bottlenecked by finding the right verifiable proxies. They would argue that all the verifiable proxies I’ve listed above assume the current paradigm. But figuring out what objective to verify in the first place—choosing which hill to have GPT-6 research—is what one might call research taste. How are you going to train for that? Before 2018, almost nobody was saying that minimizing loss on next-token prediction was going to be the most tractable path to AI progress. So nobody would have bothered to build nanoGPT-style speedrun environments where the agent swarm has to minimize how long it takes to reach a certain pre-training loss. Wouldn’t there be multiple new paradigms we have to cross to get from here to superintelligence? And won’t each new paradigm require some associated proxy that we don’t know in advance? How will the automated researchers find these during RSI?

I have 2 responses to this objection. First, the transformer was visible; in fact, it was invented to beat the existing translation benchmarks. A similar story is true for reasoning models and the pre-existing coding and math benchmarks. Proxies made before a paradigm leap seem perfectly capable of capturing large step changes in performance. This is because AI learning algorithms are quite general.

Second, it’s not that difficult to come up with paradigm-agnostic outer-loop proxies. For example, how many Millennium Prize problems does your new model solve? If you train your model on data from before 1905, how long does it take before it discovers general relativity? How fast does it master totally unseen video games? And so on. All in all, I think the other objections to the impact of RSI are much stronger, and I’d be very surprised if this ended up being the reason RSI wasn’t a big deal.

The other big bottleneck people point out is data. Isn’t data one of the main inputs into AI progress? How will RSI happen without that scaling up at the same rate? My answer is that data will continue growing during RSI, but the AIs will be the ones compiling it. Producing better training data for both pre-training and post-training is itself an ML research problem, and AIs are getting increasingly better at doing it themselves.

Now, it is true that data is an extremely important input into AI progress. Together with Jerry Huang, a student at Princeton, I found that for pre-training, over the last 6 years, more than 3 times as much algorithmic progress has come from data improvements as from model improvements. However, I don’t think people realize that these data improvements in pre-training are very much in line with what you’d expect. Data empirically lead to lower loss on a held-out corpus. A similar thing seems to be increasingly true for RL. DeepSeek just published a paper where they talk about how they post-trained their new model.

They say that changes to the basic RL algorithm have not mattered nearly as much and that, quote, “Systematic improvements in the scale, diversity, and verifiability of synthesized data and environments account for essentially all of the observed gains,” end quote. The way these environments are made is itself getting more and more automated. They write, quote, “We invest in large-scale automated pipelines for data synthesis and environment construction.”

Now, I want to talk about the idea that AI progress will be bottlenecked by experiments. Earlier, I made this extended analogy between the current acceleration in mathematics and what we should expect to happen in AI research. But there is one big difference between AI research and mathematics. In math, you can simply reason your way to new discoveries with pen and paper, whereas in AI research, you need to run actual experiments to validate your ideas, and these take compute and time. As the number of automated researchers grows, each gets access to a smaller and smaller slice of the total experiment compute pie.

My answer to this is that each agent will still get access to a ton of compute, basically. Frontier labs have been more than quadrupling their compute every single year. If this trend continues, then by the end of 2027, OpenAI could give almost 10,000 agents—the number involved in solving the Millennium Prize problems—enough compute to run a GPT-3-sized training run every single day. Keep in mind that the final model trained in the original Transformer paper in 2017 took 3 days on 8 P100s, which would cost about $50 if you rented it today, or about one ten-thousandth of the training compute of GPT-3.

If top human AI researchers like Noam Shazeer and Ilya Sutskever had enough experiment compute to train a GPT-3-sized model every single day, they might make slower progress than they currently do. But it would be extremely surprising to me if they failed to make meaningful progress altogether and just sat down and started complaining about how small their cluster was. How productive do you think an AI lab would be if it had 10,000 great researchers, each with this much compute and each thinking 10 times as fast? You wouldn’t want to write a check to this new lab?

The nature of research and experimentation at the labs today reflects that researcher time is extremely scarce relative to compute. Top AI researchers individually get paid hundreds of millions of dollars a year. So the way labs use compute today tells us very little about how efficiently they could use it if compute, rather than researcher time, were the key bottleneck.

There are two main ways in which the RSI agent swarm will use experiment compute much more efficiently than researchers today use it. First, they can immediately trade labor for compute. To save valuable researcher time, experiments today are, and should be, very wasteful with compute. So instead of painstakingly setting up and carefully monitoring tons of small-scale pre-training experiments, a human researcher today might kick off a couple of huge multi-agent RL runs in parallel. But if you’re bottlenecked by experiment compute, you can just do more of the former.

When those automated researchers are doing an RL run, they could read every single rollout trajectory and immediately pause and rebuild the whole environment if some task is broken. They could be doing crazy amounts of small-scale ablations of their models. When a team of agents decides to combine their compute to do a much bigger run, they could spend many subjective decades designing the most informative pooled experiment. Once it’s done running, they could spend further subjective decades comprehensively analyzing it before designing a new experiment.

The other big way this huge agent swarm could remain productive, despite each individual copy having access to less compute, is that these automated researchers could come up with a better science for learning from small-scale experiments. Scaling laws themselves are a good example of this. The GPT-4 paper says that they were able to predict the model’s final loss using models trained with 10,000 times less compute.

One thing smarter and faster automated researchers could do is try to discover more scaling phenomena like μP, which lets researchers anticipate the optimal hyperparameters for a larger run from a model 100 times smaller. How many more similar discoveries like this are there? I remember chatting with an AI researcher who thought that learning from extremely small neural networks could be incredibly informative. He put it this way: “The gods above must work from the same principles as the ants below.”

I want to thank my friends who paired with me through significant parts of writing this essay: Max Ferenc, Gabe Freed, and Thomas Larson. I’ll see you on the next essay about superintelligence, but in the meantime, if you want to get notified when that video essay drops, please subscribe on YouTube. All right, I’ll see you next time.