ARC Prize Version 2 发布视频！[Francois Chollet, Mike Knoop]

This is not just more of the same that we've seen in the past. We now have an existence proof that computers are able to do something that they've never been able to do before in the history of humanity.

ARC-AGI 2 has just been released, and even the frontier foundation models are failing spectacularly today. We are releasing ARC-AGI 2, the next version of the ARC benchmark. ARC-AGI 2 is pretty much the only unsaturated benchmark that is feasible for regular people, so it's a very good yardstick to measure how much fluid intelligence these models have and how close we are to AGI.

Alongside that, we're really excited to be welcoming everyone to ARC Prize 2025. The contest kicks off officially now, and it's going to run all the way through the end of 2025. The structure of the contest is very similar to last year: we're going to have the Kaggle leaderboard running, we'll be testing this year on the semiprivate dataset, and we'll wrap up by testing the final leaderboard on the private dataset.

We've still got the big prize, and it's unclaimed. In order to get the big prize, you have to open-source your solution and achieve a high degree of efficiency running on Kaggle. We're really excited to see all the new ideas. I think there was a lot that came out last year, in 2024, that really pushed the frontier.

The next version of the benchmark is more challenging and extremely unsaturated. All frontier models are scoring effectively within the single digits in percentage terms. It's also the first time we've calibrated the human-facing difficulty of the tasks. We hired roughly 400 people and tested every single task, and every single task has been solved by at least 2 people. So we know it's very easy and visible for humans, while it's extremely out of reach for any system today.

This is the frontier. The ARC benchmark forces us to confront an uncomfortable truth about our pursuit of artificial general intelligence, one that the field has been overlooking: intelligence is not just about capabilities. It's also about the efficiency with which you acquire and deploy those capabilities.

Intelligence is about finding that program in very few hops, using very little compute. Look at the amount of energy that a human expends to solve one ARC task over 2, 3, or 4 minutes. It's almost zero. Compare that to a model like o3 on high-compute settings, which is going to use over $3,000 of compute.

So it's never just an economics problem. Efficiency is actually the question we're asking. Efficiency is the problem statement; it's not capability. The goalpost is AGI. That's what we're here to do. That was the whole point of launching ARC Prize in the first place: to raise awareness that there was this really important benchmark that showed something important that the research community was missing about the nature of artificial intelligence.

The test-time optimization or test-time search techniques are the current frontier of AGI. There are many ways to approach it. You can do test-time training, or you can do search—in a symbolic space, over a chain-of-thought space, over a token space, or in latent space. There are many different ways to do it, but the frontier is adapting to novelty at test time by recombining what you know into some novel structure.

Mike, it's amazing to have you on MLST. Welcome.

Thank you so much. We're very excited to be here today.

Mike, I hear that you guys have some very exciting news today. Tell me about it.

We're super excited to be launching both ARC-AGI 2 and an updated ARC Prize 2025 contest today. You can go to ARCPrize.org to learn more and enter the contest.

1. ARC-AGI 2 Challenges Reasoning

In a nutshell, what is ARC-AGI 2, and how is it different from version 1?

The way I think about it is that ARC-AGI 1 was a benchmark designed to challenge deep learning, whereas ARC-AGI 2 is really a benchmark designed to challenge these new AI reasoning systems that we're starting to see from pretty much all of the frontier labs.

One of the really cool things about ARC-AGI 2 is that we're seeing AI systems based purely on pretraining effectively scoring 0%. Some of the frontier AI reasoning systems are in the process of being tested right now, and we're expecting single-digit performance. That's a really big update over ARC-AGI 1 from 2024.

The original version of ARC was aimed at foundation models that didn't reason, while version 2 is tuned for reasoning models. What would you say to the charge that you're moving the goalposts? How is ARC-AGI 2 meaningfully an evolution of the benchmark?

The way I think about it is that the goalpost is AGI. That's what we're here to do. That was the whole point of launching ARC Prize in the first place: to raise awareness that there was this really important benchmark that showed something important that the research community was missing about the nature of artificial intelligence.

The definition that I use for AGI, and the one that the ARC Prize Foundation adopts, is assessing the capability gap between humans and computers. ARC Prize's foundation is really about driving that gap to zero. I think it would be hard to argue that we don't have AGI if you look around and can't find any more tasks that are very straightforward, simple, and easy for humans that computers can't do as well.

The fact is that we were still able to find lots of those tasks. In fact, all of the tasks in the ARC-AGI 2 dataset fit into this category: things that are relatively easy, simple, and straightforward for humans, but comparatively very difficult and hard for AI today.

I know you've done a lot of human calibration, and we'll talk about that in a minute. The fundamental philosophy of the ARC challenge is focusing on the human gap. At the same time, AI models are becoming superhuman in so many respects. Is the big story the human gap, or is it the expansion of capabilities that are superhuman?

This is one of the things that makes ARC special, unique, and important, I would argue. There are a lot of benchmarks in the world today, and to my understanding and knowledge, pretty much every other benchmark—the frontier benchmarks, basically—is trying to test for superhuman capabilities. These are PhD-plus-type skills that you need in order to succeed at the benchmark.

Humans can't solve the problems that are in these benchmarks. You have to have a lot of experience, education, and training in order to even get close to solving them as a human. I think those benchmarks are important and useful, but I think it's actually more illuminating about the nature of artificial intelligence to look at the gaps between what's simple and easy for humans and what's hard for AI.

I think that's much more of an inspiring story. It's one where it's actually necessary to target this in order to get AGI that is capable of innovation. One of the main reasons I got into AI and AGI in the first place was being inspired and excited about trying to build systems capable of compressing science timelines.

If all we have is the kind of AI we had at the beginning of 2024—based on pretraining and memorization—you'll never get there, because these systems are faithfully going to reflect the experience and knowledge that humanity has gained over the last 10,000 generations. They're not going to be capable of producing new knowledge and new technology, adding to humanity's corpus of knowledge and technology.

I wonder whether you think we're just about 5 discoveries away from AGI, because there will presumably be version 3 of the ARC challenge, and then version 4. Intelligence is multidimensional, and I can see this both ways.

Many critics of AI are almost gaslighting us. They're saying that this amazing technology you're using doesn't work, and I'm like, "Well, yes, it does work." Will the criticisms become more and more philosophical? Will people say, "Because it's not biological, or whatever, it's not the same thing"? Or do you think we're meaningfully about 5 steps away from AGI?

I think this is why benchmarks are important. I had a similar question when I was starting to get back into AI in 2022 and trying to understand the world: Are we on track for AGI? How far off are we?

It's really hard to get a sense of the capabilities of all these systems purely by using them. You can certainly get a sense by interacting with them, but if you really want to understand what they're capable of and what they're not capable of, you need a benchmark to discern that fact.

This is one of the interesting things I picked up from building AI products at Zapier as well. It's very different building AI with AI than it is building classic software. One of the big differences is that, when you're building classic software, you can build and test with 5 users and know that the product can scale to millions and will work the exact same way. That's fundamentally not the case with AI technology.

You really have to deploy it at a large scale in order to assess how it works. You need a benchmark alongside that scaling to tell you whether the system is working or not.

What were the main lessons you learned from version 1 that you moved into version 2?

ARC-AGI 2 has actually been in the works for several years. François started working on it and crowdsourcing tasks for it years ago. There were a bunch of inherent flaws we ran into as we started popularizing the benchmark over the last year or so.

One of the things we learned was that a lot of the tasks were very susceptible to brute-force search. We wanted to minimize the incidence of tasks susceptible to something that has zero intelligence at all. We also hadn't human-calibrated it. Anecdotally, we relied on a couple of examples to say that ARC-AGI 1 was easy for humans.

We had a couple of STEM people who had taken the whole dataset, including the private set, and were able to solve 98% or 99% of it, but we were relying on anecdote. We hadn't calibrated that across the 3 different datasets we had.

Then all these AI frontier reasoning systems came out over the last 3 or 4 months. We've had a chance to study them and learn what qualities of ARC tasks remain very challenging for AI reasoning systems, which we can get into if you're curious. Those were the main insights and learnings we took from ARC-AGI 1 to produce an ARC-AGI 2 benchmark that I think will be a useful signal for development in artificial intelligence this year.

2. O3 Changes the Picture

Can we quickly touch on the OpenAI situation? In December, they gave you access to o3, and it got incredible performance on ARC-AGI 1—human-level performance, something we didn't think would really be possible so quickly.

It surprised me. It came out of nowhere.

This is one of the reasons I'm always hesitant to make predictions in AI about timelines. It's very easy to make predictions along smooth scaling curves, but the nature of innovation is a step function. Step functions are really hard to predict.

The best thing I can say, having spent some time with o3 and looking at how it performs on ARC, is that systems like o3 demand serious study. This is not just more of the same that we've seen in the past.

We now have an existence proof that computers are able to do something that they've never been able to do before in the history of humanity, which is really exciting. I think there's still a long way to go to get to AGI, but these things are important to understand, and we need to discern how they work from a capability standpoint in order to make sure that future AI systems we're developing and building look more like this, rather than the pretraining, pure-scaling regime we've had in the past.

I still remember the 2-week period—the sprint we had on testing o3. It was right at the end of the contest. We'd wrapped up our 2024 contest in early November, and we had a 3- or 4-week period where we were very busy judging all the final submissions, reviewing the papers, and putting together the technical report.

We were dropping the technical report on a Friday. I was really hoping and anticipating that I was going to have a nice, relaxing holiday period in December. The day we dropped the technical report, we heard from one of the people at OpenAI, who said, "We'd really love you to test this new thing we're working on. We think we've got some impressive results on ARC-AGI 1."

That kicked off a hectic, fast, and frantic 2-week period to try to understand what this system was like, whether it reproduced the claims OpenAI had made, what this meant for the benchmark, and what it meant for AGI.

We were able to show that o3, on its low-compute setting—which fit within the budget constraints we'd set out for our public leaderboard—got about 75%. They had a high-compute version that used perhaps 200 times more compute than the low-compute setting, and it was able to score 85%.

These are impressive results. I think this shows that o3 has a binary switch. We've gone from a regime where AI models have no ability to adapt to novelty to something like o3, which is an existence proof of an AI system that can adapt to novelty in a small way.

There were some interesting caveats that you alluded to. First, they did some kind of fine-tuning, and people at the time joked that it was scandalous that they were training on the training set.

This is a very poor critique. I think it misses the point of the benchmark. The people who feel this way are used to thinking about benchmarks and AI from the pretraining scaling regime, where if you trained on the data, testing on the data is cheating. That's true in the pretraining regime, but ARC is a special, different benchmark where it explicitly makes a training set available with the intention that you train on it.

That's what the benchmark expects you to do. We expect AI researchers to use the training set to teach their AI systems about the domain of ARC. What's special is that we have a private dataset that very few humans have ever seen. The private dataset does not look like the training set. It requires you to generalize and abstract the core knowledge and concepts you learned through the training set at test time.

Fundamentally, you cannot solve the ARC-AGI 1 or ARC-AGI 2 private datasets purely by memorizing what's in the pretraining set. A crude analogy would be that if I were going to teach an AI system grade-school math and then test it on calculus, that's very similar to what we do with ARC. The training set is a much simpler, easier curriculum to learn from, and the test is much more difficult. You have to express true intelligence and have an actual capability to adapt to novelty at test time in order to solve it.

All of that is fine, but there are a couple of things. They were spending $25,000 per task or more, which means they were probably doing sampling and a ridiculous number of completions. They were doing solution-space prediction, which is very interesting.

But, Mike, deep in your bones, do you think they were training on API data? Surely they were training on a whole bunch of data to do that. The extension of the question is: when they release the vanilla version of it, what performance would it get compared to their tweaked version?

We will test that as soon as it comes out, and I would love to report the results. They told us all they did was train on the training set, and I believe that's what they did.

I was amazed that, just by predicting the output space directly, they could do so well. Doesn't that almost take away from the idea that we need discrete code or DSL-type approaches if you can predict the solution space so well?

Effectively, what o3 is doing is using its pretrained experience and recombining it on the fly in the face of a novel task. It does this through a regime called chain of thought. This is all informed speculation; we don't have confirmed details. It's my personal assessment of how these systems work, particularly o1 Pro and o3.

If you compare them with systems like R1 or o1, those systems basically produce a single chain of thought and use that chain of thought to ground a final answer. That's distinct from how systems like o1 Pro and o3 work. They have the ability to do multisampling and recomposition at test time of that chain of thought.

This allows them to build novel chains of thought that don't show up anywhere in the pretraining—not in the existing experience—and allows these systems to search more effectively over a larger solution space based on what was in the original pretraining.

Fundamentally, these systems are a combination of a deep-learning model and a synthesis engine put on top. I think the right way to think of them is that they're really AI systems, not single models anymore.

The critique in the community is funny. Gary Marcus is now saying, "It can't draw pictures of bicycles and label the parts," whereas we see o1 Pro and o3, and it really does seem like a dramatic move toward intelligent systems.

Can we quickly talk about the testing methodology? The real work you did was to get a whole bunch of human subjects. I think you had 400 test subjects, and at least 2 people needed to solve every single task. You had to design the experiment and balance the complexity of the tasks. How did you do all of that?

This was one of the biggest things we wanted to fix with ARC-AGI 1. We never had a formal human-calibration study on how humans actually perform on these things. We relied on anecdote.

We set up a testing center in San Diego and recruited people from the local community, from Uber drivers and single mothers to UC San Diego students. We brought them in to take ARC puzzles. We have some really cool photos of these testing sessions, with dozens or hundreds of people taking ARC tasks on laptops.

Our original goal for the dataset was to ensure that every single task we put into ARC-AGI 2 was solvable by at least 1 human. What we actually found was an even higher standard: every single task in the new version 2 dataset is solvable by at least 2 humans under 2 attempts.

These are the same rules we give AI systems on the benchmark, both in the contest and on the public leaderboards. I think this is a pretty good assertion of a straightforward comparison: Are these tasks easy and straightforward for humans? Yes. Are they hard for AI? Yes.

As I said before, frontier systems are generally getting close to 0% or single-digit percentages on these tasks.

The idea, though, is related to Moravec's paradox: while we can select problems that are easy for humans and hard for AI, we haven't got AGI yet. I was looking through some of your challenges, and I felt that some were very difficult. It would have taken me 5 or 6 minutes of deep thought to solve them.

Are you finding that it's still easy to find things that are easy for humans and hard for AI, or are you scraping the barrel a little bit?

"Easy for humans, hard for AI" is a relative statement. The fact is that these ARC-AGI 2 tasks were solvable by humans under a $5-per-task solve-rate budget. They were solvable in about 5 minutes, while AI cannot solve them at all today.

So, yes, you do have to think about them. You have to put in some thought to ascertain the rule, but the data speaks for itself. Every single task in the version 2 dataset—from the public training set and the public evaluation set to the semiprivate set and the private evaluation set—is solvable by at least 2 humans under 2 attempts.

These frontier systems can't solve these things at all, or they can solve them only with a very expensive budget—thousands of dollars per task, as you pointed out earlier.

You guys have been cooking. You're already working on version 3 of ARC. What can you tell us about that?

The way I think about the different versions is that ARC-AGI 1 was designed to challenge deep learning as a paradigm, while ARC-AGI 2 is designed to challenge these AI reasoning systems. I don't expect ARC-AGI 2 to be as durable. ARC-AGI 1 lasted for 5 years, and I don't expect ARC-AGI 2 to last quite as long.

I hope it will continue to be a useful signal for researchers over the next year or 2. We've been working on ARC-AGI 3, and the right way to talk about it is that it's going to challenge AGI systems that don't even exist in the world today.

3. ARC Prize Builds for AGI

Can you tell me about the Foundation you're setting up?

This is one of the big, exciting things to come out of ARC Prize 2024. When we launched it, it was very much an experiment. Our ambitions weren't quite what they are now. When we went into 2024, our main goals were simply to raise awareness that this benchmark existed.

What I personally found was that I kept getting surprised by the community around ARC. I remember a specific moment when the o1 preview came out and there were thousands of people on Twitter demanding that we test this new model on ARC. That wasn't my mental model of what this benchmark was or what its community would be, and it was incredibly exciting.

That moment happened again when we ended the contest, and then again when we launched the results on o3. It showed that there was real demand for what ARC provides—benchmarks that ascertain the capability gap between humans and computers.

We set up the Foundation to be a North Star for AGI and to continue producing useful, interesting, durable benchmarks in the spirit of asking what is simple, straightforward, and easy for humans but remains impossible or very difficult for AI. We're going to carry that torch until we get to AGI.

As you can see, all of the large AI labs are focusing on reasoning. I'd like to think that ARC was at least a small part of that, and you folks are very focused on open source as well. Mark Jen said specifically on the OpenAI podcast that they had been thinking about ARC-AGI 1 for years.

There you go.

Well, exactly. Tell me a little bit about that. There's the industry impact, but you guys are really focused on open source as well. How do you see those two things?

My overriding philosophy at this point is that AGI is the most important technology humanity is going to develop. If we're in an idea-constrained environment and still need new ideas to get to AGI—which I think ARC-AGI 2 shows is true—then we should be designing the most innovative ecosystem and environment across the world that we possibly can.

This is one of the reasons we launched ARC Prize internationally in the first place: to reach solo researchers and inspire researchers to work on new ideas and try to get past the pretraining regime. We knew it needed to be something beyond that, and even beyond what we have today.

If you look at a healthy, strong innovation ecosystem, it's going to be one that is very open, where there's a lot of sharing and a lot of diversity of approach. That's in contrast to an ecosystem that is very closed, secretive, dogmatic, and monocultural.

Those values of openness and sharing are what the ARC Prize Foundation stands for, in order to increase the chance that we can get AGI soon.

4. Rebuilding the ARC Benchmark

So, talking about version 2 of the ARC challenge, can you give us an elevator pitch for it?

Sure. ARC-AGI 2 is basically a new version of ARC that keeps the same format but tries to address the main flaws we saw in ARC-AGI 1.

For instance, in ARC-AGI 1 we knew there was some redundancy across tasks. We saw that very early on, as early as the 2020 Kaggle competition. ARC-AGI 1 was also far too brute-forceable. Back in 2020, after the Kaggle competitions, we looked at all the tasks solved at least once by one entry in the competition.

We found that half of the private dataset could be solved just through basic brute-force program-search methods deployed during the first competition. That means half of the dataset didn't give you a very good signal about AGI at all. The other half required enough generalization that the benchmark was still useful and lasted several years after that, but it told you from the start that there were significant flaws.

That's expected, by the way. When I started creating ARC back in 2018 or 2019, I was flying blind. I was trying to capture my own thoughts and intuition about what it means to generalize and what abstraction and reasoning are. That turned into this benchmark, but I couldn't anticipate what kinds of AI techniques would be used against it.

As it turns out, a lot of it could be brute-forced. ARC-AGI 2 completely addresses that. You cannot score higher than 1% or 2% at most using brute-force techniques on ARC-AGI 2, so that's good news.

Other than that, we generally tried to make it a little harder. What we saw with ARC-AGI 1 was that it was very easy for humans to saturate. If you're a STEM graduate, for instance, you could very easily get 100%, or within the noise range of 100%—something like 97% or 98%.

That meant we weren't getting much useful bandwidth to compare AI capabilities with the capabilities of smart humans. If you make it a little harder, you get more range. If you're not very intelligent, you'll score lower; if you're very intelligent, you'll score higher. You aren't likely to completely saturate it until you're at the very top end of the distribution.

That's what ARC-AGI 2 is. It's the same format and the same basic rules. We're using only core knowledge, with input-output pairs of grids that are at most 30 by 30. But the content is very different.

You're not going to find tasks where you only have to apply 1 basic rule that could be encoded in advance, such as gravity, falling objects, or symmetry. All the tasks are very compositional. You have multiple rules, more objects, and generally larger grids. The rules can be chained together and interact with one another.

That makes it completely out of reach for brute-force methods. As it turns out, it also makes it out of reach for the base large-language-model pretraining paradigm.

You're saying that you've made the tasks more compositional and iterative, and harder for humans. Could you give me a little more detail on that?

If you think about it, there are different dimensions of things that AI models can do, and there are different dimensions of things that humans can do. Have you explored that diversity, or could you give me a breakdown of the task characteristics?

In ARC-AGI 1, you had many tasks that were very basic, where you just had 1 rule. For instance, you might have a few objects and have to flip them. That's an example of a task that's easy to brute-force because flipping is something you can acquire through pretraining as a concept, or you could hard-code it in a brute-force program-search system.

If that's the only rule you have to apply, and you only have to apply it once, that's not compositional. It's easy to anticipate and easy to brute-force.

A compositional task is one where you have more than 1 concept, and typically they're going to interact. For example, you might have object flipping, but the objects are also falling. You have 2 rules to apply to each object at once.

That could still be found through brute-force program search if gravity and flipping are key elements in your domain-specific language. So you want to create tasks where the rules are chained to a sufficient level of depth that there's no way to find that chain just by trying every possible chain, because it would become too expensive.

Humans can still do it because humans aren't trying every possible combination of everything they know on every problem they see. They have a very efficient, intuitive way of searching for a theory that explains what they see.

My co-host Keith had the idea of doing a recursive version of ARC. The thought occurred to me that, even though we do systematic compositional reasoning, we still have some kind of cognitive limit. If we nested, say, 4 levels of ARC challenges within the same problem, wouldn't you find very quickly that humans just couldn't solve it?

If you concatenate 2 ARC tasks, for instance, you get something that's much less brute-forceable and much harder because there are more rules going on. It's not quite what I would call compositional, though, because even though you have 2 rules at once, they're not interacting with each other. You can solve them separately and concatenate the solutions.

I don't think it's a bad idea at all. It would work as a way to make ARC more difficult, with the caveat that you're not actually testing for depth of compositionality.

One issue is that it would only really work once. As soon as the person developing the AI system notices that the task can be decomposed into subtasks, it's game over. I think it's more interesting to have multiple rules at once that are chained together or interact in some way.

For instance, one rule might write information on the grid that needs to be read by a second rule.

5. Measuring Fluid Intelligence

What performance do the frontier models get on ARC-AGI 2?

What we saw was a big gap between models that don't do any kind of test-time adaptation—any kind of test-time search or test-time training—and models that do.

The base LLMs, even models like GPT-4.5, are basically scoring 0%. One of them—I think it was R1—scored slightly above 0%, perhaps 1%, but that's within the noise range of 0%.

Any model that cannot do test-time adaptation and does not possess fluid intelligence starts effectively at 0%. In that sense, ARC-AGI 2 is a very strong signal that you have fluid intelligence. ARC-AGI 1 could already tell you that, but less perfectly.

On ARC-AGI 1, if you don't do test-time adaptation, you can still get up to roughly 10%. On ARC-AGI 2, that's effectively 0%. It's a better test.

When it comes to models that do test-time adaptation, we tried some of the entries from last year's Kaggle competition—models that were doing test-time training in particular, or some kind of program search. The best model, the one that actually won the Kaggle competition, can do, I believe, 3% on ARC-AGI 2. If you take an ensemble of the top entries from the competition, you get to 4%.

We also estimate that o3 is currently the state of the art in terms of an AI model that exhibits fluid intelligence. We haven't been able to test o3 on low-compute settings across all the tasks we wanted to test, but we tested it on a subset and can extrapolate. It sounds like it's going to score about 4% on the test set.

That's not very high. There's a lot of room to go higher, and we haven't been able to test o3 on high-compute settings at all. The model that scored 88% on ARC-AGI 1—I can make a guess based on what we saw from o3-low and other models. You might get up to 15%, perhaps even 20%, if you were maxing out the compute setting and spending $10,000 per task, for instance.

But that would still be far below average human performance, which should be closer to 60%.

That 4% that o3 gets on ARC-AGI 2: Do you think of that as fluid intelligence, or do you think of it as a potential gap in the benchmark? Presumably, you could have designed ARC-AGI 2 in such a way that, if you selected the correct set of human-calibrated challenges, o3 would still score 0%.

Absolutely. You could adversarially select against o3, and then o3 would score 0%. It would be very easy to go from 4% to 0%; you would just need to change a few tasks. We didn't try to do that.

I do believe that 4% shows that you have a nonzero amount of fluid intelligence. That's something you could also get as a signal from ARC-AGI 1. The sign that you see fluid intelligence in these models is the performance gap between the huge, pretrained-only models that don't do test-time adaptation—which score effectively 0%, maybe 1%—and the models that do test-time adaptation and score nonzero amounts: 3%, 4%, or 5%.

That means that about 95% of the dataset provides useful bandwidth for measuring how much fluid intelligence the model has. You weren't getting that with ARC-AGI 1. ARC-AGI 1 was more binary: if you don't have fluid intelligence, you'll score very low, below roughly 10%; if you do have it, you'll score significantly higher, and getting above 50% would be very easy.

Because the measure saturated so quickly as soon as you had nonzero fluid intelligence, you didn't get the useful bandwidth that you're getting with ARC-AGI 2. ARC-AGI 2 should let us answer the question of whether a model is as fluidly intelligent as the average human, which you couldn't get from ARC-AGI 1.

I guess it's an economics question at this point. If you spent, say, $1 billion or $500 million, you could saturate ARC-AGI 2. I'm not sure whether you would agree with that, but if that's not the case, what are the specific things missing from o3 that are stopping it from doing better?

It's never just an economics question, because intelligence isn't just about capabilities. It's also about the efficiency with which you acquire and deploy those capabilities.

If you spend billions and billions of dollars, maybe you can saturate ARC-AGI 2. But that would already have been true back in 2020 using extremely crude brute-force program search. If you have a domain-specific language that's Turing-complete, then for every ARC task there exists a program that will solve it. It may not be very long, and all you need to do to find it is iterate over all possible programs in order of length. The first one you find is the one that will generalize, because it's the shortest and most parsimonious.

If you spend unlimited resources, you already have AGI in that sense—purely in the skill sense. You can always try every possible program until you find one that works. But that's not what intelligence is.

Intelligence is about finding that program in very few hops, using very little compute. Look at the amount of energy a human expends to solve 1 ARC task over 2, 3, or 4 minutes. It's almost zero. Compare that to a model like o3 on high-compute settings, which is going to use more than $3,000 of compute.

So it's never just an economics problem. Efficiency is actually the question we're asking. Efficiency is the problem statement. It's not capability.

Intelligence is knowledge-acquisition efficiency. O3 did very well on ARC-AGI 1, and now that it does so badly on ARC-AGI 2, the whole point of your definition of intelligence is that, given some base knowledge, you efficiently recombine it and produce new skill programs.

You're saying that, in the absence of the base knowledge in version 2, there is no intelligence. Therefore, is o3 not actually as intelligent as we thought it was?

I think o3 is one of the first models—perhaps the first model—that does show fluid intelligence. The results on ARC-AGI 2 are telling you that it's not human-level fluid intelligence.

I would consider o3 a kind of proto-AGI, with 2 big flaws to be corrected. One is efficiency. Efficiency is part of the problem statement—in fact, it's the central point. As long as you're not as efficient in terms of data efficiency, compute efficiency, and energy efficiency, it's only a temporary solution. We'll find a better solution in the future.

It's also not quite human-level. If it were human-level, you'd expect it to score something like 60% on ARC-AGI 2. We don't know what the exact number is going to be, but it's probably around 4% or 5%.

Do you think that general intelligence is a category or a spectrum?

General fluid intelligence is both. There's a huge difference between having memorized a bunch of static, handcrafted skill programs and being able to adapt to novelty to a nonzero extent.

That's a binary distinction: either you have fluid intelligence or you don't. ARC-AGI 1 could answer that question for a system. But once you have nonzero fluid intelligence, the question becomes how much you have and how it compares with humans.

That's related to the recombination of the skill programs and knowledge you have, and to the depth of recombination. If you do no recombination at all, you don't have fluid intelligence. If you do some recombination, you do have it, but then the question is how deeply you can recombine.

Using a program-synthesis analogy, how big a program can you write on the fly to adapt to a new problem? And how efficiently and quickly can you write it?

So it's binary, but it's also a spectrum. ARC-AGI 1 was trying to ask the binary question: Does the system have any fluid intelligence at all? ARC-AGI 2 is more about measuring how much fluid intelligence you actually have compared with humans.

How long do you think it will take for version 2 to be saturated, and do you think it will survive until version 3 comes out?

That's a question where you have to take resource efficiency into account. If you're asking how long it will take before we have a system that can score higher than 80% on ARC-AGI 2 using less than $10,000 of compute, I would say probably a couple of years.

It's very difficult to make predictions here. If you're just looking at current techniques and scaling up current techniques, I think it could take a while. ARC-AGI 2 is really out of reach for current techniques.

Of course, we're not limited to current techniques. In 2025, we're probably going to see new breakthroughs, in the same way we saw new breakthroughs last year. These breakthroughs are very difficult to predict. I was personally very surprised by the performance o3 achieved on ARC-AGI 1 last year. That came as a surprise.

Maybe we'll have new surprises this year, but I would be extremely surprised if we saw an efficient, human-level solution on ARC-AGI 2 by the end of 2025. I would basically rule that out. By the end of 2026, maybe—which is why we have ARC-AGI 3 coming, of course.

On the analysis of failure modes, I'm sure you saw the blog post I read that went through all the different failure modes of o3. Of course, it was the solution-space prediction that made it more surprising to me.

My take was that I was really impressed that, even when it failed, it was because the solution space got too large or it was making minor mistakes. Broadly, it got the direction of many of the problems quite well. Tell me about the failure modes on version 2.

We weren't able to test o3 as much on version 2, but I can tell you about the failure modes we saw on version 1.

There are many, but generally, this is a model where reasoning abilities can decrease exponentially with problem size. If you have more objects in the scene, more rules, or more concepts interacting, you see an exponential decrease in capabilities.

It's also a model that needs to write a kind of natural-language program describing what it's seeing, the problem, and the sequence of steps needed to solve it. In that sense, it's 100% a natural-language program.

As a result, if you have a task where the rule is very simple for a human to grasp but nonverbal—where it's very difficult to put it into words and it has no verbal analogy—that's much harder to solve for this chain-of-thought model.

Another major challenge is compositionality: having multiple rules interact. There also seems to be a locality bias, where if you have to combine bits of information that are spatially collocated on the grid, that's easier for the model than if you have to do the exact same thing but the 2 pieces of information you need to synthesize are far apart.

It also seems that the model has trouble simulating the execution of a rule and then reading the results. For instance, if you're solving an ARC task and you grasp a certain rule, then start applying it—say, continuing a line—and afterward you have to take another rule and use it to read a bit of information that you've written while executing the first rule, that is completely out of reach for these chain-of-thought models.

How multidimensional do you think intelligence is?

One school of thought—and I think you might subscribe to this—is that the universe is almost made up of Platonic rules disconnected from the world we live in. Then there's this kaleidoscope idea where they get combined together, and that's what we see.

Another school of thought is that there will always be another dimension of intelligence. We'll always need ARC-AGI 4, 5, and 6, and there will always be something missing at each step of generality that you cross.

With each step of generality that you cross, you gain a nonlinear amount of capability. After a few steps, you're so overwhelmingly superhuman across every possible dimension that you can say, without a doubt, that you have AGI—in fact, superintelligence.

Intelligence is, in a sense, multidimensional. What ARC is trying to capture is really just the fluid-intelligence aspect: the ability to recombine core-knowledge building blocks.

In my definition of intelligence, intelligence is about efficiently acquiring skills and knowledge and efficiently recombining them to adapt to novel tasks and situations that you cannot prepare for explicitly. The ability to take a bunch of building blocks and recombine them through program synthesis is one aspect of that, probably the most central aspect, which is why we're focusing on it with ARC.

But it's not the only aspect, because this assumes that you already have this pile of knowledge available. It overlooks the acquisition of these abstractions and also the acquisition of information about the task.

In ARC, we provide all the information about the task at once. In the real world, you have to collect that information. You have to take actions and set goals to discover what your environment is about and what you can do within it.

You have to do these things efficiently, of course, and that efficiency aspect is very important. Intelligence was developed by evolution; it's about adaptation. When you're exploring the world, you're taking on risk. You might get killed by a predator, for instance.

You want to gain the maximum amount of information and thereby power over your environment by taking on a minimum amount of risk and expending a minimum amount of energy. That's not something you can measure or capture with ARC-AGI 1 or ARC-AGI 2 alone.

6. The Search Behind O3

Can you expand on the significance of solution-space prediction with o3? That suggests to me something close to Rich Sutton's idea that it's nearly a blank slate and very empiricist: we take the data in, and the neural network does all of the work.

I always imagined that we would need some kind of structured approach that took the core knowledge into account. Do you think it's actually simpler than we thought, and that directly predicting the output is preferable to trying to predict the steps?

Trying to directly predict the output versus trying to write down the steps to get the output are not entirely separate things. Once you've written down the steps, you can execute them, and what looks like transduction is not actually a true transduction model.

It's much closer to a program-synthesis model, where the system is searching for the right chain of thought to describe the task and list the sequence of steps needed to solve it. Once you have the chain of thought, you can use the model to execute it and get the output.

From the outside, if you treat the entire system as a black box, it looks like transduction. But the same would be true of any program-search system. What it's actually doing—and the reason it's able to adapt to novelty so well—is synthesizing this chain of thought, which serves as a recombination artifact for the knowledge and skills the model has.

That recombination artifact is adapted to the particular task at hand. It's much closer to a program-synthesis model.

The community found this confusing because, in the last interview, you described o1 Pro as a kind of explicit search process. What seems to be the case is that there is some reinforcement learning during pretraining, and then maybe it does some sampling at inference time. Are you saying it's as if it's doing a program search, or are you saying it's somehow explicitly doing a chain-of-thought program search?

It's searching over the space of possible chains of thought and finding the one that seems most appropriate. In that case, it's entirely analogous to a program-search system, where the program you're synthesizing is a natural-language program—a program written in English.

It just seems strange to do autoregression on a language model and characterize that as a search process.

A model like o1 Pro or o3 isn't just autoregressive. It has a test-time search step, which is why it can adapt to novelty much better than the base models that are purely autoregressive.

That's why ARC—even ARC-AGI 1—has completely resisted the pretraining, purely autoregressive scaling paradigm. From 2019 to 2025, we scaled these models by roughly 50,000 times, from GPT-2 to GPT-4.5. Even on ARC-AGI 1, we went from 0% to something like 10%. On ARC-AGI 2, you're going from 0% to 0%.

Meanwhile, if you have a system capable of test-time adaptation, such as test-time search—o1 Pro or o3—then you get much better performance. There's a huge performance gap.

In general, you can tell the difference between a model that doesn't do test-time adaptation and one that does by looking at this generalization gap on ARC. You can also look at latency and cost.

A model that does test-time search is going to take much longer to give you an answer. If you look at o1 Pro, for instance, it's taking 10 minutes to answer your queries. It's also going to cost much more because of all the work it's doing.

I could download the DeepSeek R1 model and run it on my machine. As far as my machine is concerned, it's just a normal LLM doing greedy sampling autoregressively.

That's right, which is why it doesn't adapt to novelty and scores basically 0% on ARC—maybe 1%.

So you're saying there's something different about o3, and that it's qualitatively different?

That's correct. It's qualitatively different from all the other models that came before it. It is actually a model that has fluid intelligence. It has a nonzero amount of fluid intelligence, whereas R1, for instance, does not.

Categorically, it's doing some kind of active search process at inference time?

That's what it looks like. Of course, I don't actually know how it works, but that's what I would speculate.

Would you be shocked and surprised if it came to light that it was just doing autoregressive greedy sampling?

Honestly, I think that's very unlikely because it's completely incompatible with the characteristics of the system that we know about and that we were exposed to when we tested o3.

Do you think there will always be human gaps?

Probably not always. Today, there are very clear and significant gaps. We're not actually that close to AGI right now.

Eventually, as we get closer and closer, there will be fewer and fewer gaps. At some point, we'll have AI systems that are overwhelmingly superhuman along every possible axis you choose to look at.

I don't think there will be gaps forever.

Tim, thank you so much for doing this. I appreciate it, and I'm looking forward to seeing it in a couple of weeks.