ARC-AGI-3 winning team - Millennia of minds, compressed into words.

But we have the benefit of millions of years of evolution, right? And so it's almost a bit unfair that we're expecting AI algorithms to be able to do all of that.

Jon Kotar

Exactly.

I think it boils down to one of the big open questions in the field: Is language critical to intelligence?

Jon Kotar

I will say, at least when I'm playing the games myself—and I think that goes for all of us—I'm using language.

My name is Jon Kotar. I have a background in physics and mathematics, and I've been working in the industry for about 15 years, with the last 5 years specializing more toward AI. It was immediate to him what the goal was. He recognized the pattern, and I showed him the game. Within 3 seconds, he finished the first level.

Often, the agents start thinking that reducing the energy bar to the minimum is the goal, or that stepping 10 times in a region is the goal. For a human, it's clear that that's not the actual goal.

Stephano

My name is Stephano. I study computer science and machine learning. I did some research in reinforcement learning at EPFL.

I guess the million-dollar question, though, is: Do you think it's possible in principle to do really well on ARC-AGI-3 and be no closer to AGI?

Stefano

Yes, I do think it is possible.

I don't know. I guess the main idea is to have a bunch of bright people in the room and do good research together. The transformers can't plan, but they can do a very good job of pretending. Essentially, that is, in a sense, indistinguishable.

36% might be misleading as a number if you don't look behind it. What it really measures is action efficiency.

D Smith

My name is D Smith. I studied electrical and computer engineering. I specialized in reinforcement learning for the past 5 or 6 years, and I've been doing research on the ARC-AGI-2 and ARC-AGI-3 sides, with some quant work in between.

Yes, perhaps that tells us something about the benchmark. There's some leakage of human priors into the games.

Michael

Hello, I'm Michael. I do a master's in data science, and I'm super excited about ARC-AGI. I love solving it.

In the example that we saw, we have a maze, right? There's no innate primitive in a newborn that says, “This is a maze.”

This is one of the official ARC-AGI games. This is one of the first games that was released. It's called LS20. Here you can see a 64 × 64 frame, which is typically how the game observation is presented to the agent. You get an initial frame, and each of these pixels can take 1 of 16 colors.

Now you need to figure out what the goal is, and even how the player moves—whether it's a specific player or object that's moving around, or some other, more abstract game. You need to figure this out.

Very often, our harnesses will break here. They actually got the idea right: You need to move to the exit with the 2 things aligned. But then they think they should win here, when actually you need to move up 1 more time. It's quite common that they fail to beat the level from this point because they can't think that they can move there. They think, “No, but that's outside the maze, so you can't move there.”

And a human would just give it a go and see what happens.

Michael

Yeah.

But somehow we can't get our harnesses to also do that. The best recipe we have today to build intelligent systems is scaling up these large language models. I think Sam said you should definitely not be trying to train LLMs yourself. I don't recommend it.

Dario Amodei

Why not? I think the easiest is just to go work at Anthropic. That's the easiest. I have a sense I cannot prove. I think this is the time things start to get weird.

Yeah, I think he's right about that. Most fundamentally, I want to understand intelligence: how the brain works and how to build intelligent machines. It's a big mystery.

I guess when I was young, I was excited about both computers and brains. With computers, I feel we now have a clue how they work, and I studied computer science. That mystery is kind of gone, but the mystery about brains is very much alive.

I have a strong sense that most of society is bottlenecked on cognitive labor, insight, and basically intelligence. So if we have more intelligence, that would be a big unlock for many things.

You won the preview competition, and that was using a different method. Then you found that it didn't work very well when the main competition came out, and you moved toward this.

D Smith

Yes. For the agent preview competition last year, I actually tried something completely different. The goal of that competition was basically to test whether there were obvious solutions that broke the mold—broke what they were actually trying to accomplish. There were, and the solution was basically just to brute-force actions.

What that solution did was basically search through the large action space for ARC-AGI-3. We have more than 4,000 actions, which makes this a difficult problem, but you could theoretically search through all of that, or just randomly search through actions.

What I did for Stochastic Goose was basically search through a large number of possibilities, but I tried to only search through actions that resulted in a frame change. Initially, if you took an action that did not do anything in the game, nothing would change—not even the timer bar. That allows you to model that behavior and see, “Okay, does this action change the timer bar?” If it doesn't, we downvote that action in the future for that given frame.

There were also only 3 games, so you have to be selective about how you improve things. You can't pretrain because you'll just overfit. Exactly what I did was use an action model that learns which frames are valid for a given transition. Then it was more about the engineering of being able to learn within 100,000 actions, because that was the maximum action limit we could get within the time limit we had.

That seemed to have worked well. It solved, I think, 18 levels across the 3 games, and I think it solved 2 of the games as well within the time limit that was provided to us.

But they then hardened the competition specifically against that. In the new competition, the games are much harder. The timer bar moves even if you use an action that's valid but doesn't really change anything in the game, and more importantly, they introduced action efficiency.

This makes it very difficult to brute-force. You have to be very direct with how you explore, and that's where LLMs come in. Even though it's slower, being able to somehow guide the exploration helps a lot. The possible combinations are too many to just brute-force, and obviously your score goes to 0 quite quickly.

They've made it so that if you go, let's say, 2× or 3× above the human baseline, you're already close to 0. Even though it's slow, it just helps. Otherwise, it's kind of intractable for us.

We tried other methods, such as directly predicting actions—more of a transductive method—where you just have the input frames as context in a long sequence model and predict the actions. But that also doesn't generalize well, and it doesn't really make intuitive sense.

If you play the game—for example, the first game here, the maze level—you would intuitively play 1 or 2 actions, then think about your path and go to the end position. But if you have to think at every step with the same computational capability, or the same budget, then you might be misrepresenting where you should go at the start. At the end, for straight lines, for example, you don't have to think that much; you can just batch those actions.

That's where the coding-agent idea came from. We also had 2 good literature results where they scaled using Opus models. One was Argentica, and the other one was the RGB agent, which also showed good results. Given no compute constraints, you could use closed-source models, and we took that as inspiration.

You mentioned transductive as well, which is quite interesting because, roughly speaking, I think of transduction as making a prediction about this specific test instance. It's quite an interesting discussion whether or not this is transduction because, even though it's chain-of-thought, it looks like a form of induction in the sense that it's a rationale that could be cross-applied in the future.

You could use the memory in the agent. You could do some kind of library transfer and make it inductive. But at the moment, if it's only for the sole purpose of this particular problem, would you call it a transductive method?

D Smith

I would call our ARC-AGI-2 solution more transductive, and this slightly more inductive, exactly as you mentioned. You can actually read the reasoning trace and understand when it's understanding the game and making progress, and when it's not.

It has this reasoning chain-of-thought, which is in English, so you could reasonably understand it. I would actually say it's more inductive. Previous attempts, as we mentioned, were where the agent just directly predicted the actions. That would be more transductive, and it doesn't seem to work at the moment.

Obviously, there are a lot of ideas to try, and I'm sure the community will come up with something interesting to make that work. But for now, this seems to be the way for us.

Yeah, I think the action efficiency makes this problem really interesting. There's so little time to learn, and if you want to get 100%, which is the grand prize, then you really have to outcompete a human, who can take a lot of time just to think. Time is not a constraint for the solutions.

So I think that's really interesting. The reflections and looking back at your past experiences make LLMs a very flexible model for this solution. Interestingly enough, even though there shouldn't be any prior, a lot of the game priors are already included in LLMs.

For example, here we're looking at a maze, and a maze is something that every LLM, even the small ones, will know and recognize from images or ASCII grids. So even though not all the priors are stripped away, there's still enough game prior that the LLM can lean on. Those aren't encoded in any reinforcement learning model, or a pure Stochastic Goose solution wouldn't have that encoded anywhere in it—that the maze is a thing—and that really helps to direct the reasoning and actions of the model.

The Benchmark With No Instructions

So, one of the main constraints was basically 2 weeks. I joined the competition late—2 weeks to do it. There are a lot of things that can be improved, but I think it was a good initial solution.

The main constraint was that we only had 3 games—3 public games and 3 private games—to evaluate on. So we had to make sure that, whatever the design was, we didn't assume too much about the environments and the games. On the private leaderboard, or even on the public leaderboard beforehand, you could see that I wasn't close to the top because it was super easy to overfit.

The basic idea was that we couldn't pretrain; we didn't have enough environments. So we thought, let's try test-time training like we did for ARC-AGI 2. But how do you do that? You can try reinforcement learning, which I have a lot of experience with, but it's not that obvious how to do reinforcement learning. Every time you get a new game at test time, the only real reward you have is a level transition. Once you transition a level, you never go back, so it's not like you're trying various different routes to optimize for passing a level. You just have to pass it once.

It didn't really make sense to use pure reward-based RL. I did a lot of research in the past on curiosity and unsupervised RL, and I thought perhaps that's a better approach where we don't have explicit rewards. We're just optimizing for curiosity, like exploring new areas that the agent hasn't explored before.

I tried a bit of world modeling. The basic idea is that the game isn't deterministic, so you can use a world model to perhaps take the input frame and the action and try to predict the next frame. If you can't predict it well, you have the policy explore that more, and that's the reward you're optimizing. That was a promising approach, but I couldn't get it working in 2 weeks.

What I then assumed was that any frame change is interesting. You want to explore things where the state changes; you don't want to explore things where the state stays the same. That was the basic derivation of the path I followed toward getting the action-model implementation for Stochastic Goose. Then it was down to basic reinforcement learning questions, like how do you get it to learn within 1,000 time steps? You have 100,000 time steps, but you need to start taking useful actions within about 1,000 time steps.

That meant using a replay buffer, hashing the experience so it doesn't go over the memory limit, doing some prioritized experience replay, and handling some basic engineering to get it to work for that competition.

Yeah, you mentioned exploration as well. I guess there's a bit of an elephant in the room, which is that Chollet is talking about the acquisition and synthesis of abstractions, and when reinforcement learning folks talk about exploration, it seems to be in quite a surface-level, superficial way—in terms of entropy or things changing.

Do you think that's in any way against this idea that we can acquire deep abstractions about the domain?

The Benchmark With No Instructions

Yes, definitely. For Stochastic Goose, it was not at all what I guess Chollet ultimately wants as a good solution for ARC-AGI 3. I can't speak to what he actually thinks is a good solution, but that was purely to see if I could maximize this competition within 2 weeks.

I think the other things we're doing on that side are perhaps more in line with what that initial vision is. For example, you have these reasoning tokens, which have an abstract representation of objects that we then manifest or write down in English-language textual tokens.

I would say that's more abstract. You identify objects, try to find out what the mechanics and dynamics of the game are, and determine what the goal is. That might be more abstract. It's also kind of like a neural-guided search, and the Python programs it creates are executable Python programs that extract and build some sort of simplified world model and search over that world model using actual algorithms, like breadth-first search.

There is understanding debt, right? You're building this really, really complex thing, and after a while—have you noticed in Claude Code that they don't even show you the code anymore? You can expand it, but by default, a lot of people don't even look at the code.

Do you think that you need to be familiar with the deep abstractions in your codebase in order to build mental models and evolve and extend it? Do you get lost in no man's land?

The Benchmark With No Instructions

Yes, this is an active discussion within our team. We don't have a clear answer yet, but I think you need a deep understanding of some important parts. Some other parts, like, let's say, a web viewer, you can vibe-code more easily because if it breaks, it's fine. But the core logic—for example, the implementation and evaluation—is especially important. We have to make sure it's working.

It is the case that we're gradually understanding less and less of our own codebase, and we're struggling even with reviewing some of the changes. You might use Codex to help review some of it because it's such a broad change, or you need to split it up. But things are moving so fast that you can't just manually write everything.

If you manually write everything, you can't keep up with the rest of the team. So, I guess this is an active discussion. It's difficult.

For me, my career has been going on a bit longer, and for most of my career there were no coding agents. That means I also have a bit of an opportunity to still make use of patterns that have been useful in the past. I think one important one that, as a team, we are more and more learning to use properly is requirements-based engineering.

The Benchmark With No Instructions

So we will formally write requirements, really following detailed prescriptions, like numbered requirements, and specify how they are tested. We might still have the coding agents helping there, but it's mostly handwritten. We review that as a team, and from there we can much more confidently hand it to a coding agent to implement than if it's just a single prompt.

Yeah, I love all of that. But I suppose there's always been this thing, right? In software engineering, we have specifications, documentation, user stories, requirements, and all these things.

There was waterfall versus Agile. For me, the difference between waterfall and Agile is this admission that there is epistemic uncertainty, right? Reality pushes back. We discover contours in reality that we didn't see before, and we need to adapt the requirements over time.

My worry is that when you crystallize the process, there's more to tacit knowledge in developer teams than the requirements. There's the generative process that created the requirements in the first place. When you only write software for the purpose of satisfying the requirements, the ability to evolve the requirements gets deteriorated.

The Benchmark With No Instructions

Yeah. What is always the case—and that was the case in classical, pre-coding-agent development as well as it is now—is that you don't write requirements and have those requirements frozen, carved in stone, and then write code. It is an interplay.

When you start implementing, you might realize, “Wait a minute, this combination of requirements is not possible.” You don't then just implement something that approximates the requirements. You go back—and we also tell our coding agents to do this—and say, “These requirements don't work. They need to be adapted,” and you make a conscious choice there.

We will also very often go back with a coding agent and say, “Can you very carefully check that every requirement is satisfied?” In fact, we have it write specific text for each requirement, like, “This is how it is being satisfied in the code.”

We try to really write the tests around the requirements rather than just having random tests for every line of code. Then you get that classical interplay that you also have in classical engineering, which you can do with your coding agents.

This is a kind of AI-human co-creativity. Humans have the taste, they have the intention, and they have something that coding agents don't have. The bull case would be: wouldn't it be amazing if you could stick the requirements in and the agents themselves would understand, “Oh, I see what they meant,” and could evolve the requirements?

I think in the short term—I can't say for the long term—at least what we're trying to do is find the areas where we can see the model still failing, like high-level conceptual ideas about the problem. Then we try to make sure, when reviewing code or creating code, that we ask specifically around those areas and make sure it's correct and understands us well.

The Benchmark With No Instructions

But yeah, I guess it's always going to be a game of cat and mouse where we're going to perhaps hand over more to the AI assistants. In some areas, it might just totally fail, and we need to take responsibility there. We need to make sure we understand what it's doing and that it understands what we want.

Longer term, I guess if you're talking about AGI or superintelligence, it's up for debate. [Laughter] It is certainly accelerating us in doing experiments, because something that would just sit in your scratch pad, you can paste into a prompt and see if it sticks. You can definitely experiment overnight with a new idea.

I think the real added value that I see of the lab right now—and what the AIs and coding agents are missing right now—is literally the attention to detail. This task is very long. If we have a long context, the execution of one game might take an hour just for it to solve one level.

In principle, you could point all the research at this and say, “Hey, just improve the score,” right? But that doesn't work, because it cannot really understand where the agent fails. It's not as simple as Andrej Karpathy's autoresearch challenges, where you just minimize a loss and there are just parameters. You need more insight into this task.

Mike Knoop

I think that's where a lot of improvements came from: not from vibe-coding another 2 million lines of code with a coding agent trying to optimize this task, but from reading deeply into the program and the logs, understanding what the issues are, and improving those one by one.

We observe a similar failure mode to what we observe when they play games. For instance, if you give them an autoresearch task, they might overfocus on details and fail to see the big picture. They don't zoom out and see things from far above, and they might start optimizing some hyperparameters for a 0.01% improvement while there is some other big improvement that could be made at another abstraction level.

We see a very similar failure mode in games. If, at the beginning of gameplay, the agent gets locked into the wrong hypothesis, it's extremely hard for it to escape that. It convinces itself that that's the right path and the right hypothesis to follow, and it's not able to escape it.

It's quite interesting that across models—smaller models and frontier models—we observe similar failure modes. They aren't able to jump between abstraction levels, zoom in on the details, and then see the higher-level picture.

At the same time, I just wondered: how do you guys think about the abstraction mountain? For Chollet, it is bottom-up. You start with the core knowledge—agency, spatial knowledge, stuff like that—and synthesize upwards.

Whereas LLMs are interesting because they learn these fractured, entangled representations that are quite high-level but also quite generalizable. You can stitch them together, repurpose them, and canalize them in different contexts, but it's always a little bit lossy.

If you're doing this library learning and transfer, you always have this issue of, “Well, yeah, there's this abstraction over here, and it kind of works, but it's not quite what I want. I can still use it anyway.” Is this kind of reasoning something that should be first-class, or do you think it could just be implicit in the harness if we had a memory system?

Mike Knoop

You were just describing how our LLMs will indeed have, let's say, the core-knowledge prior as something somewhat vague but not fully testable, yet still usable. The thing is, if you replace LLMs with humans, I think it's still true.

We also don't have formal core-knowledge priors that are fully exact, that we apply to the world and use to describe the world fully and exactly. We have vague, trained, overall life understandings that evolve over time.

If you want human intelligence, that does seem to be the paradigm you want there. I think the LLMs get the bonus that they can also reason very well, and much more implicitly than we humans can in code. If they want to get exact, they can.

But I think if you want to understand the world, certainly beyond these simple games, you will need more than that vibe understanding, basically.

I'm guilty of always suggesting what Chollet thinks. I don't really know what he thinks, but I've got a pretty good simulation of Chollet in my mind. I think he's influenced by this kind of nativist-psychology-type thing, and he thinks that a lot of the reasoning we do is almost Platonistic.

Somehow, the laws of nature imprint these primitives into our minds, and we compose these primitives together for certain classes of problems. We can do abstract System 2 reasoning, and for those types of problems, we compose these things together.

But you're absolutely right: there are so many things in the world that are actually really complicated, like navigating relationships or even pathfinding in a complex environment, on the Tube network or something like that. We do a little bit of both, but there's at least a pocket of pure reasoning. I think that's what he thinks.

Would you consider LLMs pretrained on the internet, and then, if you apply them to a new pattern using in-context learning, a form of skill acquisition on the fly? With reasoning as well, would that not be a form of perhaps skill acquisition on the fly?

It has a lot of core knowledge, but there's also a different part where it has to adapt on the fly. You can give it some example that might not be on the internet, and it might adapt. Sometimes it fails, but it is still a distribution. At least you get some adaptability.

For example, when you started prompting it to think, like with chain-of-thought prompting, it started improving because it had more time to reason and adapt to what you were asking. If you train it using reasoning, you can actually do much more, because now it has more time to reason, figure out what you're actually asking, and form new abstractions for solving a specific problem case, as opposed to just regurgitating what it's seen on the internet.

Mike Knoop

LLMs trained to do reasoning are intelligent because intelligence is just adaptation. I know this now, and I need to combine what I know over several steps to solve a particular task, in the partial-knowledge regime where we have a verifiable function and can do hill climbing.

What we see, though, is that they combine fractured and tangled representations. They get the right answer but for the wrong reasons. They create a spaghetti monster, whereas we take a more valid path and can acquire and reuse abstractions.

The models themselves don't seem to work bottom-up. They have an understanding and representations, but they're very fractured and entangled. That's still useful, but it's a kind of statistical, low-level intelligence. I think you could, in principle, still do abstraction from fractured, entangled representations, but we're kind of in statistical land. It's performance, not competence.

I don't have strong views on this, but I have a feeling that we're doing the same with our minds. If you look at the evolutionary process that led to us, where we are at the moment is that we've also evolved to work well with spatial information. That's one of the core priors in our brain, along with some level of reasoning.

That might also be why intuitively understanding quantum physics is so difficult: we overfit to our world and how we've evolved. I'm still not sure whether the human brain is also just this core prior, and whether pure reasoning is some guided search to find answers based on what we found through the evolutionary process to work well—to survive in the world and propagate.

Mike Knoop

It's the million-dollar question, isn't it? Our brains think symbolically, but our brains are actually just a bunch of neurons. Is it a post hoc confabulation, or is it actually a causal basis function of our cognition? If it is just a confabulation and it's not the basis, then maybe AI doesn't need it to be the basis either.

But, Chollet—I mean, I don't know if you guys want to talk about what Chollet is doing—but he's building this approach where we actually acquire and synthesize program graphs using neural networks as guidance.

Mike Knoop

It's a tricky one because, for me, coming mainly from a more classical machine-learning, Bayesian perspective, it feels like this should be the right way to go. Certainly, when I learned about these things, and very broadly also swept things like DreamCoder into that regime—although it's not quite the same thing—it definitely felt like it was more natural to think of it in that way, and perhaps more natural to the way we reason.

At the same time, it is not competitive at this point with LLM-based approaches. Who knows what happens in this competition, but we don't see those approaches necessarily being the best on ARC-AGI-1 and ARC-AGI-2.

And we really don't see how we could use them on ARC-AGI-3. That could also simply be because they haven't had the same attention or the same resources put into them. But, yeah, there is that mismatch. Why don't they work if they match our intelligence so well? I'm trying not to use the word symbolic AI in general, because I think the more abstract way to think about this is constraints. Chollet's been using the language of constraints, and we wrote an article about creativity.

I think that creation, whether it's intelligence in a known domain or creativity in a transformative, unknown domain, is all about respecting constraints and acquiring the constraints of a problem domain. This is why harnesses, tools, and code work so well, because you're increasingly imbuing domain constraints into the learning and acquisition environment. So we find ourselves in this halfway regime where we have a constrained, coherent execution environment, but it's only really steered and legible in the sense of these functional descriptions that we're talking about. Rather than having some low-level, bottom-up competence, we have a kind of intermediate competence, described in a similar way to the blind man and the elephant. From many perspectives, using many functional descriptions, we're using that as the basis of constraint building and knowledge acquisition.

What I find really interesting about the core knowledge prior and what you said is that we have these representations, and they're very fractured, but somehow they work. I think there are 2 levels of abstraction—2 levels of intelligence that humans have. There's the core knowledge prior, and then there's what we learn through life. The games that have been tested with humans have been tested with humans who were not infants for our challenge. There's a lot more that we accumulate over generations through education and so on, and those abstractions are way more powerful.

In the example that we saw, we have a maze, right? There's no native primitive in a newborn that says, "This is a maze." But this has been, over generations of problem-solving by humans, a very well-established concept. If we can inject this into an LLM, we can skip many levels of abstraction. I would say that's why, when we play these games, some of them for the first time are also very difficult for humans. If this is done well and there's nothing like a maze in a game, and it really boils down to the core knowledge prior, then it's very difficult even for a grown-up to synthesize the correct rules.

However, most often—and that may be a failure of ARC a little bit in its design—these games usually have some sort of, "Oh, you're shooting a ball," or you're solving a maze, or you have an enemy, right? These are, as pure as they're trying to be, never going to be pure because they're made by humans. That's why we can do so well with LLMs, I think, because a lot of these—like the maze is the canonical example—can be really well done with LLMs. They have these fractured, fractionated understandings. But isn't it interesting that when humans use LLMs, we can make them act as if they understand because we can say, "Well, you can put a framing in the prompt. You can say, 'Think about this problem like it's a maze,' or 'Think about this problem like it's tic-tac-toe,' or something like that." What you're doing in the prompt is basically setting constraints on its generation.

Mike Knoop

Yeah. And then the more you work on a software engineering project, it doesn't quite understand. You put more constraints in, you update the prompt, and it converges. After a while, you don't need to repeat yourself. It understands. So it's almost as if, when you get LLMs in the mode, when you make them track this perspective where they respect the constraints, then it understands. But the million-dollar question is: how can we just make it understand autonomously? How can we give it a novel domain and do what we do, which is say, "Ah, this is basically a maze. This is how you need to think about this," and get it into the understanding?

A good way to look at this is to look at the function that they learn during training, because an LLM becomes very good at understanding language and operating on anything that's language-related. All of the work that we do in the harness is trying to put this novel domain into language. We're trying to represent the game, or whatever domain you're trying to make the LLM act on, in a way that's language-related, so it's closer to its pretraining prior and allows it to act on it. If you're not able to do that, since the LLM has this kind of prior from pretraining, it's way harder to make it act on a game because it's such an out-of-distribution domain compared to what it was trained on.

Yeah, I also want to comment a bit. You mentioned, indeed, that you often—I think this is easiest to do with coding agents—use your analogy that you kind of have to say, "Yeah, but you need to prompt it a few times before it really gets what you want."

Mike Knoop

Yeah.

But I don't see—let's say, having worked a lot with instructing human teams of software engineers, for example—I don't really see a huge difference there, in the sense that you can't expect to just tell someone, "In 2 sentences, I want something like this," and then get the right thing. That's a failure on your part not to specify what you want. And we see, indeed, that requirements engineering—requirements-based engineering—takes a lot of thought. We might spend days getting the right requirements, but then it is, I would say, reasonably well capable—

Mike Knoop

—of doing the right thing to begin with, basically, but—

Yeah, it's also our job to specify it, right?

Mike Knoop

I call it perspectival blindness. You hire a new person, and you build an internal wiki and specify all of your learnings, and you think, "Oh, I've distilled this. I've crystallized it. The new starter will just be up to speed within a couple of days." It doesn't work like that. There's almost a weird, tacit, almost causal graph to experience that has a provenance, and it's only when the new starter plays with all of your tools, tries different things, and has counterfactual representations and whatnot.

So, yeah, it's a very human thing as well. The general thesis is that knowledge is quite path-dependent. It's not about where you end up. It's not about these abstract functional descriptions. Sometimes it is, if they're so incredibly idealized, like calculus or something, that they're the basis of evolution on their own. But so much tacit knowledge actually requires the path of how you got there in order for you to continue the path.

At first, the scoring was based on the second-best human attempt for efficiency, and that was very skewed toward a good solution, right? Then it was changed to a median, I think, but it was second-best per level as well. It was also almost impossible even for the best human to solve a game.

Mike Knoop

Yeah. So it was based on every level. It was really difficult for even a single educated human to score 100% on this benchmark, right? We don't know much about what the failures are. We have played these games and tested them ourselves, and sometimes we don't get them. It's not that I struggled with some of them and eventually solved them, but sometimes I would spend an hour on some of them to solve them. So you can't really say this is something that's immediate. It's also difficult for humans, and I think it's really useful to look at these counterfactuals: what human engineers miss versus software engineers, as you just said, or, more generally, what humans actually fail on in these tasks, rather than, "Oh, a human can solve it if you give it 10 attempts," or something.

Really interesting point. This is the human difficulty calibration. And ARC-AGI-2, for example, a task was let in if any 2 humans could pass it. This is a bit of an ideological thing because I think Chollet believes in this notion of general intelligence. In his mind, someone with general intelligence, because they have the core knowledge, could, inside a certain number of steps in the combinatorial closure of these core knowledge primitives, solve any of these problems.

The counter to that is that, as we've just been describing, maybe human intelligence is quite high-level, fractionated, pattern-based, and quite specialized. Some humans can solve certain tasks because they have a certain perspective, certain experiences, and whatnot, and they can combine those fractured representations to come to an answer. But maybe they're just in possession of those representations and another human isn't.

Mike Knoop

I have a friend who's a professional esports player. I showed him one of the games, and he completed the first level without spending an extra move that he didn't have to.

It was immediate to him what the goal was. He recognized the pattern, and I showed him the game. Within 3 seconds, he finished the first level with, let’s say, superhuman performance. And, okay, he has been playing games professionally for 5 years, right? He’s been trained on this, but exactly as you say, there’s something we can bootstrap on. We do bootstrap on it, and it does work. So maybe replicating that with LLMs is enough.

Mike Knoop

Yeah, I think gaming is a really good example. A good friend of mine is called dDK, and he does commentary on Counter-Strike and stuff like that. When I watch him do commentary, it seems super situational. In Quake 3, you have to time the mega health, then there’s the red armor, and then there’s a position over here. What you get is the emergence of these complex, situational phenomena in the game.

This doesn’t seem anything like what Chollet is talking about, because there are 2 worlds, right? There’s the emergent-complexity world, and then there’s the reductionist, core-knowledge world. I think in this world, we can still use intelligence, and we can still acquire abstractions and descriptions for these high-level phenomena. But what do we do? Do we analogize them in terms of low-level knowledge we already have, or are they something new? It seems like a different modality.

No, I don’t really have a good answer to that. I think I can only base my answer on these examples, and I completely agree with you that there are some abstractions like that. Conway’s Game of Life is still path-dependent, right? You can still trace a path from the low-level description to the high-level phenomena, but the reason it’s unintelligible and surprising is because the path is a long one and it’s computationally irreducible.

You have to perform every single intermediate computation to get there. It’s confusing for us because we can’t actually analyze or understand the path between the causal stuff that made this high-level phenomenon. We just see it on its own, disconnected, and many phenomena in the world are like that.

Some sort of emergence, where you just have to call it emergent because you can’t trace it. The computational graph of that is just not tractable for you.

Mike Knoop

That’s true. But another feature of emergence is compression. With the glider, for example, we almost don’t need to know how it came about, because the new description encompasses it. It’s a complete description of the new phenomenon, so it’s almost like the start of a new piece of knowledge. A lot of our knowledge is like that: it’s such a good compression of what went underneath that we can start from it.

So, for example, with calculus, within the closure of calculus or probability theory, we can do a lot of things. We don’t need to know how it came about. Is that a new layer of knowledge that we don’t care how it came about? It’s just saying, do we need to start from the bottom, or, in many cases, can we say, “This has come about, and we can just use it higher up the abstraction mountain”?

Mike Knoop

Yeah, I think ARC is an example of the case where you can do this at least to some level, because we see the Frontier LLMs scoring quite well on these, and they wouldn’t be able to if these core priors were breaking it.

But I think what is really nice about ARC is that there are some games that are so different and are still easy for humans, but somehow break this concept. You talk about this closure, and I gave the example of a maze. You can maybe change the core-prior concepts, and they would still make sense to a human, but they would somehow adversarially break this calculus-like, maze-like representation that helps solve the game.

You can move them around a little bit, and suddenly you break it. You still have a valid game for a human, but not for the abstracted LLM intelligence. That’s what we see with some of the difficult games. I think it’s really difficult not to move the priors too much so that you can still make it human-solvable, and sometimes ARC unfortunately fails at this.

I want to talk a little bit about ARC-AGI-3. The first 2 versions of the ARC Challenge were quite abstract because Chollet was trying to idealize intelligence on its own in the most abstract, legible way. Now ARC-AGI-3 introduces, in my opinion—and we can talk about this—the concept of agency.

Agency, in my definition, is the ability of an agent to have goals, plan, and realize those goals. The more ambitious the goals are, and the more you realize those goals, the more agency you have. There’s a kind of low-level, no-nonsense definition of agency, which is just a thing that can sense and act. If I’m doing computer programming, that’s what an agent is.

But the more cognitive-science definition of an agent is one that has future-pointing control. I have these big goals in the future, and I can realize them, and that makes me an agent. I think ARC-AGI-3 introduces agency not just in terms of realizing goals, but also acquiring goals over time through interaction and adapting them.

Not only do the levels themselves require interaction to be solved, but all the levels change. These goals dynamically change, so you cannot learn them directly.

On the concept of agency, I would argue that one of the interesting things we saw is that, when I came into this challenge and started working on this problem, I thought it would be difficult for the agent to have any idea what it should be doing at all when it tried to solve the first level.

But by now, we’ve got at least 1 solve on all the public games, at least the first level. It may be by trial and error or by luck, but still, LLMs somehow seem to be able to figure out what they need to do. That might be through the different biases that are already encoded for humans who have been building these games.

Somehow, it is possible to generate these hypotheses. They may not stand for long, and they might be random, but there are signs of life that LLMs can already do this, I would argue.

Certainly, there’s that Anthropic biology paper, and they talk about all the capabilities of language models. One thing they said language models can do is plan, and planning is required for agency, in my opinion.

We know that language models can’t plan because planning is a type of computer-science algorithm that is not possible in the substrate of transformers. What they do is a kind of simulated planning, right? They can simulate planning up to a certain depth. Do you think that matters?

The Benchmark With No Instructions

Maybe one important thing here is to realize that there are 2 types of planning involved in playing an ARC game. At some point, when you understand the rules and what you need to do to win, you need to plan a path to get there. That’s more of a traditional game-theory planning algorithm.

You might use a search or Monte Carlo algorithm, or one of many such things, which the model is perfectly capable of because it can write Python code and run the Python code. That’s 1 level of planning.

But the other level of planning is that, when you enter a level and see it for the first time, you need to figure out how you’re going to figure out the rules. “That looks like it might be a player. Let me try to move in that direction. Ah, okay, pushing up moves it up.”

You need to balance exploring and exploiting. For example, when do I start trying to win? How long do I keep playing around? That’s a whole different level of planning, which you can’t encapsulate in Python code.

We see our agents managing that quite effectively, so in a sense, yes, maybe transformers can’t plan, but they can do a very good job of pretending. Essentially, that is indistinguishable from what might be formal planning in a computer-science sense.

I hadn’t quite incorporated that. You’re absolutely right. Even if they intrinsically can’t explicitly plan, they can call tools. They can write code that can plan.

There’s a bit of a galaxy-brain side discussion there about whether they can effectively explore the space of programs to do planning, or whether it’s enough that they can statistically generate programs that do enough planning for the circumstances they’re used in. Maybe that’s too galaxy-brain for this, but they can basically plan in essence. That’s what you’re saying, right?

The Benchmark With No Instructions

Yeah. What it comes down to is that, whatever is under the galaxy-brain understanding needed to understand what’s going on, they seem to be able to do it to a degree that lets them play these games, essentially—whatever it is they’re doing and whatever it really means.

Okay. In which case, where’s the gap then? If they can plan, is it a representation problem? What’s the big gap?

Maybe it’s first good to discuss what the actual gap is, because indeed, if we see, for example, what ARC is publicly communicating, they show the benchmark where even Frontier models are not able to play the games to any significant degree.

The Benchmark With No Instructions

So this score is under 1%, which either means they’re not beating any levels or they’re being very inefficient about it, doing way too many actions. However, if you look at publications, you can see that the score can go up to, at best, 35% or 36% if you give them a proper harness.

So the point is they need some guidance. You need to tell them more, like, “Okay, you’re in a game. You’re going to do this. You can write code.” But with that guidance, they can actually do pretty well, and that guidance is not game-specific. It’s not like, “Oh, maybe this is a maze, and you need to think about this.”

No, the model still needs to do that explore-versus-exploit balance. The model still needs to do all the move planning, but it gets some kind of guidance on the general thinking patterns that might get it there. What we know is that these harnesses—we can, with a little bit of work, make them do anything.

It’s a little bit like, for example, if I already know how to solve a problem, I can use GPT-3.5 and it’ll give me the right answer because I’m giving it guidance. So it’s a little bit like a heuristic search problem where, if you guide the search enough, if you place enough constraints in there, then statistically, if you sample enough times, it’s going to get the right answer.

The Benchmark With No Instructions

Yeah. So I think that very much is Chollet’s view of this. That’s also why the scores are under 1%, because they don’t allow harnesses for that leaderboard. His point is, yeah, you shouldn’t need a harness. A human doesn’t need a harness.

It’s a tricky one. I, for myself, don’t know if I really fully buy that argument, in the sense that the reason a human can do this without needing a specific harness—I mean, that’s fine—but most of the people, especially people who are capable of playing these games, have played a lot of games, be they computer games, video games, or games in real life. So they’ve already learned the thinking patterns that we are now encoding through a harness much more directly.

I could also imagine, if that hypothesis is true, that it works that way, that if you were to take those 1%-scoring things and do a reinforcement learning loop specifically to teach them gameplay, but without hard-coding any harness, they might actually do a lot better. And does that count? Does that mean they can then play it? Well, I don’t know.

Is that the same as arguing that if you went to a deserted island where there were some tribesfolk who had never used computers before, do you think they would be able to do ARC-AGI-3?

The Benchmark With No Instructions

I think not immediately, but if you give them 100 of these things, one per day, and you reward the people doing well, I would assume that at some point they can. And I do think it’s fair to say they probably need a lot fewer training resources—I don’t know, the amount of compute required to learn it—than a frontier model would use right now.

And even when we say they can reach 36%, that is true, but it costs like a few thousand—

Which is a lot more, although it’s hard to, of course, want to use a lot more compute than the human beating these games is spending. So, yeah, does that really count as “the models can play games now” if it takes them so much effort?

I would say that is a real gap. And, also, let’s say being able to do this with frontier models—no, sorry, with small open-source models—would demonstrate a lot more, which is indeed what we’re doing in the Kaggle competition.

The Benchmark With No Instructions

Thirty-six percent might be misleading as a number if you don’t look behind it. So what it really measures is action efficiency. Correct me if I’m wrong, but it’s the ratio of the human baseline divided by the number of actions the AI has taken—whatever model it is on the level, or a human, whoever the player is—and then squared as well.

So this plays really adversarially to anything that’s a little bit action-inefficient. So 36%, actually, in this case, doesn’t mean that this approach solves 36% of the games. It solves way more of the games—at least from the training set, I would say—but it just solves them inefficiently.

I think this is really important to emphasize: the current frontier models are able to solve something like half to two-thirds of the training games, actually, till the end, but just not as efficiently.

What is the hardest thing in ARC-AGI-3? Is it the goal acquisition, or is it simply the action efficiency?

The Benchmark With No Instructions

From what we see on the training set, the testing set—the private set—is said to be harder. We don’t know anything about it; nobody outside of the organization has seen it. So we don’t know how hard it is.

But at least on the training games, we see that the LLMs can acquire the correct goals and pursue them somewhat effectively. The question is whether this also holds for the withheld private test, but it seems that goal setting is not the bottleneck.

Rather, it’s action efficiency and the accumulation of knowledge over a very long context, because you need hundreds of thousands, if not millions, of tokens to solve this. Keeping consistent knowledge of everything that has happened over such a context is a major engineering challenge at this moment.

What is harder in ARC-AGI-3 compared to the other ones is this interplay between exploration and solving the game. In ARC-AGI-1 and ARC-AGI-2, you would get all of the information in a static frame when you were given the puzzle.

Instead, in ARC-AGI-3, you are given the game, but just from the first frame, you cannot understand what needs to be done. So you need to start interacting with the game, and through that interaction you gather information about what the game is about and start understanding how to solve it.

At the same time, you need to try to understand what the game is about and try to solve it. This interplay is very hard to explain to the agent—how it should do it in an effective way such that it is general and generalizes across all games. So I think this is probably one of the key parts that is hard about ARC-AGI-3.

What we want is abstraction-based exploration.

The Benchmark With No Instructions

Yes. And that’s, I think, the very hard part. I guess here is where the similarity with ARC-AGI-2 and ARC-AGI-1 is: you kind of get the right abstraction level at which to do this exploration and kind of understand that, oh, the thing moving around is probably an agent that I can control, and it’s not just patches of pixels moving around.

If you’re able to get this right abstraction, then it’s usually trivial to solve the game. But, yeah, making the agent acquire this right abstraction level is the main challenge.

How difficult is it to acquire the goal? Are you finding that these 27-billion-parameter models can reasonably infer what the goal is, or is it much more complicated than that?

The Benchmark With No Instructions

In some of the good runs, when they get the goal right at the first trial, it’s very easy. If they find the right hypothesis for the game, then they can go on and solve a lot of levels.

But if they try once, get it wrong, and then give another hypothesis, and it’s not the right hypothesis again, then it’s very hard to get them out of the loop. We often find that they get stuck in very unintelligent goals that it’s very clear are not right.

For instance, often the agents start thinking that reducing the energy bar to the minimum is the goal, or that stepping 10 times in a region is the goal, which for a human is kind of clear is not the actual goal. But it’s very interesting that they’re not able to see that there’s no way that’s the actual goal. Yeah.

Yes. And can we contrast rewards and goals? We can use various different things. The most naive implementation is just to use level transitions and then the ARC-AGI score for that specific level transition.

So if you took more actions than a human would have taken, your score would be lower, but you use that per level.

The Benchmark With No Instructions

And then you can add various other types of rewards to improve your objective.

And did you use reward shaping?

The Benchmark With No Instructions

Yes. So for our RL training pipeline, we do reward shaping. We have 25 games. We actually have a lot more games that we’ve generated ourselves, and we can, for example, train on them to make the engine better—for example, exploration, finding goals, and achieving those goals—through end-to-end RL on just a bunch of games, basically.

And the thing is, this is much more difficult to do than ARC-AGI-2. In ARC-AGI-2, this was the standard approach: you would pretrain on a lot of puzzles, and then you would do test-time training as well on the set of puzzles.

But to do this in ARC-AGI-3, you need to train over, as Nik mentioned, something like 100,000–200,000 tokens, which is extremely difficult to do. So we try to improve this by, for example, adding reward shaping, training on shorter sequences, and trying to generalize to longer sequences.

And I guess what makes this so interesting for labs as a problem is that this is exactly the kind of research that is currently at the frontier of multi-turn, long-context reasoning. It simplifies the problem: we don’t have to worry about environments or other complexities like safety. We can just focus on the core problems and try to see if we can make progress there.

And that’s also perhaps an area where we could actually contribute some new research, which might, I guess, be open-sourced at the end of the competition, or we’ll see what we want to do with that research.

Basically, on your reward-shaping question, we add various rewards: level rewards, ARC-AGI scores, and whether the code gets executed or not.

The Benchmark With No Instructions

The length of the reasoning steps matters, because if the reasoning is too long per step, it might take too long to actually solve the game, and you only have a certain amount of tokens per game. We try to improve all of that.

You know, the weird side effect of ARC-AGI 1 and 2 is that it left me with the impression that intelligence is overrated. What I meant by that was, yes, we could solve problems and do hill climbing when we knew what the problem was. Maybe you would agree that in ARC-AGI 3, when we know what the goal is, it becomes a hill-climbing problem.

But it feels like the challenge is abstraction. It's almost unfair that Chollet is doing this action-efficiency thing. In my opinion, what he should do is: you do it the dumb way the first time, then compress that into reusable knowledge in your library. The next problem that comes up should ostensibly use what you just learned in the first game, and then you become more efficient. So you become more efficient over time because you're compressing knowledge. That kind of feels like the goal to me.

The Benchmark With No Instructions

Yes. There's actually been discussion in the community about having that as one of the goals for ARC-AGI 4. You play a game, solve it with a large compute or action budget, and then you have to play it again and do a speedrun through it to improve it.

I think that is important, but I also think action efficiency is a practical step to reduce brute-force solutions. It counteracts that: you have to explore in a more stringent way, and it makes sense in real-world environments as well, like computer-use and coding agents. You can't just explore every possibility. You have to do it in an intelligent way and improve using some heuristic. I think it makes sense not to always have all the actions available to you.

Okay, but even on ARC-AGI 3, when you do have the goal and it's a hill-climbing problem, I guess you still can't brute-force it because you get penalized on efficiency. In principle, though, you could if you had enough computation.

The Benchmark With No Instructions

You could, but they updated the games to be more robust against that. Level 1 could, in principle, be brute-forced within a certain number of actions. For level 2, the computational power you normally need is enormous, so you can't really, with a reasonable compute budget, even get past level 2. Then you have levels 3 and 4, depending on how long the game goes.

In principle, you could brute-force anything, but practically, with the compute budget we have, you won't be able to do that.

Could you describe what it is about ARC-AGI 3 that makes it almost impossible to brute-force?

The Benchmark With No Instructions

The main thing is the action space. We have 8 main actions, but there's also a mouse-click action with around 4,000 possible places you can click on a 64 × 64 grid. The number of actions you normally take is also quite large. It might be anything from a few hundred to 1,000 actions across the games. That branching factor is just enormous if you're not acting intelligently.

Is anything in ARC-AGI 3 badly designed or gameable? The preview had some elements that were a bit brute-forceable, wasn't it? You were talking about that earlier.

The Benchmark With No Instructions

Yes, exactly. I guess that was the main point of the previous competition: to show whether there was something you could easily exploit. Indeed, there was, like the Stochastic Goose algorithm and a few other algorithms. The top 4 places were basically brute-force algorithms that searched over a large space of actions but did some basic form of filtering, and you could get a very good score. I could solve 2 games and almost solve the third game on the private set.

The games were also too easy, so they increased the difficulty level. Another issue was the timing bar, which only changed when you actually executed a valid action. You could easily learn which actions were valid and which were not.

On the new set of games and in the ARC-AGI setup, I don't know if any of you have noticed anything that is easily exploitable or badly designed. I think it's implemented much better now, and that's why the scores haven't shot up initially. It's still at around 1%, so I think it's still a challenging setup.

I guess the million-dollar question is: do you think it's possible, in principle, to do really well on ARC-AGI 3 and be no closer to AGI?

The Benchmark With No Instructions

Yes, I do think it is possible. But I guess that's the point of the benchmark as well: it doesn't have to prove that once you solve ARC-AGI, you actually have AGI. I think, as Chollet has mentioned over and over again, it just disproves that something that can't solve this benchmark is AGI.

So, I think it's definitely possible, but getting 100% on this benchmark would definitely require some major improvement, like a better base model or something. We're nowhere close to getting 100%, which is the grand prize that is also made available at the end of this competition.

Do you think people will get near 100% anytime in the near future?

The Benchmark With No Instructions

I guess that's a debate. Some of us believe it might be possible. I don't think we're going to get there this year, but I don't know if there are any other opinions on when we might.

The Benchmark With No Instructions

You can split the problem, at least, into whether someone will reach it in the ARC-AGI competition, where you're playing 110 games in 9 hours on a single RTX 6000, versus whether someone will do it with a frontier model.

Getting 100% is essentially impossible because it means you cannot make a mistake even once. If you have the wrong hypothesis on one level, that's it; you're done. No single human is likely to score 100%, so I'd say maybe 90% is more realistic. You could say that this model is capable of doing this, and I feel that is actually possible even within the constraints of the ARC-AGI competition.

I do think it's going to take more research and probably simply better base models than will be achieved by November, when the competition ends.

Do you think there's some notion of an asymptotic optimal score? If you were the perfect intelligent agent, you would kind of converge on this score. Does that even make sense?

Mike Knoop

Probably, yes. I do think that score would probably be 100%. The game should actually be designed so that the optimal score is 100%. If it's not 100%, it essentially means there's guesswork involved.

Didn't you say before, though, that you could only get 100% if you already knew? If you were an agent that genuinely needed to acquire knowledge to solve the task, and you had to take the minimum number of steps to acquire that knowledge, do you think that would converge on a certain score?

Mike Knoop

Yes, that would probably be something like 90%, simply because it isn't reasonable to expect perfection. We just saw the LS20 game: there are 2 places you can move to, and you just can't guess the right one every time.

The thing is, you're being compared to humans who also had to do the same exploration, and you get the median score. If you're averagely lucky, you're going to do it, but you have 110 games, so you're going to get unlucky on one of them.

Yeah, and I should mention that even with our simple Qwen model, we can sometimes already get 100% on certain games. So it is certainly doable to beat those human baselines.

What we were saying earlier is that one view is that intelligence is this quite crystallized process. There's a concept of IQ, for example, and if you have a certain amount of IQ, then it's predictable how well you can generalize and how efficient you are.

I guess the data might just not back that up. The data might say that it's not random, but it's very specialized, and there are huge individual differences in capability in machines and in humans. Would that make them reassess their whole idea of what intelligence is, specifically based on ARC-AGI 3?

Mike Knoop

On ARC-AGI 3 specifically, I think we have a clear benchmark where we know what humans, who are general in some subdomain, can score. Whatever that domain is, we can say that score is the median score.

Even with just ARC-AGI 3, if we do badly—let's say the final submission score is 5%—I don't think that would update their views, because humans can achieve that. Maybe they would argue that if the results did converge and become more regular—if there were a new class of algorithms that consistently solved the problems in some predictable amount of time—Chollet might think, “Oh, that's the algorithm of intelligence. It's not just guessing anymore,” or something.

I think the goalposts are going to move further. I think they will update their model. In some way, I think they already have.

By changing the games, because they thought some of the games they made weren't brute-forceable, and they actually were. Then there were some simple models, and they also changed the metrics, updating them a couple of times both to be fairer and to represent human performance better.

So, certainly, they have been updating their model a little bit already. It's tough to say what's going to happen, but I think the goalposts are moving all the time. If we create an agent that can solve all of this, we're still going to discover new areas where it's missing something and where it isn't targeted by the dataset.

I’ve heard—maybe they’re just rumors—but I think there are other ARC-AGI competitions coming up. They’re building ARC-AGI 4 or something for next year, I think.

Mike Knoop

So, certainly, they have more to add to the story than just ARC-AGI 3.

Do you think they won’t release ARC-AGI 4 until people are doing well on ARC-AGI 3, or will they just release it anyway?

Mike Knoop

If you look so far, it does feel like they—I don’t know if it’s intentional or just the speed at which they developed them—but it does seem pretty clear. At least, I don’t think they knew what ARC-AGI 3 would look like when they built ARC-AGI 1. They might even have thought, “Oh, it’s going to be 10 years before anyone beats ARC-AGI 1.”

It was like 5 years in the end.

Mike Knoop

Yeah, sure. But the point is, at that point you think, “Okay, now they can do this. Does this really mean intelligence?” Well, probably not. But I think you also need to see what happens in ARC-AGI 3, and the ways in which it gets solved, to understand where the gaps still are.

Because again, no one is claiming that if a game can be played—if you can do a 64×64 game—you’re intelligent in every sense. It’s just one more step and one more challenge that they’ve beaten. You can imagine, maybe, for example, ARC-AGI 4 has games like playing Doom—not Doom, of course, but a new version of Doom. That would require you to work with a much vaguer representation of the world, and it would be a completely new challenge again.

But I imagine it’s going to take a little time to see what’s happening here. I also have no idea what they’re doing in the background right now.

Yeah, I have no idea, actually.

Mike Knoop

Yeah.

Yeah. She was quite forthcoming about ARC-AGI 3.

I’m wondering if it’s the idea you discussed, because the community has discussed this a lot, and I think there’s been some back and forth about having to play a game, then having to acquire the skills and the knowledge, and then replaying it at, let’s say, almost maximum efficiency—like speedrunning it. So that’s one intuition that’s been going around in the field. I guess only Greg and the team will be able to tell us next year.

The thing I really liked about ARC v1 and v2 was that, first of all, it was a grassroots community, and loads of people without lots of funding could just focus on the kernel of intelligence. Now it’s just ridiculously complicated, and it’s almost de-intellectualized the field, which means that loads of folks with resources are just throwing reinforcement-learning algorithms at it and almost going back into blindfold mode.

That’s a fair description of a lot of machine learning. We just have really big models, we train them to do next-token prediction, we see what happens, and we try to scale up from there. The thing that was so magical about the community was that there were just so many great ideas and really, really deep thinking.

Mike Knoop

It’s not that simple. Even if we have a lot of compute, it’s still hard to make a harness that solves these games, and training is also hard. It’s not as simple as just throwing compute at it; there are still many details that you have to get right.

But for sure, as you said, I think it raised the level needed to enter the competition. We’re seeing that many people are stuck with the kind of template solution and, as far as we’re aware, at least in the competition, not many are trying LLM approaches because they’re so computationally expensive. So it’s definitely made it harder for the average person to enter the competition, but I wouldn’t say that it’s just a matter of throwing a lot of compute at it and brute-forcing it.

It’s not like most other benchmarks, right? There’s no language, there are no instructions. Does that make it quite distinct as well?

Mike Knoop

I think that’s the interesting part about ARC: it tries to remove as much as possible the prior that you get from language or from human knowledge, strip it to the minimum, and really only test for intelligence. I think that’s a cool feature.

Language is the basis of how we do a lot of thinking. What you folks have done is, you’ve gone to language and then gone back again, so we’ve got this kind of loop. We go to language, we do some reasoning, and then we might use that to do active fine-tuning or reinforcement learning or whatever, and we’ve got this virtuous cycle.

So you took the language away, and we’re kind of putting it back again.

Mike Knoop

Yeah. I think the reason we’re putting it back is because we’re specifically focusing on language models, which have been extensively trained on language. I guess if there are other approaches, like neural-guided search, they might not use language at all. You don’t need language for this, at least human textual language.

But in our case, we’re heavily leveraging reasoning models, so it’s natural to bring them back into the domain they’re experts in. We want our harness to shape that and encourage that behavior. So, yeah, I guess that’s the main reason we’re moving back.

I think it’s also very difficult to do this. The reason—and I think I said it before already—is that language succeeds so well because there are higher-level priors encoded in it that are just so difficult to strip away from the data. Maybe that’s because humans have created these games and have an idea of what a game is. Even though they tried to remove the bias a little bit, it’s still there.

You might look at these games, and usually the backgrounds are some sort of dark color or white, rather than green. This isn’t necessarily true for all of them, but more or less it holds. You can see that these were created by humans and reflect their idea of what games are. I think it’s incredibly difficult for a human to create something that doesn’t have this in it. It might even be impossible. So that’s maybe why coming back to language works so well.

Yeah, just to quickly add to that, I’ve also seen exactly that. You could easily permute the game: you could permute the colors and rotate the images. When you do that, the games become significantly harder. So if you remove that prior—which shouldn’t actually be a prior—you can see that it becomes harder for humans to play. So, yes, perhaps that tells us something about the benchmark.

There’s some leakage of human—

Exactly.

Mike Knoop

—prior into the game.

It makes it harder for the algorithms as well.

Mike Knoop

Yes, yes. That’s one thing we found with the harness. You can provide numbers as colors, but it does worse than if you encode those numbers as specific colors that it knows, like black for background and gray for immovable areas. So if you remove those priors, it performs worse, which is interesting.

That is fascinating. I think it’s important to pass through language because initially we tried to train a neural net with reinforcement learning on a lot of games, without using an LLM at all. It’s way easier to use an LLM—you get good performance much faster—because the prior from language is so general and can be applied to so many different domains. Getting the same prior just by training would take so many games instead. By starting from language and then fine-tuning to the specific games, you can get there much faster.

It’s quite a common technique to transform something so that it falls into a representation that has more affinity with our knowledge. The encoding from just numbers to specific characters—for example, using B for blue—and telling it, “This is blue; this is this color,” automatically helps it a lot with language-based reasoning. It knows that typically, when humans play games, bright colors are objects you want to interact with or something is going to happen, while duller colors are background or walls. So that definitely helped a lot.

That’s actually really interesting, because I suppose one strategy is that you could have multiple representations. From a language point of view, you could do something really galaxy-brain, like turn it into a story where the colors become dogs or agents or people. Do you see what I mean? Is there some representation that will just make the language model come to life and make it significantly more powerful?

Mike Knoop

As far as we’ve seen so far, the models are actually pretty decent at perception. Remember, the game in the end is not a 64×64 grid; it has concepts. It has a player that’s moving around. It has a goal. It has the maze. You’re not going to play the game if you don’t get those concepts.

The thing is, the LLM is pretty good at getting those concepts. The problem is that it takes a while, and we have very little time to play the games. One aspect of ARC that we haven’t discussed yet is that we give it different views of the world. We have the ASCII grid, but we also have a very simple segmentation based on connected components.

By having that and giving it that list, it works well on some games and not well on others. But it allows the model to short-circuit a lot of this perception part. In perception, it will look at the ASCII, zoom in on part of the board, and make an ASCII representation there.

Oh, that looks like this, and do it on another one, and spend a lot of tokens that way.

What about from a vision point of view as well? Do you think language is actually necessary for reasoning? Do you think, in principle, we could train a vision model to go directly from the input to the output? I think it boils down to one of the big open questions in the field: Is language critical to intelligence?

Mike Knoop

I will say that, at least when I'm playing the games myself—and I think that goes for all of us—I'm using language. These games are set up so that, indeed, with the object prior, a 4-year-old could do it, but I don't think a 4-year-old would be beating most games. When I'm actually thinking through a strategy myself, I'm thinking either out loud or in my head, but there's basically a narration, a narrative in language: “Oh, okay, we have to get the key, but the key is blocked. How does that work?” And how you decouple all that, I'm not sure.

Language is almost like an emergent phenomenon. It's still almost determined by its physical instantiation in the world that we live in, but it's something that has evolved over a very, very long time. So, in principle, we could train an end-to-end vision model, and if it saw an insane amount of data—almost all of the distribution—it could potentially acquire the same abstractions. But language is almost like a shortcut because you can bootstrap those abstractions.

Mike Knoop

Yeah, it's also tricky because it could, but it might end up internally developing its own language, because you need things like reasoning chains. In our case, it needs to write code, which is a language itself.

That is a good point. So I did some initial experimentation. For the first experiment, to show that we can generalize, I created an ARC environment that was procedurally generated, with some new objects and new objectives. In fact, you can get the behavior: it's just a vision model with a Mamba backbone, and you can get it to learn to figure out the objects and place them in the right order, without using language, using only a latent representation.

I think it is possible in principle, but what that experiment showed is that I required about 5,000 different permutations just to solve that one setup. On real ARC games, it's going to be more than that if you want to train from scratch. That's why the prior helps a lot: You need fewer environments to actually get it to perform well. You could just do it with a harness, and if you train, the hope is that you need fewer environments to improve on and generalize, which is the most important part.

It does lean on this more general point that we think humans are so intelligent, but we have the benefit of millions of years of evolution, right? We're using all of that trial and error implicitly. So it's almost a bit unfair that we're expecting AI algorithms to be able to do all of that.

Mike Knoop

Exactly.

How do you square the circle between releasing lots of pre-built tools that are quite specialized and also being Bitter Lesson-pilled? Is that a contradiction? What does it mean to be Bitter Lesson-pilled?

Mike Knoop

As a lab, we subscribe to the Bitter Lesson. I guess the idea behind the Bitter Lesson is that often detailed and specific solutions lose compared to just more data and more compute, and going large-scale on a problem.

I think, looking at the past, even if the Bitter Lesson wins in the end, it's always useful to try the more domain-specific approach first. For instance, AlexNet, the paper that won ImageNet and in some way started the deep learning revolution: I think a lot of the intuition that brought them to use convolutional neural networks for image recognition was motivated in some way by all of the tricks that had been done before for image recognition.

So I think that even if it's true that now we're doing pure engineering on the harness, we're understanding the details of the problem that will then allow us to do a more end-to-end solution and allow us to do something that generalizes, hopefully, more. I think this understanding of the details and specification is probably a necessary step.

Yeah, I want to add that I'm maybe a little bit less Bitter Lesson-pilled than most AI researchers, mainly because I was doing it before the Bitter Lesson became the baseline. I do think, at least certainly where ARC-AGI 3 is right now, I would be willing to make a bet that the winning solution, be it ours or someone else's, of this competition will actually not be a Bitter Lesson solution.

If you look at ARC-AGI 2 now, it's been pretty much Bitter Lesson-ed, and it's being won by creating huge amounts of training data. I don't think there's a clear path just yet for this competition, and what's needed now is really the basic design. I do think it will involve training, but probably not to the degree that all you need to do is have a good training data set and then go for it.

Yeah. It's fair to say that Chollet does not want it to be Bitter Lesson-built. Every single design step has been away from the Bitter Lesson, and he would love it in a world where it wasn't the case that it was Bitter Lesson-ed. I might be willing to make that bet for this year. If there's another iteration next year, or another one the year after that, I could imagine that at some point it is Bitter Lesson-ed. But I don't think the models are there to be able to do that this year already.

Mike Knoop

Yeah, it's so amazing that even in 2026 we have so much uncertainty about this. We still don't know beyond a reasonable doubt whether it might not be Bitter Lesson-pilled in 2 years' time. It's important to bring things into distribution.

What ARC Prize has done for ARC-AGI 3 is that they specifically say the training problems, or the public problems, are not representative of the testing problems. That makes it even harder to create a distribution on which you could meaningfully train, because you still have to extrapolate in some way.

So maybe that's also why overfitting just doesn't help, because you have to generalize to new problems. As you say, they are specifically targeting this not to be just mass-trainable. But maybe if we can extrapolate the distribution well and guess whatever is in the testing distribution, then this could be done. But this is just a hypothesis.

I do think maybe, in a sense, it is going to be Bitter Lesson-ed, but the classical Bitter Lesson is just that you make a training set so big that everything is in there, and then bake everything into your weights, which is kind of what happened for ARC-AGI 2.

I think this one does need something like that, but what you need to get into the model—and I think that is maybe much more what Chollet would also like to see—is the basic thinking patterns. It needs to know the general ways in which to approach a game and how to solve a game without having memorized every game in particular. If we can manage to do that, we'll feel like we've really done something beyond just beating a bunch of games.

How do you reconcile the fact that you're trying to build more powerful AI but you're worried about it?

The Benchmark With No Instructions

Reconciliation is difficult. You cannot do AI safety research in a vacuum. You need to be clued in about the best techniques and be able to train good AI systems to study them at all.

The way I would say it is: If you want to make bridges safe, it's going to help to know civil engineering and how to build bridges. Similarly for AI, but that runs the risk that you then participate in the race and make that racing situation worse, because progress in AI capabilities runs ahead of progress toward AI safety. So I want to be very careful with this.

I think now I'm very happy with the team we currently have and the level of technical discussion, common interests, and actual work being done. So that's all very positive.

What's gone wrong? Is it just that it takes forever to get there somehow? It takes longer and is more expensive than you think it will?

The Benchmark With No Instructions

Yeah. I guess what's wrong is that broad AI progress is very fast, and it feels a bit tough: Can you ever catch up? I think next year might not look that different from today, but I would say 5 to 10 years away might look completely different.

Yeah.

The Benchmark With No Instructions

I think I'm a believer in the software singularity. I think software advances will keep being made through agent engineering and stronger AI systems in the world of atoms generally. But I think eventually we'll replicate this success with things like industrial robots and self-driving cars. I think at some point the Moravec paradox kind of falls, and then we start seeing powerful AI capabilities everywhere.