29.4% ARC-AGI-2 🤯 (TOP SCORE!) - Jeremy Berman

You can describe every single ARC-AGI-2 task in 10 bullet points of plain English, most of them in 5 bullet points. I think that actually gets to the heart of ARC, right? Everything is quite simple. It's not very hard. I think this is also how we do it: when we look at these ARC graphs, we're coming up with these bullet points in our head and checking them. “Okay, this was right; this was right.” Python doesn't have these features. It's just not as expressive as natural language.

MLST is sponsored by Cyber Fund. Link in the description.

Tim Scarfe

I get, actually, even more fundamentally, that the ideal system would be: we have a set of data, our language model is bad at a certain thing, and we can just give it this data. Then, all of a sudden, it keeps all of its knowledge and also gets really good at this new thing. We're not there yet, and that, to me, is a fundamental missing part.

Really what you want is a more expressive program. That's why I switched from Python to English, which is a much more expressive program. You can always teach a language model a skill, right? But it's the meta-skill—the skill to create the skills—that is AGI. To me, that's reasoning. Reasoning is that meta-skill.

To put it another way, I think if you fundamentally learn the skill of reasoning, you should then be able to apply that skill to learn all the other skills. That is the meta-skill. Tweak whatever weights you need to align the model to reason, and from there you have a foundation from which you can actually build general intelligence.

Tim Scarfe

Okay, folks, hot off the press: many of you will have seen last week that Jeremy Berman, a research scientist at Reflection AI, is now the winner of the ARC-AGI-2-Pub leaderboard, the public version of the leaderboard. He's using an evolutionary approach. Remember, last year in December, he published a similar approach, generating Python functions and then refining those functions in an iterative loop.

His new architecture generates descriptions of algorithms rather than code and iteratively refines them in an evolutionary sense, discarding the ones that don't work. He's now at the top of the leaderboard. It's a really cool and elegant algorithm. By the way, he works for Reflection AI, so he's doing reinforcement learning with verifiable feedback, and he's trying to address the biggest gap in AI at the moment: we want systems that can synthesize new knowledge and new understanding.

Current systems just get trained with a whole bunch of data, and they only know what they've been trained on. They can't think outside the box by creatively synthesizing new knowledge.

Prolific are really focused on the contributions of human data in AI. And the reason this is important, it's actually the dirty secret of Silicon Valley, the extent to which human data is used to evaluate and fine-tune AI models. The reason for that, as we discuss in today's show, is that current AI does not understand the world in a grounded way. It doesn't have a deep abstract understanding of the world. Which is why the only way that we can make AI work effectively is by grounding the generation and supervising the training of AI models with human data. Prolific are putting together a report on how human data is being used in AI systems and they need volunteers. You can just go and fill out this form to help them produce this report and you will get privileged access to see the report before anyone else. The link is in the description.

Oh, there was an amazing part in, I think, your first paper where you said, “A parrot that lives in a courthouse will regurgitate more correct statements than a parrot that lives in a madhouse.”

Thank you. My sister, who doesn't know anything about language models or AI, pointed that out and said it was a great line. So at least I have that.

Tim Scarfe

I've already used it. I credited you, but I'll be using that quite a lot. Well, Jeremy, it's amazing to have you on MLST. I've wanted to have you on ever since you released your first blog post. It was December last year. I was at NeurIPS at the time, and you actually got the highest score on the public ARC v1 leaderboard, just before the famous o3 launched. Do you remember when they did this ridiculous $200-per-task thing and knocked you off the board?

Jeremy, can you tell the audience a little bit about yourself? Maybe we should start with your first ARC solution?

Yeah, sure. I actually have only been working in research for about 8 months. Before that, I had a company right out of college. I got into Y Combinator, and I've been running a company for the last 4.5 years as CTO. I've always been very interested in reasoning in the brain.

I picked up Jeff Hawkins's book A Thousand Brains, and I read that. At the same time, I was coming into language models, and something just clicked inside of me. I knew I had to be working on this. I believe that artificial general intelligence will be the most important invention of, hopefully, my lifetime.

So I decided to drop everything. I stepped down as CTO. The company's still going well, so it was a difficult decision. I got in touch with Mike and François because I thought ARC-AGI was such an elegant way of describing the problems with current language models and the difference between them and the human brain. I just dug in. That was my first independent research project, and I ended up getting the top score on that. That was really great.

After that, I got recruited into François and Mike's AGI lab in India, where I was working on program synthesis. As you described earlier, over time I've become more convinced that language modeling with reinforcement learning will yield generalization far beyond what we see today. I decided to move to a company that was focused purely on language models, and that's where I am now.

I'm currently working on reasoning and post-training at Reflection, where we're building frontier foundation models.

Tim Scarfe

Very cool. Maybe we should save that for a bit later. One of the take-home messages in your new approach is that, rather than producing explicit programs, you are evolving descriptions of programs. François is a neurosymbolic guy. He thinks that we need to have a symbolic substrate where we represent the kinds of problems that we can do, and we need to do this kind of compositional form of intelligence.

So we need to be working in the symbolic layer, but perhaps guided by deep learning models. Maybe we should get to that in a minute.

In your first solution, it was an evolutionary approach using Claude 3.5 Sonnet, and you had about 4 iterations, I think. Essentially, you were working on the ARC challenge and producing these programs through evolution. Maybe for folks who don't know about the ARC challenge, could you introduce that and get into your solution?

Sure. The ARC challenge is kind of like an IQ test for machines. It's a set of input-output grids, and the whole point is to figure out how to transform input grids into output grids given a common transformation rule.

What's interesting is that these are really easy for humans, right? The average human gets around 75% accuracy on ARC v1, and at the time the best language models—GPT-4 and Claude 3.5 Sonnet—were getting maybe 5%. Basically, you have a few training examples, and you're trying to extrapolate the transformation rule on the final test example.

I approached this—I was actually inspired by Ryan Greenblatt, who had a solution earlier, which was to generate a ton of Python programs that would encapsulate the transformation rule. Python programs are great because they're deterministic, and you can pretty quickly check whether or not the Python program works, which is really cheap. So it's cheap to verify, and you can be relatively sure that if the Python program works on all the training examples, it'll work on the test example.

I started with his approach, but then I noticed that the language models actually struggled on first attempts. Even if you ask the language model 1,000 times to generate Python programs, they were always off by small amounts on easy tasks, which I thought—presumably it's in their distribution; they should be able to solve this.

What I found is that by taking the top-performing programs and then running them in a revision loop—asking Sonnet 3.5, “Hey, here's what you got wrong. Here are the cells you got wrong. Here's your original Python program. Improve it.”—that started to really work well.

Then I thought, why not just increase the depth? Why not ask it 10 times to revise until I'm happy with the solution, until it passes some sort of accuracy threshold? That's how I was inspired by it.

I didn't think of it as evolutionary at first. I was just thinking broadly about what would work, and over time I understood that there was something a bit deeper going on here: evolving solutions is a powerful technique generally, and I think it's actually going to play a role in future technologies. But that's generally guided by intuition.

Tim Scarfe

Yeah, I had Ryan Greenblatt on the show.

I'm a huge fan of his. He's a very, very smart guy. And I asked him a similar question because he did this iteration, right, where you have a certain depth of iterations. I guess one approach is that you have a shallow method, right? So you just try 200 different variations.

In your blog post, you said there's a Goldilocks zone where you want to have a certain number of tries of different variations of things, but you also want to be able to refine your solution because that allows you to do this kind of composition. And composition is very, very important for problems that require iteration. Indeed, the second version of the ARC challenge—I think the tasks were selected so that they had at least a couple of iterations in them—which meant that they needed to have this depth. Can you talk about that trade-off?

First of all, ARC V2 is, in a sense, fundamentally different from ARC V1 because of what you're describing. They're compositional. There are many rules that you have to go through, and this is partially why I found that my solution on ARC V1 did not perform well. There's a constant trade-off between how deep you go, how many revisions you take, and how broad you start out.

The problem with going deep and not going so broad is there are some edge solutions where you'll never get to, right? But then, of course, most solutions end up being somewhere within the bounds of your first broad attempt. So that's generally the trade-off, and the trade-off is different for ARC V1 and ARC V2.

Interestingly, I found that for ARC V2, it was more important to be broad. I think this is surprising to a lot of people. Partially, this is because the models now think, and that's great. So the models actually do a lot of the deep revision for you in their thinking block. This is a fundamental change from when ARC V1 existed and when I just started out in the field.

I think I'm a bit embarrassed by a lot of the things that I wrote in my first post because it was 2 weeks before o1 was released, and everything about o1 changed how I think about these things. Before, you could kind of simulate or emulate thinking, to use the quote you described with the stochastic parrot. I think before you actually taught the machines—the language models—to think with reinforcement learning, you were almost doing this stochastic guessing that was not a very efficient revision loop, basically an internal revision loop.

So you needed to artificially create that revision loop with code. But in V2, I was able to use a very powerful thinking model, which has a lot of the deep revisions for it. So I found it was best to increase entropy, let it explore the space itself, and then add a revision loop on top of that. But the revision loop is less important in V2.

Tim Scarfe

Yeah. So the first one was Claude 3.5 Sonnet. That didn't have this thinking thing built into it. In your prompt, you told it to think step by step. And there, I think you were inspired by Ryan Greenblatt's prompt, right? So you had a whole bunch of ways in there for representing the board state, and you said, "I want you to think now, and I want you to know that this is an abstract reasoning challenge, and I want you to think from first principles," and it would go through that, and then it would give you the answer. But you're saying that on the RL-trained models, it was significantly better at doing that?

Right. Exactly. You could think of RL-trained models as having built-in revision loops. They're trained to explore the space in a deep way, thinking for themselves in a general way, so you really don't need to prompt thinking models to think step by step. They already do it.

Tim Scarfe

Yeah, I wanted to challenge you on this a tiny bit, right? You said in your—I think it was in the second version of the blog post that you just released last week—that at the moment, the models can do domain-specific thinking. They can do math thinking, and they can do code thinking. What we want to do is imbue the core mechanisms of thinking into these models, and I'm a little bit skeptical.

I feel that these models, because they're not Turing-complete, because they're not symbolic—you know, similar to what François believes. I'm sure you read that "On the Biology of a Large Language Model" paper as well. They were talking about these circuits that we can find in papers that do things like multiplication and addition. What we saw was that they are quite patterned. They're quite templated. They're not thinking in a very general sense.

And my suspicion is it will always be that way because the models don't have semantics. They're nonsymbolic and so on. Do you think we could ever make them truly think in a general way?

Yeah, I think fundamentally, taking a step back, the fact that our brains can do it and our brains are generally running similar algorithms, to me, means that we will eventually be able to inject general reasoning into language models. I don't think there's a fundamental reason why neural networks can't behave like biological neural networks. So that's the higher-level point.

And then, zooming in, right now the models are as bad as they're ever going to be. There's generally more compute going into pretraining than there is reinforcement learning. And of the compute going into reinforcement learning, a subset is going into specific general reasoning. So I think that over time you're going to see the models get better and better at general reasoning.

But I guess a question I would have for you is: do you think there's a fundamental difference between the way the brain works, where there's some sort of symbolic nature to the brain, and it's not possible to inject that type of nature into an artificial network?

Tim Scarfe

Yes. Yeah. I mean, you mentioned Jeff Hawkins. I interviewed Jeff. He's absolutely amazing. And, of course, his HTM algorithm is computationally stronger than a neural network. It's Turing-complete, and our brains, even though they are finite, run a Turing-complete algorithm, which means our brains know how to expand their memory, right? Our memory—we can go and write things on a whiteboard, and we can go and get another notebook. And that is a special type of algorithm which is not traversable with stochastic gradient descent.

So the rough argument is yes, there is a difference there. And I also wanted to touch on this RL with verifiable rewards thing, which is that we do that at training time. I'm very excited in the future about an active-inference version of that, like an agentic version, where we're actually doing this kind of transductive active fine-tuning in an agentic way.

I take an action, I get some new information from the environment, and I update my weights, and that would be truly adaptive. That would be intelligent. But what we do now is we do all of this stuff at training time, and the resulting frozen artifact is still an LLM. It still has just a bunch of patterns in there.

And I think that while that can uplift reasoning in many ways, I don't think it has the intelligence. And according to Chollet, intelligence is simply the ability to search through the space of Turing programs, right? I don't think that's what's happening with these LLMs at the moment.

Alexia Jolicoeur-Martineau

I think you're generally correct that it's not happening at the moment, but I still think fundamentally, I don't think there's a fundamental blocker physically for why they won't be able to do it in the future. It's possible that SGD—stochastic gradient descent—is fundamentally an issue, and I think we're going to overcome that.

I guess what I would say is that artificial neural networks have the structure capable of being as smart in every way as a human brain. And I subscribe to François's definition of general intelligence as well.

Tim Scarfe

Yeah. I think we mostly agree. I mean, let's look at AlphaZero or MuZero or something like that. They did this training loop where they were actually updating the value network and the policy network, and then it was frozen, and they did some kind of Monte Carlo search. So they were achieving adaptivity through exhaustive search during the actual games, and in an ideal world, we would have this adaptivity that's actually updating the weights.

Now, I believe the only reason we can't do that at the moment is just computational tractability, right? We have these huge models. We couldn't possibly have a dynamically updating model for every single person that's using ChatGPT. It would just be ridiculously slow. But I think you and I agree that if that were possible, that would be an entirely different kind of form of intelligence.

I don't think that's so intractable, actually. My guess is that, with NVIDIA putting $100 billion into OpenAI, Sam Altman's plan is to produce a gigawatt of compute a week, something like that. I actually don't think, with ever more efficient algorithms, that is crazy far off.

I mean, right now you could buy a GPU, have it running in your house, and it could be running OSS 12B, right? And fine-tuning is relatively trivial compared to the entire process for pretraining. I actually think that is totally within the realm of possibilities in the next 10 years.

Tim Scarfe

I think that actually is potentially where this goes. You know far more about this than I do, but I think the reason why fine-tuning is so expensive is that we have this continual-learning problem. When you fine-tune a model on OpenAI, they’re not just fine-tuning it on the data you give them. To stop this catastrophic-forgetting problem, they presumably have to sample in a bunch of the original training data and maintain the distribution and so on. If they did this for everyone, it would be insane.

But I am excited about it just like you are because I interviewed the ARChitects, and I think they got first place on the private version last year. They were doing this transductive active fine-tuning. They actually said that this is a curious oddity with transformers: if you start with an almost-virgin 8-billion-parameter transformer, it almost doesn’t matter what it knew about before. You could pretty much start training it from scratch on the ARC challenges.

They did a whole bunch of augmentation and active fine-tuning, and they built an intelligent artifact. Intelligence is domain-specific, as per François Chollet, and they actually built this system that was adapting per task and solving the tasks. They were updating the weights, and it was beautiful. So that was an existence proof, if nothing else, that this thing could work.

Alexia Jolicoeur-Martineau

And that was on a Kaggle notebook.

Tim Scarfe

Yeah. In 10 years, this is going to be, like, the Apollo mission computer. I think what you’re describing—I’m actually not even totally convinced that continual learning is fundamentally the blocker. But if it is the fundamental blocker, that’s incredible, because we will solve continual learning. That’s something that’s physically possible.

Yeah, so I’m inclined to agree. I’ve long dreamed about there being a Docker for language models. In Docker, you can freeze-dry a state of, let’s say, a Linux operating system with an application and its security updates. You have these immutable layers, and the composability that we often talk about could actually happen at the architectural level. We could do dynamic model merging between different layers and whatnot. That would be very exciting.

But just to come back to what you said before, I’ve never really heard this before: you’re distinguishing forgetting and learning when we talk about catastrophic forgetting and continual learning. Can you sketch out that distinction a bit more?

The way I think about it, you have a neural network, and it has all these weights inside of it. Any time you update those weights, you are pushing some weights out, and presumably you are pushing out some correct answers that you’ve previously trained, and they are getting pushed out. The benefit, I think, fundamentally, of symbolic systems is that doesn’t happen. Symbolic systems are deterministic: when you get the right answer, you can be sure you have the right answer, and you stash it away into your library of correct solutions. This is the problem with continuous structures.

This is also why I think it’s important to draw from the brain, because a similar thing happens with the brain. I believe the brain is much more composable than neural networks biologically, but I think there’s no reason why we won’t be able to figure this out. Again, it could be as easy as ending up freezing experts. The freezing of the layers, I think, is an underexplored area.

I think we’re going to go through basically this RL S-curve, and then the next S-curve is figuring out how to make language models composable—figuring out how to make it so that, ideally, we have a set of data for something our language model is bad at, we can just give it this data, and then all of a sudden it keeps all of its knowledge and also gets really good at this new thing. We are not there yet, and that, to me, is a fundamental missing part of general intelligence.

Tim Scarfe

Yeah, completely agree. It sounds like we have very similar intuitions, and Chollet talks about this as well. Interestingly, in his “On the Measure of Intelligence” paper—it was actually about the measure of intelligence—he’s never really spoken about the mechanics of intelligence. He talks about it casually. He says we need the ARC priors, and we need to—those are like the basis functions—and we need to do library learning and library transfer, and do some kind of symbolic compositional process to adapt to novelty. He’s sketched out the mechanics of it, but he’s never actually formally spoken about it. I assume that’s what he’s building at his company.

There was a famous guy called Jerry Fodor. In 1988, he had this connectionist critique. He had this beautiful paper, and he was basically saying that symbolic systems have systematicity and productivity. Systematicity is this compositional thing: it’s the ability to generalize between “Mary loves John” and “Mary loves Jane.” You have semantics, these symbolic relations, and they have certain computational properties, like variable binding and quantification over potentially infinite domains. We intuitively understand that symbolic things have very interesting properties. What we’re trying to do is—we know neural networks are really good, and we want to somehow graft this capability onto neural networks.

Yes. And I actually think neural networks are, in some ways, a superset of symbolic systems. Generally, you should be able to encapsulate a symbolic system with a neural network, in the same way that I think you can do the same thing with the brain as well. There’s nothing fundamentally blocking that, but once you have this symbolic system in the neural network, it might catastrophically forget when you fine-tune it. I guess that’s where we disagree a bit. Everything you’re describing is totally possible, but when you come to train it again, there’s no guarantee that it keeps the same structure.

Tim Scarfe

I think it’s possible because a neural network is not Turing complete. So I think, in principle, it can’t do many of these things, but you can build a controller. You could just build a very simple kind of wrapper, just as you did with your solution. You had a bunch of code, and it was doing this—basically compositionality in code on top of the neural network substrate—and that gives you many of those things.

For example, we often talk about library learning and library transfer, and I’m not sure if you’ve seen Eric Pang’s solution. I’m speaking to him in Hong Kong in a couple of weeks, but rather than the DreamCoder approach, where they do this explicit library learning, he was doing it in an implicit way using the LLMs. I think there’s a whole spectrum: you don’t have to do it explicitly. You can use neural networks and do some kind of implicit composition, and get many of these features.

Also, I want to say that generally, when I speak about language models, I assume that they basically have a Python terminal that they can run.

Okay. So, I guess two things. The first is, if you have a large enough neural network, I think generally almost everything—you could represent a symbolic system, but of course it’s not Turing complete. Given a neural network plus the ability to write programs, I think we’re basically at the human-brain equivalent.

Tim Scarfe

So yes, that is a hybrid system, and that certainly is significantly more powerful. I’m just regurgitating my co-host, Dr. Dugar, because this is his favorite point; he always likes to make it. But he says that’s true, but stochastic gradient descent does not find the algorithms that allow the systems to behave as if they are Turing machines.

Yes.

God knows how it happened in our brains. There is some quirk of evolution or something where we suddenly got the merge operator, or God knows what happened, and we've got this incredible Turing-complete algorithm in our finite brain. So we're getting into that trainability thing.

Maybe there is an out. There is a set of weights that we might find one day, and it can access a Python tool and do all of those things. Is its capability now effectively searching the space of Turing machine programs? I think there are lots of problems there, like how it would know which ones halt and which ones don't, and how it would be able to efficiently search that space. It feels like there's a gap now, but I agree with you that it's significantly stronger than not being able to use the tools.

Yeah, but you think that the human brain is running a Turing system?

Tim Scarfe

Yes, I think the algorithm that runs in our brain is a Turing machine algorithm. A Turing machine has a controller, which is a finite-state automaton, and then it has read-write access to these two potentially infinite tapes. The algorithm that you put in that Turing machine is very difficult to find. I don't disagree with that, but why wouldn't we be able to find that algorithm for neural networks, right? Why wouldn't we train neural networks much bigger than the brain? We put a lot of compute toward them. Do you just not think that finding the same algorithm is possible with SGD?

Tim Scarfe

I think with SGD, the fascinating thing is that, if you look at all of the FSA algorithms, a tiny sliver of those algorithms are capable of controlling a Turing machine and expanding their memory and so on. It's in the space, and I don't know if you saw that amazing paper by Kenneth Stanley, the “Fractured Entangled Representations” paper. He had this beautiful diagram, and he said that SGD finds the algorithms over here and neuroevolution algorithms find the ones over here.

It just so happens that the neuroevolution algorithms find ones that have these factored representations, which means they find representations about the world that are grounded in the world, that carve the world up at the joints. If only we could find those things. When I spoke to Schmidhuber, he said the same thing. He said it is actually possible to find the right weights in a neural network to make it effectively Turing-complete, with some caveats and so on. But when we do SGD, because there are all of these shortcuts, right? It's a bit like Goodharting. It will always just find the wrong thing.

I need to think about that a bit more.

Tim Scarfe

Okay. On the first one as well, you were generating Python programs explicitly, and because of all the things we were just talking about, I'm a big fan of that. I feel intuitively—and I think you did that—there's something special about Python programs, and then you did this iterative updating of those programs and converged on the right one. You also had this amazing diagram in your first blog post where you visualized the space of all the possible programs and showed what was happening in every iteration.

In the first one, I used Python programs because Python programs are deterministic, and it's really easy to verify whether or not they're correct. Did it run, and did it run on the training examples and produce the correct outputs? So it's a perfect program, right? It is a program.

The problem is that Python programs are brittle in that there are many things that are very difficult to describe with Python, ARC grids in V2 being one of them. You have some grids that are very easily described by Python, but then almost the majority—the overwhelming majority—in ARC-AGI-2 are very hard to describe in Python. The correct Python formulation is lines and lines and lines.

Really, what you want is a more expressive program. That's why I switched from Python to English, which is a much more expressive program. You can describe every single ARC-AGI-2 task in 10 bullet points of plain English, most of them in 5 bullet points.

I think this actually gets to the heart of ARC, right? Everything is quite simple. It's not very hard. I think this is also how we do it, too. When we look at these ARC grids, we're coming up with these bullet points in our head and checking them: “Okay, this was right, this was right.” Python doesn't have these features. It's just not as expressive as natural language.

I think another way to put it would be that you have this inductive-transductive trade-off, right? You could think of language models as being trained inductively, and then they have an inductive bias, and you almost want to let that inductive bias express itself fully, in a way. The way you do that is to give it the full power of how it was trained.

I think this is the same thing with humans, too. If I told you to solve with Python programs, you'd do a way worse job, even if you were an expert at Python. I think fundamentally it's more general, and it leads to more general and better solutions. The accuracy is much higher when you use natural language.

Now, the problem is that you actually have to verify whether the instructions are correct. You can't run natural language on ARC grids. This was the fundamental problem with the solution. This is what made iteration challenging, especially because, for each grid, for each training example, you have to run the natural-language instructions, and it takes a really long time, especially with this thinking model.

I originally started with a weak model. It's the checker model; it's the checker agent. Let's just use GPT-5 mini or nano. It did terribly. So I ended up finding that it was actually more important that the checker was stronger than the actual instruction creator, which I think is interesting.

That just highlights the trade-offs with using natural language. You can express programs that you want to run much more concisely, but then they're not runnable programs. You actually have to check them inductively. That was the trade-off, but it was worth it for ARC-AGI-2.

Tim Scarfe

Yeah, so fascinating. For the audience, we've been using transduction and induction to distinguish predicting the solution space versus predicting a program. I had this discussion with Clément Bonnet—we need not detain us now—but I think in traditional machine learning, transduction means that the test example is a function of your prediction.

I had this discussion with the ARC-AGI architects as well. When you have this natural-language description, natural language is more expressive, which simply means that there are more degrees of freedom. This is the beauty of LLMs: there's this huge space that you're traversing around, and when you use natural language, you can just traverse to more places in that space more easily.

So it seems like it would be a win, and I'm really fascinated to find out whether that is just a huge component of your solution, because on Eric's solution, he's still predicting programs and still doing very well. So I'm not sure about that.

The other thing is, I wasn't entirely sure whether you are actually using a transductive method. In your solution checker agent, is it directly going to the solution space, or is it generating a program and testing it?

Mike Knoop

In the checker, it takes in the natural language, and then it outputs a grid. That's all it does. It just outputs a grid.

Tim Scarfe

Okay, cool. So you've moved to a transductive modality. Did you see any errors in that? Did it sometimes produce the wrong grid?

Mike Knoop

All the time.

Tim Scarfe

Yeah.

Mike Knoop

Yes.

Tim Scarfe

All the time.

And it's worth noting, actually, the Python solution was—I obviously tried my V1 solution on V2, right?—and it wasn't so much worse. But part of what I wanted to do with V2 is show that, as language models get more powerful and we get to use thinking models, we can start using more general solutions.

I just thought there was something elegant about using natural language, and it also happened to be that there were problems that I could tell the Python functions we were never going to get. There are basically no problems that my ARC-AGI-2 solution won't get. So when Grok 6 comes out, or GPT-7 comes out, you can use my V2 solution and it will win; it will beat ARC. That is not the case for my V1 solution.

Tim Scarfe

The other important thing is you're now using Grok 4, which is very, very powerful. I assume you chose Grok 4 because it just happened to be the winner on the leaderboard for the base ARC-AGI-2. How much uplift is coming from that? For example, if you used Grok 4 on your solution last year, how much better would it be?

I actually don't think it would be so much better, for some reason. This is what I talk about in my blog post: these language models are very spiky in certain things where they were trained heavily.

I think what happened with Grok is there was a distribution of similar-shaped tasks—grid tasks, just reasoning in a general direction—that allowed Grok to have a special capability in this area. I actually tested each model. I tested Grok versus GPT; I didn't just go by the leaderboard.

Grok definitely outperformed. The problem is that, for my V1 solution, you also have to generate code, and Sonnet 3.5 is really good at thinking about and generating code. I prefer Sonnet to Grok for code generation.

My guess would be that, if you use my V1 solution, it’s highly possible that Opus 4.1 would be the best. I haven’t tested that. It would be very expensive to run on Opus 4.1, but maybe it’s worth testing.

I think the general idea is that these networks are very spiky when you get into specific domains. It actually matters very much which model you use, and ARC is a great example of this. The leaderboard is super spiky in ways that other benchmarks are not.

Tim Scarfe

I did an interesting interview at Europe's last year with the Google guys, and they were talking about adaptive temperature in language models for reasoning. There’s this constant trade-off: with reasoning, we want to be quite constrained, right? We want to go down a particular pathway, and we want to be constrained by our knowledge. When we’re being quite creative and flexible, we want to be able to go in different places.

I’m really interested in creativity, for example, and I think creativity is very similar to reasoning, as Chollet talks about. You’re composing these constraints together; there’s this fecundity of knowledge, and you need to respect it as much as possible, because if you don’t respect it, you’re not grounded anymore. It feels to me intuitively that code is great because it means I’m actually respecting the constraints, the semantics are correct, and it’s grounded in the real world.

Do you feel in any way that, by using these natural-language descriptions, you’re creating something which might, by dint of chance or search, find the right solution but isn’t correct and verifiable? Yes. Okay. Yes, tell me more.

Yes, for sure. I think generally, when models think in natural language and output natural language, they’re higher entropy. The second you start prompting with code, they go into code mode. There are a lot of papers that show that, just by prompting it in a certain direction, it activates certain weights that are naturally lower entropy.

That was part of what I wanted. I actually wanted to introduce entropy because, still, on most ARC tasks for V2, the models don’t get close. My solution was the top, and it’s at 30%. I wanted to inject as much entropy as possible, which is partially why my prompts are so broad.

I could definitely improve my accuracy on a few tasks by making the prompts more specific, but I wanted to constantly berate it: more entropy, more entropy. I actually found that to be a positive, not a negative.

Tim Scarfe

Interesting. On the efficiency of the solution, the o3 model from OpenAI was about $200 per task. I think it was sampling, right? They just sampled it a bunch of times, so they had a basic verifier. Is that correct?

I don’t think we ever figured that out.

Tim Scarfe

So you think it could be? When I interviewed Chollet, he was being quite—not cagey—but it seemed like he was suggesting they were actually performing a search algorithm. I think the OpenAI guys said on Twitter, “No, they were just doing very basic sampling.” I think they even published the code they used. I’m not quite sure.

Mike Knoop

Yes, I also spoke to the OpenAI guys about this, and I’m not sure, after all of this, what they were doing. I think it’s probable that they were doing sampling.

Tim Scarfe

It’s hard to imagine they weren’t doing something else. They were certainly doing sampling. I’m not sure what else they were doing, or whether the model was fine-tuned. My best guess is that they were sampling and it actually was not a fine-tuned model.

Oh, very interesting. I remember there was that big hoo-ha at the time that it was scandalous they were training on the training set. But anyway, that’s one side. There’s also the thought that, if they did something like what you’re doing—this approach of iterative refinement with verification at every single step—would they have done even better?

Mike Knoop

For sure. I think OpenAI generally wants to do the right thing and wants its solutions to be very general and broad. This is the sense I get, and I think it’s part of their culture.

I spoke to the OpenAI guys. They did include the training data in that o3 model, but I think that’s fair game, right? They also—I don’t think they fine-tuned on it, right? It’s just part of the corpus that went into pretraining, which to me is fair game. This is totally fine.

Tim Scarfe

Okay, very cool. So, on ARC-AGI-1, their efficiency was $200 per task. What was your efficiency?

On ARC-AGI-1, maybe $10, something like that. I need to check. In order of magnitude, maybe—I need to check.

Tim Scarfe

Yeah. Talk to me more about this. Eric Pang’s solution came in slightly worse than yours, but I think it was a fair bit more efficient. I’m going off my memory now. Was it about $8 per task? Was yours about $30 per task on ARC-AGI-2?

Oh, this is the V2. My latest solution was around $30 on V2 and $8 on V1.

Tim Scarfe

Oh, okay. I think Eric’s solution was slightly more efficient, and he was indicating that it was because he was doing library learning and transfer. I was left thinking, first of all, that it’s interesting you got better results. Is that because there isn’t much transfer? Where does the library transfer come into this?

The broader question is: if you were to make your solution significantly more efficient, what would you do?

I had a version that does library transfer. Basically, I would save the traces from training and try to pull those in during test time. Out of simplicity, because I was getting such high scores with the simple solution, I wanted to just push the simple solution. If someone’s going to beat my score, I might bring that back in.

That will for sure improve the score, and it’s useful. There’s a lot of transfer efficiency. I just found what I was doing very elegant, and so I actually like to keep it. There are no third-party dependencies or anything like that, but that for sure helps accuracy.

I think the fundamental reason why I got higher—and I could match his efficiency and still get higher—is because I was using natural language. Natural language is a much more efficient area to play in. That’s at least what I found.

Tim Scarfe

Yeah. I just wonder how close you think we’re getting to the Pareto optimum of this approach. To give you a few examples, we interviewed the AlphaEvolve team, which was fascinating, and maybe you can contrast it with those guys.

Sakana AI yesterday—Robert Lange was the first author—released what I think was called shrinker. That was a similar kind of evolutionary programming system, and they had some cool features in there, like using bandits and UCB. I guess we’re getting to the point where we’re going to really figure out the most optimal way to do this. By the way, they were also switching between different foundation models.

I think improvements will be logarithmic. Using the language models we have today, I would not expect anyone to break, let’s say, 40%. But you could probably make my solution twice as efficient.

You wouldn’t get more than a few percentage points more accurate, is my guess, but you could make it a lot more efficient. There are a ton of efficiency gains to be made.

Tim Scarfe

We’ve been dancing around this a little bit. Chollet’s measure of intelligence was all about resisting memorization, and there’s this question now of to what extent we’re actually building systems that we might call intelligent. He says that intelligence is simply the efficiency of knowledge acquisition, and I’m really on board with that.

I think it’s fair to say at the moment that your solution and Greenblatt’s solution are quite ephemeral and stateless. When you have a new task come along, you start again from scratch, which means it’s not really adapting, acquiring new knowledge, and transferring that knowledge.

Maybe you would agree that, in the spirit of Chollet’s measure of intelligence, at the moment it’s more of a searching approach. What do you think we would need to do to make it more adaptable?

Right. I think test-time fine-tuning would be the way to fundamentally make it adaptable. But I also think Chollet hits a core problem with language models: their reasoning is domain-specific.

In my blog post, I described how, when you train a language model to reason about math, for some reason most of the reasoning circuits it gains live in the math weights. Then you try to train it on science, and it gets some generalization, but not as much as you would want—and I think not nearly as much as what humans get.

Humans have this generalization engine that is our reasoning capability. This is the fundamental hole in language models today. In fact, I generally agree with François Chollet when he says you can always teach a language model a skill, but it’s the meta-skill—the skill to create the skills—that is AGI. To me, that’s reasoning.

To put it another way, if you fundamentally learn the skill of reasoning, you should then be able to apply that skill to learn all the other skills. That is the meta-skill, and we need to figure it out. You need to do whatever you can—kick out whatever weights you need to—to align the model to reason. From there, you have a foundation from which you can actually build general intelligence.

I don’t know if that was a higher-level answer to your question, but what I’m focused on is really just fitting all of reasoning into these models. I don’t really care what else is left. I just want all of reasoning in.

Tim Scarfe

Yes, I pretty much agree. You probably know that I’m François Chollet’s biggest fan, so I’ve obviously been a huge fan of his for years. By the way, he’s just released the third edition of his book Deep Learning with Python, and I recommend folks read Chapter 19. You can actually read it online for free, and he sketches out this entire vision.

It’s so exciting to see it so beautifully articulated, because there is a bit of an elephant in the room in the scene at the moment. I think so many people just don’t have such a crisp understanding. The only departure I make from Chollet—and from you, Jeremy—is that I think Chollet really focuses on behavioral tests of intelligence. It’s reasoning if it can pass the test and actually get the right answer.

I think we need to go further. This is where I was talking about systematicity and symbolic AI. I think how you got there is important, because it’s possible to get the right answer for the wrong reasons. If we have a system with semantics, so we actually know what these symbols mean and we’ve composed them together in a principled way, then we can not only get the right answer for the right reasons, but also make the system evolvable—to have an efficient epistemic base that allows us to go on to acquire new knowledge in the future.

That, to me, points to the need to have a mechanistic view of how we’re acquiring this knowledge. Would you agree with that?

Yes. I think about it a bit differently, so let me know if what I say is aligned with what you think.

Tim Scarfe

Okay.

To me, pre-training is kind of the opposite of what you described. I view there as being 2 types of knowledge. There’s knowledge that is memorized, like the capital of New York or the Spanish language, and then there’s knowledge that is deduced. That’s physics, special relativity, and general relativity. From axioms, you can deduce these things, and it’s a causal tree.

Then there’s another type of knowledge: what is the capital of North Dakota? That is a knowledge network. It’s not deductive; it’s not a tree. I think pre-training treats all knowledge as a knowledge web. It’s embeddings that are connected, but there’s no guarantee that you have the correct causal relationship between things.

This is where memorization comes in, and I think this is actually where compression fits into intelligence. My view is that intelligence is compression, in that you should be able to deduce—you should be able to build a knowledge tree based on almost nothing. You can deduce so much of math. You can deduce special relativity from the very roots of physics.

Einstein was extremely intelligent because the hints that he needed to come up with special relativity were zero. He could start from almost nothing and build up this deductive tree. I think it’s almost like reinforcement learning, and reasoning is the process of pruning our knowledge network and replacing it with this tree.

Until we have weights that represent the actual deductive nature of knowledge, we won’t actually get generalization. I don’t know if this fits in, but this is kind of how I think about reinforcement learning: it’s replacing a knowledge web with a knowledge tree.

Tim Scarfe

Yes. Yes. This is brilliant. We’re getting to the center of the bull’s-eye here. I remember reading in the first version of your blog post that you were talking about how we need to do this kind of deduction where we synthesize hypotheses, then test them, and do this kind of generate-test loop. That is what creativity is. It’s what reasoning is.

When I first read Chollet’s paper years ago, I didn’t understand whether he was talking about acquisition or synthesis, and I now understand he’s talking about synthesis. Reasoning is like LEGO: you build this kind of tree, this epistemic tree. Actually, this is what we do.

There’s a difference between knowing and understanding. Knowing is at the high level, whereas understanding is actually like this whole structure. Just imagine this big block of LEGO, and you’re tracing down the structure with all the building blocks of how you got there.

I think even when you teach kids at university, what you’re really doing is teaching them facts, but then they synthesize their understanding over time. They’re doing this composition and getting there the way that they get there. We need to build systems to do this.

There’s the perennial problem you were talking about: in deep learning, what we do is start with this big pattern network and sparsify it. I think reasoning should be more about synthesizing from building blocks. When you synthesize, you can actually do types of reasoning that are not in the training data. You can build things that simply aren’t there. You can think about things and figure them out.

Do you think of that as a gap?

Yes, I think that’s exactly right. Then the question is: can you build the system with language models or not?

Tim Scarfe

Yeah.

I think you can build them with language models. The fact that we’re slowly increasing our ability to synthesize new information is a testament to what I’m saying. Reinforcement learning—and reinforcement learning with verifiable rewards—is fundamentally ensuring that whatever circuits led to the right answer must be consistent with the deductive tree.

It’s basically: can you replace all of your pre-trained weights with weights that are coherent with the environment? The problem is that there are so many weights from pre-training, so it’s very difficult.

One of my hot takes is that pre-training, in many ways, slows down reasoning. It makes it harder to reason. The analogy I draw is that you have consultants who know the names for things but couldn’t deduce the thing, and then you have Feynman, who can deduce anything. Reinforcement learning is turning your consultant into Feynman.

This is what I’m most interested in. It’s interesting because you get to play at both sides. You get to play with the pre-training: maybe we shouldn’t include these things and should let the model figure them out in reinforcement learning. There’s no guarantee that if we pre-train it like this, it’s going to have the proper deductive circuitry. Maybe this is best left for post-training.

This is a hot take. This is not what people currently think. I think people think, “Let’s jam as much information as we can into pre-training, and then we’ll reinforcement-learn when we need to.” But I think this could be incorrect.

Tim Scarfe

Yeah. I pretty much agree with you, with the caveats we discussed previously: if we could do that on the basis of representations that are actually grounded in the world, rather than things that just happen to give you the right answer for the wrong reasons, that’s absolutely true.

Just a bit of a curveball. In the first version of the article, you said you were inspired by Yann LeCun’s JEPA—Joint Embedding Predictive Architecture—and he’s also a big advocate of energy-based models, which are really cool.

I don’t know if you’ve seen the recent couple of papers applying them to transformers, where, essentially, it’s a step toward probabilistic models where you have uncertainty quantification, can do counterfactuals, and actually have to solve an optimization problem at inference time. You can do adaptive computation, and it’s all very exciting.

But I still have some reservations. Do you think architectures like that are exciting?

I think they are exciting. I’m a bit less excited about them, not because of their merits, but because I think I was underrating transformers when I wrote that. This was really before I started being an actual researcher and building transformers—actually coding with them. Since then, I’ve had a new appreciation for language models.

I think where I was coming from was that it seemed like language models were overfitting to the next token, right? JEPA is so interesting because all of a sudden you have them predicting concepts, and fundamentally, we care about concepts. The words don’t really matter; it’s the concepts that matter. But I think language models do operate on the conceptual level in the hidden layers, and that was something I slowly came to the realization of.

I think there’s a lot of potential in JEPA frameworks. I think they’re really cool, and I hope people keep pulling on them. But I think most of the benefits I thought came from JEPA exist in language models. I just didn’t see it at the time.

Tim Scarfe

You know, a lot of people get an earworm where they just get obsessed with an idea and think about it all the time. What is that thing for you?

If we do have these language models with weights that are aligned in what I guess is the tree of deduction, it seems like we’re still missing one more thing, which is creativity, which we touched on, right? You can have the correct deductive tree, but then how do you search through all of the possible premises you can add to this tree? How do you find the right ones?

I think there are a few experiments that I’m looking forward to doing. One of them is ablating pretraining data at will and then basically using reinforcement learning, building an environment to have the model regenerate that information. For example, let’s say you had the ability to ablate special relativity and all of the physics that came from special relativity from your pretraining data. That’s a gold mine of an environment, right?

Because now you can prompt the model, you can do everything, and you can really try to get it to deduce special relativity. My hunch is that part of the reason why models are not yet great at coming up with novel solutions and information is because they don’t have the circuitry of invention. I think that is actually a circuit that needs to be developed, and we don’t have the environments to develop that circuit yet.

I actually just saw today—I think OpenAI released a math paper. I skimmed it, and they almost came up with a novel conjecture or something like that. I think that’s exactly what was in my head: I want to be able to build environments where the model’s never seen something and basically tries to deduce these new things that are outside the distribution. Over time, it learns and practices and builds this invention circuit.

I think it’s two things: reinforcement learning to make sure that the knowledge tree is consistent, and then making sure that the circuitry is able to pull from its entire corpus of understanding of the world and bring that in to fuel the innovation engine.

Tim Scarfe

Yeah, I think we’re so close. I think the only slight disagreement is whether there could be such a thing as an invention circuit. It seems to me, just like that LLM biology paper and whatnot, that it would be very patterned in the weights. It would be templated; it would be specific to certain domains. I feel that we can do it, but we would need to build a controller on top.

I also feel, by the way, that creativity is very domain-specific. What I mean by that is, with this Kenneth Stanley view that there’s this big phylogeny of knowledge, I’ve noticed that when I hire creative professionals—an editor, for example—they can’t edit my show, even if they’re really good at editing other people’s shows, because they simply don’t know anything about machine learning. I’ve just discovered the crazy degree to which creativity is domain-specific.

I just wonder whether there is an algorithm for creativity, because I’ve got this epistemic lens of creativity, which is that it’s simply about accessing all of the ancestors in my tree and doing some composition. It’s like building LEGO and so on. If I want to use something from the tree over here, this other branch, maybe it’s compatible and I can bring it in, or maybe I just need to start a new branch, or maybe I need to jump over to the other branch. I’m not sure whether I’m applying the same algorithm when I’m doing that.

I think you actually are, and it’s not sufficient to just be creative in this case. You need to be creative and knowledgeable; otherwise, your creativity can’t build the tree. I think your editors can’t build the tree because they don’t have the deductive footing. That would be my best guess, but that’s an interesting perspective. I want to think about that.

Fundamentally, creativity is knowing which axioms to include in the next branch of the tree, right? You’re at level 5 of the tree. How do you get to level 6? It’s knowing which assumptions to pile in to get to level 6, right?

Tim Scarfe

It’s so beautiful that we’re writing an article about creativity at the moment. I believe that in order to be creative, the depth of understanding of the tree is very important. Per Chollet, intelligence is the efficiency of how much of the history you can acquire.

A university professor understands the tree very deeply, and that actually makes them unintelligible to a normal person. When Stephen Wolfram talks about the Ruliad all the time, people have no idea what he’s talking about. He’s actually being very expressive; he’s talking about things at a level of abstraction that can refer to anything, but it’s beyond most people’s cognitive horizon.

There is something to be said for that. When you can respect the history deep down into the epistemic tree, the creative stepping stones you take, because they respect the history, actually have more evolvability, right? You’re still grounded in the real world; you’re not becoming incoherent. So there’s something there about how really knowing things deeply is important.

Yes. For the record, I would define understanding as being able to—I mean, I think understanding is a spectrum. On one end, it’s memorization, which is zero understanding, and on the other end, it’s the ability to deduce and also to deduce correctly.

I agree with what you said. It’s not enough just to have the right proof; you actually have to have understood the tree. That is understanding, and then intelligence is just how many things you understand. So it’s really just how wide and high your garden of trees is.

Tim Scarfe

Yeah, we’re pretty similar. I would say intelligence is the efficiency with which you can acquire the tree, reasoning is building the tree, discursive reasoning is executing the tree, and understanding is simply possession of the tree.

Alexia Jolicoeur-Martineau

So intelligence to you is the speed at which you can build the tree, not how many trees you have?

Tim Scarfe

Yeah.

Alexia Jolicoeur-Martineau

Or not how large your tree is?

Tim Scarfe

Yeah. Yeah. Yeah. I think understanding is how much of the tree I have.

I think that’s correct. Yes. Yes. You could have a very intelligent child who doesn’t know a lot about the world but has the ability—the potential—to build a tree.

Tim Scarfe

Yes.

Alexia Jolicoeur-Martineau

Yes, I think that’s correct.

Tim Scarfe

There’s a spectrum of understanding. Language models famously don’t understand the tree very deeply, so they only understand the tree a few levels down. When language models are doing autonomous generation, the reason why we have to do so many different generations and select the best one is because it’s not grounded. It doesn’t understand the tree very deeply.

We can overcome that because we understand the tree deeply. We can put a prompt in there that constrains their generation, and now we can make them act as if they understood the tree when they didn’t. But we just need to build models that do understand the tree deeply, and then we can trust them to generate autonomously.

Yes, that is a very good way of saying what I think, and that is what I’m focused on. That’s a really good way of putting it, which is forcing language models to develop these deep trees from the ground up. I think you can only develop it from the ground up, so we need to come up with new techniques and new environments to grow the trees instead of pretraining, which is pre-filling—it’s not random, but it’s the web; it’s not a tree.

Tim Scarfe

Yes, there’s also the vexed issue of what happens at the bottom of the tree. Chollet argues that this spiky core knowledge—these are knowledge primitives that are so fundamental that, inside the deductive closure of those primitives, we can talk about anything. You can still get lost in different parts of the tree, I suppose, and there are some issues of intelligibility when moving between them. But if you understood the tree deeply enough, you could go anywhere.

Maybe there are different trees. In physics, there are different levels of description and different levels of understanding the universe. Do you think it’s one big tree or lots of trees?

Alexia Jolicoeur-Martineau

I think it’s kind of gated by the laws of whatever you’re doing. I looked into this a few times, and I know there are some axioms that you need to take for granted when, let’s say, deducing some forms of math, right? They would be their own tree because you can’t get to one part without the other.

If you take this for granted, then you can't deduce this, but you can deduce this. And so I'd say, basically, they have to be logical chains, and, of course, there might be multiple chains. Maybe everything is grounded in logic. Logic is the fundamental—I guess that is true, actually. Logic must be the fundamental block of a tree, right? Everything comes from logic, because if we didn't have logic, we couldn't have trees. I guess it would be all one tree, and it's a logic tree. I guess there are people that subscribe to logic trees, and then there are people that are illogical, and they don't have their trees.

Tim Scarfe

Jeremy, it's been an absolute honor to have you on the show. Just before we go, are you hiring, or is there anything you want to say to the audience?

For sure. Yeah. At Reflection AI, we're building open intelligence models. We're hiring across the stack: pretraining, post-training, and large language models. We have a lot of GPUs. If you're interested in pretraining or post-training, we're in San Francisco, New York, and London. Definitely apply on our site, or you can just hit me up on Twitter.

Tim Scarfe

Amazing. Jeremy, I've really enjoyed this. Thank you so much.

Jeremy Nixon

Awesome. Thank you.

Tim Scarfe

Just keep doing what you're doing, man. I really, really think that you're onto something here. I mean, obviously, there was the minor discussion about how we're actually going to do this, but I think the direction is quite clearly set. I know it's such a vexed issue, though. I'm interviewing a bunch of cognitive scientists in Japan next week, and you can really go down the rabbit hole on this.

For example, I'm a big fan of externalism, like enactive cognition, and there are also all these philosophical views where consciousness is basically a property of certain types of physical material. What we're talking about here with understanding, if you actually abstract it into physics, is certain types of causal graphs. You can argue that certain types of cognition actually require certain types of physical instantiation, where, in the middle of that graph, you have material capable of producing consciousness. So some components of understanding are phenomenal; they're conscious, right?

When you start taking it to this philosophical level, there's almost no end to it, because you'll always have people who argue against functionalism and say that cognition must be physically instantiated in a certain way. I'm not sure where I am on that, because even the cognitive scientists are saying, “Okay, well, guys, we have to admit that, even though we can make all of these arguments—I mean, fuck me, these LLMs, they're doing so well.”

But, yeah, I think it'd be a fascinating discussion, because a lot of neuroscientists are internalists. They basically, like Jeff Hawkins, think that all of the shit happens in the brain, and we have these sensorimotor circuits, and we've got this master algorithm in our neocortex, and that does all of the things. I think there's something to be said for that, but I think it does ignore quite a lot of the other field.

Yeah. Well, it all comes down to whether we can build machines that actually have a deep, grounded understanding of the world. Let's assume that this doesn't actually have to be physically grounded, right? If the representation is grounded in the sense that it's a faithful description of what is happening out there, and we can do this creative reasoning on that understanding, then what's to stop us from inventing new things? I mean, that's basically the thesis, right?

Yeah. Beautiful stuff, Jeremy. Thank you so much, man. I really, really appreciate it.