Francois Chollet - ARC reflections - NeurIPS 2024

I've never been in the purely symbolic camp. If you go back to my earliest writing about why we need program synthesis, I was saying we need deep learning-guided program synthesis. We need a merger of intuition and pattern recognition with discrete, step-by-step reasoning and search in one single data structure.

I've said very repeatedly for the past 8 years or so that human cognition really is a mixture of intuition and reasoning, and that you're not going to get very far with only one of them. You need the continuous kind of abstraction that's provided by vector spaces and deep learning models in general, as well as the more discrete, symbolic kind of abstraction provided by graphs and discrete search.

So why do your detractors see you as a symbolist when you're clearly not?

I'm not sure. I've been into deep learning for a very long time, since basically 2013. I started evangelizing deep learning very heavily around 2014, and back then the field was pretty small. Especially with Keras, I think I've done quite a bit to popularize deep learning and make it accessible to as many people as possible.

I've always been a deep learning guy. When I started thinking about the limitations of deep learning, I was not thinking in terms of replacing deep learning with something completely different. I was thinking of augmenting deep learning with symbolic elements.

What is your definition of reasoning?

1. Reasoning Means Novelty

I don't really have a single definition for reasoning. I think it's a pretty loaded term, and you can mean many different things by that. But there are at least 2 ways in which I see the term being used, and they're actually pretty different.

For instance, if you're just memorizing a program and then applying that program, you could say that's a form of reasoning. Let's say in school you're learning the algorithm for multiplying numbers, for instance. While you're learning that algorithm, and then when you're tested, you're actually applying the algorithm. Is that reasoning? I think yes, that's one form of reasoning.

It's the kind of reasoning that LLMs and deep learning models in particular are very good at. You're memorizing a pattern, and at test time you're fetching the pattern and reapplying it. But another form of reasoning is when you're faced with something you've never seen before and you have to recompose, recombine the continuous building blocks you have access to—your knowledge, and so on—into a brand-new model and do so on the fly.

That is also reasoning, but it's a very different kind of reasoning, and it internalizes very different kinds of capabilities. I think the important question about deep learning models and LLMs in particular is not whether they can reason. There's always some sense in which they are doing reasoning. The more important question is whether they can adapt to novelty.

There are many different systems that could just memorize programs provided by humans and then reapply them. What's more interesting is whether they can come up with their own programs and their own abstractions on the fly.

Broadly speaking, I think programming from input-output pairs will be a widespread programming paradigm in the future, and that will be accessible to anyone because you don't need to write any code. You're just specifying what you want the program to do, and then the computer programs itself.

If there's any ambiguity in what you meant—and there will always be ambiguity, especially if the instructions are provided by a nontechnical user—you don't have to worry about it because the computer will ask you to clarify. It'll tell you, “Okay, so I created basically the most plausible program given what you told me, but there's some ambiguity here and there. What about this input? Currently I have this output. Does that look right? Do you want to change it?”

As you change it iteratively, you are creating this correct program in collaboration with the computer.

Is there just a massive new type of architecture we need to build for this?

2. Lifelong Learning Needs New Architecture

Yeah, I think we're going to need a completely new type of architecture to implement lifelong distributed learning, where you have many instances of the same AI solving many different problems for different people in parallel and looking for commonalities between the problems and commonalities between the solutions.

Any time they find sufficient commonality, they just abstract these commonalities into a new building block, which goes back into the system and makes the system more capable and more intelligent.

I think I've got it now. So what you're building is a globally distributed AGI on the basis that we find a good solution to ARC.

Well, I can't really tell you exactly what we're building, but it's going to be cool.

Yeah, it sounds pretty cool. What's your theory on how o1 works?

3. o1 Turns Search Into Reasoning

We can only speculate. I'm not sure how it really works, but what seems to be happening is that it is running a search process in the space of possible chains of thought, trying to evaluate which branches in the tree work better, potentially backtracking and editing if the current branch is not working out.

It ends up with this very long and sophisticated, and plausibly near-optimal, chain of thought, which represents basically a natural-language program describing what the model should be doing. In the process of creating this program, the model is adapting to novelty.

I think something like o1 is a genuine breakthrough in terms of the generalization power that you can achieve with these systems. We are far beyond the classical deep learning paradigm.

So we're now at the end of the ARC Prize, and you just released a technical report about it. Can you reflect on the competition?

4. ARC Prize Changes the Narrative

I think we learned a lot, and overall it's been a big success. In 2024, we've seen a huge shift in the narrative around AI. Previously, the mainstream narrative was that we could just train larger models on more data—100× larger models, 100× more data—and get something that's basically AGI.

More recently, in the past year or so, there's been this realization that we were going to need something akin to System 2 reasoning: it is not something that will simply emerge from pre-training larger models on larger datasets. You need to add it to the system somehow.

Of course, you can use test-time search, program synthesis, and these sorts of techniques to do it. I think ARC Prize was really part of that narrative shift.

It's also part of the reason why ARC Prize has been very popular, with lots of teams entering, lots of people talking about it, and using it as a kind of reference for whether we might have achieved AGI or not. Which, by the way, is not what ARC is intended to be. It's not intended to be an indicator of whether we have AGI.

It's really just intended to be a tool—a research tool that gets you to think about the right problems and focus on the right directions. The reason why it's been successful is because there was a latent demand for something like this.

Many people had this intuition that plain LLMs weren't going to get to AGI, that we needed something more—either as a replacement for LLMs or some kind of superstructure around them—that would implement System 2.

And there was this intuition floating around, and I think lots of people just latched onto ARC Prize as a concrete sign that their intuition was right.

So there were 2 flavors of the ARC Prize. There was a compute-restricted one, and there was the main one. Can you reflect on the difference between the 2 in terms of the entries?

Sure. We had the main track for the competition on Kaggle, and this was only for submissions that were self-contained. The reason is primarily that we needed to keep the private test set fully private. We cannot send it to a third-party server via an API.

On this track, people are submitting notebooks, effectively, that have to run on a VM in less than 12 hours, and the VM has just 1 P100 GPU. This is equivalent to a total of roughly $10 worth of compute per submission.

Then we had the public leaderboard, which was targeted at frontier models. It was evaluated on a different set of tasks because we cannot evaluate on the private test set without leaking it to a third-party server. Instead, it is evaluated on an entirely new set of tasks, which we call the semiprivate test set.

It is semiprivate because it is not published anywhere, so it is not public. But it is also not entirely private, since we are actually sending the task data via the API to OpenAI, Anthropic, and so on. Each submission on that leaderboard can use up to $10,000 in API credits. So that is 1,000 times more compute than the private leaderboard.

On a per-task basis, it is actually a little bit less because we are evaluating on more tasks. We are evaluating on the 400 tasks from the public evaluation and the 100 from the semiprivate evaluation, which is really what we are looking at. On a per-task basis, that is about 200 times more compute.

What was really remarkable, and frankly quite shocking, is that the scores you end up seeing on the public leaderboard track what you are seeing on the private leaderboard. In both cases, we are at about 55%.

This tells you that it is not just about throwing more compute at the benchmark. Compute is really a multiplier for ideas. Of course, if you have infinite compute, you can solve the benchmark in a very stupid way, such as via brute-force search. But having better ideas gives you dramatically more leverage for your compute.

That is why we end up with solutions consuming $10 worth of compute that are doing around 55%, while solutions consuming $10,000 worth of compute are doing exactly the same. They are just not nearly as compute-efficient.

Which was the most successful method that worked across the board?

5. Induction Versus Transduction

There were really 2 categories of approaches that worked well. One is deep-learning-guided program synthesis, which is my favorite approach personally and what I have been advocating for many years.

Most people nowadays are doing deep-learning-guided program synthesis using LLMs. They are using LLMs to generate code and to iteratively debug code. Some people are trying to do deep-learning-guided program synthesis using building blocks from a DSL. I think this is a very underexplored approach, but I think it should be an effective one.

The other category of approaches has been test-time training, where you use an LLM to directly try to predict the solution given a task description. You are looking at a set of demonstration pairs and then looking at an input grid, and you are directly trying to generate the output grid. This is a process that we call transduction, as opposed to induction.

Program synthesis is program induction. You are trying to write down the program that will map the input grids to the output grids. In transduction, you are just trying to directly predict the output grid.

If you try to do this with LLMs and stay within the classical deep-learning paradigm, where you have a big model that is pretrained on tons of data and then, at inference time, it is static, you really cannot adapt to any meaningful amount of novelty. You are stuck memorizing patterns and, at test time, fetching and reapplying the patterns that you have memorized.

To go beyond that, people have started using test-time training. The idea is that you start by pretraining a base model. It knows about ARC and ARC tasks. Then, at inference time, on each new task that you see, you try to fine-tune the base model on the demonstration pairs, basically trying to recombine the knowledge contained within latent space into a new model adapted to the task at hand.

If you do not do this test-time adaptation, LLM-based transduction models are stuck below roughly 10% accuracy. But if you start doing test-time training, you unlock a dramatically higher level of generalization, and you can go well into the 50% or 55% range, probably even 60% soon.

The big question is whether that is in the spirit of the challenge or not. I have done some interviews on this transduction with active fine-tuning, where you take the test instances, do some dataset generation and augmentation, fine-tune the model, and do really well.

Certainly, as a methodology to broadly generalize to lots of tasks, it is good. But the problem is that it still has human supervision. You have stressed from the very beginning, in your measure of intelligence, that we need developer-aware generalization. That simply means that we cannot have a human supervisor specializing the thing for every downstream task.

We need to make a system that can itself generalize to tasks that the developer of the system was not aware of. By that metric, do you feel that it is not in the spirit of ARC?

No. I think it is a completely legitimate way to approach a challenge. I also think it represents a very significant breakthrough in generalization power and in the ability of these models to adapt to something they have not seen before.

I do not think the supervision we are talking about is really done by humans. You are using the demonstration pairs to fine-tune the model, so this is actually fairly autonomous. Of course, this needs to have been programmed by a human who is aware of the task format. But the same would be true for a program-induction-type approach.

I think it is very much in the spirit of the challenge. Further, I think it does demonstrate a legitimate breakthrough in generalization.

Pressing on the legitimacy thing, it stands to reason that we do some kind of active inference.

Of course. Yes.

We are always adapting to novelty, building new models, and so on and so forth. So what is the difference between transductive active fine-tuning and what we do?

I am not sure what we do exactly. Of course, we are doing active inference. What does that mean exactly? What algorithms and data structures are we leveraging? We do not know, so I cannot really tell you what the difference is.

I will say one thing. When you are doing test-time training with an LLM, you are letting a gradient-descent process do the knowledge recombination. To adapt to novelty, it is necessary to take the knowledge that you have and recombine it in some way. There are multiple different things you could be doing to achieve that.

You could be doing a program search, where the thing that is adapted to the new task is a program, and you are building this program via a search process. You could also do something like what the o1 model from OpenAI is doing, which is very similar to that. You are effectively doing a search in the space of chains of thought, and you are writing down this chain of thought, which is basically a natural-language program for the model to execute.

You are doing this search via an AlphaZero-style search process, like a tree-search process. That is one approach. You can also use discrete program search to write down the program, and the program is the artifact that models the task at hand.

Or you can try to modify the weights of the model and modify its representations to create a new model adapted to the task. In this case, the artifact adapted to the task is the model itself. That is what test-time training does, and it does so via gradient descent.

My take is that I do not think humans adapt to novelty by recombining what they know via gradient descent specifically. I think the level at which we represent knowledge, especially in the context of solving ARC puzzles, is much more abstract and symbolic in nature. The way we combine it is much closer to function composition than to what you can achieve with gradient descent.

In short, I do not think it is a good idea to try to use gradient descent as a replacement for a programming process.

I’m actually more of a fan of what a system like o1 is doing than trying to do this test-time training.

We’ll get to o1 in a second, but there are folks who are really bullish on this transduction thing. I think you and I agree that the reason it doesn’t work in principle is because language models are just finite-state automata. They don’t have this compositional generalization, even in principle, right?

But we know there’s evidence that transformers on their own can’t do basic things like copying and counting and all of these kinds of things. Some people are bullish because they think we could improve the architecture so that it could do those kinds of things, and then a transductive approach might work in the future. Would you rule that out?

No, I think that’s entirely plausible. As you point out, even given lots of data, there are many algorithmic tasks that you cannot train a transformer to do. Or even if you can, it will learn a solution that does not generalize very well. It will work on inputs that are pretty close to what it’s been trained on, but if you try an input that’s very far away from the training distribution, it will just fail.

People who think we can move past these limitations are saying that we can make architecture tweaks. They are right. It is always possible to take a deep learning model and modify the architecture to bake into it some strong structural prior about the algorithmic problem that you’re trying to solve. Then you can actually use gradient descent to find a solution that will generalize.

The way this works is by asking a human engineer to first understand the task at hand and convert that understanding—that symbolic understanding—into a better architecture: an architecture that, in some important ways, is isomorphic to the causal structure of the problem.

Of course, if you want to autonomously adapt to novelty, you cannot require a human engineer to intervene in writing your architecture. The process has to be fully autonomous. The question is, can you create an architecture-search or architecture-generation machine that will take a problem, identify the key elements of the problem structure that you need to bake into your architecture, and then generate the architecture?

If you can do that, then sure, maybe you can leverage that to achieve trans-structuralization. But I think that problem is at least as difficult as program search in the general case.

It’s also pretty overfit, as it turns out. On the public leaderboard, we are evaluating on the public evaluation set, but we’re also evaluating on this semi-private evaluation set. The reason we’re doing that is to test for overfitting. Some solutions might be overfit to the public evaluation, and that was actually the case for their solution.

The score was something like 10 percentage points lower on the semi-private set. Other solutions—solutions based on program synthesis in particular—didn’t feature that drop at all. They were actually scoring exactly the same on both sets. So it tells you that whatever they did was, in some important way, overfit to the data that they had.

Can we touch on that as well? You said that when you take an ensemble of all of the original Kaggle 2020 competition results, it got to about 49%.

That’s right.

Tell me more.

6. ARC 2 Fixes Benchmark Flaws

In the very first ARC-AGI competition on Kaggle, back in 2020, the highest score by a single submission was only 20%. That was the winner, Icecuber, and it was doing basic brute-force program enumeration.

But if you looked at all of the submissions in the competition and assembled them together, you would see a high score of 49%, which would have been state of the art until very recently. That was, again, 4 years ago. What that tells you is that about half of the private test set is easily brute-forceable, because every single entry in the competition back then, in 2020, was doing some kind of brute-force program enumeration.

That is not AGI. That is not the sort of solution that we are looking for. The fact that doing this kind of stuff at scale could get you 49% is a very strong sign that the benchmark is flawed.

Today, if you look at the 2024 competition, the state of the art for any single submission is about 55%. You could say, “Okay, so we are very far from solving the benchmark, since the bar is 85%.” We also know that humans can solve very close to 100%. If I showed you the private test set, you’d probably do something like 97%, 98%, or 99%.

If you take an ensemble of everything that was submitted in the competition in 2024, you would get to a high score of 81%, which is pretty close to what we are looking for. I don’t think anything that was tried in the competition this year is really close to AGI in a meaningful sense, and the ensemble of everything is still not close to AGI.

It just shows that scale—brute-force compute scale—will eventually crack the challenge. What that says is that the benchmark is flawed and is now close to saturation, and that we need something else. That’s why we’re working on ARC 2.

ARC 2 is not exactly a novel idea. It’s not a reaction to the results that we got in 2024. I first publicly announced ARC 2 in early 2022, so a while back. It was a reaction to the 2020 competition results. I was aware that the benchmark had flaws: it was not quite diverse enough in terms of task diversity, not every task was quite unique, and there was some amount of redundancy. It might not have been challenging enough, either. So I wanted to do V2.

Back in 2022, in partnership with Lab42 in Davos, we started crowdsourcing a bunch of new tasks. Since then, we’ve kept making a lot more new tasks, and we’ve started filtering them and trying to analyze which ones were difficult for humans, which ones were difficult for AI, and so on.

We’ve collected a lot of human testing data as well. We hired people to actually try to solve the tasks, and that gives you a lot of information about how many attempts they use for different tasks and how many people solved each task. You can turn this information into a kind of human-facing difficulty rating. Then you can try to cross-correlate it with what AI can do.

We’re going to be releasing ARC 2 early next year. It’s going to address all of the flaws of ARC 1. It’s going to have slightly more data and leverage 3 sets: the public evaluation, of course; the semi-private evaluation; and the private evaluation.

One problem with the first few competitions on Kaggle is that we’ve been reusing the same private test sets across every competition. Anytime anyone made a submission, they could immediately see their score on the private evaluation. This can lead to some amount of information leakage about the private test set over time.

In fact, there are very well-documented techniques for how you can start to reconstruct the contents of the private evaluation, given enough submissions. We want to avoid this in the competition next year.

The way we’re doing this is that we’re going to be evaluating on the semi-private evaluation during the competition. When you submit something, you get your semi-private evaluation score, and only at the very end of the competition, to create the final leaderboard, are we actually going to run the submissions on the fully private test set.

Another nice thing about doing that is that it enables you to make direct, apples-to-apples comparisons between the Kaggle leaderboard and the public leaderboard with all the frontier models. Of course, it’s not going to be the same amount of compute, but the scores are going to be apples to apples. You’ll be able to say, “Okay, so my AI on this test set is scoring as well as, I don’t know, o1 Pro or whatever.”

The Kaggle 2021 49%—when you take the aggregate—first of all, which ones were more brute-forceable? I suppose one way of looking at it is that, certainly with the mosaic patterns, there’s an exponential number of combinations in the solution space. But of course, in the compositional space, it might be brute-forceable.

We’re also starting to see with the induction and transduction methods that they can solve different types of problems. Maybe when you do human evaluations as well, humans can solve different types of problems, and there are all these overlapping Venn diagrams. How do you think about that?

7. Induction Meets Transduction

Yes. It’s actually one of the most interesting findings of the 2024 competition: program induction and doing transduction with deep learning models, typically an LLM, lead to solving substantially different sets of tasks.

And so this was surprising to me, but in retrospect, it makes sense, especially if you start looking at the tasks and analyzing what makes them different. There are some tasks that are very perceptual in nature. They are effectively pattern-recognition problems, and this is the kind of task that you solve well with transduction methods. There are other tasks that are much more algorithmic and much more discrete in nature. You cannot provide an easy solution based on pattern recognition, but it is very easy to write down an algorithm to produce a solution.

Conversely, if you look at puzzles that are very perceptual, it is very challenging to write solution programs for them. The program would have to formalize a lot of perceptual concepts that make intuitive sense for us but are actually very difficult to express in program form. Imagine trying to recognize the letter A, for instance. If you are just looking at letters, you can immediately see it. If you have some pretraining visual knowledge of the shape of letters, this is a trivial problem: you just recognize letters using pattern recognition. But if you try to write down an algorithm that would take a handwritten A and tell you whether it is an A, for instance, it is actually a tremendously difficult problem.

A couple of thoughts on that. First of all, on that Kevin Ellis paper—the induction-transduction paper—maybe for political reasons, it feels like they wanted to find an inductive, explicit function. In their ensemble method, they first searched for inductive functions by brute-forcing it, and then fell back to transduction. The thought occurs to me that, just because we cannot write a Python program to recognize a digit, surely such a Python program could exist. Shouldn't we be thinking about making models generate such a program? Maybe the reason they cannot is because nothing like that is in distribution.

I think the reason why it is difficult is that we are talking about an input space that is structurally continuous and where decision boundaries are fuzzy, basically. You are talking about a problem that is fundamentally a pattern-recognition problem, and neural networks are intrinsically a good data structure for approaching this type of problem, whereas discrete symbolic programs are not. I really think it depends on what problem you are looking at. There are problems where vector spaces are the right data structure and problems where symbolic, discrete programs are the right data structure.

I guess an interesting thought experiment is that there must exist a Python program that does what an MNIST model does.

Sure.

What is the simplest possible representation of that program? Would it still be ridiculously complicated?

I think the simplest representation of that program would look a lot like what the current neural network is doing, to be honest.

Right. Yeah, I think that is the clincher, isn't it? There is no way of decomposing it into a much simpler version.

Yeah, I think it really depends on the nature of the problem. For some problems, program synthesis is just a bad idea, and perceptual problems are certainly in this category. Conversely, for some problems, trying to use a pattern-recognition machine is just a bad idea—problems that are algorithmic in nature.

How do you think we could effectively combine induction and transduction methods?

Well, the way Kevin Ellis and his team are doing it in their paper is that they start with induction and fall back on transduction when it does not work. I think that is a very smart strategy because induction is formally verifiable. You can try to run your candidate program on the demonstration pairs that you have access to and see, first of all, whether it runs and whether it gets you the right result. If it does, you can have a fairly high degree of confidence that it is going to generalize.

Whereas, when you are doing transduction, it is more like guessing where the answer might be, and you do not really have any way to verify that it is the right guess. One thing you can do is increase the sample size. You can make many independent guesses and then look at which answers come up most often. But you are basically making the assumption that wrong guesses are all going to be wrong for different reasons, so you will end up with different wrong answers, whereas correct guesses are all going to be correct for the right reasons. The correct answer will show up more often. But you really have no way to make sure, with 100% certainty, that your guesses are correct. So it is much smarter to start with induction, because then you can have a high degree of confidence that the solution you have is the right solution, and then fall back when it simply does not work. Basically, induction is the method you should prefer, and you should only use transduction if it is not working.

Should we think of them as being completely different? Hypothetically, if you used a shared model for doing induction and transduction, do you think there could be some crossover between them?

Absolutely. In fact, this is something that some people in the competition have tried. They are using, as you said, the so-called Omni-ARC approach, where the team is using the same model to solve a range of different ARC-related tasks. That includes writing down the program, interpreting programs, doing transduction, generating more input grids, and so on. All these different tasks are handled by a single model, and that does lead to learning better representations for the concepts that you find in ARC.

Can you give me some more intuition on that? In a sense, you can get the network to think about the symbolic version at the same time as the solution space.

The basic intuition is that, if you look at the same problem from different angles, you are more likely to come up with the true shape of the problem. This is especially true if your data structure of choice is a neural network, because neural networks have a tendency to latch onto noisy statistical regularities. If you are only targeting one problem and using only one input modality, you are much more likely to overfit to elements of noise within that problem.

But if you are forcing the same representations to work across many different views of the problem, it acts as a regularization mechanism. First of all, you get better information about the problem because there is some knowledge transfer and information exchange between things like trying to predict the output grid and trying to generate more input grids, for instance. But it also acts as a regularization mechanism, where the noise that you might be learning with one of the modalities is countered by what you are learning with another modality.

Another approach from Clément Bonnet, I think his name is, is searching the latent space better. Not necessarily brute-forcing, but actually searching the latent space for quite a while before you present a prediction. What do you think about that?

Yeah, this is actually one of my favorite papers submitted in the competition. It is a very original idea: you learn a latent space of programs, and then, at test time, you need some adaptation to the problem at hand. You need some recombination of your knowledge. While some people use test-time training and some people use search, what Clément Bonnet and his colleague Mark Farnay [?] are doing is very original.

They are learning this latent space of programs, and then, at test time, they are doing gradient descent in latent program space. They are essentially moving around latent program space to find the point that best matches the task. I think this is a great idea. There are many ways, I think, to improve the idea, but it is a very original take on test-time adaptation that is neither search nor fine-tuning. So I like it a lot.

Would you still call that process thinking? Is thinking a System 2 process, or would you call that thinking as well?

It is a form of test-time search, except it is not discrete search. It is based on gradient descent. So, sure, I do not see why you could not implement some form of System 2 processing with that.

Yeah. I wonder whether that breaks the analogy with human thinking—that it is doing perceptual deliberation.

It's quite an interesting category.

It is doing deliberation in latent space, yes.

Yeah.

I think one way to improve the process is that you could also try to decode your latent programs back into a symbolic, discrete form, and then you can start doing local discrete search around the decoded programs. The benefit of that approach is that you would have the ability to actually run the programs and verify whether they work.

As long as you stay in latent space, even if you're doing gradient-descent-guided search within latent space to find the best possible point that represents a target program, you are very much limited to guessing. You have no way to assert that what the latent space is telling you matches the reality on the ground. So the ability to decode back into real program space and run these programs would be a very good addition to the system.

Are there any potential issues with that approach? I guess it helps if the latent structure is quite homogeneous and the modes are easy to find. Could it be improved?

What do you mean by that exactly?

He does gradient search over the latent space—

Yeah.

—and finds some optimal position, and then does the inference from there. But wouldn't that work very well if it were quite a convex space, but not so well if it were a very heterogeneous space?

Of course. In order to be able to do gradient descent, you need a relatively smooth surface. But I think that's why they're using a VAE. I'm not sure if they've tried just directly learning a program embedding space where one point is one program.

As it turns out, this is not what they are doing. They are using a VAE, and the reason why is because, when you're using a VAE, you are learning much more structured, much smoother latent spaces. I think this is key to making test-time gradient descent work.

So, other unexplored avenues of ARC. Maybe a better way to ask this is: if you yourself spent a year working on ARC, what would you do?

I would be doing deep-learning-guided program synthesis. I think the way people are doing deep-learning-guided program synthesis today is wrong.

Everyone is leveraging LLMs, which of course makes sense because they are these very powerful tools. They contain a lot of useful knowledge, and they can be applied to any problem. We've invested billions and billions of dollars into creating these tools, so not using them would feel like you're missing out on a lot of power.

But I think it is not the right approach to treat program synthesis as just token-by-token code generation. I think the right way to think about a program is as a graph of operators, and program search—program synthesis—is basically a tree-search process. I think you're better off trying to use deep learning models, and in particular LLMs, to guide that tree-search process. This is not something that many people are trying today, but I think this would be closer to the right approach.

Another thing that people are not doing today but should be is that, if you look at the way humans solve ARC puzzles, they are not trying many different solution programs in their mind. They're only trying a few. I think humans have the capability to first describe a model with their thinking and use this model, basically describing a grid, for instance, in terms of the objects it contains, their properties, and their relationships with other objects, with a focus on causal relationships in particular.

You can use these descriptive models to constrain the search space when you're finally looking at input-to-output programs. That's the reason why we only need to consider a handful of programs before finding the correct one. In that sense, it might be possible to do enough modeling of the task to almost entirely remove the need for search.

It's so interesting what you just said. We should meditate on that just for a second. First of all, loads of people I've interviewed this week, especially in the neuroevolution space under Geoff Clune, for example, are latching onto LLMs because they say we need to have a measure of interestingness or novelty. And LLMs, because they're trained on all of the data in the world, capture our instincts really, really well—our intuition and so on. So they're a great way of generating programs.

But you said, though, that it's not such a good idea just to generate the program. The next step of evolution is guiding the search, and I think we're starting to see this enlightenment in the use of LLMs in many commercial bits of software.

For example, the original use of an LLM in an app was that you just had a chatbot and stuck it in there. Now things like Cursor, for example, are exposing a low-level API and using tool use and so on. The LLM is actually guiding the low-level API interactions in the app. So you're advocating for a similar evolution here, where the LLM actually guides the discrete search process rather than just generating code.

That's right. The idea is that, by creating your program via this iterative discrete search process, you actually have the ability to make targeted modifications to your program graph that will be significantly harder to make, I think, if you just treat the program as a sequence of tokens.

Also, you can change the nature of the space in which you are making additional decisions, right? Graphs are just the natural data structure to represent programs. Programs are not sequences of tokens.

Another thing we're seeing is an intermediate solution. Kevin Ellis did this thing called remixing, where you have 100 handcrafted solutions, and for every single ARC task, you have a generator that can generate new instances so that they can generate sprites and layouts and so on.

Yes.

Then they did this kind of expansion where they used retrieval-augmented generation, and it was doing an implicit form of library learning. It was mixing all the solutions together. Is that an intermediate to what you're talking about?

Not quite. I think that's a separate avenue. The idea is that, in order to get LLMs to perform well on a dataset, you need to expose them to as dense a sampling of ARC space as possible. And of course, there aren't a lot of ARC tasks available, so you have to make new ones.

A very easy way to make new ones is to leverage an LLM to extract the programmatic concepts found in, for instance, the training set, and then remix them into new tasks. But at the end of the day, this leads you to severe overfitting, and this is exactly what you are seeing with your solution.

Because there's been a bit of an interesting evolution where even Kevin, in DreamCoder, had an explicit concept of awake and asleep—dreaming. And in his newer work on this, learning by example pairs, he's kind of made that dream-sleep process implicit. It just feels that maybe we could achieve some of the same stuff by coming up with a proxy or implicit version of it.

Mm-hmm. Yeah, yeah.

Any other avenues for ARC that you're interested in? By the way, I think you spoke with George from Symbolica, and he had this kind of program-verification approach that he said he discussed with you.

Yes. What we discussed is what he described to me: basically, this idea of using a symbolic process to turn a problem definition—a task—into a deep learning architecture, and then training that deep learning architecture. I think that's a very, very original approach. I don't think there's anyone else, to the best of my knowledge, working on something similar.

I'm very curious about what he's going to be doing with it. It sounds fascinating.

Yeah, absolutely. Do you think, just talking about benchmarks in general, we need to incorporate the compute budget into the benchmark?

Yes, absolutely. I think this is going to be a very pressing need in the future, and in particular next year for the publicly available ARC.

It's always possible, with test-time compute—like test-time search, test-time training, and so on—to buy higher performance at the cost of more compute. Typically, you're going to see a logarithmic relationship at test time between compute and performance. It kind of means that, if you want an apples-to-apples comparison between two systems, you have to look at the compute budgets.

You can only compare systems that are using the same amount of compute. For instance, if you look at the o1 model from OpenAI, you cannot really attribute to it a fixed score on ARC-AGI unless you're also kind of limiting yourself to a certain amount of compute.

It's always possible to logarithmically improve your performance by just throwing more compute at the problem. Of course, this is true for o1, but even before that, it was also true for brute-force program search systems. Assuming that you have the right DSL, extremely crude, basic brute-force program enumeration can solve ARC at a human level.

It would just take hundreds of millions of dollars of compute to crack the entire dataset that way. So it's an extremely inefficient and extremely stupid idea, to be honest. But in theory, it's possible, right?

Philosophically speaking, do you think there's always a commensurate relationship between the amount of compute that is taken to do a task? What I mean by that is, when we use language and cognitive tools, we might not think we're using compute, but the universe presumably expended a lot of compute in order for those things to emerge. So in some sense, is it really possible to compress the amount of compute that we use?

I think so. I think humans are tremendously compute-efficient, and you see this in the fact that, let's say you're solving ARC tasks, for instance. You can solve the entire private test set and only expend a few calories, basically. You could say, “Okay, but it's because we're just using extremely little energy per operation that our brain does.”

But it's actually not true at all. If you're comparing transistors and neurons, for instance, you find that neurons are tremendously more energy-hungry than transistors. It just so happens that we're managing to solve extraordinarily hard problems using a comparatively small amount of neural operations. We're tremendously energy-efficient compared to current AI, and we're going to have AGI when we get to the same level of energy efficiency.

What's your opinion on using a programming language like Python, a Turing-complete language, versus using a DSL in these approaches?

I think using a DSL, like for ARC, for instance, is fundamentally limiting. No matter what you do, no matter what base language you're using, you should be able to learn the functions that you're applying from the data that you have. In fact, you should be able to do this as a lifelong process.

Every time you find a new task and you're solving it, in the process, you're going to be coming up with useful abstractions, maybe abstractions that relate to problems you've seen in the past. So you're going to want to turn that into reusable functions, reusable building blocks, and store them, so that the next time you come across a similar problem, you can reapply the same building blocks and save compute—solve an equally difficult problem in fewer steps.

No matter what you do, you want to learn the language that you're going to be using. Of course, that could mean learning the DSL. That could also mean using something like Python, but within it, increasingly writing higher-order functions and classes and other reusable building blocks to enrich your language.

I wondered if you are softening your position at all. You tweeted, “It's highly plausible that fuzzy pattern matching, when sufficiently iterated many times, can asymptotically turn into reasoning, and it's even possible that humans basically do it in that way, but it doesn't mean it's the optimal way to do reasoning.” Is that a shift in your position?

Compared to what position? I don't think it's a shift.

I suppose it's saying that, first of all, it's possible that we think in this way.

I think we do, yes.

Oh, interesting, because I would have thought that, because of the Spelke view on things, you would have thought that we do this high-level reasoning and we don't do the fuzzy matching.

8. Consciousness Enforces Consistency

I think the fundamental cognitive units in the human brain are actually fuzzy pattern recognition. That's the core thing that you do. When you're doing something that's more akin to reasoning or planning—when you're doing basically System 2 processing, this sort of slow, logical, step-by-step processing—what you're really doing is iteratively applying your intuition, but in a structured form.

By the way, this is exactly what deep-learning-guided program synthesis is about, which is the approach I've been advocating for since 2017. What are you doing when you are doing deep-learning-guided program synthesis? You are building a program, so basically a graph of operators, but you're building it step by step. At each step, when you're choosing what to edit in your graph, where to add something, or what branching decision to make, you're applying your intuition. You're applying a guess provided by a deep learning model, right?

So you're iteratively guessing to create this highly structured, symbolic, discrete artifact: this program. When you're running this program, that is System 2. I think this is basically the way we do System 2 as humans: we are iteratively guessing, iteratively applying fuzzy pattern recognition to construct an artifact that is, in fact, symbolic in nature.

Let's say you're playing chess, for instance. When you're calculating in your mind, you're unfolding some moves step by step, but you're only going to be doing it for a few of the moves that are on the board. How do you select which moves to look at? You're applying pattern recognition. When you're simulating one move into the future, you're not going to be simulating the entire board. You're going to be focusing on some areas. So again, that's pattern recognition.

Sometimes, by the way, any sort of pattern recognition is basically a guess at heart, so it might be wrong in some way. In chess, you're calculating, and some of your intuition about future states of the board is wrong. Then you play the move and realize, “Oh, oops,” right?

I think this is basically how humans implement System 2. So this is not a shift in my position. I've been thinking about these ideas for quite a while. In fact, this is the basis for my current favorite theory for how to interpret consciousness: this idea that, in order for something like System 2 to arise from iterated fuzzy pattern recognition, that iteration sequence needs to be highly self-consistent.

Everything you add, you need to double-check that it matches what came before it, you know?

Yes.

If you're just iteratively pattern-matching with no guardrails whatsoever, you're basically hallucinating; you're dreaming. This is exactly what happens when you're in a dream state, by the way. You're just repeatedly intuiting what comes next, but with no regard whatsoever for consistency, for self-consistency with the past.

I think this is the reason why any sort of deliberate, logical processing in the brain needs to involve awareness, needs to involve consciousness. Consciousness is this sort of self-consistency check. It's the process that forces the next iteration of your intuition, of this pattern-recognition process, to be consistent with everything that came before it, right?

The only way to achieve this consistency is via these sorts of back-and-forth loops that are bringing the past into the present and bringing your prediction of the future into the present. So you have this sort of nexus point in the present, this thing you're focusing on, and that nexus is basically your consciousness.

Consciousness is the process that forces iterated pattern recognition to turn into something that's actually reasoning, that's actually self-consistent.

So I think the issue was that you haven't changed your position. It's just a case of people understanding what your position is. I think people dichotomize System 2. Symbolic people think it's all this discrete world, and then the alternative, the connectionist approach, is that it's all fuzzy matching.

I've never been in the purely symbolic camp. If you go back to my earliest writing about why we need program synthesis, I was saying we need deep-learning-guided program synthesis. We need a merger of intuition and pattern recognition together with discrete, step-by-step reasoning and search into one single data structure.

I've said very repeatedly, for the past 8 years or so, that human cognition really is a mixture of intuition and reasoning, and that you're not going to get very far with only one of them. You need the continuous kind of abstraction that's provided by vector spaces and deep learning models in general, and the more discrete, symbolic kind of abstraction provided by graphs and discrete search.

So why do your detractors see you as a symbolist when you're clearly not?

I'm not sure. I've been into deep learning for a very long time, since basically 2013.

I started evangelizing deep learning very heavily around 2014, and back then the field was pretty small. Especially with Keras, I think I’ve done quite a bit to popularize deep learning and make it accessible to as many people as possible. I’ve always been a deep learning guy, right? And when I started thinking about the limitations of deep learning, I was not thinking in terms of replacing deep learning with something completely different. I was thinking of augmenting deep learning with symbolic elements.

So you commented as well in this tweet that you were just talking to me about consciousness. You suggested that all System 2 processing involves consciousness.

Yes.

Explain more what you mean by that.

Any sort of explicit, step-by-step reasoning needs to involve awareness. And conversely, if there’s any cognitive process that you’re running unconsciously, it will not have this strong self-consistency guarantee. It will be more like a dream, a hallucination.

It’s basically the idea that if you’re just iteratively guessing, unless you have this strong self-consistency guarantee, you will end up drifting and diverging. Consciousness is the self-consistency guardrail, basically. This is why you cannot have a System 2 without consciousness.

What is your definition of reasoning?

I don’t really have a single definition of reasoning. I think it’s a pretty loaded term, and you can mean many different things by that. But there are at least 2 ways in which I see the term being used, and they’re actually pretty different.

For instance, if you’re just memorizing a program and then applying that program, you could say that’s a form of reasoning. Let’s say in school you’re learning the algorithm for multiplying numbers. When you’re tested, you’re actually applying the algorithm. Is that reasoning? I think yes, that’s one form of reasoning.

It’s the kind of reasoning that LLMs and deep learning models in particular are very good at. You’re memorizing a pattern, and at test time, you’re fetching the pattern and reapplying it. But another form of reasoning is when you’re faced with something you’ve never seen before, and you have to recombine the current building blocks you have access to—your knowledge and so on—into a brand-new model and do so on the fly.

That is also reasoning, but it’s a very, very different kind of reasoning, and it underlies very different kinds of capabilities. I think the important question about deep learning models and LLMs in particular is not whether they can reason. There’s always some sense in which they are doing reasoning. The more important question is whether they can adapt to novelty, because there are many different systems that could just memorize programs provided by humans and then reapply them. What’s more interesting is whether they can come up with their own programs, their own abstractions, on the fly.

And what would it mean for a system to come up with its own abstractions?

To start coming up with abstractions, first of all, you need to be solving novel problems. Solving a novel problem means that you’re starting from some base of knowledge, some building blocks. Then you’re faced with a new task. You’re recombining them into a model of the task, applying this model, and it works.

In the process, as you solve many problems, you’ll start noticing that some patterns of recombination of the building blocks that you had happen often, right? When you start noticing this, it means you can take them and abstract them, refactor them into a more reusable form. Then you can add this reusable form back to the set of building blocks that you have access to, right?

Next time you encounter a similar problem, you’re going to be able to solve it in fewer steps, expending less energy, because you have access to this higher-level abstraction that fits the problem.

Is there a way of measuring the strength of reasoning?

Again, you would need to start by defining precisely what you mean by reasoning. I think you can define, for instance, generalization power, which is basically the amount of novelty that you can adapt to.

Yeah. It’s quite interesting because I suppose you define it in terms of performance, as in, how good is my model? Another way of describing it is to imagine that we think of reasoning purely as traversing the deductive closure: just composing together knowledge we already have in new configurations, and then making that leap in solution space because we found a new model that works really, really well.

Is there an intrinsic way of measuring the type of model rather than its generalization power?

No. I think you really have to observe what the model does. You cannot just inspect the model and tell how strong it is at reasoning.

So there’s no intrinsic form of “This is good reasoning”?

Given 2 models of a problem, for instance, which model is better? Can you just look at them and tell which one is better? I think it’s very much goal-dependent, right?

You cannot really evaluate a model, like a simulation of a thing, for instance, if you don’t have something that you want to do with it. But if you do have a goal, then you just look at the causal factors required to achieve that goal, and the best model is probably going to be the simplest model that retains these causal factors.

Francois, what are you doing next?

9. A New Research Lab

I just left Google a few weeks ago, so I’m starting a new company, a new research lab with a friend. I can’t really share much for the time being, but we’re going to be tackling program synthesis, and in particular, deep learning-guided program synthesis. We’re currently building the team.

Amazing. Are you looking for people to join?

Yes.

Tell me more. What are you allowed to tell us?

What I can tell you is that I’ve been talking about some of these questions, like the best way to get to AGI by deep learning-guided program synthesis, methods surrounding ARC-AGI, and what the next benchmark after ARC-AGI might be.

I’ve been thinking about this question on the side while at Google, where my full-time job was developing Keras, right? And now I feel like the time has come to focus full-time on the research question, so make the research not a side project, but the main thing.

And is the focus creating the new benchmarks or beating the benchmark?

I think both. I’m a pretty strong believer in the idea that you need to co-evolve the solution together with the problem. That was actually the motivation for creating ARC-AGI in the first place: to have the right challenge that forces you to focus on the right questions, on the main bottlenecks to achieving strong generalization with AI.

I don’t think ARC-AGI is the last benchmark. Of course, there’s going to be a v2 of ARC-AGI, but that also isn’t going to be the last benchmark. I think we’re always going to need new benchmarks, exploring new things that are hard for AI and easy for humans.

Is it cheating in any way if you do work on your own benchmark?

I don’t think so. The benchmark is meant as a tool for research. Again, something like ARC-AGI, for instance, isn’t really meant as a binary indicator telling you, “Oh, do we have AGI or not?” It’s really meant as a research tool.

It’s a challenge that forces you to work on the right questions, directs your attention to the right problems, and helps you make progress. So you could say that ARC-AGI is basically a compass towards AGI. It’s not like a test for AGI.

There’s a spectrum of possible solutions to ARC. For example, because you know what’s in the private test set, you could just put the answers in directly, and then the—

Of course, but I’m not going to be entering ARC Prize in any way. Obviously not. I’m running ARC Prize anyway, so why would I enter it?

Right.

But on that spectrum of generalization, there’s the moonshot, which is going for extreme generalization. Or there’s one notch below that. What are you going for?

Yes.

What are you going for?

I would like to build AGI. I would like to build something with human-level capability.

And what would that mean? What kind of things do you think you could achieve with that kind of AGI?

The most obvious thing is solving programming, right? If you solve AGI, then you can just describe what you want to a computer, and the computer will build it for you. If it’s really AGI, then it will scale to the same level of complexity, the same level of codebase complexity, that you can do with a human software engineer. And it’s probably not going to stop there.

What’s the role of humans in software engineering when we start to get there?

Well, we’ll see. I think we’re going to start creating entirely new tools, entirely new interfaces, to work with this technology when it’s ready. We are still pretty far from it. We’re talking about something that doesn’t quite exist yet. I don’t think frontier models—not even o1—are quite at that level.

Right now, programmers are very technical. Do you think that programming might be democratized in the future?

I think so, yes. I think in the future, anyone should be able to basically develop their own automation processes based on their own domain-expert knowledge of the problems they’re facing. Everybody should be able to program—not really in the sense of writing down code, but describing to the computer what they want to automate and how they want to automate it, and the computer will just do it.

A big thing in software is tackling complexity at different scales. It’s certainly what you’ve been doing for your entire career. Do you think we’ll always have this problem, that we’ll always be on the boundary of this incredible complexity?

What do you mean by that?

Even if we democratize one or two steps up the hierarchy, wouldn’t we still always just build software which is really complicated?

Yeah, quite possibly, but the idea is that we’re going to be able to offload that complexity to an external complexity-processing AI.

So we will transition into a future where we no longer understand the code that’s being run in any way?

Absolutely. I think to a large extent, this is already true. If you look at any sizable code base, there is no single software engineer that actually understands it all. We’re always limited due to a fragmented understanding of what we’re doing, which is fine as long as we have a good grasp on the high-level goals and constraints of the system.

So where should the source of agency be there? You’re describing a blind-man-and-the-elephant-type challenge, where loads and loads of developers have their own perspective on a very small part of the system.

Sure.

But when we have the AGI version, how could that change?

Broadly speaking, I think programming from input-output pairs will be a widespread programming paradigm in the future, and that will be accessible to anyone because you don’t need to write any code. You’re just specifying what you want the program to do, and then the computer programs itself.

If there’s any ambiguity in what you meant—and there will always be ambiguity, especially if the instructions are provided by a non-technical user—you don’t have to worry about it because the computer will ask you to clarify. It’ll tell you, “Okay, so I created basically the most plausible program given what you told me, but there’s some ambiguity here and there. What about this input? Currently, I have this output. Does that look right? Do you want to change it?”

As you change it iteratively, you’re creating this correct program in collaboration with the computer.

So these future systems will have program synthesis as a core component, an explicit component.

Yes.

But how will humans interface with it? Are we still going to describe things with natural language, gestures, images, and things like that?

It could be natural language. It could also just be drawing interface elements on your screen. You could always try to generate a high-level representation of the program that’s being generated, at a level where it can be visualized and understood by a non-technical user. It could show, for instance, a kind of data-flow graph and ask the user for input about it.

Very cool. One of the characteristics of LLMs at hyperscale is that, in a sense, it’s not that difficult because they just have to— it’s like a CDN. They just copy all the weights and move them all over the place. You’re describing something which is very sophisticated. It might be a little bit akin to a globally distributed database, where the skill programs move around all the different nodes and so on. Is there just a massive new type of architecture we need to build for this?

Yeah. I think we’re going to need a completely new type of architecture to implement lifelong distributed learning, where you have many instances of the same AI solving many different problems for different people in parallel and looking for commonalities between the problems and commonalities between the solutions.

Anytime they find sufficient commonalities, they just abstract these commonalities into a new building block, which goes back into the system and makes the system more capable and more intelligent.

I think I’ve got it now. So what you’re building is a globally distributed ARC, on the basis that we find a good solution to ARC.

Well, I can’t really tell you exactly what we’re building, but it’s going to be cool.

Yeah, it sounds pretty cool. How do you think that folks like OpenAI are going to start incorporating not only test-time inference, but some of your ideas, realistically, into their system?

How might frontier models incorporate program synthesis, for instance? I think something like o1 is already doing precisely that. When you look at what o1 is probably doing, it is writing its own natural-language program describing what it’s supposed to be doing, and it is executing this program itself. The way it’s writing this program is via a very sophisticated search process.

So this is already program synthesis in natural-language space. There are other ways you could leverage program synthesis. You could do program synthesis in latent space, potentially, kind of like what Clement Bonnefil and friends are doing. You could also just be generating actual programs. Why use natural language? Sometimes you might want to use an actual programming language.

I think we are definitely seeing a shift towards leveraging more and more test-time compute, and that is going to accelerate. It’s a fundamental trend.

On the transductive active fine-tuning, though, that’s a little bit more architecturally difficult, isn’t it? Because my model is always being fine-tuned. I can imagine they might build something a bit like Docker, where there’s the base layer, and then there’s my fine-tuning layer and another fine-tuning layer.

Sure.

And it’s very fragmented.

The difficulty in applying test-time training in actual frontier models is not so much the infrastructure. It’s definitely true that current serving infrastructure is absolutely not set up for per-task fine-tuning, but you could re-engineer it for that.

The main bottleneck is actually the task format. You can only do test-time training if you have pretty clear inputs and targets. You basically need input-output pairs, right? For ARC, obviously, you have that, but for most problems, you don’t.

Very cool. What’s your theory on how o1 works?

Well, we can only speculate. I’m not sure how it really works. But what seems to be happening is that it is running a search process in the space of possible chains of thought, trying to evaluate which branches in the tree work better, potentially backtracking and editing if the current branch isn’t working out.

It ends up with this very long and sophisticated, plausibly near-optimal chain of thought, which represents basically a natural-language program describing what the model should be doing. In the process of creating this program, the model is adapting to novelty.

So I think something like o1 is a genuine breakthrough in terms of the generalization power that you can achieve with these systems. We are far beyond the classical deep-learning paradigm.

One school of thought, which I think you agree with, is that there’s some kind of active controller at inference time, so it’s actually doing multiple trajectories in an isolated way and then—

Yes. It’s doing search.

Okay. Some people think that it does process supervision and whatnot at training time, but at inference time, it’s all just one forward pass.

No. That is certainly not plausible because of the amount of compute that’s being spent at test time. It is very clearly doing search at test time.

Very interesting.

I think it is trained at training time to reproduce the best available chain of thought for the current problem, kind of like AlphaZero-style training, basically.

But it’s also doing search at test time in chain-of-thought space. This is kind of obvious. The telling sign is just the compute it’s expending—the amount of tokens and the latency.

Are there any other telltale signs that this kind of thing is happening? For example, it might have explored a particular area, and then after the consolidation, that is now gone. So when you talk to the model, it’s almost like it’s forgotten part of its thinking.

Yeah. Honestly, this is a little bit too specific. I don’t have any insider information about what o1 is doing, so I can only speculate.

Okay. So people like Noam Brown are really bullish on this new scaling law for test time, and certainly I love o1 Pro. I think it’s really, really good. It’s qualitatively a big improvement. What do you think there?

Sure. So the test-time scaling law is basically this observation that if you expend more compute, if you search further, you see a corresponding improvement in accuracy. That relationship is logarithmic, by the way, right? So accuracy improves logarithmically with compute. While this is not really new, anytime you do test-time search, you will see this relationship.

If you’re doing brute-force program search, for instance, you will find that your ability to solve a problem improves logarithmically with the amount of compute. If you have more compute, you can just search further into the space of possible programs, and logarithmically you find more solutions. So anytime you do any kind of test-time search, you will see this relationship.

What is your current go-to model, and what do you use it for?

For the most part, I’m using Gemini Advanced, and I’ve actually just started using the new Gemini Flash, the latest one.

Me too.

I’m paying for Gemini Advanced. I’m also using Claude 3.5 Sonnet. I think it’s very good for programming. So these are the two I’m using.

What’s your programming workflow with LLMs?

I don’t use LLMs all that much when I’m programming. But typically, if I’m currently facing a problem that I feel might be a good fit for LLMs, I will just open my browser and prompt the LLM. I’ll ask it for a function that does X, Y, Z. It usually doesn’t work on the first try, but after a little bit of debugging and nudging, I think it’s a big time saver.

What kind of failure modes do you see when you’re programming?

With LLMs, you mean? Well, the failure modes are different based on the model that you’re using, right? I think in general, Claude 3.5 Sonnet is the best one.

Yeah.

Sometimes you might have code that’s there for absolutely no reason, like variables that are not used or assumptions that are being made by the code that are not verified by the data that comes in. So it’s pretty clear that the code is generated in terms of statistical likelihood. There’s no effort to actually make it self-consistent, make it correct, try to execute it beforehand, and so on.

I think there’s actually a lot of room for improvement there. You could imagine LLM-based software developer assistants in the future that actually do all these things as you prompt them—that actually write you the code, but then try to debug it themselves before actually showing it to you.

What’s your opinion on LLM agent systems, which are all the rage at the moment?

Right. Agents have been all the rage for quite a while now. People started talking about agents being the future almost 2 years ago—like a year and a half ago. So far, agents have not really taken off.

The fundamental problem here is that LLMs are not quite reliable. If you look at one forward pass of an LLM, you can think of the LLM as a guessing machine, right? The guesses it makes are much better than random, obviously. They’re very useful guesses. They’re directionally accurate, but they have some probability of being wrong.

When you look at an agentic workflow, you are chaining many of these guesses. The likelihood that you will end up not where you would like to be gets dramatically higher as you chain more guesses like this. So this is the big bottleneck. Agents are just not reliable. They just don’t have a sufficient level of autonomy.

People say that with better models this will get fixed. I think it’s an empirical question. I’m waiting to see when agentic workflows actually start working. I don’t think we’re there today.

I’ve softened my position a tiny bit on this. I agree that, because of the ambiguity problem, they are misdirected, and when you chain this, they’re very misdirected. But there is something to be said for just having more computation at hand.

I interviewed the people who did The AI Scientist paper, and certainly if you take Claude 3.5 Sonnet and say, “Generate me an entire paper,” it will be banality beyond belief. It’ll just be—

Sure.

…a sketch. What they did was decompose it into lots of agent workers that, using the Google Maps analogy, just kind of zoom in and zoom in and zoom in many times over. It produced dramatically better results.

Yeah. This makes sense. I think this basically tracks the idea we were talking about earlier of System 2 being something like iterated System 1 with strong guardrails. The guardrails are very important.

In this case, the superstructure is provided by the human programmer. The human programmer is breaking down the problem into the right subproblems and orchestrating the whole thing in the right way. Then each subproblem can actually be solved by guessing and producing a good enough guess.

Amazing. When’s ARC-AGI-2 coming out?

Early next year, probably. We are currently finalizing human testing. As I mentioned, every puzzle is going to be solved by a bunch of humans, so we know it’s solvable, and we have some data to be able to tell how difficult it is for the average human.

The goal is going to be to have 3 sets that are difficulty-calibrated. So if you get a score on the public eval, and you’re not overfit to that dataset, you can be very confident that you’re going to get a very similar kind of score on the other 2 sets.

When you tested with humans, you wrote about this in The Measure of Intelligence: one school of thought in intelligence is that there’s this g factor, and another school of thought is that it’s very specialized. Did your experiments reveal that a group of humans generally performed quite well across all of the tasks, or did you see huge specializations?

No, it’s absolutely the case that there are people who are just more intelligent, and they are just better at solving hard tasks. You do see that, yeah, in the human testing data.

What about on the long tail? Do you see specialization in the types of tasks, or is it fairly flat?

Yeah, I think it’s pretty flat. Either you’re good at it or not.

Interesting. Francois Chollet, it’s been an honor to have you on the show. Thank you so much.

Thanks so much for having me. It’s been great.

Amazing.