Test-Time Adaptation: the key to reasoning with DL

This time. Fine tuning is a new paradigm to deep learning, right? It's something completely outside of the deep learning paradigm. What's the most efficient way to learn at test time? That's a very interesting question. How did you encode the problems? The whole point of ARC is they're going to trick you. Whatever specialization you put in the input, you can create a problem that's adversarial to that tokenization scheme or special representation scheme for arc problems because it's so arbitrary. The problem is so different, right? It's a very new problem. But the really cool thing that you get out of this is you train this model to be a very weak, dumb contextualizar maybe. Right. But that's what you're tuning for. Transformers, even in a trivial sense, cannot do counting or copying. They just can't do it. You count up to 100 and you just say, can you count these numbers up? And it just fails abysmally. What we do, which is really interesting, we prompt everything into the forward pass all at once. We're looking for people that are interested in changing the paradigm going into test time. Compute that, like working in small and nimble teams, tackling really big problems. Tufa Labs is a very, very exciting new research lab that's just started in Zurich. They are looking for amazing ML engineers to join their team. It's a very small team. If that sounds like you, go to http://TufaLabs.ai. You might remember when we interviewed Mohamed last year, he's part of MindsAI along with Jack Cole and Michael Hodel, a couple of legends, of course. They were acquired by Tufa Labs. They got the highest score on the ARC challenge, about 58%, and they've just released their paper where they spill the beans on how they did it. It's called don't throw the Baby out with the bathwater. How and why deep learning for AAC. Now, the prevailing view has been that neural networks lack the necessary capabilities for abstract reasoning tasks, and at least to some extent, they proved that wrong. Consider this particular ARC puzzle the task requires inferring complex transformational rules from minimal examples. A challenge where vanilla llms like GPT-4 get no better than around 10%. Now, Mohamed thinks that ARC puzzles are fundamentally perceptual reasoning problems. They incorporated the optimizer directly into the evaluation process, which allowed the overall approach to develop new abstractions during test time, and their methodology introduces two principal techniques. Number one, we've spoken about this a lot on the show test time, active fine tuning, or what we've been referring to as Test Time Active or Transductive Fine tuning, where you generate synthetic training data derived from each puzzle's examples, and you fine tune the model as you go. The second approach is what they call augment inference. Reverse vote, where you apply transformations to input puzzles, generate predictions, reverse the transformation, and implement a voting mechanism to identify a consistent solution. They found that the latter improved performance by 260% with test. Time active fine tuning providing an additional 300% improvement, which is how they. Yielded the highest score on ARC. About 58%. Now, another thing they found is that the model architecture scale has a greater impact than pre-training for building new abstractions. Larger models are simply more expressive, enabling better abstractions and reasoning during inference. Now we are counting down the days until the version two of the ARC prize. We are launching it on MLST on Monday, and I hope you're excited because we've got Francois coming over and he's going to tell us all about it. Suffice to say that all of the frontier models are going to be going down to negligible performance on it, and I'm excited to show you. See you on Monday and enjoy the show.

Mo, you've been on the show before, but basically you won the ARC challenge with Jack and Michael. Technically, you didn't win because you didn't choose to share your solution, but certainly in terms of the leaderboard, you guys did the best solution. Welcome back to MLST.

Thank you so much for having me. It's always a pleasure, Tim. It's really nice seeing you here in beautiful Vancouver. I feel like you're kind of a guest to me in Canada here. It's really nice seeing you.

Thank you. Amazing, amazing. So congratulations on the incredible result. Tell me more.

Thank you so much. We've been obsessed with the ARC challenge for a very long time. Jack and I have been working on it for 2 years as part of the same team. Michael has been working on it for 2 years as well, by the way.

We've always thought that this benchmark was going to get more and more important, and this is the case now. At NeurIPS, lots of people know about the ARC challenge—lots more people know about it. There's been a lot of great popularization happening, and I'm really happy to see that.

We continued developing our methods based on the same kinds of philosophies that we had, and I'm really excited to dive deeper into those. I'm also excited to give you some of our results that we're planning to share very soon in the paper.

By the time this goes out, you probably will have released that paper, so you can probably tell us some of the headlines.

The paper is not going to be super interesting technically because we've shared our ideas before. We've shared them on MLST, and we're always sharing. If you look at the current top 10 on the ARC leaderboard, I don't know how many, but maybe close to 80% were using similar ideas—very similar ideas in certain cases.

We're always happy to share, and we've been pretty open about that. We're happy to see everyone converging on test-time fine-tuning and the voting mechanism that we found to be super useful.

Very cool. Why don't we break down the key innovations? Maybe you can't talk about all of them, but what are some of the key innovations that led to your win?

I'll preface this by saying that lots of the current methods are using very similar things. Again, I'm very happy to see that everyone has leveled up. I'd be remiss to say it's not because of the amount of sharing we were doing. I'm very happy to see that, and I'm happy to see what new innovations are going to come up now that we've all leveled up here.

I think we should start with test-time fine-tuning because it's really the big one. I like framing it in this way. There is a sense, or you could say, that test-time fine-tuning is a new paradigm in deep learning. It's something that's completely outside of the deep-learning paradigm: you're changing parameters at test time. That's not really how deep learning normally works, and that's a fair thing to say.

But there is a way to look at it in which it exactly fits the deep-learning paradigm, and that's how we arrived at it. We really see ARC as a perceptual problem. You have an interpretation problem. It's subjective in a way because you have these biases. You're looking at the problem, and you have an almost infinite set of possible transformations from inputs to outputs. It's really hard to narrow down what you should look at.

Imagine a riddle where there is a box, and inside it there is another object. There are different types of objects, and it's really hard to find the right level of representation to even start searching around for solutions. You kind of have to take it all in at once and hope that something pops up.

It's very similar to looking at an image. There is an almost infinite set of different colors and different lighting conditions, and they're all the same thing. You have to contend with this infinity before you can abstract over the image itself and say, “Okay, now there are 4 apples,” and start counting. First, you have to identify the apple across infinite lighting and infinite coloring.

That perception part is really important in images, but we think it's really important in ARC as well. You could easily imagine that, once you have the right level of abstraction, search becomes much easier. Searching for the right function is very easy if I know this is the relevant object I need to count, or I need to do something around this specific object, or I need to map one object to another.

It's very important to get the right level of perception. What's the best way that we know of to tackle a novel perceptual problem? It's the deep-learning paradigm. If you want to learn a new perceptual skill—say, classify mugs—you start with an untrained network and train it on a bunch of images of mugs.

We take that idea and apply it to ARC. The claim with ARC is that the examples are completely novel at test time. The logical thing to do is to take seriously the idea that ARC's most difficult aspect, if not the most difficult aspect, is this perception problem. Then you apply that paradigm at test time.

It's known to acquire skills really well, so you apply the whole paradigm to acquire skills at test time.

There are things that neural networks can do that we cannot write programs to do, just because they are perceptual. François Chollet distinguishes between perceptual problems and Type 2 problems, or whatever. Many of those problems are perceptual problems.

One really interesting thing, though, is that, contrary to people like Kevin Ellis, you guys are fans of solution-space prediction, so you don't create intermediate Python functions.

Is that the case?

Yeah, that's true. That's a pretty huge strategy right there. I have this intuition, and Chollet does that: there's something special about Python programs. The special thing is mostly that they have this kind of compositionality, which means they can be composed together. They can be decomposed into small parts. You can construct a library, and you can take bits together, and so on.

We intuitively feel that neural networks, for whatever reason, don't have this compositionality. Maybe we can make neural networks that do in the future, but right now they don't. So people who are really bullish about neural networks, and certainly I think you guys are into theories as well, think that they can do this kind of compositionality; they just need to be coaxed in the right way.

Yeah, that's a very poignant point. It's absolutely true that neural networks, by default, are not compositional, and you need to do a lot of work to get them to be compositional. That's not easy to do. We did do that work, and it took us a lot of time to really understand that point: by default, they're going to just learn statistics, and it's not elegant because they're not composable.

But I think—and I'll speak to this later—that, for a certain domain, you can get the biases deep enough. The biases are really important, such that you're able to tune the reasoning part really easily. In that sense, you do get compositionality, but it's not an elegant solution. Without the test-time aspect, without putting the biases deep enough, you don't get that.

So now, what do you do, and how do you do that more efficiently? These are all things that we want to explore in the future, and we're going to do that at Tufa Labs with the Python programs.

There's something, again, that's really important. You can perceive the problem. If you perceive it, you can take action in Python or in the neural space. I think that's fine as long as you see the right level of abstraction, you have that dynamism inside the model to be able to find some correlations from input to output, and then match those correlations and enforce them in the other input-to-output relationship. Then you can make sure it generalizes. After that, you can either output in Python or in a direct-output manner.

Let me press on this a little bit, because I think this is potentially the most exciting part. We have an intuition that Python programs are compositional. I interviewed Laura Ruiz the other day. She's working at Cohere, and she did this paper showing that you train influence functions on neural networks, and then you get them to do fact retrieval or reasoning tasks. You can see how many of the source documents light up, essentially, based on the task.

When doing reasoning tasks, it had a very diffuse activation in terms of the source documents. What she noticed was that it was looking at code. When calculating the slopes of lines and so on, it was looking at things on Stack Overflow, and it was looking at procedures for how to perform some reasoning. It was applying that in a very diffuse way to a specific problem.

The fascinating thing is that even though it's not explicitly generating code, you can fine-tune it on code, and code has some form of compositionality. You can coax the neural network to do an approximate form of compositionality, even though you're doing solution-space prediction. That is fascinating.

One of the things that we do for pre-training is code. I think there's another way of saying what you're saying about code. By the way, I know about that work. It's really amazing work.

With code, it's really hard to predict the next token. You have to contextualize a lot more. You have to know: What are we doing? What is the exact name of the variable? What is the exact relevant variable here? With language, that's not the case. With language, you can shortcut. You can cheat easily, right? You can use an inaccurate word, and it'll probably be fine. With code, you have to be very precise, so you have to be very contextual.

I think that's another way of looking at the influence of code. Code pre-training has been shown to improve reasoning across many domains in a few new papers now, so this has been reinforced. But, yeah, training on code is really interesting.

I also want to go to this idea of tuning the reasoning. You mentioned that while the neural networks are solving this reasoning task, they're looking at code, right? I think framing the problem also matters in a very important way here.

First of all, we established that test-time training gives you the generalization ability. So what do you have to do? What's the best way to learn at test time? That's a very interesting question. What we do is prompt everything in context.

We're just speaking about contextualization ability, and that's a really important thing for us to maximize for ARC. If you're prompting things into ARC, in the ARC format, you need the model to be as dynamic as possible. You have your inputs and outputs, inputs and outputs, and now you have this test input. The problem is so different; it's a very new problem. So contextualization and the ability of the model to be steered are very important.

What we do, which is really interesting, is prompt everything into the forward pass all at once: input and output, input and output, input, and then the test input—the new input—all in the forward pass at once. It's one way to look at it as a measure of the novel contextualization ability of the transformer.

The really cool thing that you get out of this is that you train this model to be a very weak, dumb contextualizer, maybe. The modeling ability in the forward pass is not that good, but that's what you're tuning for. It's kind of like a meta-model, in a sense. That's what you're pre-training for.

The model is going to learn reasoning patterns. It's not going to learn exact transformations, like in Clem's work. In Clem's work, the function learns just the transformation. But our model, by putting everything in the forward pass in pre-training and training it over many different ARC riddles, is being told, “The thing to learn here is this meta-task of looking at the context and then generalizing from it, or doing your best at modeling from it.”

So you have this weak meta-model, and tuning that meta-model is a much easier task. The thing is, you have a predictive model that's going to be kind of wrong. You can just tune tiny pieces of it to get the reasoning to click. That's really key.

It's work that's not super present in the literature. I didn't survey it very well, but there was one interesting paper, the MLC paper by Brandon Lake in Nature, which was looking at reasoning. They don't do test-time training or anything like that, but implicitly, there's a way to look at it where it's exactly this. It's exactly doing what we're doing.

For that paper, they have this input-output test that they want to learn. They retrieve similar things, put them in context, and then put in the new thing. That's their forward pass, and then they train over that, which helps them do much better. They're kind of training over that.

That's kind of test-time tuning if you take that whole paradigm and put it at test time. We train over that, but at test time, and it's the framing of the model as a meta-model. The tunings that you have to do are much smaller. I think that's maybe one of the biggest differences between our work and Clem's.

Can you just summarize that again, just so the audience understands it? You said that you're doing more training of the model before you do anything at test time. Can you explain that?

Yeah. We pre-train the models on ARC. You start with a language model—maybe Llama or something like that, or a T5 variant, LongT5. So you start with a T5 language model.

What kind of language model is it?

It's not a normal autoregressive language model. It's an encoder-decoder.

So that's an encoder-decoder?

Yeah, so it's an encoder-decoder, but it's pre-trained, so we don't train it from scratch. It's pre-trained on language, which you think has that contextualization aspect a little more. We also train it further on code, so it gets a stronger emphasis on this contextualization ability. You want to train the best kind of forward-pass-steerable model—a dynamic model that can change a lot based on its input.

It's not. That is possible with Transformers, but only to a certain extent. So that's what we start with.

Even that is very interesting, though, because no one else has done that. You're starting with an antique model from about 2020, which is an encoder-decoder, something that doesn't really exist anymore. Those were originally used for machine-translation tasks about a million years ago. It's presumably trained on a tiny corpus compared to modern models that might be trained on, I don't know, less than a billion tokens, probably.

Yeah, yeah.

So you start with that. But the benefit of it is, I'm guessing, it's a tiny model. It's in the millions of parameters—340 million parameters or something?

Exactly. Yeah, it's a tiny model.

Start with that. Then what?

Then we have this pre-training recipe, which involves code and autogenerated ARC tasks in various ways. Jack did lots of really good work there—Jack Cole. There's this magic recipe, but the bottom line is that it has lots of code and lots of synthetic tasks.

By the way, the total number of new tasks isn't big, right? It's not like we're able to sit down and generate so many synthetic riddle generators. The total number of new concepts in pre-training is small. But there's an important thing that we think is happening during pre-training, even with the few concepts being trained.

So we have a recipe that has some code and some synthetic tasks, and then we get our pretrained ARC model.

Okay. Now we go into the test time, where we feed everything into the forward pass, like we were doing in pre-training. All of the instances are in the forward pass, and then we get the test input—the new input that we need to predict the output for—which is also in the forward pass. Then we need to predict the output.

What I was saying about tuning the reasoning is that the framing of the problem is really important, right? Everything in the input, you need to learn: How do I compare the input and then get at a very almost-okay function—but probably not okay in very important ways—that's already very close to your solution?

So now, at test time, you can tune and search over the reasoning. You tune the reasoning, rather than looking at it in a different way.

First of all, that compares to the MLC work [?]. It compares to lots of ideas where, when you frame the problem in a meta way, you give the model more to learn. It kind of is always the best thing to do in the forward pass, and then you have to scaffold, tune, and do everything else. But you're halfway there with that step, so the tuning is much less work at that point.

At test time, you call it the forward pass. You put the test instances in there and represent them in a really clever way that helps the language model—the seq2seq model—do its thing. Then you do some green blatting for want of a better term. You do lots of sampling until you get ones that fit all of the specifications. Do you do any augmentation in the forward pass?

Yeah. The forward pass is just prompting the model with the representation of the board. The representation isn't super special; it's just a plain representation. So we put a plain representation into the model.

We don't do any filtering, if that's what you mean by green blotting. It's very transductive: you just produce the output directly. So we can't do any filtering, actually.

Right. But do you do loads and loads of sampling until you get ones that you think are good?

Yeah. We have a method of voting that we also introduce in the paper. Voting is especially suited to ARC, I think, and it follows this idea: There are many ways that you can be wrong about an ARC task, but there's only 1 correct solution. So you try all of these ways and hope that the majority vote goes to the only right way to do it.

Tell me more about the voting.

I think that summarizes it pretty well. We do augmentation. There are many ways to sample things out of a model. One way to sample something out of a model is beam search, right? Or you could do temperature-based sampling, which I don't think is a good idea for ARC because, again, if you just get bad RNG, you're kind of done for.

But beam search is really interesting because ARC is very special: there's only 1 right way to do it. You can imagine all of these beams, and the model is uncertain about a certain next token. That's totally fair, so we're going to take a beam and try all of them.

Iteratively, if you've made the wrong decision, you're going to be lost. In the pixel space, if you're outputting and you've made the wrong decision, you're going to be very lost because you don't know what to do—the mistake has already been made. Obviously, these models can't backtrack. So how do you really continue after a mistake? It's unclear.

Things are ambiguous, so the probabilities are going to disperse, and these beams are going to fall. But if you've done the right thing, you're going to become more and more certain of the next token.

That's 1 aspect, 1 way that we get the samples. But we have a variety of ways to get samples for voting. We go into it a little bit more in the paper, which will hopefully be out.

Yeah, so that's the voting. That was the sampling. How do you do the voting?

It's just majority vote.

Okay. You said that there's only 1 way of getting it right, but aren't there uncountably many ways of getting it right?

Well, there's only 1 way with the human priors and with our core knowledge of getting it right—hopefully. Otherwise, it's an ill-designed riddle, right? For an ARC riddle, there's officially only 1 correct answer.

Of course, you can have different systems with different biases that think this is the right way, and so on. But hopefully it's very clear, given human biases and core-knowledge priors, that there's only 1 way of getting it right.

We should explore that. The ridiculous example is that I could have a program which says, “If you see this specification, then give this answer. If you see this specification, then give this answer. If you see the third one, then give this answer.” So it's explicitly encoding the answers without any generalization or anything like that.

You're making the argument that, because we have a certain structure of priors that can be composed in a certain way, you think it's very unlikely that the model will find a solution other than the ones humans would agree on in that compositional space.

That's what we hope to do with the training. With the pre-training, that's what we hope to encode. We hope to encode 2 things.

First, we want to encode the priors really well. I also really want to get into prior encoding and what that means, because there's lots of theorizing about that. There are lots of people saying, “Okay, you're memorizing certain reasoning patterns,” and so on. Let's pin that for later.

But that's what we're hoping to do with the pre-training. We're hoping to get the right priors and the right kind of core knowledge into the model in such a way that it only outputs the right solution.

The second thing we want to do is make the model steerable. For a new riddle, we don't want it to rely on the priors too much. We want it to actually produce the right one that generalizes.

It's this very difficult balance. You want the model to be contextual, but you also want the priors to be deeply embedded so that it can use them to search.

So there's a trade-off between flexibility and correctness. The reason I ask is that, in Wen-Ding and Kevin's paper, they said that—because they did an ensemble with induction and transduction, although they favored induction—and I say transduction, but they're both transduction—they had a 9% false-positive rate on the explicit function generation. That meant they were creating functions that gave the correct answer but were actually wrong.

It worked on the test specification, but it was actually wrong. Have you seen that?

We don't output programs, so we haven't seen that. But you can have transductive—in my definition of transduction—intratransductive Python programs. As you said earlier, if this is the question, give me this answer; if this is the input, give me this answer.

You can have that with Python programs. You can have overfit Python programs, which just means that they didn't get the right prior from this one, this one, and this one to be able to generalize to new ones.

Exactly. So how much of an issue was that for you guys?

We kind of sidestep this whole thing. We find different ways of giving feedback to the model—in other words, test-time fine-tuning—to hope that the built-in model has the best chance to generalize to the new test riddle.

Basically, the answer to that is everything we do at test time. We do test-time tuning, which means you start with the initial guess from the model based on ICL—in-context learning—or whatever you want to call it, the contextualization ability of the model. Then you tune that.

And what’s happening is that there is an implicit model inside—an implicit transformation function, you could say, inside the model—that is getting tuned. It’s giving a guess, and then the guess is incorrect in major or minor ways. It’s getting feedback through gradient descent, updating that implicit model through the weights, and then giving another guess, and another guess. So that’s test-time tuning.

The hope is that ARC has a perceptual problem, and that neural networks can learn these generalized perceptual problems if you put them in-domain and tune the reasoning in that way. So I guess that’s how we deal with it.

How did you encode the problems?

It’s very plain: numbers as text. There’s nothing special there. Absolutely nothing special.

This is a very interesting question. In my opinion, you shouldn’t generate any special representation for ARC. The whole point of ARC is that it’s going to trick you. Whatever specialization you put in the input, you can create a problem that’s adversarial to that tokenization scheme or special-representation scheme for ARC problems, because it’s so arbitrary. The main point of ARC is that it’s raw. You get the problem raw, and then you need to be flexible about combining it, because it’s going to be novel and raw.

This is another thing: VLMs are really bad for ARC. Maybe this is giving away a lot of alpha to people, but I think that, for a big majority of people, the first thing they try is, “Oh, yeah, this is a visual problem. Let’s use a VLM.” But that’s a really bad idea.

That gets to some of the stuff I was seeing at NeurIPS, where people are coming up with lots of datasets where VLMs do really badly. Take the six-finger problem. If you have a hand with six fingers and give it to a VLM, asking it how many fingers it has, it’ll tell you five. Why? Because the VLM is a fixed-representation machine.

This really gets to the core of ARC. If you want to understand ARC, you need raw representations going into the transformer rather than pre-encoding some stuff. The fixed, frozen encoder in most VLMs will take some kind of perspective—some preexisting perspective—on the ARC problem. But you haven’t seen the rest of the input-output pairs yet. You don’t know what the right framing is. You’re guessing, and there are many ways that you could be wrong.

It’s the same thing with the six-finger example. The VLM only knows certain things, and it’s going to give you, in its output tokens, a representation of only that framing of certain things, like whether it’s a hand or a leg. It’s not flexible enough to be able to recombine. It doesn’t know what the question is.

Also, with a VLM, you’ve already encoded the image before asking the question, “Do I need to count the fingers in the image?” So the transformer model and the VLM encoder can’t condition on that.

A couple of things on that. I mean, Greenblatt did GPT-4V, the vision model, but I’m not entirely sure how that works. The best way to do a visual model is to have it be multimodal. So it’s a language model and a vision model all in one, and it can do some kind of crossover between them.

There’s the question of whether having a vision model actually helps with ARC, or whether it’s better to think of it as a reasoning problem, skip the interpretation part, and give it a structured representation. Maybe we should start with that. Do you think, in principle, that if we had a strong multimodal model which could transfer between the reasoning domain and the visual perception domain, it would work? Or was the other point of your statement that it’s not flexible enough? Could we not do test-time training and all of this transduction stuff with a VLM as well?

Yeah, you could. I just don’t think it’s elegant. I think it could get you to 60% or whatever, but it’s not going to get you 100% of the way there unless you change the architecture. I’m making a statement about a specific architecture, just to be clear. Multimodality might help.

The thing that I’m advocating for is that you need a flat representation and an intermixing. You could have vision patches intermixed with language tokens, and they all condition on each other. That seems reasonable.

But if you have a pre-encoding that is frozen and you don’t test-time tune, that’s just a bad idea. If you do test-time tuning, maybe that works. But again, it’s a crutch. It’s not reliable. It’s better to really focus on contextualization and on being flexible, because that’s the point of ARC, right? It’s a very, very good benchmark because of that, because it found a way to test it.

I think one thing that’s really good about our approach is that there’s a very different way of seeing it. If you want to measure the contextualization ability—the true contextualization ability—of these models and architectures, one thing you could do is just prompt it in the forward pass and see if it’s able to solve the problem. That’s a really cool way of attempting to solve ARC. Obviously, that’s what we’re doing.

That allows you to ask certain research questions, and I’m really excited to answer them at Tufa Labs. It’s not just about solving ARC. You could start framing things around measuring what transformers are doing, or their contextualization ability. I hope more people start doing that as well.

One of the really cool things about test-time compute—I don’t know if you saw this—is that Hugging Face has just released this kind of o1-type thing, and they’re showing that they can make a 1-billion-parameter Llama solve the same types of math problems as a vanilla 8-billion-parameter Llama model.

What it seems to do is give you the ability to make a small model behave as if it’s a big model, and that seems to be a superpower.

I’m not familiar with that release from Hugging Face, so I can’t comment on it. I can give you some more details about how our models scale on the hidden test set. I think that’s really interesting, and it’s something people really want to know.

So what do the scaling laws look like for the hidden test set, which is completely uncontaminated? The public set is not completely uncontaminated; the hidden test set is.

I think there is a potential objection that, because we’ve now hit the hidden test set a lot of times, there’s some unintentional information leakage. From an information-theory perspective, I think the information leakage is very, very low.

But I’ll tell you what we’re trying to do with the information leakage: we’re trying to make it algorithmic. There is information leakage about the type of things that work on ARC, and that was the hope, but I don’t see it as significant at all, honestly.

The ARC dataset was actually available for 100 submissions a day for a couple of years, and people didn’t seem to gain too much from that. We didn’t gain too much from that. We honestly weren’t using it that much. I’d say it’s very minimal.

Yeah, because François Chollet thinks that the reason he needs to make the next version of the ARC challenge is that, if it’s not overfit now, it kind of is, but just in a latent way.

When you look at the ensemble of all the various approaches to ARC, even back in 2020, the ensemble was getting about 49%. He said that, if you did a targeted attack—imagine we’re security researchers; that’s what they do—they look for sources of entropy and mix things together in a targeted way. Even though it hadn’t really been attacked in this way yet, he said it’s only a matter of time until it was.

Yeah, I think that’s fine. I think it’s good to refresh the dataset and also maybe to remove the brute-forceable ones.

I spoke to François a little bit, and he mentioned that even in his talk. He mentioned the aspect of, “Okay, I’m going to find out the brute-forceable ones,” because that was the main approach in 2020, “and I’m going to remove those.” I think that could be a good thing to do.

I do think the domain is so huge and the possible variation in the grid is so big that it’s totally fine to use it. But again, from an information-theory perspective, the bits are not that much. I mean, who knows.

I’m totally fine with a new dataset that is harder. I think we’re up for the challenge. The refresh is good. The guys I spoke to are doing really good calibration with human testing to make sure that the datasets are well calibrated. So, yeah, it’s going to be fun.

Chollet is quite bearish about some of the test-time-compute strategies that are being used. Could you reflect on why that is and what you think he would rather people did?

Yeah, he spoke about that a little in his talk. So he’s bearish on test-time compute, or on the specific strategies that people are using for test-time compute? I think both.

I asked him, “What would you do?” because he’s doing his own startup now. He has advocated almost for something that resembles the original DreamCoder. So he thinks that we should have. So first of all, he's a big program space guy, so he doesn't he thinks that there are inherent limits in compositionality and whatnot using neural networks. So we should do the program-space thing, whether it’s a DSL or an actual programming language. I can’t remember exactly what he said—let’s say programming language. He says that we should have a neural-guided search, which is what the original DreamCoder did.

The reason I don’t like DreamCoder-type approaches is not because of DreamCoder specifically. I spoke to even Kevin Ellis about this, and I think their current approach is interesting when you consider that he was the first author there. I think there are 2 kinds of things that you have to focus on, especially when talking about DreamCoder: the output space. That’s what Kevin Ellis mentioned. It’s too restrictive, right?

This lambda calculus is very inflexible. In my opinion, generating Python programs is hard. If I ask you to write the Python program for Syrian rebels, it could take you up to 30 minutes, even if you’re a good programmer—maybe 10 to 30 minutes. But if I ask you to color them in, you can do that instantly.

That gets to Piaget’s theory of incremental development. You can be in a game without being able to perfectly describe the rules of the game, but you can act in the game. You can color the things, represent the thing, and act in it. But being able to fully specify the rules of the game and write them down is a higher level, according to Piaget. That exactly hits it home for me. So that's the first problem with Dreamcoder.

The second problem is that it just doesn’t focus on perception, right? It could, and if it did, maybe it would do well, but it doesn’t. I think the main thing should be: how are you tackling being flexible in the perception space? Then you can go on to do whatever you like—to learn programs in the sleep phase and compose them. All of that stuff is fine, but first, the first question is: how are you tackling perception? So that’s exactly right.

What’s also interesting is that Kevin has abandoned using DSLs. He has, with open arms, embraced language models. Why is that? Well, it’s because, as you were just saying, Python code is Turing-complete. You can represent any concept; it’s incredibly flexible.

The reason why he didn’t generate Python code in the first place was because there was this intractable search problem. It wouldn’t have been possible. Now, with language models, we can, because language models encode our knowledge and our notions of what’s interesting—even basic priors that we couldn’t put into words, like the complexity of a program, intuition, creativity, and all of this kind of stuff.

So now we can generate these programs, mix them and remix them, and do all of this kind of stuff. Everyone at NeurIPS is doing the same thing. I guess the question to you is: this was a restricted-compute benchmark, so of course you made certain trade-offs that you wouldn’t make otherwise. Now at Tufa Labs, you’ve got all the compute in the world and all the time in the world. What would you do differently?

That’s a great question. I think the most interesting thing to me right now is the thing I mentioned about prompting everything in the forward pass at once, measuring the transformer’s innate ability, and then tuning the reasoning—those types of lines. There are so many angles to tackle. It’s about having the compute to do many experiments, tackle the different angles, and get people on board who can help.

They’re really interested in tackling these things and thinking along these same lines. Transformers are really bad at ARC. That’s the statement that surely—or language models are really bad at ARC. And they are. But let’s measure it and try crazy ideas to get them to improve. That’s what I’m after.

Tell me about this paper that you’re writing with Jack and Michael.

Jack and I are working on a paper to outline test-time tuning. There have been papers that outline the technical content of what we have, but we wanted to put the paper out there for people to cite as the original source. People right now, by the way, are citing our MLS podcast.

No way.

Yeah, lots of papers.

Really?

Yeah. The test-time paper by MIT, by Ekin, cites the podcast. Lots of people cite that MLS video podcast, which is not great, right?

Good for me.

Yeah, it’s probably because you didn’t have it in writing anywhere.

Folks, you can continue to cite the MLS podcast as much as you want. I mean, that’s actually amazing. You should have a kind of a counter.

I hope people do that more, but I also want to put a paper out there so that people can cite the written work and we can track it better. I think this is a great format. We shared a lot about our method, and people got inspired by it, implemented it, and put a lot of papers out. So we want to put our paper out there.

Technically, it’s very similar. The stuff we’re doing now isn’t far away from the stuff everyone else is doing. Some people are complaining about us not open-sourcing, but again, the stuff is already out there. Some professor told me this: I’m more worried about OpenAI not open-sourcing than MindsAI.

I think that’s the key thing. Another key thing is that we’re working on more papers, not just that one, that explore different angles. We’re going to be putting lots of stuff out there in the very near future. I’m super excited about that.

Quick point on that: why did you not open-source the paper, given that you know the approach and have already put more out there?

We were in a very tough position from many different angles. The requirement for open-sourcing was a little too extensive. They required weights, but also training code or training examples, and the entire test-time code.

There were a lot of things where we felt like, with Michael’s DSL—which so many people are now using and have cited for everything—and the related work for test-time tuning and voting that everyone is using to get the top scores, we’ve contributed a lot to the community. It was a little too much, especially for us targeting that 85%, right?

The incentive structure really didn’t make sense if you look at the prize money and the potential gain. I think lots of people were making the comment that the incentive structure, if you don’t get 85% from the first competition, just wasn’t great. But we’re super excited to do that moving forward. The competition team has told us that they’re working on that.

Absolutely. They’ve taken that feedback. Just to be clear to the audience, what was the incentive, given that you got around 56% or 57%? If you did open it up, what would you have got?

That’s a very good question. By the way, just for reference, we started out at 33% at the beginning of the competition. We did all of the work to get it to 55.5%. We even got a 58% score on the hidden set, but it wasn’t on the leaderboard because the time had passed.

What we would have gotten was $25,000 USD, after taxes. You know what I mean? Not a huge incentive. But again, I just want to say it was a great competition, and the guys did a lot to put the word of ARC out there and into the world.

They’ve taken that feedback, and we have also upleveled the community and shared a lot. I feel like we’re both happy, and we don’t need to dwell on it too much. We can just look forward to next year.

You guys are now working with Tufa Labs in Zurich. As I understand it, Tufa has acquired MindsAI. Is that right? Tell me the story.

Yeah, we are now. The whole team is now working out of Tufa Labs. We have a lot of funding and a lot of compute coming in. We’re going to be purely focused on ARC for around the first year, exploring different angles to ARC.

Again, we’re working on not just this paper but another one and a couple of other angles that we hope to turn into papers. We’re going to be putting stuff out there, and we’re hiring, and we have compute coming.

We also have plans for things to do after ARC. Again, we spoke a little bit about this, with the compositionality of these large language models. It’s not there. How can we get them there in a more general format?

There are lots of System 2 goals. That and test-time computation, obviously, are very ripe areas for research more generally. So we’re going to be exploring all of that beyond ARC.

All of that is very exciting. The compositionality problem—Clem was saying the same thing. That is the golden ticket. If we can solve the compositionality problem, I think reasoning opens up.

Presumably, though, the approaches that you’re going to be working on now are going to be slightly broader than what was appropriate for this particular benchmark. You’re going to be doing a few things differently. What particular strategies and approaches are you going to be looking at now?

Different creative ways of doing test-time computation are really interesting to me, too. First, I’ll say you always have to target 100% with ARC. You can’t optimize for a certain competition—or, for something like ARC, it’s a really bad idea. It’s not going to give you a lot of score, just because of how private the dataset is, and you can’t make guesses about it.

It’s formulated in a really nice way where you have to go for broke. Whether you are working in a small team with very little funding or with a lot more funding, I think it’s kind of the same. But what we get now is different angles, right? A research angle and the competition angle, and these are super well aligned, by the way. They mesh together really well.

So, just more angles of that. Also, thinking about different benchmarks around ARC, I have some really good ideas there. ARC is a special format, I think, that allows you to benchmark certain things. Maybe we can go into that at another time.

What do you think will happen when Chollet releases the new version of ARC? One thing that Kevin and Xena said, which I didn’t predict, was: has ARC become a lame-duck benchmark now that Chollet has said he’s going to invalidate it next year? “Invalidate” meaning he’s making a new version of it.

It’s almost like, are you wasting your time working on ARC 1 when everything changes with ARC 2?

I think ARC 2 is going to be the same format as ARC 1, right? I think what’s going to happen is it’s just going to get harder.

What you said to me was that he’s employing loads and loads of humans to design and select tasks that are at the appropriate level of difficulty. Some tasks are insanely difficult, and most humans don’t get them right, while some are too easy. Even then, there’s an interesting overlap between what’s easy for a human and what’s easy for a computer.

Chollet, of course, is a big believer that they are a proxy for intelligence, that there should be some kind of g factor, or general intelligence, between them. The other thing I wasn’t entirely clear on is whether he is protecting ARC 2 through diversity—reducing information leakage through sheer diversity—or whether he is talking about creating a dynamic benchmark that is almost entirely impervious to overfitting.

I will say this: I was talking to Chollet, and I asked him, “Are you going to have a dynamic benchmark where you iteratively—or have some kind of iterative framework where you can ask for more?”

He said no. It’s going to be the same format, just different data and, very likely, more difficult in whatever way they are calibrating for.

I think one thing that Chollet speaks about related to your first question, about getting the right level and so on, is this idea of idiosyncratic results. I mean one-off riddles—riddles that are so creative. I think it requires real creativity to come up with those riddles. They’re very novel; you can’t label them.

If you were working on a labeling approach for ARC and trying to find the right label, it would be hard to categorize them. They’re one-offs. I think what he’s going to do is have a lot more of these idiosyncratic riddles, and I think that’s interesting. I think that’s a good thing.

But I also think that the original ARC formulation was already good enough, and it’s already enough for us to come up with new methods. Even if it remained the same, I think there are still lots of angles where you can measure generalization. Transformers improve on it even with this fixed thing.

I do think it gets solved with scale. I think ARC 1—give us 4 3090s, or 2 3090s and more time on the problem next year—and absolutely we get to 85%. So, for ARC 1, I do think it scales. I have some data on that with the hidden test set, and it does scale in a very interesting way that you wouldn’t expect.

I’m okay with more idiosyncratic riddles. Maybe that would tune our signal for what generalization is a little bit better. They’re putting a lot of hard work into that, and I’m excited for it.

If they do listen to this, I have one request, or hope: that they also open up V1 again so that we can have a benchmark for progress on the methods. We know the scores really well on V1. If we can still submit to V1, even though now there’s a V2 and it’s very different, that would be really good to have—maybe as a validation set or another benchmark. You know what I mean?

Yeah. I mean, surely this measure of intelligence was so fascinating because he was talking about this adaptation—being able to create a new skill program in response to novelty.

If you think about it, it’s not a foregone conclusion. This is actually a very difficult thing to do from a psychological point of view, because you’re trying to create a general benchmark where, given the base knowledge of the average person, they would be able to perform this generalization.

You gave the example of the one-offs: “Why are manhole covers round?” These ridiculous riddles that they used at Microsoft to hire people. Only 1 in 1,000 people would get them right. The information gain on that is basically zero.

So you need to create a set of problems that the average person, with their basic priors, would be able to solve, and ARC does that really well. I think Chollet doesn’t get enough credit for how cleverly he selected challenges that would work well for this. Of course, now he needs to diversify across that.

One other quick question: in your experimentation, did you notice patterns? Which types of tasks are you failing on, and which ones are you doing well on?

That’s a good question. Professor Melanie Mitchell has a great benchmark concept: ConceptARC. She’s been thinking about this for a long time, and she has lots of great ideas there.

One thing I’m noticing is that I’ve spoken to lots of people who are using neural methods on the benchmark, and they all say that counting in ConceptARC is the lowest, right? It’s just a thing. If you try to add more counting-based riddles, or priors—in terms of feature engineering and that kind of thing—we haven’t tried that, but it’s not good.

For some reason, counting is really abysmal for neural networks.

Yeah, and I can tell you why. I interviewed 2 guys at DeepMind on Monday, and they’ve studied this in self-attention Transformers. They actually say it’s because of representational squashing. I won’t spoil the surprise—you can watch the interview—but there are a couple of problems with Transformers.

Because of the way they’re set up, almost all of the attention gets focused on the first token in the limit, actually, if you scale Transformers all the way up. There’s also another problem with the softmax function that creates this kind of directedness. For certain types of reasoning tasks, you want it to be directed, and for certain types of creative reasoning, you want it to be diffuse.

They were basically saying that Transformers, even in a trivial sense, cannot do counting or copying. They just can’t do it. You can give it a trivial example where you say, “Can you count these numbers up to 100?” and it just fails abysmally straight away.

Copying is an interesting example because even if you use tools, people say, “It’s okay, you can just use tools. You can stick it into a Python tool.” If you can’t even copy the tokens into the tool, then you can’t do that.

The upside of this is that all we need to do is fix this problem, right? Whatever the problem is in these architectures, if we can make them copy and count, then maybe all of these problems will just disappear.

Yeah, it’s exactly that. You need to dive really deep into the architecture and see where the problem is. In this case, it can be very clear what the problem is, right? The softmax is a max—a kind of cheat-code way to achieve something similar to a max—and you can see how that would bottleneck things.

A really brilliant postdoc told me this: you don’t want to do the processing of these things in a single layer. You want to do them as you go up the layers. If you’re adding or counting all in one layer, that means you’re overfitting, right? Why? Because you’re not really running an algorithm. You’re just taking this feature, this feature, and doing it in this one layer.

But if you just have one layer that goes and attends to this thing and updates itself, and then goes and updates this thing, that’s the general algorithm that you want. It’s not this heuristic kind of MLP in one single layer that just gets this feature, this feature, and this feature and does everything in one layer. That’s not what you want.

There are lots of ways of doing that that I’m, again, super excited to explore. I think you hit the nail on the head there with what you just described.

Mo, it’s been amazing to have you here. Thanks so much for coming on.

Thank you so much.