Subbarao Kambhampati - Do o1 models search?

So much so that we now have phrases like “fractal intelligence.” In fact, I think Andrej Karpathy basically was saying LLMs have fractal intelligence. What the fractal intelligence is, we basically don’t know. When they work, they work; when they don’t, they don’t. That’s fractal intelligence, and that sort of shows—which is good.

Still, we had nothing like this before, but part of the science of LLMs has to be saying something more than “jagged intelligence”: saying, “Here is the level to which you can depend on their results.” In reasoning and logic, there are ways of formally characterizing the limits of reasoning, like limited-depth, limited-lookahead reasoning, and so on. None of them seem to work for LLMs.

Hmm.

The question then is: What would work? We have to figure that out.

The bitter lesson is over, and efficiency is going to matter. I completely agree with that. I’ve been arguing this for a long time, too, so think about the following thing: The first time we sent humans to the moon, cost wasn’t a consideration. We wanted to show that we could do it. NASA was the one doing it.

The second and third times may be okay, but by now it’s Elon Musk sending people to space and, supposedly, possibly to Mars too, because the cost matters, right? Essentially, once it’s been done, then you start caring about the cost that you’re paying. Computer science is actually quite a bit about the unsexy parts of cost, just as it is about doing things that haven’t been done before.

There are people who say, “Well, if it’s not retrieval, then it is reasoning.” So what say you?

That reminds me of this old Monty Python thing, I think Life of Brian, where this guy does something that looks like this: If this is to prove that somebody is a witch, if she’s made of wood and she floats on water, then…

Ah.

How do you know she is a witch?

She looks like one.

Yeah.

She looks like a witch.

Bring her forward. I’m not a witch. I’m not a witch. But you are dressed as one. They dressed me up like this.

Well, who do you think? Who do you think?

And this isn’t my nose. It’s a false one. Well?

Well, we did do the nose.

You know, random connections, and then saying, “She’s a witch,” and you say QED. That looks like reasoning because it’s not just retrieving something like “she’s a witch,” but we know that it’s not sound reasoning.

Tufa Labs is a new AI research lab I’m starting in Zurich. In a way, it is a Swiss version of DeepSeek, and first we want to investigate LLM systems and search methods applied to them, similar to o1. We want to investigate, reverse-engineer, and explore the techniques ourselves.

MLST is sponsored by CentML, which is the compute platform specifically optimized for AI workloads. They support all of the latest open-source language models out of the box, like Llama, for example. You can just choose the pricing point, choose the model that you want. It spins up. It's elastic autoscale. You can pay on consumption, essentially, or you could have a model which is always working, or it can be freeze-dried when you're not using it. So what are you waiting for? Go to CentML.ai and sign up now.

Microsoft essentially cannot any longer control OpenAI if, in fact, AGI has been achieved. That was one way they could avoid being beholden to Microsoft. But now they’re trying to say, “We’ll remove it so that we’ll get more money from Microsoft.” That’s kind of—I don’t know what that says. Are they looking for money, or have they realized AGI isn’t actually going to come anyway, so why bother with that clause—

Yeah.

—other than surprise.

So much has changed since our last conversation at ICML. Can you give us a bit of a rundown of what’s happened?

When we were talking in Vienna, I think we were talking about the reasoning abilities of large language models. In particular, I think of large language models as autoregressive, token-by-token prediction models, which are pretrained for that, and they also do that at inference time. It was clear, I think, as we were talking about at that time, that, from my perspective, those models did not have reasoning abilities. They’re amazing at supporting creative work, where they can give you ideas and you can run with them, but they will give you an answer as soon as you hit Return, and they’re not guaranteed to be correct.

One of the interesting questions, of course, is that reasoning tends to have a higher complexity in terms of the time needed and other ways of actually changing LLM substrates to do that. A couple of things happened. Obviously, we’ll get to o1 in a second, because that’s the bigger thing that happened. But an interesting way of looking at that whole direction is through what’s been called—there are 2 parts—inference-time scaling and post-training.

The first ideas that were tried, and in fact we talked about this when I talked about LLM-Modulo, were: to the extent that LLMs are essentially quickly generating candidates but with no guarantees, maybe you can make them generate tons and tons of candidates, and then either do majority voting, or self-consistency, or something like that, to see if you have the better answer. How do you check the better answer? There are a whole series of ways. There might be external verifiers. There might be LLMs themselves trying to partially verify. There are problems with that that we talked about, but they have tried that too. That’s one type of inference-time scaling.

1. Prompt Augmentation Drives Chain Of Thought

A related idea there—an interesting idea—is that it’s been known from day 1 that if you’re trying to give a reasoning task to an LLM as a prompt, and then it gives a completion, and you check its completion for whether it contains the solution, the probability of that happening, in general, can be made higher if you can find the right kind of prompt augmentation. In addition to your reasoning thing, you add some magical tokens, and that seems to increase the probability. This has been seen in multiple scenarios.

Originally, this idea was bandied about as chain of thought, and the very first version of that is essentially zeroth-order chain of thought, where the magical token will always be the same one, irrespective of the task and the LLM: “Let’s think step by step.” That sort of worked partly because the human data had those specific types of tokens. The LLM outputs that, and then that jogs its pattern-matching to actually pick up other solutions, and so on.

Then came the task-specific chain of thought, the one Jason Wei and co. did. There, humans give task-specific advice as to how to solve the problem and then hope that LLMs will actually solve it. This can be connected with inference-time scaling because you’re adding chain of thought and also essentially making it generate multiple candidates and then actually picking from them.

Chain of thought by itself has, again, problems, just as LLM verification has problems; chain of thought has problems. In fact, at NeurIPS, we had a paper called “Chain of Thoughtlessness,” which we’ll talk about later. By itself, it has problems, but as part of the toolbox of increasing the time spent during inference before you blurt out one answer, chain of thought, together with picking from many samples, has shown some promise.

One variation of that—and, in fact, something that I’ve been pushing more recently—is that chain of thought was originally confused with the possibility that it might be anthropomorphic. We tend to tell ourselves, “Okay, let me do it this way,” and so on. People are hoping that LLMs are doing the same thing. Mostly, they were just imitating whatever “Let’s think step by step” data they had found in the training data. Somehow, people thought that if you make them imitate human thinking, then maybe they will do better. Those are the first 2 ideas, and neither of them actually went that far.

Another idea is that you realize it’s just magical tokens that you’re trying to add, and you just have to figure out what the right magical token is. It’s sort of a SQL function. You’re trying to figure out a task-specific, LLM-specific magical token that increases the probability. This is a learning problem. It’s an extra learning problem. There are 2 general approaches that have been tried.

The first approach essentially was to say: Before giving the answer, the LLM has to tell itself a few things. Some, like “step by step,” make sense to us, but it can actually give itself a gobbledygook string, another gobbledygook string, and then that kind of probes its conditional probability of completion in such a way that it might actually come up with the correct solution. The question then is: Where are these tokens coming from? One of the first ideas was that humans would supply these tokens through chain-of-thought advice.

That wasn't going anywhere. Before going there, OpenAI did the following thing: maybe we will ask humans to solve specific problems while thinking aloud. There was actually a paper from about 1.5 years ago saying, “Let’s think step by step.” This went under the whole issue of process supervision, and people were asked to record what they were telling themselves, et cetera.

This is the worst form of psychology, unfortunately, because we don’t actually know how we think. But they tried this, and one of the things is that it’s extremely costly. My joke is that they improved the GDP of Nigeria because Nigerian Turkers were being asked to solve tons and tons of these problems and then think aloud. That was very costly.

A separate, similar idea was that there are a bunch of problems for which there are systematic solvers. For example, for arithmetic, there are arithmetic solvers; for search problems, there are A* search sorts of things; and for planning, you have planners. In general, any systematic solver would be manipulating some data structures until a certain termination condition is reached, and then it outputs the solution.

Imagine making it output the trace of the data-structure manipulation operations. All you needed, hopefully, was some extra kind of tokens coming out before the solution. This stuff can be thought of as a derivation, and the idea people had was: let’s train the LLM with a huge number of these derivational, synthetic derivational traces and the solution.

Remember, this only works for problems for which there are synthetic solvers and systematic solvers. You’re just trying to make the problem solvable in a general sense without having to call those solvers. That was the idea. There have been a couple—three or four—efforts along these lines. There’s Search Farmer from Facebook, Meta, there’s Stream of Search, and just last week there was a Google DeepMind paper that also talked about internal versus external planning to do multi-bot game solving. All of these essentially use variations of this idea.

You have to realize that all they’re doing is making the LLM, before outputting the solution, output some additional tokens that will jog its memory to hopefully produce a better solution. This is the hope. People tried it, and sometimes it actually works; it improves performance.

There is no good reason to systematically say that this would make sense, because it’s almost like if you’re trying to teach your kids how to reason—very small kids—and you do some hand movements and then say, “Think like that,” and give the answer, you would see the junior also doing these hand movements, thinking like that, and giving the wrong answer. LLMs can do that. They’re essentially imitating whatever the derivational piece is, which may not even actually make sense, but sometimes they have shown some promise.

This has basically become the most recent idea called inference-time scaling, where essentially you do this and also generate multiple suggestions and then pick from them, et cetera. This comes very close to what I think o1 is doing, but with a big difference. As you know, nobody knows what o1 is doing, and it’s become like we all sit around the ring and suppose. I like to say Noam Brown sits in the middle and knows. But they don’t want to tell us what they’re doing.

My guess—I mean, everybody has a guess, and my best guess as to what o1 might be doing—is that it’s again using this prompt-augmentation idea. But the question, of course, is where are these prompt augmentations coming from? First, we talked about one prompt augmentation for everything. Second is human-given prompt augmentation, which is chain of thought. The third is a synthetic derivational trace that gives these tokens, and maybe you try to say this back.

A much better idea is to ask, “What should I be telling myself to improve my outcome?” That’s a kind of reinforcement-learning problem. Imagine an AlphaGo agent sitting there and thinking, “What action should I take, one after another, such that my win probability increases?” It does a whole bunch of these board actions, and then, at some point, it gets a signal saying it won the game or lost the game. You do this a gazillion times, and then you can bring this reasoning back through the sequential decisions, computing their Q-values. You ask: under what board positions, or for what actions, is it worth doing? That’s the Q-value.

If you take the AlphaGo analogy and apply it to LLMs, the LLM board position is essentially the context window with the prompt and all the other things that you have put in. The action is the token that you are generating. To make things simple, I like to think of it as a big LLM—let’s think of GPT-4—and a small LLM with a reduced vocabulary. All the small LLM is trying to do is generate these prompt augmentations. It tries them, throws them out, and then they are given to the other LLM as part of its context. It gives extensions, then it tries one more. At some point, it checks whether the solution is correct.

How does it get the solution? You could have generated huge numbers of synthetic examples beforehand, again using solvers. It is pretty much known that OpenAI did this. It’s no longer humans solving problems because that’s too costly. These are systematic solvers solving planning problems, constraint-satisfaction problems, and various sorts of problems for which they have the problem and the answer. Then the LLM, together with this prompt-augmentation engine, tries to solve it. If it happens to reach the correct solution, you can propagate the signal back.

This is reinforcement learning in pseudo-moves. The actions are not Go actions if the prompt is about Go; they are essentially these prompt-augmentation tokens. One nice thing is that, instead of learning the Q-values, you can essentially change the weights of the smaller LLM in the right ways so that it outputs the right kinds of tokens given the context window.

If you do this approximately, you have approximate Q-values, and then this is the pre-training phase. In the pre-training, there is the LLM training, followed by this humongously costly post-training phase. They’re not telling us how costly it is—a humongously costly post-training phase in which they spent billions of dollars. At that point, you have the o1 model, which is now ready for inference time.

At inference time, once again, they’re doing inference-time scaling, except now they have the Q-values. You can improve the Q-values with online MCT kinds of approaches, the kind of thing that AlphaGo did. That’s where we can actually see that they’re doing it, because they charge you for these reasoning tokens.

If you run o1, it takes the prompt and gives the answer. In the old GPT-4, the amount of money that you have to pay them is proportional to the number of input prompt tokens plus 4 times the number of output tokens. In the case of o1, it does this whole bunch of stuff—essentially these pseudo-moves whose Q-values it has improved—and tells itself those things. It never shows that to you, but they are all counted as output tokens.

You might have, let’s say, 50 input tokens, 100 output tokens, and maybe 5,000 reasoning tokens. Suddenly, you start paying a lot more. One funny thing that happened was that when we started playing with the o1 preview when it came out, in 2 days we spent about $8,000. In fact, I had to get special permission from the university because they normally don’t reimburse beyond a certain amount unless you have separate permission and so on. That’s basically one of the ways this works out.

The interesting thing, of course, is that the way we describe this, it is based on an LLM, but a significant amount of additional work has been done, right? Essentially, you are doing something like an AlphaGo-style post-training phase, followed by an AlphaGo-style MCT online computation. At that point, I would think it could make sense.

Not surprisingly, in our results, we found that for the normal PlanBench, it does much better than the state-of-the-art GPT LLMs, including Claude and so on. But then, of course, you can go to the next level. It has its own issues. We can still talk about the fact that it doesn’t scale beyond larger problems. It can make mistakes, it has problems with unsolvability, and there are no guarantees about the solution.

But it now makes more sense to me. Again, I don’t know if this is how it works, but I think this is a reasonable way o1 could be working. If it is the way it’s working, this is the first time I can make sense of how reasoning can emerge, because you are at least having these pseudo-actions whose Q-values you are learning. Nobody ever said reinforcement learning cannot do reasoning.

RL can do reasoning. It's just that now it's an interesting thing where I keep using the stone soup analogy. You can make soup with stones if you start adding carrots, tomatoes, and all that stuff. At that point, it will still taste like soup. The question, of course, is who gets the credit? That's an interesting question that we would think about.

That is the long arc of what happened, in my view, in the last 4 months or so since we discussed this. One of the other interesting things is that part of the mystique of LLMs was that you'd write the prompt, hit Return, get the answer, and it didn't cost you too much.

Yeah.

That was how everybody was using it. With o1, of course, the post-training itself is extremely costly, but they are not charging us for that. They are charging us for the reasoning tokens, which you never see, but you pay for them. You just have to take their word that a huge number of reasoning tokens were generated, and they are going to make you pay for that.

As far as I can tell, at least in academia, very few people have actually been doing experiments evaluating o1 because it costs a lot, essentially. People are still going with autoregressive LLMs because they're cheap. One of the interesting things is that you can do reasoning, but the usual computational-complexity issues that we blithely forgot in the era of autoregressive LLMs—and hoped would somehow disappear—will come back.

If you want to improve accuracy, you have to actually do reasoning. This is pseudo-reasoning, in my view, but it still costs. That becomes an interesting question: when is it useful to use a general-purpose system versus a hybrid general- and special-purpose system versus an extremely specialized solver? That's something we haven't talked about before, but now it will become costlier, at least for industries. In fact, there's this whole movement about compound AI systems, and that's basically the kind of thing that people think about.

Very shortly after o1 was released, you quickly, as you were just saying, spent $8,000. You put a paper together called “Planning in Strawberry Fields: Evaluating and Improving Planning and Scheduling Capabilities of LRM-o1.”

Yeah.

LRM.

Yeah.

So you basically said they are positioned as approximate reasoners rather than mere approximate retrievers.

2. Testing O1 On Planning

We don't know the actual details of what they're doing, so there are 2 parts. One is what is objectively verifiable: we tested o1 on the same PlanBench problems, and it did quite well on Blocks World. I think Claude was already at 66. These things were at 99 or something. They basically saturated. More impressively, they did better on the mystery domain.

Yeah.

Given what I explained to you earlier about the possibility that they are training themselves with synthetic data, maybe they have unintentionally trained on the mystery domain, which we have available outside. So we actually generated truly new random mystery domains. Performance was lower, but it was still not like the 0.5% on the old ones. It goes up to, I don't remember the exact numbers, 20% or 23% on some of these problems, which is obviously a good sign that they're actually able to solve them.

The other part—as to why they are approximate reasoners rather than retrievers—is based a lot more on my reconstruction of what they could potentially be doing. They are doing reinforcement-learning-based post-training as well as online Q-value updates, using pseudo-action moves. I call them pseudo-action moves because you could do RL for normal Go or any specific board game. This one is just language games, where the game basically consists of a language context window, a prompt augmentation, a new completion, and then one more prompt augmentation.

This is what they call a string of chains of thought. But that's basically adding a bunch of prompt augmentations and seeing what happens at the end. If it winds up being correct—in the sense that it winds up containing the correct solution for your training data—then that's sort of like AlphaGo getting a victory signal after a bunch of moves. It then just needs to do credit assignment for the moves, and that's what RL is essentially good at doing.

If you're doing that, it is reasoning, and it's approximate reasoning because it is not actually using problem-specific actions. It is using problem-independent language-prompt actions.

Is it possible that you might be wrong about that? Is it possible that we're giving them too much credit and what they're actually doing is just this massive generation of trajectories all in a single forward pass? Maybe they do something like process supervision, so they do some clever RL pre-training stuff, but at inference time—

Obviously, again, this is the sad part of the o1 thing. I must tell you a funny thing: I was talking to somebody who said they were having conversations with the OpenAI guys, trying to sound them out as to what o1 might be doing. At some point, one of them said, “I think you may have to wait until the Chinese replicate what we did to actually figure out what we did.” That's the level to which the science of OpenAI has gone.

The only reason it is possible that I might be giving a lot more credit to the sophistication of the method they might be using is that, as I said earlier in the description of how things shifted from LLMs to inference-time scaling to this sort of o1-style method, the general inference-time scaling methods are not comparable. Inference-time scaling just hasn't been as good.

The other very important thing that you have to keep in mind is that while o1 takes more time, it doesn't take hours. A second of online computation time is way more expensive from a business perspective than days and months in the pre-training phase. Some of the inference-time scaling methods actually spend a lot more time than o1 does, and they still are not getting, as far as I know, to that level of accuracy in general.

That makes me think that unless you do a significant amount of post-training to get approximate Q-values up front, you can't improve just by MCTS. Think in terms of the AlphaGo analogy: if you only did MCTS, it would take much, much longer per move before you could get any level of accuracy or confidence. But one of the things AlphaGo does is a humongous amount of pre-training, where it learns an approximate policy that it then rolls out to improve the Q-value estimates it has.

That's possibly the reason why I think it makes sense. Of course, I also think that the normal inference-time scaling methods don't make too much sense to me. The one closest to a pure MCTS method that I have seen is this paper from Alibaba called Marco-o1. They have this MarcoPolo group or something, and they called it Marco-o1. Marco o1 essentially trains itself on chain-of-thought data, which is basically derivational data, and then on top of that it does an online MCT-style computation to improve the Q-values further.

They are much smaller, and they are not as impressive in terms of performance gains as o1. Those are the reasons I think the full picture requires post-training as well as inference time. The thing that you and I see is the inference time. But the thing that OpenAI can spend tons and tons of money on is the post-training, which happens before they actually deploy the model. That's where it is getting these approximate Q-values, is my guess.

Again, as I said, it's a strange thing to be involved in. We should be looking for the secrets of nature, because nature won't tell us. But we are now looking for the secrets of OpenAI, because OpenAI won't tell us. Hopefully, many of the efforts already underway to replicate this sort of thing will tell us more.

As of now, that's the thing. I cannot be sure exactly what they're doing. Everything that they have said publicly is consistent with my hypothesis. That's about the only thing I can say. There's nothing that's inconsistent with my model—or, I should say, my speculation about how o1 is working.

In that Strawberry paper, there's an appendix where we wrote down the speculation, and that is still consistent with everything they have said.

Yeah.

That's the only thing I can say.

Yeah. I like the sound of it. It makes me more excited about using it because it makes me feel that there's more sophistication behind the system. But a lot of this comes down to reasoning.

Mm-hmm.

I'd love to hear your definition of reasoning, but there are people who say, “Well, if it's not retrieval, then it is reasoning.” So what say you?

Let's actually look at it in 2 parts: the first part and the second part.

3. Reasoning Needs Guarantees

The definition of reasoning itself is a good place to start from, I think. I know that this whole AGI crowd basically tries to say that AI is going to be like humans. The problem is, we don't have a good definition of what human reasoning is.

Since the Greeks, our civilization has gone forward not by saying, “How do we define what humans do?” but by defining sound reasoning patterns. Aristotle's syllogisms, logic, probabilistic logic—the entire field of computer science and civilization depended on having formal notions of reasoning, where there is correctness, there are incorrect things, and so on.

It reminds me of this old Monty Python thing—I think Life of Brian—where this guy does something that looks like reasoning. If this is to prove that some woman is a witch, then if she’s made of wood and she floats on water, then she must be like a duck. So you make random connections and then say, “She’s a witch,” and you say, “QED.” That looks like reasoning because it's not just retrieving something—“She's a witch”—but we know that it's not sound reasoning.

In general, I prefer to think in terms of this: ultimately, these systems are going to be deployed, whether you like it or not. Civilization didn't depend on fallible humans being able to make mistakes while we looked the other way. We actually have to have guarantees, at some level or other, about the soundness and completeness of reasoning.

I go back to essentially the definitions of reasoning from logic and so on—the formal definitions of reasoning. I try to avoid getting into this question of what human reasoning is, because that's a big mess. Cognitive scientists don't know it, we don't know it, and psychologists don't know it. So I try to give it a wide berth.

That's the part as far as what I believe about reasoning. That's why we looked at planning problems, for which there is a correct solution, and constraint-satisfaction problems, which have a correct solution. If you say the system is a reasoning system that can be deployed, it should have some guarantees.

You can say that humans can make mistakes, but one of the things I keep saying is that if you are being paid to make decisions and make mistakes, there are penalties for you. In the end, you can be put in jail. Until we figure out whom to put in jail and how to put them in jail when AI systems make mistakes over which they have no actual guarantees, we are better off thinking in terms of formal definitions of reasoning and then seeing to what extent AI systems are coming close to them.

This has basically been very connected to how AI has developed up until now, anyway. This discussion also brings us back to the issue of retrieval versus reasoning. I think you're talking about a couple of these papers that keep coming out, basically trying to say, “Look, LLMs aren't exactly retrieving anything that they've been told. They're not just memorizing and retrieving, so they must be doing something else.”

I would say that Monty Python logic isn't actually retrieving anything. It puts together a whole bunch of things, but that's not reasoning either. Between retrieval and what I would consider reasoning, there can be an entire universe of things that still won't be considered reasoning, as far as I'm concerned, because there are no sorts of guarantees.

From the beginning, we knew that. If you go back to many of these papers, these claims go back to essentially the autoregressive LLMs. By the way, the researchers are still very busy. I think we are one of the few papers on o1. We have an evaluation of o1 being presented at this NeurIPS workshop.

Most people are still trying to make sense of autoregressive LLMs themselves because that is still, as we talked about last time, a very impressive System 1. We never had System 1 in human civilization, and trying to understand what these systems are doing is useful.

They go back to that and say, “Look, they're not actually doing exact retrieval. They're doing something else, and we'll call this something else reasoning.” That's not a satisfactory argument. First of all, everybody always knew that LLMs are not databases, right? They don't retrieve, essentially. They actually have a hard time memorizing and retrieving.

When they memorize something, it's not by deliberation; it doesn't happen deliberately. It happens fortuitously. It's surprising that they sometimes wind up memorizing long passages, because essentially everybody agrees that they are some kind of n-gram model rather than databases in the way they are trained.

Given that, it's very clear that they will never retrieve. The fact that they are not retrieving should not be seen as an indication that they are reasoning. It can be seen as an indication that they are not retrieving, but we knew this already. The part that people seem to hint at is that since they are not retrieving, maybe they are reasoning.

No, that doesn't make sense, because again, you have to subject it to what you would consider the evaluations for sound reasoning procedures, and they fail just as easily as before.

If you come back to this chain-of-thought paper that I was mentioning, which we just presented at NeurIPS, in the case of chain of thought—in the Jason Wei-style chain-of-thought papers—the idea is, let's say you take something like last-letter concatenation, which is a really small toy problem. You give, say, K words—N words—and the system is supposed to take the last letter of each of these words and concatenate them into a string.

For example, “large,” “big,” and “rows”: E-G-E is the output you're supposed to give. That's basically the string. What they said was that if you just told an LLM, in the prompt, “You're supposed to take the last letters and concatenate them and give the answer,” and then tested it, its performance wasn't as good.

If they then told it, “Here are some examples of three-word last-letter concatenation problems, and then four-word last-letter concatenation problems,” and gave it a couple of these examples before asking their questions, its performance improved. That looks like reasoning. Somehow, it is able to follow the procedure.

The problem, I think we talked about this last time too, is the problem with ersatz empirical science: you shouldn't stop when you get the answers that you're hoping for. You should see whether you can break your own hypothesis.

4. Chain Of Thought Fails To Generalize

What they did was give examples of three- and four-word problems, and then test on three- and four-word problems. If you expect a system to be doing any kind of reasoning or procedure-following, once I tell you what last-letter concatenation is and give you an example, you should be able to do it for 20 words, 30 words, and so on. It's just mechanically taking the last letter and concatenating.

What we show is that if you increase the number of words, the performance plummets close to zero. This also happens in planning problems, not surprisingly. It happens in last-letter concatenation, and it happens in planning problems, which shows that it is doing something that seems to have improved its performance on problems of the size for which you gave the examples. Its pattern matching of some kind is helping there, but it's not in any way generalized reasoning that would generalize with respect to length, for example.

One interesting way I've been thinking about this is that it's sort of “the glass is nowhere near full” versus “the glass is already wet.” It's that sort of optimism versus pessimism. People tend to think that since it's at least solving the three- or four-word problems with higher accuracy because I gave it this chain of thought, that's showing reasoning abilities.

The question is, we don't have a good understanding of what the boundary is—where it will actually give the correct answer. So much so that we now have phrases like “fractal intelligence.” In fact, I think Andrej Karpathy was basically saying that LLMs have fractal intelligence. What fractal intelligence is, we don't know. When they work, they work; when they don't, they don't. That's fractal intelligence.

That sort of shows—which is good—that we had nothing like this before. But part of the science of LLMs has to be to say something more than “fractal intelligence.” It has to say, “Here is the level at which you can depend on their results.”

In reasoning and logic, there are ways of formally characterizing the limits of reasoning, such as limited-depth and limited-lookahead reasoning, and so on. None of them seem to work for LLMs. The question then is, what would work? We have to figure that out.

Instead of that, once in a while there are these papers saying, “Look, we probed LLMs using mechanistic interpretability techniques and found that they basically aren't acting as though they're doing retrieval.” But that's already understood. I think the mechanistic interpretability work is still very interesting. It may actually be part of the solution to figuring out what LLMs are doing.

But the argument that since it's not retrieval, it must be something like reasoning, is still quite unsatisfactory to me. What I'm saying is that it isn't reasoning, because whatever it was doing before you conducted your mechanistic interpretability study, it's still doing after the study, and it still has these limitations both before and after your study.

We don't actually know how to characterize what it is that they're doing, and that's the part where we are stuck right now.

Is it possible that everyone is right? What I mean by that is, I spoke with some DeepMind guys earlier in the week. There's a great paper about “Softmax Needs Glasses,” talking about how sometimes we need directed attention for reasoning, and sometimes we don't. There was another great paper talking about the utter limitations of transformers for counting and copying.

Laura Ruis—I’m speaking with her on Sunday—has a paper out where she's looked at reasoning traces. Sometimes they are just retrieving facts from documents. Sometimes they're doing procedural information generation, which you might liken to a reasoning process.

Mm-hmm.

I guess it's a little bit like this fractal intelligence thing: it might be the case that, in certain circumstances, these models are doing something that we would think is reasoning, and sometimes they're doing retrieval, and sometimes they're doing something else.

Yeah. No. Actually, I think Laura Ruis's paper is one of the ones I had in mind when I was describing this issue of mechanistic interpretability earlier. I think it's a good paper in terms of having developed an interesting set of techniques to actually see what is going on in the way LLMs are outputting their tokens.

But the thing that is unsatisfactory to me is basically 2 things. First of all, everybody knew that LLMs are not doing retrieval alone. That was well known way before, right? Nobody believes that LLMs are just doing retrieval. The question is, what else are they doing, and is there any clean characterization of what they're doing? I did not see that. I actually looked at that paper, and I think they've done good work, but I'm still hoping that there would be an interesting characterization.

There are lots and lots of groups trying to look for a characterization of what this fractal intelligence might be right now, but we haven't gone further than that. In terms of everybody being right, there could be this whole blind-men-and-the-elephant phenomenon in play to some extent. That part is possible because we are actually trying to piece through a large number of parts of this puzzle, including the reasoning part, including what they're even trying to do, including what sorts of techniques seem to improve their accuracy, and so on.

I think that's part of science. Basically, my sense is that eternal discontent is part of science. I am much more worried about being too optimistic that we figured it out than I am about being somewhat more discontented that we haven't yet figured it out. I want to err on that side, not because I think we know more than before when GPT-3 came out.

On the other hand, I think all the camps know. The people who thought GPT-3 was AGI know that that's not the case, and the camp that thought GPT-3 was just a stochastic parrot has to know that it's more than that by now, okay? So that is a collective improvement in our intelligence, but there are still a large number of pieces that we haven't figured out yet.

Yeah. On Laura's paper, she was using influence functions. I'm not sure if that would be classed as classical interpretability or mechanistic interpretability, but I think mechanistic interpretability is largely about finding circuits in neural networks, and even that's an interesting discussion.

To me, it's more of a general idea of figuring out a way of probing the inside of what LLMs are doing. I think of that as mechanistic interpretability. There are very specific techniques that have shown great promise, such as the autoencoder stuff, et cetera. But I think all of these are essentially trying to interpret what they're doing at the circuit level and make sense of their external behavior.

There are 2 ways of making sense of what LLMs are doing. One is just external evaluation. That happened already, and we know that they're not doing any kind of guaranteeable reasoning. There are enough results showing that they seem to do promising things in some cases, and there are also results showing that they seem to be very brittle: you change a prompt a little bit, or you change the problem specification a little bit, and they'll die.

Again, we are talking about autoregressive LLMs, not o1-type things. That's a whole other thing that we haven't yet started analyzing in the same way. Once you figure those things out, my sense is that trying to get a sense of what they're doing just from the outside versus also probing the internal circuits—if you start doing the internal circuits, I think of that generally as the mechanistic interpretability style.

Okay, okay. But isn't it interesting, though, that she found that code- and math-based procedural documents appeared disproportionately influential for tasks requiring reasoning? Larger models show an even stronger reliance on general procedural data for reasoning. The presence of code data in the pre-training mix seems to offer abstract reasoning patterns that the model can generalize from. These are interesting observations.

Again, I don't want to make this a very specific criticism of a particular paper, because that's not fair to them or to me. But I do want to say that there is a distinction between factual tasks and reasoning tasks.

Yeah.

LLMs have been used for both, and they have troubles in both. For factuality, I would think the only sorts of things that will improve them are techniques like RAG, where you just give them the factual data and ask them to summarize.

For reasoning tasks, for arithmetic and so on, I would expect that these are the kinds of things where the exact results don't exist. I would be equally troubled by the fact that people have shown that, if you take something like LLM multiplication—this was way before all of Laura's work—they tend to be correct in multiplications for popular digits and less correct for non-popular digits.

It's mind-blowing that there are digits that are popular versus non-popular, but that is an interesting point: the LLM's final performance is a complex combination of the data that they've been trained on and some additional pattern-matching abilities that they are using on top. But that's not sound reasoning, so we still don't quite know where it breaks. The fact that it gets to be correct for popular digits and not for some other digits is particularly interesting to me.

By the way, while we are on that subject, some work has shown that even with o1, we looked at o1 more on the planning side, but some people—I think Tom McCoy and co.—did more work. They tried the Caesar cipher sort of thing, the AMBERS thing, and they basically also found that o1 does better on some of those things. But they also still found that there are data dependencies, in the sense that its accuracy was higher in the regions where there was more pre-training data.

Again, I think that's consistent with my view of what o1 might be doing. There is an LLM that was pre-trained on some corpus, and there is this smaller LLM that is generating these pseudo-action tokens that will make it output things.

One of the interesting things is actually the difference, I'm told—again, we don't know for sure. I'm told that when the original o1 models came out, there were o1 Mini and o1 Preview. The difference, I'm told, was that one of them—I think o1 Mini—was using the smaller LLM as the base LLM, and o1 Preview was using the larger LLM as the base LLM.

I don't know; they didn't say this second part, but I would assume that if I have a pseudo-action-generator model, it is working on a bigger LLM with higher capacity, so it can generate more interesting completions, versus a smaller LLM that has less interesting completions. That makes a difference in terms of how much the RL-based training can get your accuracy up.

Yeah. I've noticed some interesting things. I've now paid for o1 Pro. I was very skeptical of o1. As you say, the base model is an even weaker version of GPT-4o.

I hate that model. I hate the style of it. I think it's dumb, and I must admit it's mostly because I'm anthropomorphizing it: I hate the style, so I think it's dumb. Humans are very brittle even on the RLHF. We like assertiveness, we like complexity. There are certain styles that we like, and we don't actually see the content.

But with that to one side, I don't like the model.

With o1 Preview and Mini, it doesn't really want to think. Most of the time it won't think, and you get an even dumber answer than you would with GPT-4o.

Mm-hm.

5. O1 Pro Changes The Vibes

However, o1 Pro—the vibes are different. It thinks more, and it gives you something qualitatively on a completely different level.

It doesn't look like dumb ChatGPT anymore. It feels very, very different. But there are still some issues with it. Certainly, for situations where you are dealing with ambiguity, doing programming, or something like that—

Mm-hm.

—I actually like having a dumber model because it's a didactic exchange, right? I'm saying, "No, you misunderstood that. Let's do this. Let's do that. We're working on this thing together." What o1 does is say, "Well, on the one hand, you can do this, and on the other hand, you can do that." It gives you a range of options, and I'm like, "Well, wouldn't it be better just to either go and dance with the model or better specify what you wanted in the first place?"

So, again, there are 2 issues. First of all, o1 Pro just came out, I think, last week, right? This was exam week for me, and we haven't spent any money yet on o1 Pro. I played with it from outside, but we haven't done any API-level studies, which is the kind of thing we did with o1 Preview.

But one thing I noticed, looking at the Twitter exchanges of the usual suspects trying various things with it, is that one of the things we saw in o1 Preview is exactly what you're saying, and it looks like o1 is still doing it. They're good at digging in to try to explain why the answer they gave is the correct answer.

One of the funny things was that I used this one particular 3-block stacking example, which is unsolvable. In fact, this showed up in The New York Times as an example of why GPT-4o actually fails on that. When o1 Preview came out, Noam Brown, in one of his long tweets, said that Rav [?] had said this in an ACL talk: that this problem can be solved. O1 Preview actually does solve this instance, and that's good.

People have actually said that o1 gets the wrong answer. Multiple people have said this, and they've posted screenshots. It gets the wrong answer, but it argues with you as to why the answer it is giving is still possibly correct.

This particular problem involves a situation where there is no way of solving it without moving C, because it turns out that it gives an answer where C moves. Because of gravity, it falls down. Then it tries to argue with you that there are games where people will say that unless you are intentionally moving C, if the natural process makes it fall, it's not considered moving.

That's a very interesting thing that we've seen in o1 Preview too. When we give it unsolvable instances, normal LLMs just die, because they've been RLHF'd to death. They think that if we give a problem to them, there must be an answer. So basically, they'll give you something.

Most unsolvable problems—this is why this was an unsolvable instance that I showed to GPT-4o before—are actually solved correctly more often by o1 Preview. That's a credit to it. That's why it's actually a more approximate reasoning model—an LRM, in my view—than an LLM.

On the other hand, when it gives a solution for an unsolvable instance, it'll argue with you that it is still right. I made this joke in the strawberry paper: we've gone from hallucinations to gaslighting. It tries to argue that you were wrong. Just like what you're saying, this is, on the one hand, what you want to do might be worthwhile, but on the other hand, this is why what I'm doing is worthwhile.

In fact, I think this guy, Colin Fraser—I believe he's one of these people on Twitter who keeps playing with these models—said he gave it the classical surgeon problem, the boy who gets into an accident. O1 Pro went through this whole thing: this is a classical puzzle that brings gender stereotypes into account, et cetera, and then gives the answer that the right way to think about it is—

In this version, he changed the puzzle so that the mother and the boy are driving, the mother dies, and the doctor says, "I can't operate on the boy." O1 apparently still says, "We should realize that the doctor is the second mother of the boy," and it tries to argue that position.

So, interestingly, this is overthinking and actually trying to dig down. One of the interesting questions that we don't know—we haven't played with this—is to what extent its explanation and its reasoning are connected.

Mm.

In humans, this is—I'm not trying to anthropomorphize what it's doing—if there are 2 different phases, right? If in phase 1 it comes up with a solution, and in phase 2 it needs to explain it, if it doesn't have to look at what it did to get to the solution, the explanation is just to dig its heels in and try to say the solution is correct.

People tend to do that. Sometimes we'll come to some solution, and then we'll try to come up with an explanation as to why what we did might be right. LLMs had this problem to begin with because they completely assume these are completely different things. I'm always worried about LLM explanations. LRMs seem to be even more sophisticated at this sometimes, but it's mostly anecdotal. I haven't really done systematic studies on this.

So, I don't have visceral opinions about any of these models because, to be honest, I don't use them in my day-to-day life most of the time. I write English well enough that I haven't yet seen an LLM that does a better job of things than I do, and I haven't yet found useful things where I would need an LLM's help. Maybe I will at some point use LLMs and LRMs.

I don't use them, so I don't have the kind of anecdotal experiences that you have. I'm mostly focused on specific systematic studies, with multiple instances of planning problems. We extended PlanBench to look at unsolvability, longer-length problems, scheduling problems, and so on, to evaluate them. Those are the things that give me a better sense of what o1 can and cannot do.

Yes, I must admit I've updated a little bit. I was always in the same camp as you when we thought of them as approximate retrievers, and I'm now starting to see something.

Yes. No, again, my point is that there are 2 different ways of thinking about it. One is that it's not that LLMs became that. How you define LLMs has to be a discussion that we should have. That's why I keep talking about the Stone Soup metaphor, not because I want to play down the importance of o1. It's a great thing, but you do have to decide who you want to give credit to.

My reservations about the reasoning abilities of LLMs were that they were autoregressive, teacher-forced training systems. That was true from GPT-2.5 all the way to GPT-4o. OpenAI knows this. They know it well enough that they no longer call this GPT-o1. You know that. It's called o1. It's a completely different model, and they know that it's not GPT.

All you can say is that it was developed by some of the same people who also developed LLMs. But we can't define LLMs to be whatever it is that OpenAI is producing. We have to have theoretical definitions. My sense is that autoregressive LLMs still have all the problems, but they also have all the advantages because they're very fast. They're amazing, fast System 1s.

o1 is a reasoning model because it adds reasoning post-training as well as reasoning at inference time. Nobody said that wouldn't be doable. It's great that they're able to do it in a very general sense, but I don't think there was ever an argument that AI systems could not do reasoning. After all, AlphaGo is basically a reasoning system. It was just a deep and narrow reasoning system.

The question was whether something more general and broader, but not as shallow as LLMs, is an LRM. That's a good step in the right direction. But it doesn't change what I thought about LLMs, which is that autoregressive models are different. In fact, they have advantages that o1 lacks.

6. Efficiency Becomes The Bottleneck

For example, the cost of LLMs can be much lower. It is indeed much lower. One of the studies, one of the things that we learned in the strawberry paper—for example, the “Planning in Strawberry Fields” paper—is that in some cases, you have to think of computer science as eventually being about efficiency and cost too, right?

If you give a particular instance of a problem to o1 and pay this many dollars, versus giving the same instance to an LLM with a verifier in this inference-time scaling approach—what I would call LLM-Modulo, which is a general approach that we've been pushing—

The LLM modulo approach, which uses an autoregressive LLM to generate many candidates and an external verifier—or even an LLM-based verifier or a learned verifier—to check them, can actually be cheaper than o1 just doing 1 candidate with the same accuracy.

That becomes interesting because part of what is interesting about human civilization is that, on one hand, we are general-purpose reasoners. But on the other hand, we also know that every job requires a tool, and we do that too. The fact that we do everything that a particular specialized tool does can be extremely inefficient in terms of the time that we are spending. That is going to be the case for these reasoning models too, to some extent, because o1 actually costs quite a bit right now.

How much that is going to change is anybody's guess. In fact, there was Shep Houck, Houck writer, the LSTM guy.

Yeah. Coming here on Friday.

Oh, great. That's great. So you should ask him too. Yesterday, I was in his talk, and he basically had one slide that said, “The bitter lesson is over, and efficiency is going to matter.” I completely agree with that. I've been arguing this for a long time too, so think about the following thing.

The first time we sent a man to the moon, cost was not a consideration. We wanted to show that we could do it. NASA was the one doing it. The second and third times, et cetera, may be okay, but by now it's Elon Musk sending people into space and supposedly possibly to Mars too, because the cost matters.

Essentially, once it's been done, you start caring about the cost that you're paying. Computer science is actually quite a bit about the unsexy parts of cost, just as it is about doing things that haven't been done before. We are now in the second phase, where we are actually going to care about how much we are spending—in terms of the pre-training cost, in terms of the inference cost, et cetera—and whether there are better approaches that we could be using.

This has been the case with computer science before too, and it became less of an issue for a while because LLMs were just systems where there was no real inference-time cost. Even though pre-training was very costly, inference time was very cheap, and so we didn't have to worry about it. Now we will worry about it.

One of the funny things—the elephant in the room—for our PlanBench problems on o1-preview was that the normal classical planners that are meant to solve these problems solve them in a fraction—I mean, such a small fraction—of the cost. They work on our laptops and solve all the problems with 100% guarantees.

Yeah.

Right? So the question is: I realize they are completely specialized only for that problem, but on the other hand, you have this very general-purpose thing that has cost as well as inaccuracies. We start worrying about the trade-off. At what level in this generality-cost spectrum are you going to find a home? That is going to be a very important thing, and I think that's sort of what Shep Houck writer was hinting at when he said, “The bitter lesson part is over”: you do actually need to worry about the cost you are spending to achieve a goal.

The first time you're achieving that goal, nobody cares about the cost—

Yeah.

—because it's never been done, so you're doing it and you get all the credit. But the umpteenth time it's being done, because it becomes a normal day-to-day thing, then the efficiency aspects matter.

A few things on that. First of all, with o1 Pro, I think it's worth $200 a month, and you can call it 100 times a day. Of course, the API is very, very expensive.

Mm-hmm.

But I'm already spending over $1,000 a month on Claude Sonnet 3.5. You raised an interesting point. First of all, the utility of an o1 model is a bit of a weird model, right? It's useful in certain specific circumstances, and if anything, because of the verbosity, the distractors, and the context, it's not really a model that you want to be using most of the time.

But that raises the pragmatism, architecture, and efficiency issues that you're speaking to. I spoke with some guys this morning, and they have built a kind of neuroevolution approach to designing multi-agent systems. At the moment, we hack in the tool use. Do we use a debate pattern? Do we have a small model and prompt it a lot, or do we use a bigger model? We're all just hacking together these multi-agent architectures, and some of those architectures will even be doing the kinds of things that you're speaking about.

Rather than trying to convince you that it got the right answer, there might be a supervisor agent which does some reflexive checking, and there might be another agent which generates the symbolic planning code and runs it on a tool.

Mm-hmm. Mm-hmm.

So we're building these big, complicated things, and I think that's the process we need to figure out now: building the systems that actually use this technology in the best way.

Yeah. I sort of agree, but one thing I want to point out—a distinction—is that there are these 2 notions of using these kinds of models.

When you do a subscription model, $20 or $200, I would argue that is by definition human in the loop, with the model being an assistant to you. It's a very different way of evaluation, where you were unhappy with the previous model because it was wasting more of your time and it wasn't worth it. For you, this one was helping in whatever you were doing, and you are happy with that. That's 1 particular type.

In general, I've always thought—and I think we talked about it last time too—that large language models, and large reasoning models now too, are all intelligence amplifiers. There's no question about that. If you want to use them, you use them, and people are able to find uses for them. That's great.

The part I'm talking more about, and that's been most of our work, is that there would be scenarios where these become the end-user-facing systems, where they'll make the decisions. They will just say, “This is the answer,” and then you're going to execute this plan. So the robot will execute this plan, or this is the travel plan for which I'll buy the tickets.

You don't get to come back in and say, “Oh, I don't like this travel plan.” That's what you do in the subscription model. But the one I'm talking about, basically the API access, is what all the startups that are trying to build additional tools on top of these models are going to use to provide specific autonomous functionality.

That's where I'm talking about the actual computational cost versus benefit for a certain level of accuracy at end-user time. Both of these are very different kinds of uses, and I have no question at all in my mind that all LLMs, and definitely LRMs too, are just great intelligence amplifiers. But that's not what my worry is.

My worry has always been that people are trying to put this in end-user-facing situations where they'll actually make the decisions and some executor just executes them without pushing back. When that happens, the guarantees matter, and the brittleness of the reasoning matters.

If you are in the loop, you would never blindly use the assistant's ideas. If you have an assistant, you may fire the assistant if they are giving mostly bad ideas, but you will never blindly use the assistant's ideas, right? The buck always stops with you. That's a very different way of using LLMs than when the LLMs are the ones that the patient talks to. There's no doctor between the LLM, LRM, and the patient, in which case their accuracy matters, and the cost of getting to a certain level of accuracy matters.

These are 2 very different uses, and I'm much more interested in the second use than the first use.

Can I push back just a tiny bit? First of all, I completely agree with you that, used autonomously, these things don't work.

Mm-hmm.

They don't work for all of the reasons that you said. But that's not how they're being used, and they're not being used like that because they don't work. What we are seeing is that all of the successful reimagining of applications with language models is completely interactive, so they have a human in the loop, and the human is supervising, augmenting, redirecting, and so on.

The next step that we haven't seen yet, but we're starting to see, is having autonomous agent-based systems with multiple levels of reflection, checking, and so on. For example, it could be a bunch of agents generating programs. It could be contributing to a library of programs. The programs are being supervised not just by you, but by other users of the application, and the whole thing just grows, and it's a living ecosystem.

So there's some diffused form of human-supervised verification.

And maybe in the future, the humans might be increasingly taken out of the foreground.

I think that's a very sane way of using it, but I'm afraid that's not the only way it's being used. So, actually, there are 2 issues. One is, if that's the only way, I'm very happy because it's like a tool, and you would use it, and the onus is still on you. Finally, the buck stops with you because you are in the loop, right?

But most of the imagined uses, at least from where I sit and the kind of startups that I hear from and the kind of papers that I'm even reading, are all about autonomous uses. And that's where I'm looking at the fact that there is more promise than before. It was very brittle before. It's less brittle now, okay? But it is less brittle at the expense of cost.

It's actually interesting that the evaluation strategies for both of these are quite different. Evaluating assistive technologies is very different from evaluating autonomous technologies. And it's not that assistive-technology evaluation is any easier, in fact. You can basically say the evaluation is just whether people are buying it and keep paying for the subscription; that's proof that people seem to be getting some value out of it. But it's actually pretty hard to correctly evaluate assistive technologies, and that's a whole area.

Ultimately, all of this is what we are interested in: irrespective of whether you believe AGI is coming next week, next decade, or next century, everybody in AI eventually wants these autonomous abilities to actually make intelligent action with guarantees, sort of, right? And that's basically where I think we'll get to. But prematurely saying that whatever currently is there is already working is what a bunch of us are worried about, and that's what we're pushing back on.

With humans in the loop, it's a completely different thing. Even for code generation right now, there are 2 different uses, essentially. It's not just idea generation for the human. If it's idea generation, it's great, because somebody else's job is on the line. The buck still stops with the actual programmer.

So I think that the autonomous one is the one that I care about, at any rate, and that's the one that I'm worried about: the premature declarations that they're already autonomously intelligent. But I'm generally very happy that this technology exists as a human-in-the-loop technology.

It's kind of interesting for me, sitting here, to hear you say that you actually, as a user, seem to be more of a regular user of LLMs than I ever have been. It means something to me when you say that you like o1 more than you ever liked o1 Preview, and you were kind of okay with GPT-4, maybe, but now you like o1 a little more. So you're basically getting value out of it, but you still have the red switch. You can decide not to take its answer, you know.

o1 Pro.

Yeah, okay. o1 Pro. Okay.

The only difference is that there seems to be a qualitative improvement when it thinks for a long time. I wanted to get your take on something else. We're seeing—I mean, you had your LLM-Modulo architecture, and then we've got this huge approach of test-time scaling, this kind of green-blatting approach. So you green-blat the model, and you get it to generate loads and loads of Python functions.

Mm-hmm.

And in a way, this is the sort of thing that we like because we like programs.

Yeah, the ARC thing. Yeah, yeah, yeah. green-blat approach. Yeah, run green-blat. Okay, fine.

But we're seeing that in lots and lots of different ways. So, doing loads and loads of inference, and then we've got these Python functions, and maybe we do library learning and remixing. We're in the world of code, so we're using code. We're generating an explicit function. We can verify it. We love that. We're in a very happy place.

But now we're seeing an interesting shift. Certainly on ARC and on several other papers, people are moving towards this idea of transductive active fine-tuning. And that simply means, rather than generating an explicit Python function and doing that loads and loads of times, let's just generate the solution directly, just using the neural network.

Mm-hmm.

And this is a step away because we like programs.

Yeah. Yeah.

Because programs are Turing complete, and we understand what they mean and everything. And now there's a whole load of people who say, "Actually, the neural network can just do whatever the program does. Let's just let the neural network output the solution directly." What do you think about that?

7. Programs Compete With Direct Solutions

To be honest, I haven't followed that work as closely, so my answer is somewhat more generic. I would be surprised. I would have the same bias that, in fact, there's the old saying: Why write programs when you can write programs that write programs? That's the version that we're talking about. Basically, you want to generate higher-level code that generates the solutions.

This has always been the conceit of computer science. I don't actually know specifically the work that you're referring to in terms of just going back and directly going for the solutions. Honestly, in the context of inference-time scaling, one interesting question is that you generate loads and loads of candidates. The candidates can be either direct solution candidates or code candidates, either way, and then you still have to have a verifier. If it's code, you need to have a code verifier. If you have a solution, you need to have a solution verifier.

One of the interesting questions is, where are these verifiers coming from? One of the more effective ideas that we've been pursuing is that you can essentially generate verifiers. Of course, there are symbolic verifiers that might be there for specific things, and we can use those in LLM-Modulo-style frameworks. But you could also use learned verifiers, where you essentially learn, discriminatively, what is a solution versus what is not a solution.

A third idea is to generate the code for the verifier and then correct it. Actually, at least in our case, it seems to be promising. We are working on some things that are going to come out soon. But basically, I still think that, especially in the context of LLMs, it's a very different thing if you're not having LLMs in the loop at all. That's a different question.

But if the LLMs are there, one of the things they're actually good at doing is outputting code as well as solutions. In that case, the code can output lots and lots of classes of solutions, which can be verified by the code. If you correct it once, then it will work for a longer time, in essence.

So I would still think that, at least for the inference-time-scaling verifiers case, that seems to still be a good idea. I don't quite know the specific context from which you were saying that people are saying directly guessing solutions would help. I'm not sure whether they still have an LLM in the loop, or they're just saying, "We'll just directly train a separate neural network."

Well, I'll sketch it out. Solving ARC: they have 2 Llama 8B models. One is generating Python programs, and they green-blat it. The other one is trained separately just to output the answer grid directly.

And in both cases they do inference-time compute, so either generating lots of Python programs or doing active fine-tuning of the direct-solution one by augmenting the test-time examples. What they found is, on the Venn diagram of their success rates, for some problems the program works really well—the green-blat approach. For some problems, certainly things like mosaics and spatial, perceptual-type stuff, the transduction works really, really well.

Okay.

And this is kind of weird, because if you think about the space of functions that the neural network could reason about, they should be the same. So I don't know whether it's just because of limitations in the neural network, characteristics of the problem, or something that you've seen.

To me, interestingly, again, it depends very much on the space of solution configurations versus the space of code configurations.

There are many problems where solutions might be of less, quote-unquote, “syntactic complexity” than the code configurations. A neural network that can guess a string may not be able to guess something that looks like a syntactically correct Python program, right? LLMs actually can do the latter. And so it is interesting that if you can do that, and if you still go back to a neural network to directly guess the solution, it may be a more useful step. Again, the stuff that we are doing for the verification thing is still in the initial stages, and we haven’t actually checked whether this kind of trade-off would exist. So I have no more insights specifically on why that might be happening.

Wonderful. What are you doing at the conference this week?

It’s fun. I’m just here today, and I think we did this Chain-of-Thoughtlessness paper. When we wrote it, I kind of said, “It can’t follow procedures, so I should be able to show it.” But now I actually explain the whole thing the way I explained it to you here in the beginning: essentially, go from prompt augmentation.

Schopenhauer said, “Life must be lived forwards, but only makes sense backwards.” Papers also only make sense backwards. After a while of writing, you actually look at it and say, “What I really want to say is that the reason chain of thought is not a great idea is because you really want to think in terms of prompt augmentations, and humans coming into the loop becomes less important.” That’s what we did, and then I’m actually going to this compound AI systems thing and having a great time. There are about 16,000 people, and I’m running into lots of old friends and so on.

Yeah. One of the best moments from the last interview was when you were talking about that paper, saying that you can teach someone to catch 2 fish, or 3 fish, or 4 fish.

Yeah, that one. I mean, that’s basically because it doesn’t quite know how to generalize. I made the point that, essentially, you have to give it examples for 4-word problems, then give it examples for 7-word problems, then give it examples of 9-word problems, et cetera, and then try to improve it.

The conceit is that, when you say this, people will say, “Oh, it must be doing procedure generalization.” The interesting thing, again—we had this conversation last time too—is that the way I look at this, I’m skeptical only because I have some additional background. One of the things is that John McCarthy, who was one of the founding fathers—the guy who coined the name artificial intelligence—basically said, “The holy grail of AI is an advice-taker program.”

Yeah.

Advice-taking is AI-complete. If chain of thought is able to make LLMs take advice, that would be pretty impressive. I went in thinking that there had to be holes there, and that is where the “1 fish, 2 fish” thing comes in.

The more interesting thing, I think, is de-anthropomorphizing LLMs and trying to think of them as alien entities for which arbitrary prompt augmentations can generate good behavior. One example of this that people should be thinking about is jailbreaks on LLMs. Jailbreaks are when you give a normal prompt and then give a particular, carefully constructed learned sequence. Zico Kolter’s original paper—his group’s original paper—shows that the sequence makes no sense to humans, but it will make most LLMs provide a deterministic behavior, like saying, “Got you,” or something of that kind.

Essentially, that should tell us that they’re not seeing language, and so the prompt augmentations don’t have to make sense to humans in the loop. That’s okay. In some sense, the only chains of thought that sort of made sense to humans were giving this false impression that somehow LLMs are doing things the way we do, but that’s not the way it is. We might as well go with what they can do and optimize directly, which is what inference-time scaling and post-training methods seem to be doing.

Yeah. The one thing I get stuck on is that we can criticize individual LLMs. I mean, yeah, they are approximate retrieval engines. My co-host, Keith Dargar, is always at pains to point out, theoretically, that they’re not Turing-complete.

Mm-hmm.

They’re finite-state automata and all of this kind of stuff. But the thing is, it all breaks down when you talk about LLM systems. Even with the chain-of-thought thing, I could have another supervisor model that could generalize the prompt to go to 5 fish, 6 fish, and so on. We can easily build systems that overcome all of these criticisms. So at some point, does it just seem like we’re making criticisms that can be easily overcome?

8. Compound Systems Change The Game

No, actually, it’s a very good point, sir. After this, I’m going to this compound AI systems meetup, and I’m a big believer in that whole direction. But there are some people who don’t want to believe that. The usual LLM aficionados don’t.

In fact, it’s a very interesting thing that OpenAI was at pains to point out that o1-preview was a model, not a system. It’s not you and me saying it; it’s them saying it. They would like to say there is this one-size-fits-all model that will do it, and so it is reasonable to take their word for that.

But in parallel, I also like the compound systems work. LLM-Modulo is a compound system, and it basically improves on the limitations of LLMs. I’m completely fine with it. Again, it doesn’t matter to me as long as I can give guarantees and use it in safety-critical scenarios. I don’t have that bias. But if you are saying a single model will do it, I will take you at your word and then see whether or not that’s true. That seems like a fair thing to me.

Why do you think Google has completely embraced hybrid systems, while OpenAI is really clinging to this single model that does everything?

I think they’re slowly changing that, but I think there was a reason. Again, to some extent, I can understand it. The idea is this anthropomorphization again: we only have 1 brain. It’s not that we have a brain for eating and a brain for something else—just 1 brain, right? So it would be nice if what we are trying to do would somehow be this one-size-fits-all, general system.

At the same time, there’s also this issue of whatever I do, I want to provide guarantees and safety so that it can be used in safety-critical systems. The problem is that modern AI, neuroscience, and cognitive science are not one and the same, right? Everybody understands that. Neural networks themselves are not really that well connected to the brain; essentially, they’re biologically implausible, and LLMs are definitely not biologically plausible.

There’s nothing wrong with that, just like we say planes don’t have to flap their wings. We don’t try to make sense of planes and birds in the same sentence because they both fly. Other than that, the mechanics are different. The flight equations are not at all exactly the same. That’s going to be more of the case with LLMs too, and as long as we realize that, it would be good.

I think OpenAI originally hoped—my sense is that a bunch of these people were hoping—that we would get 2 birds with 1 stone: we would get AI systems as well as understand how the brain works. But I don’t really think anybody believes that part, honestly. You might use these systems to improve our understanding while doing neuroscience. I think Seung Kim basically says that, obviously, these systems help in doing neuroscience research, but they’re not actually telling you how the brain necessarily works.

That might explain why OpenAI and some of these people are sticking to this approach. But the kind of conversations I’ve been having on the sidelines at the conference already—the companies, the startups, and so on—they’re already going much more into hybrid systems and much more into compound systems.

Yeah.

That would basically not be a single system. OpenAI is also slowly coming up with these fine-tuning models. They have this RL fine-tuning stuff for your specific kinds of scenarios, et cetera. So it will be interesting to see.

Just going back to your original idea, I think compound systems are very different. The individual role that LLMs have to play is much less demanding.

Mm.

In fact, one of the fun things is that we can do LLM-Modulo with normal LLMs, or LRM-Modulo with, instead of an LLM, what I call o1. The generation of candidates is costlier, and we actually show in the Strawberry paper that we can further improve the performance of o1-preview on some of the problems, even though we couldn't change how much time it takes to think.

We could improve its performance and accuracy quite significantly simply by calling it multiple times with better criticisms of the problems and answers it gave. So that is still using them in a system. LRMs themselves can be used in a system. But I think o1 and OpenAI itself just want to call them models up until now. Let's see what happens.