超越表面统计（Apple 研究员）[Iman Mirzadeh]

I think if someone wants to take one important message from this talk, it would be understanding the difference between intelligence and achievement. The field is currently focused very heavily on achievement, numbers, and accuracies instead of trying to understand what an intelligent system means and what it means for a system to understand and reason instead of tying it to a certain number for a certain benchmark.

I think you would agree that we need to have better abstract world models, right? We need to have better representations. How is that going to work?

To me, it looks nearly impossible to build an intelligent system that operates without an abstract model of the environment, the world, and knowledge. But right now, there are many questions that need to be answered before that. One of the issues that I have with the literature right now, including my own work, is that we don't even have basic answers to these questions.

After AlphaZero came out, chess became more popular, and the quality of chess improved—not because people used chess engines and simply memorized moves, but because they tried to understand what AlphaZero and other chess engines were doing. Grandmasters actually use these tools a lot, but they don't memorize the moves generated by chess engines. What they do is develop theory: they work on novel openings, novel moves in certain positions, and novel strategies. They learn from these tools, not by simply memorizing. They understand and create new knowledge and new theory.

In image and computer vision, we had benchmarks like ImageNet and all those other benchmarks. We saturated them and thought, “Okay, vision is solved.” But now we see that self-driving cars are not becoming a reality because the real world is very difficult. There isn't a specific, frozen cut of reality with a fixed number of examples. You have to build an agent that understands and reasons. That's why I think it's important to focus not on the exact number.

You know, we all accept that there are limitations in reasoning in LLM systems. Some people say it's just because there's a missing feature or a missing tweak: we just need to modify the Transformer architecture, and then the models can do copying, counting, and reasoning. Other people say all we need to do is let them access tools. For example, if I want a system that plays chess, it can just use a chess computer. What could possibly go wrong with that?

That's one of the most common arguments I hear and debate with many friends and colleagues, including people at NeurIPS. I don't personally have any issue with using tools. Humans also use tools; everyone uses tools. But going back to our previous part of the discussion, it's not about solving a task or achieving a certain accuracy on a task. It's about understanding, creating novel knowledge, creating novel goals, and achieving those goals.

There are 2 arguments I have against tool use. First of all, it's not just 1 tool. If your only goal in the world is to do well at chess and beat some humans, then yes, you can look at a chess engine as a tool. You give moves to it, get moves back, play chess, and win. But it won't be just 1 tool for your whole life; it will be many tools.

If you look at reasoning tests such as solving a mathematical or logical problem or planning, they involve many steps and many states that you need to navigate and learn. Even with tools, you won't necessarily use only 1 tool. You will need many tools. At some point, you might need to use 5 or 10 tools, and the tools may become complicated. They might not simply be a case of giving input and getting output; they may be complex. Essentially, it reduces to the same problem. What if I need to plan 10 different actions? At that point, you still face the same problem.

For chess, there is also the argument that using a tool is fine, but you may not be able to know whether the system that uses the external tool understands the task. You could use a chess engine, play, and win, but you won't necessarily understand why you made those moves, why a certain move is appropriate in that position, or what constitutes a good or bad position for planning.

I'm a huge fan of chess. I watch chess competitions and follow what's going on in the chess world. After AlphaZero came out, everyone thought that chess as a sport and competition would lose its importance because there was a system that could beat any grandmaster in the world. What happened instead was that chess became more popular and the quality of chess improved.

It didn't improve because people used chess engines and simply memorized moves. It improved because people tried to understand what AlphaZero and other chess engines were doing. People even wrote a book about understanding AlphaZero's moves. Famously, AlphaZero sometimes pushes from the left or right corner, making moves that had not been seen before.

You could say, “I don't care; this is a move that gives me an advantage and wins.” Or you could understand why the move works. Those kinds of moves improved chess. Grandmasters use these tools a lot, but they don't memorize the moves generated by chess engines. They develop theory: they work on novel openings, novel moves in certain positions, and novel strategies. They learn from these tools, but not by simply memorizing. They understand and create new knowledge and new theory.

I think this distinction between how humans use chess engines and how people think we should use chess engines is very important. If someone claims that an LLM can use a chess engine as an external tool, I'm fine with that. If you want to develop that system, that's fine. But my metric for measuring how good that system is would be whether, after using the chess engine for a while, it comes up with a theory.

One example is the principle in chess that, during the opening phase, you should control the center. Controlling the center gives you control over more squares, allows you to access more squares in subsequent moves, and prevents your opponent from developing their pieces. Developing pieces and controlling the center are principles of chess.

If you start with any system and a chess engine, and after a while the system comes up with why controlling the center is important, then using a tool is fine. A model could be trained on all the games and learn, as a distribution or through statistics, that White's first moves are often e4, d4, or c4—the central pawns. But the reason behind that is that the player wants to control the center.

If the system comes up with that new knowledge and understands why it matters, then using a tool is fine. But if it simply uses a chess engine, gives and receives moves, and has no understanding of what's going on, then all it has done is achieve a high chess rating without really knowing what's happening.

How puritanical should we be here? LLMs act as if they are reasoning in many circumstances, but they are prompt-dependent. You give them the wrong prompt or put distractors in there, and they fall apart; they don't work. How do we interpret that? Do we say it's not reasoning because of the brittleness?

The short answer is yes. The way I look at prompting, and the reason I don't necessarily think that exploring different prompts helps us really understand or improve these systems, is that I look at LLMs as systems that know many distributions. Prompting looks to me like conditioning those distributions or nudging them to steer the model in a particular direction.

You can use a prompt to do almost anything you want, but that doesn't mean the system understands what's going on behind the knowledge. My problem with the classical way we train these systems is that we look at everything as a distribution. When you learn a distribution, our loss function is designed to learn a distribution that minimizes the distance between the model's distribution and the distribution inferred from the data.

Once the model learns that distribution, that's all it cares about. By construction, these systems won't be able to understand what lies beyond that distribution. We train the models to minimize a loss and stay within the boundaries of a distribution. That's what leads to all these problems: you change something, and the system breaks; you want to measure how the model performs on a different distribution, and it can't do that. By the way, I don't have answers for how to fix this right now.

How do you expect a model to stay in a box and then ask it what happens outside the box? You constructed the system that way.

Another issue with the way we train our models is the use of cross-entropy loss. Let's say you're teaching a model arithmetic: “2 + 2 =”. Assuming 2 is a token and plus is a token, the model sees 4 in the text. All the model has to do is learn that after “2 + 2 =” it should output 4, perhaps given some context.

It doesn't matter to the system that 4 comes from understanding the natural numbers, the number line, and addition. Can we do this, or is it simply memorization? There is nothing in our loss function that says it's important to understand natural numbers as a concept that can later grow into rational numbers, real numbers, and all these other concepts.

All the model has to do is give me 4 after “2 + 2”. Because of that, we don't know whether the model can develop that kind of understanding, build a world model, or form concepts. Those are the main issues and, I would say, the consequences of how we train these systems.

You said to me that you're overwhelmed by the lack of progress in the sphere of intelligence in AI research at the moment. Tell me more.

There are 2 sides to this story. On the 1 hand, we're living in a very important and exciting moment. There has been a lot of progress, many people are investing in AI, and a lot of research is being done. From that perspective, it's very exciting.

On the other hand, in my opinion, the way we're doing research in the field is not necessarily optimal. The way we've designed our metrics and the way the systems work in general are not necessarily optimal.

The current way of doing research is that most people believe research is always an incremental process. I don't disagree with that: you will always build on top of other people's work. My issue with how research is being conducted is that, even by reading new papers and learning about all the latest work, we still don't have a better understanding of how these systems work than we did 2, 3, or 4 years ago.

We don't know whether we're moving in the right direction. Most papers these days have a hypothesis: if you do this, then this will happen. The experiment shows that if you do this, then this happens. Then another paper says that if you do this, something else happens. Eventually, another paper says that all those papers made a particular assumption, and if you have a different assumption, something else happens.

After reading 10 or 20 papers, you still don't have a coherent picture of how these systems work or what the general hypothesis is. What's good about fields such as physics is that you have a hypothesis: “This is my model of a Transformer. This is how I think a Transformer works.” Then you say, “If my hypothesis is true, these consequences should follow. I'm going to design an experiment and measure them.”

That's not how things work in our field. People look for solutions before understanding the problem. I hope people spend more time understanding things rather than focusing on solutions.

I think it's more important to understand what is happening now: what the problems are, the pros and cons of these systems, and how these systems work. Once you have at least a hypothesis or mental model of how they work, you can build on top of that.

Otherwise, you might encounter research suggesting that changing a prompt improves performance. You could say, “My model of prompting is that it nudges or conditions the distributions. If the model is interpolating over a few distributions, then this will happen.” You could then run tests of that hypothesis.

Instead, someone explores what happens if you tune a language model, and even if accuracy increases by 5%, you still don't know what's going on. After reading work that increases model performance by a few percentage points—and all the other work in that area—you still don't have a better understanding of the models. That makes it very difficult to improve the systems.

There are other aspects as well. From a theory perspective, I'm not an expert in theory at all, but I think our theory may be too rigorous for the moment. The problem is very hard, and current theory lags far behind the state of the art in practice. Researchers want to remain very rigorous and develop the theory carefully, but it may be difficult to make progress if the theory doesn't become more relaxed—relaxed enough to move forward and make faster progress.

Those are the main issues. Given the amount of interest and investment in AI, many people are working in the field, which is really great. But there are other challenges, such as the reviewing process. Peer review is really difficult and noisy. Unfortunately, I don't have a solution to propose here, but we can at least think about these problems and how to improve those processes.

Those are the main challenges in the AI literature and research community right now, in terms of research and how research is conducted.

How can we bridge symbolic AI and connectionism?

There are a couple of ways to think about it. You could have a symbolic model. I'm not saying right now that a symbolic model, versus a nonsymbolic model or planning, is good or bad. I'm thinking more about a fundamental question: given any system, is there a way to understand whether the system understands a concept and is able to reason?

In general, I think it's fine to start by integrating these 2 approaches as separate modules. Eventually, by combining the 2 fields, they may become more integrated and improve over time. But it's very important not to look at them as 2 separate systems. Otherwise, it leads to problems involving external tools and all those other issues.

There should be some notion of understanding. The model itself should be able to have a world model and understand whether it needs to update that model.

Imagine that we're talking and I tell you that the area of a square is πr to the power of ½—or something like that, π times 2r. You wouldn't accept my argument. You could prove mathematically that it's not correct.

But if I tell an LLM that the area of a circle is πr to the power of 3, because there is no module in the system asking whether this makes sense or what it believes about the area of a circle, it might accept it. Ideally, it shouldn't accept it, but we don't have such a system.

That's 1 example of why we need another component. We're missing at least 1 component that, over time, develops its own belief system and knowledge representation. When new information arrives, it may update its belief system or disagree with it.

I wouldn't accept that the area of a circle is something else simply because you told me so. If you have another proof, I'll read it, but I would doubt that the proof is correct. That's very important. But there are other belief systems that I'm open to changing, such as the best place to eat in Vancouver. I'm open to trying your suggestions.

The system should not be separated into 2 components. It should be an integrated loop.

I think it's interesting that you say humans don't always reason, but we're capable of a privileged form of cognition that we might call reasoning. You could also argue that LLMs have a spectrum of modes. Sometimes they're doing surface statistics, and perhaps in certain circumstances they're doing at least an approximate version of reasoning. There seems to be a spectrum. Tell me about that.

If you look back, we can admit that what these systems are capable of today has surprised the field. But sometimes we read too much into what these systems are capable of doing.

Regarding the claims that models can reason and that there is a spectrum, I'm not sure about that. What these systems are good at, I think, is some form of interpolation. It's not that they're simply memorizing things or that they can't do anything beyond the training data. They learn many things from different places, learn different distributions, and interpolate between the things they've learned.

I think that's what creates the illusion that these systems are capable of reasoning. If the domain is very limited and closed—for example, a specific domain where the overall space is closed—then the model's interpolation may be enough to make it seem as if the system is capable of performing in that domain.

In general, though, there is a huge distinction between the achievement and capability of a system, and the intelligence and other capabilities of a system. These are different, but we currently mix them together.

For us, if a model performs well on a coding benchmark, it means it has reasoning. If it does well on a math benchmark, it means it's doing some sort of reasoning. Mixing these things together makes it very difficult to discuss and understand the systems.

It's important to distinguish intelligence from achievement. Intelligence is about the capability of a system and how it can grow and eventually become capable, not necessarily how capable it is right now. Achievement is about measuring what the system does on a specific domain, task, or benchmark.

These 2 things are different. If a system is intelligent, eventually it will be able to do well on some benchmark. But the other direction is not necessarily correct. If a model performs well on a benchmark, that doesn't necessarily mean it is intelligent.

Intelligence is difficult to measure; it's an open question. We set a benchmark and say that an intelligent system should perform well on it. When a model performs well, we say it's doing reasoning. Those are different things.

Intelligence is about how a system is able to perform over the long run. There was an interesting figure in Ilya's talk at NeurIPS this year. It showed a regression line between body mass and brain mass and how different species scale in that sense.

My takeaway from that figure was that intelligence is not about where you are on the scale right now. It's about the slope of the figure. If you're thinking about scaling laws, intelligence is the slope of scaling, not the point at which you currently are.

I love all of this. You probably know that I'm a huge fan of François Chollet. He's always at pains to say that intelligence is not skill; it's adaptation to novelty and skill-acquisition efficiency. There's something about macro-adaptability that's important.

When we were discussing this before, you came up with your own test for intelligence, which I'm going to call the Iman Moon Test. Imagine we start with cavemen and land on the Moon. How quickly can we do that?

That came from a thought experiment. To reiterate the distinction between intelligence and achievement, imagine a caveman. If you go back in time and give MMLU or GSM8K to a caveman, I would argue that the performance would be near 0%. Current models achieve roughly 80% or 90% accuracy on these benchmarks.

Do we really believe that the current systems we have are more intelligent than a caveman? Intelligence is about the system a human possesses, not how well that human performs on a benchmark.

Even if you go further and consider great figures in history and science, such as Aristotle, I don't think Aristotle would perform nearly as well as current LLMs on MMLU. But do we really believe that the LLMs we have are more intelligent?

Intelligence is not about how much you achieve on a certain task. It's about how well a system can learn and grow when it spends time learning something.

How could we measure this? Shane Legg had a formalism for measuring intelligence, but it wasn't computable. Benchmarks just get saturated; they get gamed. How can we build a new suite of benchmarks?

I don't have a concrete answer to that. I'm thinking about it a lot, but I don't think I have an answer right now.

We could start with something that isn't necessarily objective. We could ask what kind of systems we want and what intelligence and achievement mean to us. If we start with an axiomatic definition and a set of characteristics, it won't initially be objective or concrete, and it may be open-ended. But if we start thinking about these important problems, we may eventually find a way to measure them.

Measuring intelligence is very difficult. Even for humans, it's not trivial to measure intelligence. But we can define some desired properties that we want from a system. Those properties might include the ability to perform well on novel tasks. The definition of “novel” itself isn't easy, but we can start there and build on top of it.

There is a very good paper by Jill G. Knack et al. called “On the Definition of Intelligence.” It hasn't circulated very widely in the machine-learning community because the authors come from psychology and cognitive science, but it's a very good paper. It formally defines intelligence, artificial intelligence, and the kinds of intelligence we're looking for.

We can start from there, think more about these problems, and perhaps come up with something better. In general, I think the direction should be toward measuring performance on novel tasks.

For example, if you're training a model to do coding, a better measure would be how quickly that system can learn a new programming language from scratch. An LLM could perform very well on Python or on all the languages it has been trained on. But if I create a new programming language and ask it to write a program in that language, how quickly can it learn the language, and how well can it write programs in it?

We could compare that with a human who has never seen the language before. One thing we can do is compare and contrast how humans learn and reason with how machines learn and reason. We can move in that direction, but I don't have a concrete answer.

It's very slippery. There's a great paper by Pei Wang on defining intelligence. He said that it's very anthropocentric, and different fields use different techniques.

We could have a copy of a human brain, which is the most anthropocentric approach, but that's not particularly useful. Or we could have notions of behavior and more abstract capabilities, such as being able to write Python programs. We could define functions, having systems that perform planning and reasoning and all these abstract cognitive functions.

Perhaps we could have systems with certain principles, such as emergence or other characteristics. It's very slippery. Before we move off this topic, I'm interested in the psychology paper you cited. What was their definition?

Pei Wang's definition is very much about adaptivity. That seems to be 1 of the core ideas for him.

That paper discusses many other works on studying, understanding, and defining intelligence. I highly recommend it.

The main emphasis of that paper is novelty. Its definition of intelligence is the maximum capability and capacity of a system to achieve a novel goal given some amount of time. Obviously, you can define many programs and systems that would eventually reach a goal given infinite time. But time is also important. That's why I mentioned the slope of scaling.

Do you think agency and autonomy are part of intelligence? The ability to set your own goals? Or should we think of intelligence purely as a system that has a goal and can achieve it?

I think agency and having some concept of what the system wants are very important. I was recently reading a book called “How We Learn: Why Brains Learn Better Than Any Machine... for Now” by Stanislas Dehaene. It discusses many important topics, including how humans learn and how machines learn, and it offers insights from neuroscience.

One of the things that seemed particularly interesting to me was that active engagement is 1 of the pillars of human learning. It's very difficult for humans simply to observe something without getting involved and learn from it. Compared with passive observation, people learn better when they set their own goals, actively engage with the environment, explore, and perhaps later exploit what they've learned.

According to the book, active engagement is 1 of the necessary conditions of learning. It's not simply something that's good to have; it's something you must have.

The implication for me is that supervised learning won't be enough if you want to build a really intelligent system. You need an agent that can ask, “What should I do next?” and ask questions. It shouldn't simply observe and say, “I learned this; I learned that.” It should be able to say, “Now that I've learned this, what do I want to learn? Maybe I haven't understood this well enough. I need to explore more. Let's explore this.”

Controlling the center is an abstract category, right? It's an analogy. Douglas Hofstadter has said that a concept is essentially a bag of analogies. Moving to the center is something we all understand at an abstract level, but what does it actually mean?

Many concepts in chess are quite fuzzy. What they mean in different situations is slightly different. So what's the difference between AlphaZero discovering move 37 and, if you play it millions of times, having move 37 emerge as a mode in its behavioral space?

You could argue that it's doing reasoning, but it's doing an emergentist form of reasoning. Rather than having an explicit notion of “the center,” its behavioral profile acts as if it has such a notion. How is that so different from the way our brains work?

AlphaZero, AlphaGo, and similar systems are able to understand the environment they're placed in. They explore the environment, play many games, and eventually explore most of the positions in Go or chess.

They can eventually make rational decisions because they understand the value of each state. But that doesn't mean they come up with the concept itself.

For a chess engine, the move improves the value of a position. But it doesn't necessarily transfer to other aspects of life. In chess, you could argue that controlling the center means controlling important or strategic positions. Once you learn that, you could apply it to other aspects of life.

It's not simply about 4 or 5 squares in the center of a chessboard. It's about the importance of specific positions in different settings. It could mean controlling an important road in a country or an important passage.

That form of abstraction—and the way we store knowledge and learn—is what allows humans to scale very quickly. Once we learn this in chess, we're able to use it across other domains. Systems such as AlphaGo or AlphaZero aren't able to do that because, to them, the world is only that particular game. They don't have these abstract representations and knowledge.

I'm not saying that learning abstractions is the only way to build an intelligent system, but it certainly helps humans scale and improve much faster than machines.

On the matter of generalizing out of distribution, we're starting to see chain-of-thought used a lot. You can take something trained on a distribution and manipulate it through directed retrieval to create compositions in some fixed way.

There's chain-of-thought prompting, and some people are doing program induction. With a few examples, perhaps with chain-of-thought, you can get a language model to induce a very rich and diverse set of programs, many of which seem to have some kind of abstraction.

We're also seeing transduction, where you do some active fine-tuning. You take the test instances and modify the existing model slightly so that you get generalization in that domain. There are many approaches that seem to work quite well for making these systems perform better. What do you think about that?

There are many ways to explore this, including chain-of-thought and program synthesis. My question, which I've been thinking about a lot, is what assumptions all these systems, methods, and directions are built on.

The assumption is that there is something going on in the model—some kind of understanding—and that we need to improve it. I'm not sure whether that assumption is correct. I'm not saying that it is correct or incorrect; I'm not sure about it.

If the assumption is incorrect, then all the things you're doing—chain-of-thought and other methods for improving reasoning—may not be helpful. We need to take a step back and ask whether there is anything there beyond interpolation between distributions.

If there is, then we can continue building on top of it. If there isn't, what are we doing? Why are we trying all these methods? It's very difficult because we're doing it in an ad hoc way. There is no coherent model of what a Transformer is doing at scale.

Right now, I'm leaning toward the view that these models don't have the correct representation. The way we build these models is limiting, so I don't really see how simply extending them would be helpful.

There's an old example: if you want to land on the Moon, you can do it with an airplane, right? If you have an airplane and want to land on the Moon, you could improve the speed of the plane, make it lighter, make it go faster, and so on. You could work on the shape of the wings or the kind of runway needed for a faster takeoff.

None of that necessarily helps you achieve your goal. I'm not saying it's impossible, but we should ask whether this is the right vessel and whether it makes sense. In general, we should take a step back and think harder about the problem.

On scaling laws, Noam Brown is talking about a new test-time scaling law. You've said it's very difficult to convince people that scaling laws don't work, because there will always be a group of people saying, “Just over the next hill—if we increase the parameters and compute by 10 times, we'll finally get there.” How do you convince people that they're wrong?

I can't really convince them. The issue with scaling laws is that if I spend millions or billions of dollars training a model with, say, 10 trillion parameters on 100 trillion tokens and show you that it can't do simple mathematics beyond what it was trained on and can't create new knowledge, someone can still claim, “That's not the right scale. If you scale it to 50 trillion parameters, something will emerge.”

Because my argument isn't theoretical, I can't say they're wrong. I won't be able to train that 50-trillion-parameter model and spend several billion dollars to prove the point. There will always be a debate about whether something will emerge.

I don't have an issue with scaling in the sense that I'm not saying it will hurt or be completely unhelpful. I'm saying that if you're thinking about scaling, you should focus on the slope of scaling rather than where you are right now.

If you want to compare 2 systems or 1 architecture with another, you shouldn't focus on what happens at 100 trillion parameters or whether something will emerge. You should ask how quickly these models can learn and do novel things.

Right now, in that sense, the slope is close to 0 for all the architectures and methods we have. We should focus on the slope rather than the points or endpoints on the slope.

How does the slope of human scaling compare with LLM scaling?

We haven't been able to measure that, but it would be interesting. There are many studies—I’m not an expert in the field—showing how humans and models respond to novel environments.

If you compare them, I think the studies show that humans can adapt to a new environment and learn it much more easily than LLMs. I don't have a specific number, but I think the slope for humans is faster, or better, than the slope for LLMs.

There could be another species with a better scaling slope, and I look forward to that.

Iman, I think you would agree that we need better abstract world models, right? We need better representations. How is that going to work?

To me, it looks nearly impossible to build an intelligent system that operates without an abstract model of the environment, the world, and knowledge. But there are many questions that need to be answered before we can do that.

One of the issues I have with the literature right now, including my own work, is that we don't even have basic answers to these questions. Imagine that you want to learn a function. One way to represent that function is with a table: the input is x and the output is y.

Another way is to represent the function in a form such as a polynomial: y = x² plus some number. We don't currently have an objective measure for deciding which representation is better. You might say the second representation is better because it's abstract or has a particular form. I also believe the second one is better, but we don't know how to quantify that.

Someone might claim that the second representation is more compressed because you don't need a table; you only need a polynomial. But I don't think that's necessarily correct. Representing something with a polynomial isn't about the number of characters in a string; it's about concepts.

To define a polynomial, you need to understand what a function means, what x² means, what x to the power of n means, and concepts such as continuity and real numbers. If you want to encode everything required to arrive at y = x² plus some number, that representation may actually be larger than the table.

If your only goal is accuracy, referring back to our previous discussion, the 2 representations are equal. You give them an input, and both produce an output. The table is even faster because it's a lookup.

We don't have a way to compare 2 representations. To summarize, in order to answer all these questions, we need to start by understanding how to compare 2 representations. Once we have a good measure for doing that, we can build on top of it. We don't currently have such a measure.

I was talking with a friend about how we could compare these 2 representations. They said they didn't have a measure, but perhaps it's similar to the idea that someone eventually formalized generalization. Two hundred years ago, generalization wasn't a formal concept; it was a word.

They suggested that 1 representation could be more beautiful than another. A polynomial might be more beautiful than a table, but we don't have a formal way to define what makes something more beautiful. These are problems we think about all the time.

So the core question is whether, deep in your bones, you're a connectionist and believe that this kind of thing could work in principle.

You've designed a symbolic test to show that certain types of reasoning don't happen in an LLM. In 1988, there was the famous connectionism critique by Fodor and Pylyshyn. They argued that these systems don't have systematicity, compositionality, or generalization. They don't have interpretability, and they can't explain why they did things or carry their knowledge in the way you're describing.

Other people, such as Smolensky in 1990 and certainly Bengio, argue that this isn't a problem—that symbolic use can emerge at some level of complexity. Do you think that, in the future, we can make an LLM do this kind of symbolic reasoning?

When we talk about symbols, we first have to define what we mean by a symbol. Is a symbol something concrete and predefined? Do you agree that the activation of a deep-learning model could represent a symbol? If you agree, that could be viewed as a formal symbol.

In that case, I don't see, at least in theory, why such a system wouldn't be able to create that kind of symbol. The model could eventually create symbols and perform computations on those symbols.

I'm not saying this is the only way or that we shouldn't have anything else. I'm not being dogmatic in that sense. The same applies when we discuss architectures. I don't know whether the Transformer is the right architecture, but in theory, given the context and assuming the model is generating tokens, it is Turing-complete. I don't currently have a theoretical reason to believe that this is a limitation.

Other architectures might be better, and we may develop better architectures. But right now, I don't see that as a limitation.

So you do think we need Turing completeness for symbolic use? Could we somehow come up with a neural network that isn't Turing-complete?

When we're defining intelligence, at some point, being Turing-complete may be a necessary condition.

We're not there yet, but perhaps.

We should talk about your GSM-Symbolic paper. It's a landmark paper. It did the rounds on social media, and millions of people in my community said that I had to interview you because it was amazing. Sketch it out for me.

We were working on understanding reasoning and exploring ideas for improving the reasoning of models, including increasing the amount of computation a model performs per output token. For that, we needed to evaluate the models, have a robust evaluation, and eventually perhaps create better training data.

We started with the evaluation by creating samples from GSM-Symbolic using templates. As a sanity check, we ran the experiments to see how close the numbers were to GSM-Symbolic and GSM8K. We observed a huge gap for some models: 20% for some models, 14% for others, and so on. There was also a huge amount of variation.

The second thing I worked on, after the initial GSM-Symbolic version, was GSM-NoOp. In that version, you try to trick the model by adding 1 clause to the question that carries no logical or arithmetic operation. If you completely ignore that clause, you can still solve the problem. That's why it's called GSM-NoOp.

That was the second experiment in the paper. We observed a very large drop in performance. Then we tried to understand what was happening by creating something in between. We created easier benchmarks than GSM-NoOp by adding 1 clause that carries an operation, then 2 clauses that carry operations, so the model can't ignore them. We examined how the models performed under those conditions.

Among all the benchmarks and experiments we designed, the most surprising thing to me was the variance in performance across models. I wish we could do some kind of cross-evaluation with students to see how they would perform if we changed the words in a problem without changing the numbers.

For example, instead of saying that Amy has 3 apples, we could say that John has 3 bananas. The logical reasoning is the same. I wish we could study how a human student would perform if we changed “orange” to “banana,” compared with an LLM.

The most fascinating part to me is the variation. Why does this gap exist when you only change the numbers? We have a benchmark called GSM-Names in which we only change the proper names in the question. We don't change the numbers or add anything to the question.

The frontier models still did quite well. I would have expected them to drop off a cliff. Subbarao Kambhampati has done experiments with planning and a Mystery Blocksworld, where he changed the names of the symbols to random things. Performance dropped off a cliff, and the o1 model managed to recover to nontrivial performance, but there was still a huge decline.

It was still quite interesting that the frontier models performed much worse but retained fairly nontrivial performance.

There are a couple of things to consider. Overall, I think GSM8K should be a very simple benchmark. We also don't know exactly what kind of data these models were trained on. It could be—but I'm not saying it is—that these models were trained on synthetic data generated from questions similar to math word problems such as GSM8K.

We also know that companies building LLMs are obtaining many human-created question-and-answer examples in mathematics and other domains. We don't know exactly what data the models were trained on. It could be that frontier models have access to better-quality data in a format similar to GSM8K.

It could also be that they've unlocked some emergent ability. But if that were true, there are other benchmarks similar to GSM8K that are slightly more difficult. Someone applied a similar idea to the Hendrycks MATH dataset and observed a large performance drop for frontier models as well.

Overall, though, it's not about performance for me. I don't prefer a model whose performance drops by 10% when the question becomes difficult but shows no variation over a model whose performance drops by only 1% but shows large variation.

The question is why a system that understands a set of logical steps tries to perform a different set of logical steps and gets the answer wrong when you change “orange” to “banana.” That probably comes from training the system to learn a distribution. There is no concept in the model saying that the object doesn't matter and that the problem is about the number of objects.

Those things seem trivial to us, but for a system trained to predict the exact objects in a question, they are not trivial. The field looks at these things as accuracy numbers rather than asking what it means for the accuracy to drop or for the variation to increase.

The field is now trying to move beyond GSM8K because performance is reaching 95% or close to 100%. Recently, there was a workshop discussing a difficult benchmark called FrontierMath, designed so that frontier models achieve only around 10%. Now the goal is to increase that from 10% to 90%.

My problem with benchmarks is that they're always a cut of reality. Once you freeze that cut, you can change the system in a way that indirectly changes its performance on the benchmark. Eventually, the benchmark will always be saturated.

It doesn't matter to me if a system gets 99% on a benchmark if nothing fundamental has changed. That system may still be unable to create new knowledge or understand what's happening.

In image and computer vision, we had benchmarks such as ImageNet. We saturated them and thought vision was solved. But self-driving cars still haven't become a reality because the real world is difficult. There isn't a specific frozen cut of reality with a fixed number of examples. Reality changes, and you have to build an agent that understands and reasons.

That's why I think it's important to focus not on the exact number.

Benchmarking is a big problem. As you say, I'm sure many frontier models have essentially memorized GSM8K. We should move toward a more generative type of benchmark, where we have a generative system, sample from it enough times, and report some kind of average.

You showed some interesting results. On the GSM-NoOp dataset, there was a change when you used 8-shot prompting, which should have allowed the model to filter out the distractors, but it didn't. More broadly, isn't it interesting that when you sample from these models many times, you see huge variation?

What's the implication? When we use language models for reasoning, should we sample 100 times and take the average result? Many of us use language models and sample only once. We assume, “It's doing reasoning, and that's the right answer,” without thinking that if we asked it another 100 times, it would give us many different answers.

There are a couple of things about sampling. Sometimes sampling doesn't make sense. In the example “2 + 2 =”, there shouldn't be any sampling; the answer should be 4. If you increase the temperature, the model might produce 5, but why would you want that? Randomly picking a number doesn't make sense.

That's why, in our study, we always used greedy decoding rather than sampling. If you're doing arithmetic or reasoning, the output should be greedy.

The other argument against sampling, majority voting, and similar methods is an example I remember from an episode of yours. If you let 10,000 drunk people leave a bar and go home, eventually some of them may reach home. But that doesn't mean they understood what was happening.

Given enough samples, you will eventually reach the destination, but that doesn't mean the system understands the problem. There was actually a paper that made exactly this argument. It said that if you sample a model 100 times on GSM8K and MATH, performance may be 20% better.

But that's not what we should be studying.

Iman, it's been an absolute honor to have you on MLST. Thank you so much for joining us.

Thank you. It was great talking to you.