Transformers Need Glasses! [Federico Barbero]

I think reasoning is just very ill-defined. A computer program will generalize arbitrarily well if you write it correctly, and we can prove that it’s perfect and so on. Is it reasoning? I’m not sure.

These models are probably not implementing algorithms. They’re implementing heuristics that, at least during training, are good enough to fit. But as soon as they go out of distribution, they generalize horribly. I like our results because we give you a quantity that you can directly measure and say, “This is happening in a Transformer.” These other results are more like constructive proofs that you can represent this class of languages, and so on. They’re very related, but they’re also very different approaches to similar issues.

Federico, welcome to MLST. It’s so nice to have you here.

Thanks for having me.

Why do Transformers need glasses?

Probably I should not be allowed to pick titles anymore because I come up with these wacky titles. The key idea behind the phrase is that they seem to be very bad at detecting whether you care about a single token. At some point, this breaks, and it’s something about how they can’t—at least in the limit, as your context size grows—be good at caring about a single token, especially if that token is at the end.

There’s a figure, I think on page 2, where you talk about the representations getting kind of squashed toward the end. Can you explain that figure?

Imagine a family of growing sequences where you repeat the last token. In the paper, for example, we ask: can you count the number of 1s in this sequence? Now you add an extra 1, so the 2 sequences have different answers. The one in which you repeat an extra 1 now has the answer \(n+1\), or \(k+1\).

There’s actually an issue with this construction. At some point, the influence of this final 1 gets lost. That’s the fundamental idea: as these sequences grow, you really care about that final 1, the one that gives you the right answer. But at some point, if you start measuring inside the model, at the level of the representations, you can see—and we also showed this mathematically—that these representations get closer and closer.

Computers have finite precision. If the representations get too close, below the precision of the machine, you’re forced to map these 2 sequences to the same thing. Essentially, 1 of them has to contain a mistake.

Let’s sketch this out. I’m not sure I remember the experiments exactly, but it’s something along the lines of generating a sequence such as 1, 0, 1, 0, 1, 0. You might probabilistically generate the 1s and 0s—say, 70% of the time it’s a 1 and 30% of the time it’s a 0—and then, at the end, ask a question. You might ask, “How many 1s are there?” or “What was the last thing?”

You’re saying that as the sequence grows longer, the network develops a blind spot toward the most recent thing.

What was surprising to me was the way you construct this sequence. Let’s stick to copying, because that’s a simpler example: can you copy the last element of this sequence? This is a completely trivial operation for a human. A human will never make a mistake, irrespective of how long the sequence is. If you’re counting a massive sequence, a human will very likely make a mistake, but if you just ask what the last element is, you’ll never make a mistake because it’s a trivially generalizable operation. You don’t need to perform any computation; you just look at the last thing.

But somehow the machine, or the Transformer—we tried this with Gemini, which at the time was the most powerful model—at some point just fails. If you give it a very long sequence of 1s with a 0 at the end, it starts outputting 1. This very nicely explains that, at some point, the fact that there’s a 0 in the sequence gets lost in the representation.

We measured this in a smaller model, of course, because we needed to have the weights and everything to do it. You can really see the representations converging toward each other.

I think most of us feel that there’s a kind of recency bias. If anything, a model is more likely to know about things that you’ve just said than things from a long time ago. When we look at some of these large models with large contexts, we see a U-shaped curve, don’t we?

Yes. We have to talk a bit about one of the main intuitions in the paper, which is related to this representation-collapse idea. In our paper, we develop it as its own idea: the way information flows through these Transformers has an inherent mechanistic bias. It’s biased toward the start of the sequence.

When you train these models to predict the next token, that next token very likely depends on the nearest tokens. In some sense, the training dynamics push the model to care about the most recent things, but the mechanics of the information flow push it to retain information toward the start of the sequence.

We believe this explains the phenomenon observed in information retrieval, where Transformers and language models tend to find it harder to retrieve information from the middle. Mechanically, Transformers are good at the start, but they learn to care about the end, so the middle is somehow lost.

This is something we should linger on. There’s a figure that shows this kind of topological information transfer through the network. The interesting thing is that the earlier you are in the sequence, the more topological pathways there are to the prediction token.

If you think about the self-attention Transformer, it has causal masking. You should introduce what causal masking is, but it’s basically there so that, during training, the model can’t cheat by looking into the future. Subject to that causal masking, the further back you go, the more pathways through the self-attention matrix there are before the model makes its prediction.

What you’re claiming is that the more pathways there are, the less likely the information is to become squashed.

These ideas are popular in graph-network literature, so they’re not coming completely out of the blue. As you correctly mentioned, the important fact is that we have a causal mechanism in place. Attention looks like a lower-triangular matrix because it only allows you to look backward.

This is really a training trick. You can have Transformers such as BERT that don’t use causal masking, but if you want to train autoregressively and efficiently, you can’t have the Transformer look forward because that makes training super inefficient.

The choice of causal attention has an implication for how information propagates. It’s exactly the number of paths that matters. Intuitively, if you’re the last token, the only way for your information to survive through the attention mechanism is for the model to attend to itself. That’s the only attention that preserves the information from that token.

But because there’s a sum-to-1 constraint, if the model attends strongly to itself, everything else has to be small. This creates a trade-off. If you want the last token to survive, you can’t really have it attend to all the other tokens as well.

I found this to be a useful explanation for many interesting phenomena. That’s why I like this intuition.

There’s an interesting literature here. Your supervisor was Michael Bronstein, right? Or still is, I guess. I love Michael; he’s amazing. He’s done a lot of work on this idea of squashing in graph neural networks, and even before that, people were talking about vanishing gradients in recurrent neural networks.

I think I read in your paper that, in the graph-neural-network literature on squashing, there’s a notion—I can’t remember the word you used—related to something like the taxicab distance. The more steps you have to travel, the more squashed you become.

The paper is taking ideas from this work on graph neural networks and building a bridge to Transformers. I used to study information propagation in graph neural networks, so this connection isn’t really a surprise.

The way people study propagation in graph networks is by relating it to spectral graph theory. It’s also closely related to Markov chains. You get natural quantities such as commute time.

Commute time is a quantity that measures the following. Imagine you’re on a graph and define a random walk. If you’re at a node with degree 5, meaning you have 5 neighbors, you can jump to any of them with probability \(1/5\), and so on. This defines a process: you start at a node and sample a journey.

The commute time asks you to pick 2 nodes on the graph, start at 1 of them, and calculate the expected number of steps required to reach the other node and come back. It’s called commute time because the random walk has to hit that node and return.

This is closely related to heat equations on graphs. That makes sense because there’s a relationship between how heat spreads and how random particles move. It’s a well-studied behavior.

Depending on how you choose a graph network—for example, a graph convolutional network—it can behave similarly to a heat equation. You can view it as a discretization of a heat equation over a graph.

This is well defined, so it’s not surprising that the sensitivity of a graph network to 2 nodes communicating with each other is related to their commute time. People have been studying graph networks in this way, and we had a paper on this showing that simply changing how the graph is connected affects how easily the graph network can make 2 things communicate or transfer information between them.

When you talk about heat equations, are you talking about things such as the second law of thermodynamics and the Navier–Stokes equations, but in the context of a graph where you have fewer degrees of freedom because you can speak to your neighbors? It’s obviously a little different.

You can literally define a heat equation over a graph. You can define a Laplacian operator and evolve it. Studying the spectrum of this operator is a well-established approach.

There are strong analogies between continuous heat equations and graph-based versions. You can view graphs as discrete types of surfaces, and then draw many parallels. For some reason—maybe not magically, because it’s very consistent—you can develop inequalities on graphs and heat equations on graphs that are consistent with continuous spaces.

You can then use these inequalities to bound how a graph network can spread information, or how its information-spreading process operates.

You cited Muse lyrics about how energy dissipates over time.

That citation is more due to Petar, so I can’t take credit.

Petar Veličković is famous for inventing graph attention networks, and he has loads of citations on that. He’s also been talking for years about the potential limitations of Transformers and looking into graph networks to improve them. Do you think there’s an opportunity to build a better architecture here?

That’s the hope. I may be biased because I started my PhD working on graph networks and then jumped ship to language models, but I think there’s a lot of value in studying graph networks. We have a fairly deep understanding of how they work. At least for some models, you can study them quite nicely because of their relationships with physics.

The hope is that we can take these approaches and build a bridge to Transformers. If language models were fully connected, this wouldn’t be as interesting. You would have arbitrary behavior, so what could you say? But Transformers don’t have arbitrary connectivity; they have a very specific topology. That gives you an opportunity to exploit these ideas.

This is why I like causal attention mechanisms. They’re genuinely interesting to study. In our paper, we make some efforts in this direction. They’re relatively small and mostly in the appendix, but I think they’re interesting.

If you can develop a spectral theory for these triangular attention matrices, you can obtain interesting or cute results. For example, we have a result showing that, in the limit, you only care about the initial token. These ideas arise naturally from spectral graph theory.

They’re not necessarily deep results, but they provide a lot of intuition for what’s happening.

There were so many things you said there that I don’t know which one to take first. Let’s touch on the last thing, which is one of the ending theorems in the paper. You said that, in the limit, as the token length increases, the model will increasingly pay attention only to the first token.

We had to make some assumptions. Everything is highly nonlinear, so the main intuition is difficult to study exactly. We were trying to understand what happens if you add more and more layers.

If you assume that you can ignore some of the contributions—which is a pretty big limitation—you can study the powers of the attention matrices. Attention matrices can never be truly sharp. By “sharp,” I mean that you care about exactly one thing.

Suppose you want to understand how much this process mixes information. Because attention can only go backward, the mixing occurs in 1 direction, toward the beginning of the sequence. The more layers you apply, the more the process comes to care about the beginning of the sequence.

There are some very nice results here. I was happy to see a paper by Kini [?] and colleagues, who work on security, where they found that when you repeat tokens, the final output eventually becomes the beginning-of-sequence token.

This was nice to see because, to me, our results explain why that happens. It can seem arbitrary: why should the output become the beginning token? Mechanically, it makes sense from a spectral perspective.

This is why I’m excited about the direction. Even if we make broad simplifications, there’s something there that can improve our understanding. I hope this work could eventually spill over into better security, better attacks, or better defenses. Understanding comes first, and applications come later.

The obvious question is that we now have models with a 2-million-token context. They don’t work perfectly, but they seem to work quite well. Why don’t we see massive model collapse in those models?

Someone has to open up these models and figure this out. Spectrally, they shouldn’t work, at least according to the basic argument.

The issue is that attention can be viewed as a type of contraction, but the Transformer also has other components, such as MLPs, that can act as expansions. There’s something about the balance between these components.

You want to counteract the contracting effect of attention. Residual connections help with this, and MLPs can help as well. Perhaps this is some kind of self-stabilizing process.

A lot of models with much longer contexts alternate between windowed attention. Maybe these mechanisms are discovered by accident through architecture search, and this helps. I also imagine that models with very long contexts learn many mechanisms to preserve mixing. For example, if you look at Llama or Gemma, many of the attention heads don’t do anything. They implement diagonal heads, which are essentially quadratic no-ops. They’re expensive no-ops.

Why is the model learning this? I think it’s trying to preserve itself. It’s trying not to mix everything at once. It’s trying to avoid overmixing in some way.

A lot of heads simply attend to the beginning-of-sequence token. That token has a norm very close to 0, so again, this is a no-op. There’s a huge opportunity to understand this better. It’s a huge waste, and model compression is already a major topic. Perhaps studying these issues can help us understand why models learn no-ops.

A few thoughts on that. This kind of theory can help us understand training dynamics and come up with principled ways to design architectures.

There’s also an element of randomness. If you take Gemini and train it 100 times, presumably some runs will work better than others because of whatever the training conditions are. Sometimes you may need to give the model more capacity, and it may decide not to use that capacity.

Training dynamics also don’t make sense as a completely global process because they’re input-sensitive. Certain inputs may activate more of the network, and perhaps that gives the model more capacity.

Yes, that’s a good way to summarize it. Razvan Pascanu, who is also on this paper, would probably say that you need a lot of heads. At the start, this gives the model many options.

The model may learn that some heads explore in 1 direction while other heads explore in another. Eventually, the heads it doesn’t need shut off, and it preserves the ones that found the best setup for some reason.

You might have 500 heads, or even many more, and this is simply a diversity strategy at the beginning. The model then tries to select the heads it cares about.

You were saying that we design these new network architectures in a slightly hacky way. There are multiple opportunities, multiple heads, multiple pathways, residual networks, and so on. I loved how you described it as expansion and contraction in successive stages.

What’s the intuition there?

The intuition comes from the eigenvalues of these triangular matrices. If you make some side assumptions, there’s a single eigenvalue equal to 1, and all the others are less than 1.

If you take powers of the matrix, you’re eventually left with the single eigenvalue that equals 1. That eigenspace survives. This acts like a contraction.

I’m being somewhat loose with the idea of contraction, but in this sense the process destroys information unless you’re in this specific starting-token direction. MLPs, by contrast, can have whatever Lipschitz constant they want, so they can expand the representation as much as necessary.

I view these 2 operations as something like read and write operations. Imagine that the attention mechanism cares about 100 different things, so each coefficient is roughly \(1/100\). It copies information into a different hidden dimension within the value vector, and everything is scaled by \(1/100\).

The MLP can then take this information and multiply it by 100, making everything strong again. Of course, you’re still dealing with layer normalization and other components, but this gives the model a way to copy information.

The more things the attention mechanism copies, the weaker each individual copy is. The MLP can counteract that by turning up the strength of the copying operation.

When you cited Caron, and certainly in most of the experiments, the sequence had fairly low entropy. It was made up of 1s and 0s or something similar. How much does the entropy of the sequence matter?

It matters. I also think the effect is related to how tokenization works. You have to be very careful when running these experiments. For example, you usually want to put a space between tokens so that the tokenizer sees the space and gives you a new token.

The entropy of the sequence also matters, and this makes the experiments more complicated. We have some evidence that the effect still holds for random or arbitrary sequences, but the details definitely have an effect.

For some reason, digits such as 9, or larger digits in general, tend to have larger magnitudes. It makes some sense given what the model is trying to do, but all of these details affect how quickly the 2 sequences converge to each other.

Can we explain what we mean by that? If I understand correctly, you have 2 sequences and then take the \(L_1\) difference of their softmax outputs. You’re saying that this converges toward some arbitrarily small \(\epsilon\).

What we study is the representation of the final token at the last layer, because that’s what you feed into the linear projection to extract the next token.

The key idea is that if you take 2 sequences and the representations of their final tokens at the last layer become arbitrarily close, you run into problems. They may then be mapped to the same thing.

What you care about is the norm between these 2 representations. You want to show that, for this family of sequences, as you keep growing them, the norm eventually becomes arbitrarily close to \(\epsilon\).

We want self-attention Transformers to have enough representational fidelity to distinguish differences in these situations.

Here’s another interesting point. You bring in the idea of numerical precision. We have these sequences, and as they get longer, perhaps they start with fairly low entropy. Eventually, depending on the numerical precision of the neural network—whether it’s FP6 or quantized—we dip below the threshold where the network can distinguish between the 2 things.

Yes, exactly. The fact that we use heavy quantization in many models is important. They go well below 16 bits.

It’s understandable that quantizing a model reduces its performance, and everyone would expect that. But this points to a mechanical issue with quantization. If you quantize aggressively, eventually you have sequences that were distinguishable at higher precision but are indistinguishable after quantization. The model is then forced to make a mistake.

Doesn’t this become catastrophic quite quickly? Your research shows—and Roon, the “Doom” debate guy, keeps saying this—that large language models can’t copy or count beyond around 100 elements. Frontier models can’t do much more than that, even before you quantize them.

Now we quantize them to 4 bits or whatever, and they collapse immediately. That’s a huge problem.

I’m not sure what the objection is. Perhaps it depends on who you are and what you’re interested in.

I’m interested in understanding what’s happening, but I also think a model should be able to do simple things that a human can do. Of course, you can equip models with tools, and that’s very successful. But one of the main points we make in the paper is that you often want to copy things into tools.

If a model can’t even do that reliably, then you probably have a problem. Counting could be solved with tools, but copying is fundamental, and you want the model to be robust at it.

You also did 2 very interesting experiments. One was with chain-of-thought, and the other has a bit of history. Do you remember GPT-3 having problems with numbers because of byte-pair encoding? If you put spaces between the numbers, it worked better.

One of your experiments interleaved the sequences. Strangely, that helped. Why?

It wasn’t really strange. We were trying to debug our theory. The intuition behind oversquashing is that if you’re trying to copy the 0 at the end, its position at the end makes the task problematic because the information about the 0 gets lost.

If you add more 0s to the sequence, the fact that there’s a 0 is not lost as quickly. Adding the 0s helps counteract the 1s dominating the representation, so the model doesn’t lose the information about the 0 as quickly.

What was particularly nice to see was that copying the first token seemed much easier than copying the last one. That directly reflects the path argument. If you look at the partial derivatives, the representation of the 0 is much more influential when it’s at the start than when it’s at the end.

To me, that showed that we had built some understanding. Without the oversquashing and path perspective, this would seem like absurdly random behavior. Why should this happen? The intuition from the experiments provides an explanation.

People should definitely look at the paper. As I remember it, there was a task where you asked the model to do something with a token at the beginning and another token at the end. As the sequence length increased, it successfully attended to the beginning for quite a long time, but performance on the token at the end dropped off very quickly.

You also looked at zero-shot and few-shot chain-of-thought prompting. The experiments were showing that, once the sequence length reached a fairly small number—perhaps even beyond 10—the model started going crazy.

If you’re counting, you would expect a monotonically increasing set of results, but the model collapsed very quickly.

The experiments were very simple. We asked the model to sum \(1+1+1\) and so on, perhaps 5, 10, 15, up to 200 times.

These models are notoriously bad at this kind of task, which I find disturbing. With chain of thought, we asked the model to break the task down, perhaps into groups of 5, and so on.

But the model quickly started diverging. It came up with nonsensical answers. That wasn’t surprising to me because the task is repetitive. There are so many 1s being added that the model gets confused about what it’s keeping track of.

There was an amazing plot with a large mode right in the middle. I think it was a counting task. Can you explain that?

We fed the model a simple prompt: “Can you sum 1+1+1?” and varied the number of repetitions from 5 to 200.

The model performed reasonably around 20 repetitions. Then it started outputting numbers that weren’t even multiples of 5. Once the sequence passed a certain length, it strongly preferred to output 100.

This suggested that the model probably wasn’t mechanically counting. There have been some recent preprints on this behavior arguing that language models probably aren’t implementing algorithms. They’re implementing what those papers call a bag of heuristics.

These heuristics are good enough to fit the training data, but as soon as the model goes out of distribution, they generalize horribly. The model outputs 100 because 100 seems like a plausible answer for a large sum of 1s. From the perspective of the training distribution, 100 is a common and reasonable answer.

That paper is “Arithmetic Without Algorithms: Language Models Solve Math Problems with a Bag of Heuristics,” right?

When I read your paper, there was a beautiful word I had to look up because I hadn’t seen it before: “subitizing.” Children apparently learn to recognize that a group contains 5 apples, for example. They learn a pattern for counting rather than actually counting, and language models may be doing something similar.

It makes sense that rough estimates are more important than precise numbers. I’m not sure I understand the difference between 100 and 103, but I can understand the difference between 10 and 100.

I don’t even know how I count. I can do it mechanically, but I don’t know how robust my internal concept of counting is. It makes sense that rough estimates are important for children and probably for language models as well.

In the introduction, you included a literature review of some of the papers skeptical about large language models. You cited Oriol Vinyals talking about how, under certain conditions, self-attention Transformers can be Turing complete.

Under what conditions can they be Turing complete?

That particular paper isn’t very formal about it, but there are other papers from people taking a more computer-science-oriented approach, looking at Transformers through automata theory.

The usual assumption is hard attention, meaning that the attention weights are either 0 or 1. That’s impossible in practice, but it’s a useful mathematical paradigm.

Sometimes people relax this to what you might call average-hard attention, where instead of a single delta function, the model attends uniformly with weight \(1/k\) over a set of \(k\) tokens. These constructions are easier to treat mathematically.

Shouldn’t it be impossible in principle for a self-attention Transformer to be Turing complete if it has finite precision and performs a fixed computation?

Those results also make assumptions about memory growing with sequence length, or about precision growing with sequence length. By memory, I roughly mean hidden dimension multiplied by floating-point precision—the number of bits available to the model. You can also multiply that by the number of tokens.

A lot of this work studies the setting in which precision increases with sequence length, perhaps logarithmically. But precision doesn’t actually increase in practice. Precision is whatever you choose it to be.

I spoke with Jürgen Schmidhuber about this. There’s also the 1995 paper by Siegelmann and Sontag showing that an RNN with infinite precision can simulate a Turing machine.

With infinite precision, you don’t have representational collapse. You can make 2 representations as close as you want and still distinguish them, because you can always increase the precision.

There’s a relationship between these results and the work on RASP, the programming language that implements operations Transformers can perform. There’s also a conjecture that if you want to implement something that corresponds to a short RASP program, a Transformer can learn it relatively well and generalize better.

There’s a nice relationship with programming languages and Transformers. But I think the main difference is that our results give you a quantity you can directly measure and say, “This is happening in a Transformer.”

The other results are constructive proofs showing that a model can represent a particular class of languages. They’re very interesting, but they’re constructive arguments. Our work is more about identifying a measurable mechanism.

The approaches are related, but they’re also very different ways of addressing similar issues.

There are many ways to approach this. I love the formal approach because it can give us solutions that are explainable, robust in some ways, verifiable, and so on.

I think Greg Yang discussed the Chomsky hierarchy and mapped different models, such as LSTMs and Transformers, onto the automata hierarchy. If I remember correctly, he put Transformers fairly low down.

I think RNNs can learn counting languages.

You also cited work by Pang on the limits of compositionality in Transformers. That must trivially be true, surely. A language model couldn’t generalize from “Mary loves John” to “Mary loves Jane” because it doesn’t have that kind of invertibility. Everything gets spread out into all of these circuits.

There’s also work showing that simply switching the order of the inputs can affect whether a language model solves a problem. For example, if you ask it to solve first-order logic with chains of implications, the order in which the implications are presented greatly affects whether the Transformer can solve the problem.

All of these results point to fairly poor generalization habits in current models.

What practical modifications could we make to overcome this collapse of representations and vanishing gradients?

Vanishing gradients are related to this, and Transformers were in some sense a solution to the problem that recurrent neural networks have this kind of bias. But we’re pointing out that Transformers implement a similar bias through their causal mechanisms.

There’s something fundamentally difficult about processing long sequences. You have to take a large amount of information and fit it into a finite number of bits. That’s simply a hard problem.

The best solution may be to find ways to compress information so that the model is left with less information to process. You want the technique to care about fewer things.

This is part of the motivation for windowed attention and related mechanisms. They’re well motivated from this perspective.

Deep in your bones, do you lean connectionist, or do you think we need a hybrid, neurosymbolic approach?

I’m a big fan of hybrid approaches. There’s a chess world championship going on at the moment, and Petar and some other researchers recently released a component in Gemini that can play chess.

It’s surprisingly strong, and it’s still a language model, but it has a specialized chess component. That seems like a principled approach.

Imagine having a base language model that acts as an orchestrator, with specialized components for mathematics, chess, and other tasks. Ideally, you could keep the base model fixed while swapping out and improving the chess component.

That sounds very exciting to me. We already know that this kind of approach can work. Vision-language models often have a specialized vision component and a specialized language component.

You could have a stream of tokens generated by very specialized components, with a base model processing the stream. The tokens could come from whatever you want. They might even come from an RNN if you found that RNNs were better at counting.

What are the trade-offs? I love the formal approach because it can give us explainable, robust, and verifiable solutions. But when you mix these components together, learning becomes very difficult.

The system is bottlenecked by how we design the interaction between the components. There’s no clean and easy way to build them together.

I agree. That’s probably why we don’t do it very often. It’s hard.

I think Transformers have fundamental limitations, so it makes sense to try to use other models as well. But it’s extremely difficult to make the combination work. Once we find a better way to do it, it may become much easier.

What’s your definition of reasoning?

I think reasoning is very ill-defined. I can talk to you about length generalization, which I view as a subcomponent of reasoning, but reasoning is a broad idea involving generalization and robust generalization.

There are different cases. A computer program will generalize arbitrarily well if you write it correctly, and we can prove that it’s perfect. Is that reasoning? I’m not sure.

Consider humans. There’s an example—I believe it was from Numberphile—involving a group of people who set out to compute π to a large number of digits. They spent a week doing it, and the first time they finished, they had made a mistake somewhere in the middle.

This isn’t surprising. Even if you know how to sort an array, good luck sorting an array with a million digits without making a mistake somewhere. Does that mean you’re not reasoning? I’m not sure.

There’s a strange trade-off here. We care about length generalization, but humans are bad at length generalization. We aren’t good at processing large amounts of data.

So perhaps if we only care about length generalization, we’re no longer talking about reasoning. That’s a long way of saying that reasoning may not be very well defined.

I’m fine with language models having a different definition of reasoning from humans because they’re fundamentally computers, or at least they’re built on computers. Computers are good at things humans are terrible at.

Counting should be possible for a computer-based system. Reasoning may involve different things for humans and machines because we care about different capabilities.

Chollet is coming this weekend. He wrote the “On the Measure of Intelligence” paper, and he never uses the word “reasoning,” which is interesting.

He defines intelligence as knowledge-acquisition efficiency. For him, knowledge acquisition is about a creative, adaptive search for novelty. He’s always careful to stress that simply performing a skill isn’t intelligent and that it isn’t reasoning.

A chess computer isn’t reasoning, but AlphaGo—or AlphaZero, I should say—is reasoning. There’s something about creativity because it’s trying things and acquiring knowledge that the creators of the system didn’t explicitly program into it.

I’m very interested in chess, so I don’t understand that comparison. AlphaZero has produced novel ideas that computers hadn’t found before. For example, it introduced ideas such as pushing the h-pawn that are now used by grandmasters all the time.

Mechanically, AlphaZero and Stockfish are doing similar things. They both use tree search, but they compute their heuristics differently.

In Stockfish, my understanding is that people have encoded many heuristics, perhaps extracted from statistics. You might want your knights not to be on the edge of the board, on average, and those heuristics bias the tree search.

AlphaZero doesn’t have those human heuristics in the same form. It finds something different, but that isn’t surprising. It’s using a function that isn’t aligned with what humans necessarily think.

Even Stockfish can find completely crazy ideas. The difference is that there’s still a human touch in Stockfish that isn’t present in AlphaZero. I’m not sure that finding novel ideas makes AlphaZero creative.

You could also ask whether a human playing chess is reasoning. Humans are much less methodical. A lot of moves appear in our heads, and we can’t explain why.

A computer performing a search can sometimes tell you why it chose a move. It can say that, deep down, the move works for a particular reason.

If we were to design an optimally intelligent reasoning system, what would it look like to you?

I would like it to behave like a computer in the sense that it doesn’t make trivial mistakes. It shouldn’t make arithmetic mistakes, for example, but it also shouldn’t make conceptual mistakes.

That’s where language models can be much better than humans. If we can leverage the fact that they’re machines while also aligning them to think in ways that resemble humans, we could create a very powerful combination.

Such a system would be completely out of reach for an individual human.

When I was speaking with Neil about the Golden Gate Claude example, I was slightly skeptical. The system took an abstract feature and showed us the tokens that maximally activated it. To me, that looked a little like keyword matching.

This is where we have a kind of reasoning chauvinism. We think that when humans do something, it’s reasoning, but when language models do it, it isn’t.

The meta-component of reasoning seems important. We could perform a search and, from a functional perspective, it would be as if the system were reasoning because it found all of these weird and wonderful trajectories and patterns.

When humans reason, we look at abstract analogies in the real world. We see that the universe has patterns, like a kaleidoscope, and knowledge can be composed and mixed together. When we look at a chessboard, we might see an analogy with something completely different, such as bananas, and perform a kind of multidomain analogical reasoning.

That feels like a creative jump, so we call it reasoning. When we see a computer doing something in a simplistic way, we say that it isn’t reasoning.

There’s an illusion of reasoning that comes from the fact that humans are extremely good at finding complex patterns that are difficult to explain. We interpret those jumps as reasoning.

Chess is a great way to think about this. There’s a famous experiment where grandmasters and strong chess players are shown a position and asked to reproduce it from memory. They look at the position for perhaps 20 seconds, then a fresh board is placed in front of them, and they have to put all the pieces back.

The grandmasters are much better than amateurs at reproducing the position. But the positions in the experiment came from real games. They were positions that naturally arise during chess play.

When the researchers gave them random positions, the grandmasters and amateurs performed similarly. Grandmasters don’t have extraordinary general memory. They have a much more compressed representation of the chess world because they’ve seen so many positions.

They can compress structures. If I have pawns on f7, g6, and h7, a bishop on g7, and I’m castled, I immediately recognize a fianchetto. That’s 1 thing to me.

Someone who has never played chess has no concept of that structure. I’ve seen the Sicilian pawn structure thousands of times, so I recognize it immediately. I might recognize that it’s a position arising from a particular opening or FEN.

I’m allocating a lot of my brain to compressing information that seems noisy, but it isn’t actually noisy. It’s something I’ve seen 10,000 times.

There’s a question about that. We have all of this experience, and then there’s this diffuse process of concept formation. Intuition isn’t reasoning; surely it’s memorization.

But it isn’t a simple case where, one day, a chess player has a flash of inspiration and defines a new piece of knowledge. From that point on, they can intuitively retrieve it.

It’s a diffuse process. You have many diverse experiences, and a concept emerges. It might happen mimetically, or it might develop across many people over a long period.

It’s somehow magical—the magical power of the human brain. You don’t know why, one day, you look at a position and come up with a brilliant move that you might not even have considered before.

But this only happens once you have an extremely strong compressed representation. Once you have that representation, you can ignore the noise. You have more capacity to compute without thinking, “These pieces are here, and those pieces are there.”

You look at the position and compute with a much clearer representation. That’s when brilliancies occur and magical moves appear.

Are you a fan of the flash of inspiration? Do these moments happen when everything suddenly clicks?

When everything clicks, it’s as if you’ve climbed a small step in your understanding. In my experience, these moments only happen once many things are in place.

You need to be deeply focused on the task. You need to sleep well, be healthy, and be in a good state. Then perhaps you can play at a much higher level one day, or perform a task at a much higher level.

I think these moments gradually accumulate. It’s a sequence of inspiration moments that build on each other. You can’t choose them, and they don’t happen unless you invest a great deal of time in something.

Karl Friston has a beautiful phrase for this: “epistemic foraging.” Over time, we discover knowledge and the cloud of knowledge increases.

Federico, it’s been an honor to have you on. Thank you so much.

Thank you so much.