Why Scale Will Not Solve AGI | Vishal Misra - The a16z Show

Anthropic makes great products. Claude Code is fantastic. Cowork is fantastic. But they are grains of silicon doing matrix multiplication. They don't have consciousness. They don't have an inner monologue. You take an LLM and train it on pre-1916 or 1911 physics and see if it can come up with the theory of relativity. If it does, then we have AGI.

Just today, by the way, Dario allegedly said that you can't rule out that they're conscious. You can rule out their consciousness. I mean, come on. To get to what is called AGI, I think there are 2 things that need to happen.

Vishal, it's great to have you in again.

Great to be back. This is one of my favorite topics: how do LLMs actually work?

I think that you, in my opinion, have done kind of the best work on modeling this out.

Thank you.

For those that did not see the original one, maybe it's worth doing just a quick background on what led you to this point, and then we'll go into the current work that you've been doing.

5 years ago, when GPT-3 was first released, I got early access to it and started playing with it. I was trying to solve a problem related to querying a cricket database. I got GPT-3 to do in-context learning, or few-shot learning. It was kind of the first—at least to me, it was the first known implementation of RAG, or retrieval-augmented generation, which I used to solve the problem of getting GPT-3 to translate natural language into something that could be used to query a database GPT-3 had no idea about.

I had no access to GPT-3's internals, but I was still able to use it to solve that problem. It worked beautifully. We deployed this in production at ESPN in September 2021.

Wow. You did the first implementation of RAG in 2021.

No, no, no. In 2020. I got it working in 2020, and by the time you talked to all the lawyers at ESPN and productionized it, it took a while. But in October 2020, I had this architecture working. After I got it to work, I was amazed that it worked. I wanted to understand how it worked.

I looked at the “Attention Is All You Need” paper and all the other deep-learning architecture papers, and I couldn't understand why it worked.

So then I started getting deep into building a mathematical model.

Now you've published a series of papers. The first one that I read was the one where you had kind of your matrix abstraction. So maybe we'll talk about that, and then we'll talk about the more recent work.

Yeah.

Perhaps we'll just start with the first one. You were trying to come up with a mathematical model of how LLMs work.

Yeah.

At the time, you were actually trying to figure out how in-context learning was working.

Yes.

You came up with an abstraction for LLMs, which is basically this very, very large matrix, and you used that to describe it. Maybe you can walk through that work very quickly.

Sure. What you do is imagine this huge, gigantic matrix where every row of the matrix corresponds to a prompt. The way these LLMs work is, given a prompt, they construct a distribution of probabilities for the next token. The next token is the next word. Every LLM has a vocabulary; GPT and its variants have a vocabulary of about 50,000 tokens.

Given a prompt, it'll come up with a distribution of what the next token should be. Then all these models sample from that distribution.

That's the posterior distribution, right? That's how LLMs work. The idea of this matrix is that for every possible combination of tokens, which is a prompt, there's a row.

The columns are a distribution over the vocabulary. So if you have a vocabulary of 50,000 possible tokens, it's a distribution over those 50,000 tokens.

By distribution, it's just the probability that the next token should be this versus that.

The probability. Sorry.

That's the idea. When you start viewing it that way, it makes things at least clearer to people like me who want to model what's happening. Concretely, let's say you have an example where your prompt is just 1 word: “protein.” If you look at the distribution of the next word—the next token—after that, most of the probabilities would be 0, but you'd have nonzero, nontrivial probabilities on, let's say, 2 words: 1 is “synthesis,” and the other is “shake.”

Right. Now the LLM is going to sample this next token and might pick “synthesis” or “shake.” Or you, as a human, will give the prompt “protein shake” or “protein synthesis.”

Depending on whether you pick “synthesis” or “shake,” the next row looks very different, right? If you pick “protein synthesis,” the terms that would have a high probability would all be concerned with biology. But if you pick “protein shake,” it'll all be about gyms and exercise and bodybuilding stuff. So “synthesis” or “shake” completely changes what comes next.

This is an example of what you can call Bayesian updating. You start with “protein”; you have a prior that after “protein,” this is going to happen. As soon as you get new evidence, then the next term is “synthesis” or “shake,” and you completely update the distribution.

Now you can imagine that the entirety of an LLM is this giant matrix where you have every row: “protein,” “protein shake,” “protein synthesis,” “the cat sat on the”—“Humpty Dumpty,” blah, blah, blah. Given the vocabulary of these LLMs—let's say 50,000—and the context window, GPT—for instance, ChatGPT—the first version had a context window of 8,000 tokens. If you look at all possible combinations of 8,000 tokens and a 50,000-token vocabulary, the number of rows in this matrix is more than the number of electrons across all galaxies.

There's no way that these LLMs can represent it exactly. Fortunately, this matrix is very sparse. Why? Because an arbitrary combination of these tokens is gibberish. We're never going to use that in natural, real life. Also, the columns are mainly 0. If you have “protein,” then you won't have lots of arbitrary numbers or arbitrary words after that. It's very sparse both in rows and in columns.

In an abstract way, what all these LLMs are doing is coming up with a compressed representation of this matrix. When you give a prompt, they try to approximate what the true distribution should have been and generate it. That's what, in my mind at least, it boils up to.

Just from my understanding, if you have a row of “protein” and then you have one with “protein shake,” is “protein shake” a subset of “protein,” or is it different?

It's different. It's a continuation from “protein.”

I see. Right. No, but I'm just saying: is the actual posterior distribution a subset?

You can say it's a subset, right? If you have “protein,” then “protein shake” and “protein synthesis” are all continuations from “protein.” So both “synthesis” and “shake” have nonzero probabilities. You can think of it as somewhat a subset.

You used this approach to describe how in-context learning works. Maybe first describe what in-context learning is, and then the conclusion that you came to from that.

In-context learning is when you show the LLM something it has kind of never seen before. You give it a few examples of what you're trying to do. Then you give it a new problem related to the examples that you have shown, and the LLM learns in real time what it's supposed to do and solves that problem.

By the way, the first time I saw this, it absolutely blew my mind. I actually used your DSL when I was first learning about it. The DSL thing is just crazy—that this works at all.

It's absolutely mind-blowing that it works. Going back to that cricket problem, in the mid-’90s I was part of a group that had created this cricket portal called Cricinfo.

Cricket is a very strange sport. Think baseball multiplied by 1,000, with all kinds of stats. We had created an online searchable database called Statsguru where you could search for anything—any stat related to cricket. It has been available since 2000.

Because you could query for anything, everything was made available. How do you make something like that available to the general public? They're not going to write SQL queries. The next best thing at that time was to create a web form. Unfortunately, everything was crammed into that web form. As a result, you had 20 dropdowns, 15 checkboxes, and 18 different text fields.

It looked like a very complicated, daunting interface. Even though it could solve, or answer, any query, almost no one used it. A vanishingly small percentage of cricket fans used it because it looked intimidating.

ESPN bought that site in 2007. I still know people who run the site, and I always told them, “Why don't you do something about Statsguru?” In January 2020, the editor-in-chief of Cricinfo, Sambit Bal—he's a friend—came to New York, and we went out for drinks. Again, I told him, “Why don't you do something about Statsguru?”

He looks at me and says, “Why don't you do something about Statsguru?” He was joking, but that idea kind of stayed with me. When GPT-3 was released, I thought maybe I could use GPT-3 to create a front end for Statsguru.

What I did was design a DSL, a domain-specific language, which converted queries about cricket stats in natural language into this DSL.

No. To be clear, you created this; it wasn't part of any training that GPT-3 could have seen?

Nothing GPT-3 could have seen. I created it. I thought, okay, this makes sense. So I designed that DSL, and then I did a few-shot learning thing. I created a database of about 1,500 natural-language queries and the DSL corresponding to each query. When a new query came in—somebody asking a stats question in English—what I would do was go through the natural-language queries, do a semantic search, and pick the top few that matched most closely.

Yeah.

I would then use that natural-language query and its DSL and send that as a prefix. GPT-3, if you recall, had a context window of only 2,000 tokens.

Yeah. So you had to be very judicious about which examples you picked. You picked those, and then you sent the new query, and GPT-3 would complete it in the DSL that you had designed, which, until milliseconds ago, it had never seen.

Yeah. And I had no access to the internals of GPT-3. I had no access to the weights.

Yeah.

But still, it worked. That's how it worked.

So it's not obvious to me, given your matrix example of a prompt and then a distribution, how something like in-context learning would work. I think your first paper tackled this problem, right? Maybe you could walk through your understanding of how LLMs do in-context learning.

Yeah. When you think about what in-context learning is, it's that, as you see evidence, you update your belief. In the first paper, I also took this cricket DSL example, and I depicted the next-token probabilities of the model as it was shown more and more examples. The first time you showed it this DSL—the natural language and the DSL—the probabilities of the DSL tokens were extremely low because GPT-3 had never seen this thing. When it saw the cricket question, it was trying to continue it with an English answer. So the probabilities that were high were all English words.

Yeah.

Once it saw my prompt, where I had the question and the DSL, the next time I had the question in the next row, the probabilities of the DSL tokens started going up. With every example, they went up, and finally, when I gave it the new query, it was like it had almost a 100% probability of getting the right token.

Yeah.

This is an example of the model updating its posterior probability in real time. It was upgrading its knowledge: “Okay, I've seen evidence. This is what I'm supposed to do.” This is a colloquial way of saying what Bayesian inference is. Bayesian updating basically means you start with a prior, and when you see new evidence, you update your posterior. That's the mathematical definition. In English, it's basically: You see something, you see new evidence, and you update your belief about what's happening.

Yeah.

So it was clear to me that LLMs were doing something that resembled Bayesian updating. In that first paper, I had this matrix formulation, and I showed that what it was doing looked like Bayesian updating.

Yeah.

Then we can come to the next series of papers.

That's right. It seemed pretty conclusive to me at that time, and then you went quiet for a while. I still remember the WhatsApp text. You said, “Erik, I know exactly how these things are working now.”

Yeah.

Then you dropped a series of papers that kind of broke the internet. You went super viral on Twitter. People really noticed. I want to get to that in just a second, but before that, I remember when your first paper came out, people would say, “These things are definitely not Bayesian. Anything could be considered Bayesian, but they're not.” Why do you think there was this reaction? I felt like there was almost a backlash just because they were being characterized as Bayesian.

Yeah. I think in this whole world of probability and machine learning, there have been camps of Bayesians and frequentists. I don't want to get in the middle of that sort of political battle, but Bayesian has become almost a political label, and people had a reaction to that. It's part of that war.

I see.

It's like the old Bayesian-frequentist battle. People just had this reaction: “Oh no, you can say anything is Bayesian, right?” So I said, okay, maybe they have a point. Maybe what we're saying is not really Bayesian. How do we prove that it's Bayesian?

Right.

First, I have to thank you and Horowitz for this. When I said that in my first paper I showed these probabilities, it was because OpenAI had, in its chat interface, an option to display those probabilities. Then they stopped. For some reason, OpenAI stopped. I'm not going to get into the open-versus-closed debate, but they stopped, so we could no longer peer inside what was happening.

We developed our own interface, which could let you look not only at the probabilities but also at the entropy of the next token.

Was this on top of an open-source model?

Yeah, you can load any sort of open-source model. But being in academia, we didn't have access to compute. Thanks to your generous donation, we got the clusters to run what's called TokenProbe. So you can go to tokenprobe.cs.columbia.edu.

Is it still running?

It's still running, and people come to it. I use it in my classes to get students to do assignments. They write their own DSLs, and they say that it really helps them understand how these LLMs work. My understanding of LLMs came from TokenProbe—just sitting there and looking at the distribution as you filled out a prompt. It's very, very enlightening.

For those of you listening, what's the URL again?

TokenProbe: tokenprobe.cs.columbia.edu.

Yeah, check it out. It's actually a very useful way to see how the probability distribution gets updated as you fill out a prompt.

Right. Yeah. But then I cheated. It was running, but I also had access to the GPUs that were powering it.

Mhm.

Along with colleagues at Columbia, one of whom is now at DeepMind, we started to think about how to really prove that it's Bayesian.

Can you just explain it? I actually don't know the answer to this.

It seemed to me you proved it in the first paper. What was missing?

Well, in the first paper, we showed it. It was empirical, and you could see it, but it wasn't mathematical.

I see, I see.

It was even obvious to me, but to convince people—you could say, you know, people who dismiss it, “Oh, anything can be Bayesian”—we had to show it precisely, mathematically.

I see, I see.

So then we came up with this idea. My colleagues Naman Agarwal and Siddharth Dalal and I wrote the series of papers together. We came up with the idea of a Bayesian wind tunnel.

Okay, so what's a wind tunnel? A wind tunnel in the aerospace industry is where you test an aircraft in an isolated environment. You don't fly it; you test it against all sorts of aerodynamic pressures. Then you see what it will withstand—what kind of altitude, pressure, and so on. You don't want to do that testing up in the air.

So we said, okay, why don't we create an environment where we take these architectures and test transformers, Mamba, LSTMs, MLPs—all architectures? We take a blank architecture and give it a task where it's impossible for the architecture to memorize what the solution to that task should be. The space is combinatorially impossible given the number of parameters, and we took very small models. So it's difficult enough that they cannot memorize it, but tractable enough that we know precisely what the Bayesian posterior should be. You can calculate it analytically.

We gave these models a bunch of tasks where, again, we showed that it's impossible to memorize. We trained these models, and we found that the transformer got the precise Bayesian posterior down to 10^-3 bits of accuracy. It was matching the distribution perfectly. So it is actually doing Bayesian inference in the mathematical sense, given a task where it has to update its belief.

Mamba also does it reasonably well. LSTMs can do one of the things. In the papers, we have a taxonomy of Bayesian tasks: Transformers do everything, Mamba does most of it, LSTMs do only some of it, and MLPs fail completely.

So is this a reflection of the data that it's trained on, or is it more a reflection of the mechanism?

It's the mechanism. It's the architecture. The data decides what tasks it learns.

Right?

So in the first paper, we had these Bayesian wind tunnels, and we showed that it was doing the job with different tasks. In the second paper, we showed why it does it. We looked at the transformers, we looked at the gradients, and we showed how the gradients actually shape this geometry, which enables this Bayesian updating to happen.

Then, in the third paper, we took these frontier production LLMs, which have open weights so that we could look inside them, and we did our testing. We saw that the geometries we observed in the small models persisted in models with hundreds of millions of parameters. The same signature existed. The only thing is that, because they are trained on all sorts of data, it's a little bit dirty or messy.

Yeah.

But you can see the same structure. The whole idea behind the Bayesian wind tunnel was that, unlike production LLMs, where you don't know what they have been trained on, you cannot mathematically compute the posterior.

Right? So, again, how do you prove it? I mean, it looks Bayesian, you know, from the first paper.

From the first paper, it looks Bayesian, but the wind tunnel sort of solved that problem for us. We said, “Okay, let's start with a blank architecture. Give it a task where we know what the answer is. It cannot memorize it. Let's see what it does.”

Do you think this provides any sort of indication of how humans think, or do you think that these things are totally independent?

No, no, it does provide an indication. Human beings also update our beliefs as we see new evidence. We do Bayesian updating, in some sense, but we do something more than that. I'll come to that. These transformers, or even Mamba, do this Bayesian updating.

But the difference with humans is that we'll update our posterior when we see some new evidence, but the way our brains have evolved over hundreds of millions of years, our optimization objective has been “don't die and reproduce.” That's been the driving force, and our brains have learned to adjust.

When we see some danger—there's something rustling in that bush—we don't go near it. We know how to react to that danger. We know how to save ourselves. We internalize that learning, and our brain cells, or our synapses, remain plastic throughout our lifetime.

What happens with an LLM is that, once the training is done, those weights are frozen. When you're doing inference—for instance, in-context learning or anything during that conversation—you're doing Bayesian inference, but then you forget. The next time a new conversation starts with zero context, you don't retain any learning that happened in the previous instance.

For instance, with the Cricket DSL that I was doing, every invocation of it was fresh. It did not remember the last time I sent a query or what the DSL looked like. That's one difference between how humans use Bayesian updating—we remain plastic all our lives—whereas LLMs are frozen.

Tell me. Yeah, yeah, yeah.

The other difference is that our objective is “don't die and reproduce.” The LLM's objective is to predict the next token as accurately as possible.

All these scary stories that you read about—how the LLM tried to deceive someone or tried to prevent itself from being shut down—that's not a function of the architecture. That's a function of the training data. It's been fed stories of Asimov or Reddit, or whatever. It'll reproduce those stories.

I mean, just today, by the way, Dario—

Allegedly said that you can't rule out that they're conscious.

You can rule out their consciousness. I mean, come on. Anthropic makes great products. Claude Code is fantastic. Cowork is fantastic. But they are grains of silicon doing matrix multiplication. They don't have consciousness. They don't have an inner monologue. They're not driven by the same objective function—“don't die, reproduce.” They're driven by “don't make a mistake on the next token,” and that's driven entirely by the training data.

You train the LLM with stories of Asimov or Reddit where, in order to survive, it's going to do this or that. It'll reproduce that. It's a reflection; it's not a mind. The results, just to say it for the 10th time, are perfectly Bayesian.

Perfectly. Yeah.

To the digit. I trained it for 150,000 steps, and the accuracy was 10^-3 bits. I could have trained it for longer, but this happened in half an hour on the infrastructure that you provided for TokenPro in the background. I could use those GPUs to train. So thank you again for that.

Coming back to it, human beings—we are Bayesian—

Bayesian.

But we do something else. When I throw this pen at you, what will you do?

Dodge it or—

Do it? Yeah.

Why will you dodge it?

To avoid being hit.

Avoid being hit. But your head is not doing a Bayesian calculation of, “Okay, this pen is coming. The probability that it hits me is this much, and it'll cause this much pain,” or anything like that.

Correct. What you're essentially doing in your head is a simulation.

You see the pen coming, and you know that it will come and hit you. Your mind simulates it, and you dodge it. All of deep learning is doing correlations. It's not doing causation.

Yeah.

Causal models are the ones that are able to do simulations and interventions. Judea Pearl has this whole causal hierarchy, where the first level is association, which is where you build these correlation models. Deep learning is beautiful and extremely powerful. You see every day that all these models are amazingly good.

They do association.

The second is intervention in the hierarchy. Deep learning models do not do that. The third is counterfactual. Both intervention and counterfactual involve some sort of simulation. You build a causal model of what's happening, and then you're able to simulate. Our brains do that. The current architectures don't.

Another example that I think will make it clear is the difference between—I’ll use these technical terms—Shannon entropy and Kolmogorov complexity.

Sure.

If you look at the Shannon entropy of the digits of pi, it's infinite.

Sure.

It's impossible to predict and learn what digit will come after. That's the definition of Shannon entropy. Shannon entropy tries to build a correlation; it tries to learn the correlation. Deep learning does Shannon entropy.

Kolmogorov complexity, on the other hand, is the length of the shortest program that will reproduce the string in question.

Yeah.

The program to get the digits of pi is very small. Thanks to Ramanujan and others, there are all sorts of really small programs that can reproduce it exactly. So the Kolmogorov complexity of pi is very small, while its Shannon entropy is infinite.

I think deep learning is still in the Shannon entropy world. It has not crossed over to Kolmogorov complexity and the causal world.

Wow, interesting. So, to what extent do you think this provides us with research directions to improve the state of the art? Let me give you a specific example. You talked about how human beings don't actually update their matrices or update their weights, but right now there's a lot of research on continual learning.

Does your work provide any guidance on how you might approach those problems? In particular, I've always had this question: We use so much data and so much compute to create these models. Is it even reasonable to think that you can update the weights and actually have a meaningful impact in real time? It just seems like you need so much more data in order to do that. Can you start answering these questions?

You can start answering some of these questions. One of the misconceptions that exists today is that scale will solve everything. Scale will not solve everything. You need a different kind of architecture, and continual learning is a difficult problem.

You have to balance the fact that you will learn something new against the risk of catastrophic forgetting.

Right.

If you update the weights and forget what was important and what you had already learned, then you're not making progress. It'll just be some sort of random, chaotic model. So solving that problem is difficult. That's one aspect of it.

To get to what is called AGI, I think there are 2 things that need to happen. One is plasticity, which has to be implemented through continual learning.

Yeah.

Secondly, we have to move from correlation to causation.

Yeah. How much is this similar to what Yann LeCun talks about with causality and planning—predicting how your action would—

It is related. He's coming at it from a different angle than the GPT model, but it is related.

The other thing is that, the first time I came on this podcast, I mentioned this test of AGI: the Einstein test.

I don't remember.

I said, “You take an LLM and train it on pre-1916 physics and see if it can come up with the theory of relativity.” If it does, then we have AGI. It's a high bar, but we should have high bars. It won't.

This is the same test that I think Demis mentioned at the India AI Summit a couple of weeks ago. It's created a lot of news. But why is that, and how is that related to this idea of Shannon versus Kolmogorov?

At the time of Einstein, there were a lot of clues that something was missing from Newtonian mechanics.

Yeah.

Right. People knew that Mercury’s orbit didn’t make sense. There was something off about it. Then there were these experiments—the Michelson–Morley experiments—where they were trying to figure out a medium called the ether through which light travels. They felt that if you bounced light in different directions, the speed might change and they could detect a change in the speed of light.

They tried several experiments. They had really precise instruments that could measure the speed, and they found nothing. They found that the speed of light did not change at all. Then there was the whole issue of black holes.

Yeah.

Then gravitational lensing. So there were a lot of these signs that Newtonian mechanics was not really explaining everything.

Yeah. But until Einstein came up with a new representation of the space-time container—

Right.

—we were stuck.

So if you had a model that just looked at correlations and saw all of these pieces of individual evidence and put them together, it would not have come up with the beautiful equation that Einstein came up with. I’m forgetting exactly what it is: Gμν = 8πG Tμν, something like that, where the equation of the space-time continuum is represented by the tensor.

So he came up with a new formulation. He kind of rejected the existing axioms and came up with a very short, compact representation of the world.

Interesting.

One equation, and from that equation everything else follows, whether you’re talking about gravitational waves, black holes, Mercury, or how GPS works. The GPS that we use every day in our phones uses the equation of relativity.

So does this end up becoming like—you almost have to ignore the majority of previous data in order to do it, which an LLM can’t because it’s trained on the majority of previous data? It’s like you almost have this kind of data gravity that’s pulling you back. Everybody said it’s X.

There’s a little bit of evidence that it’s Y, but because everybody said it’s X, the LLM will always say it’s X.

It’ll always say it’ll treat that Y as an anomaly. Actually, this is a very nice way to say it. It’s like—

It’s like—

So now, okay, now I get your Shannon entropy versus Kolmogorov complexity. One of them is—

The total amount of information there will always be bound to the total amount of information there, which is what happens right now.

Yeah, where you can actually describe another notion. You can describe everything with a shorter description with the new data, which would be a totally different model. It’s like—

You need a new representation, right?

Yeah.

You know, another way that I’ve always thought about this—and I thought you articulated it well the last time we talked about it—is that the universe is this very, very complex space and somehow humans map it into a manifold.

Mhm.

That’s less complex.

Yeah.

And then that gets written down, and then the LLM. So that’s kind of some distribution. It’s still a very large space, but it’s a bounded space. The LLMs learn that manifold, and then they use Bayesian inference to move up and down that manifold, but they’re bound to that manifold.

Yeah.

And again, I don’t want to put words in your mouth, but what they can’t do is generate a new manifold, right? That requires understanding the way the universe works and then coming up with a new representation of the universe.

And this is what relativity is, right?

Yeah, exactly.

Einstein had to create a new manifold.

Yeah. If you just stuck with the old manifold of Newtonian physics, then you would see these correlations, but you could not come up with a manifold that explained them. So you need to come up with a new representation.

To me, there are lots of definitions of AGI. The Turing test—we have already passed that. Performing economically useful work every day—you see LLMs doing that.

Do we? I don’t know. No, I mean, they are—

I mean, without human intervention.

No, no, no. That’s different, but still, it’s like a car can run faster than humans, right?

I mean, that’s a very shallow definition.

Yeah. So all these definitions do useful—

Maybe in 6 months you’ll have Claude or Gemini doing tasks without intervention, tasks that are well-defined and well-scoped. That’s possible, but to me, AGI will happen when these 2 problems get solved: properly implementing plasticity and continual learning, and building a causal model in a more data-efficient manner.

Yeah. We’re hearing people now talking about generality, like Donald Knuth, for example, in the last few days had this aha moment that apparently went viral on X. Do you think that suggests that we’re seeing generality?

No, no, no. That, to me, validates what I’ve been talking about for a while now. So if you read what he did, with the help of a colleague, he got the LLMs to solve this particular problem of finding Hamiltonian cycles for odd numbers. We won’t get into that. He got the LLMs to keep solving for one odd number after the other.

What he also got them to do is, after they found a solution for a particular value of m, he made the LLM update its memory with exactly what it learned in solving that problem. The LLMs tried many different things.

Yeah.

When something worked, they updated the memory. So that’s kind of hacking together plasticity.

Yeah.

Right. It’s learning what it has done as it went along. Again, it’s a hacked version of it. You’re not changing the weights; you’re just improving the context.

Right.

Right. But as you learned, and even after that, this whole space of Hamiltonian cycles and the associated math was well represented in the manifolds that these LLMs had been trained on.

Right. You just had to find the right connection.

And LLMs—I know, you throw enough compute at them, they will find the right connection. Knuth was able to find the LLMs’ attempts, and eventually it needed him to put together what he saw into a solution.

Yeah.

Right. So it’s like Einstein saw all this evidence, then he thought, “What will explain this?” He came up with a causal model.

Yeah. So Knuth and his brain are sort of the Kolmogorov part of it.

That’s the human, right?

And the LLMs are extremely efficient at doing the Shannon part of it. They found all the solutions by trying various things and learning more and more—a clever way to decompose it.

I’m wondering—again, I’m going to ask the same question—which is, do you think this provides some sort of insight on the next problem to tackle? Is there a mechanism that will get to Kolmogorov complexity or not? It tells us which direction, but clearly not how to do it.

Not how to do it, but even Kolmogorov complexity has largely remained a sort of theoretical construct.

Yeah, for sure. There’s no algorithm. There haven’t been practical implementations of finding the shortest program.

We know it exists. You can argue about it, but that’s where I think—it’s my bias—that’s where our energy should be focused, not on larger models with more tokens.

Can you tie the 2 things together? How does that pair with doing simulation, or is simulation totally orthogonal?

No, simulation is related, right? The simulator is the program that we create. It may not be the perfect program.

Oh, I see.

But in our heads we create this simulator: when I’m throwing the pen, you know it’s coming at you, and you duck. So you’re not computing the probabilities as it goes, but you build a very physical thing.

Conceptually.

Conceptually, but it’s the same because of the same mechanism.

It’s the same mechanism, really.

Yeah. You have to build a causal model.

Yeah.

Right.

I see. For most things, right?

So you have to move from correlation to causation. We’ve heard this term ad infinitum, but here it’s making a difference in the way we view intelligence.

How have the last 3 papers been received?

No, I don’t know. They’re well—I mean, the arXiv versions will let me tell you.

A lot of great reception, a lot of people read them. I’m just wondering what kind of feedback you’ve gotten.

I’m getting good feedback, but I’m an outsider in this field, right? That’s right—I’m a networking guy. I’m a networking guy. Why is he writing about learning and machine learning and deep learning and Bayesian inference?

But people who have actually taken the time to read those papers are giving me really good feedback. There was a recent paper by Google Research that tried to teach an LLM through some sort of reinforcement learning to do Bayesian learning properly.

Yeah.

And that's going in this direction. I think people are coming around to the view that LLMs are doing Bayesian learning. I know that some people also looked at the Bayesian wind tunnel paper on the arXiv and reproduced the experiments.

That's great.

Did they just see what was written, do the training, and see, “Yeah, this is actually happening”?

That's great. So what's next?

What's next is these 2 parallel tracks. I hope to make progress there: plasticity and causality.

Because today you've taken an existing mechanism, and you've created a formal model of how it works. Now you're actually interested in improving it, in creating a new mechanism. Do you think it's an entirely different architecture, or do you think LLMs are part of the solution?

I think LLMs are definitely part of the solution.

I see.

But there has to be something more. I was not interested in cataloging what all these LLMs can do. I was more interested in why they are doing it and how they are doing it. I think now we have a good grip on the why and how. The next step is to move them to the next level. I think we now have a fairly good understanding of what the limits are.

Yeah. Now, how do you go to the next step? Is there an equivalent kind of theoretical framework for causality that applies here, similar to Bayesian inference?

Well, Judea Pearl's whole causal hierarchy, I think—I think that's the right one.

That's a very good one. You know, the whole do-calculus approach, I think, is a good way to think about it: association, intervention, and counterfactuals.

Yeah.

It takes you from correlation to actual causation in a mathematical way.

Yeah.

That's great. All right. Well, listen, I really appreciate you coming. This is awesome. We had you here for the first paper, where you had the empirical results.

Mhm.

And then we had you back when you actually had the formal proof, and hopefully the next time you come back, you will have a proposal for the mechanism that actually provides the next step.

Hopefully.

All right. Cool. Thank you for coming in.

Thank you for having me.