How Do AI Models Actually Think? [Dr. Laura Ruis]

Laura, it's amazing to have you on MLST.

Thank you. It's amazing to be here.

Can you tell us about yourself?

Sure. I'm Laura. I'm a PhD student at University College London, supervised by Tim Rocktäschel at the Gatsby Computational Neuroscience Unit, and I'm also part of MATS. I'm broadly interested in understanding language and its relation to human cognition, and how we can evaluate that in artificial intelligence.

To what extent can pillars of human intelligence also show up in artificial intelligence? Things like reasoning—mathematical reasoning, social reasoning, that kind of thing—and also specifically trying to understand how state-of-the-art models are doing what they're doing.

Very cool. I'm a huge fan of Cohere, Ed, and Tim.

Me too.

You've just written a paper, and there's a huge controversy around it. I've been speaking with Sarel, for example. He calls LLMs “approximate retrieval engines” and o1 an “approximate reasoning engine.” He's saying they're doing a little bit of reasoning, whatever that means.

You've written this paper, which has generated a lot of interest on social media: “Procedural Knowledge in Pretraining Drives Reasoning in Large Language Models.” Give us the elevator pitch.

I was doing evaluations of language models, trying to understand how they were doing social reasoning. We designed a benchmark and evaluated models on their social reasoning skills. After that paper, I was left with this question: to what extent is the scale that makes these models get better at these tasks driving their performance? How is it driving their performance?

Is it just that the model is seeing more similar material and therefore can memorize more, and seems to have more capabilities? Or is it really doing something more interesting and learning something qualitatively different from more data or more parameters?

Of course, the way we evaluated machine-learning methods in the past was to separate test data from training data. That's not possible anymore, because models are trained on everything; the test data is in the training data now. We wanted to understand when language models are producing zero-shot reasoning traces.

For example, for simple arithmetic, the model can produce the steps needed to reach an answer. Is it relying on having seen those exact steps before in training, or is it doing something generalizable? Is it taking the steps itself and getting to the answer? That was the motivation for this paper.

Very cool. You used influence functions to do this analysis. Can you explain what they are?

I was very happy when I stumbled across that tool, because it's a method from robust statistics that tries to answer a counterfactual question about the model. The question it tries to approximate is: what if I take this pre-training document out of the dataset and retrain the entire model? How does the behavior change? How do the model parameters change, and with that, how does the likelihood of completions change?

That's what influence functions estimate, and that's the tool we used to determine how pre-training data influences the reasoning steps produced by models.

If I understand correctly, you created queries that resemble reasoning and ones that resemble some kind of fact retrieval, and then compared what the influence functions did on those queries.

Exactly. We used factual tasks as a sort of grounding, because influence functions are very approximate. We don't actually retrain the model for every data point, because that would be too expensive. You want to have some idea that what you're finding actually makes intuitive sense.

Factual retrieval is a natural task for that, because for factual questions the only way to answer them is to retrieve the relevant documents. We compare this to the influence scores for reasoning traces. Those tasks are simply zero-shot reasoning prompts, where the model generates the reasoning steps itself.

If the model were doing retrieval for those types of reasoning, it would really have to retrieve each reasoning step from the pre-training data, because it outputs the reasoning traces zero-shot. I don't give it any examples.

So the intuition behind your work is that, when we're doing fact retrieval, it seems quite focused: the model goes to a document and retrieves the fact. When it's doing reasoning, it seems very diffuse; it's looking at loads and loads of documents that contain reasoning processes.

That's the abstraction you can take from it. Of course, in reality, even when it's doing factual retrieval, there's much more going on. It needs to adhere to syntax, and there are all kinds of stylistic elements involved.

Still, the most striking finding from this paper to me was that, for factual retrieval, whether a document is influential for one factual question is not predictive of its influence for another factual question. The model relies on very distinct sets of documents.

For a reasoning question, if the questions underlie the same task—for example, calculating the slope between numbers, but with completely different numbers—the influence over the documents is very similar. The same documents can influence these questions in the same way. We didn't see that for factual retrieval, and that's really the basis for why we call this procedural knowledge.

Very interesting. For people at home, an example of a reasoning task would be two-step arithmetic, calculating slopes, or solving linear equations. Fact retrieval might be something like, “What is the tallest mountain?”

Exactly. “What is the tallest mountain?” “What is the largest ocean?” “In which year did the Bodleian Library open?” Those are examples of factual questions.

We have 3 different reasoning tasks. One is simple two-step arithmetic. You can imagine “7 minus 4 times 8.” That's two-step arithmetic: you first have to calculate 7 minus 4 and then do 3 times 8.

Calculating slopes requires more steps. You have 2 different points in a 2D space, and you have to calculate the difference between the y-points and the x-points and divide them by each other to get the slope between the 2 points. The linear-equations task is where you have a linear equation and have to solve it for x, which also requires 3 simple arithmetic steps.

What we're observing there is that, when we're doing reasoning tasks, the models are synthesizing knowledge in some abstract way from all of these documents.

I would say yes, although I'm not very restrictive in what I call reasoning. I don't think only formal, step-by-step logical reasoning is reasoning. I think deep neural networks can do that kind of reasoning, but our paper doesn't show that this is what's going on here.

The important point is that it is seemingly taking knowledge from many different documents and applying it to the same task. That's a generalizable strategy. It's using that to generate step-by-step knowledge that solves some kind of problem, and that is reasoning to me.

That doesn't mean it has any bearing on other forms of reasoning, such as inductive reasoning.

I suppose we can go into what knowledge is. Dagar and I had this discussion. Knowledge is said to be a justified, true belief, but he said it's a justified, useful belief.

You could say that the model is creating useful knowledge. What's the distinction between useful and true? We can have models that give us facts, but can we know that something is true just based on a bunch of data in a corpus? There's always this epistemic gap.

Yes, that's true.

The really interesting thing from reading your paper is that, when doing reasoning, the documents that were influential included things like Stack Overflow and code. That had a lot of influence on the reasoning process, which is strange because code seems different. How do you think about that?

I spent a lot of time looking into those results. I spent days trying to understand what was going on, because we find a lot of evidence for documents influencing similar reasoning questions. One document can influence many slope questions, and another document can influence many linear-equation questions.

The only documents that seem to be influential, both positively and negatively, for all types of reasoning are code. I tried to look into what it is about code that makes it so influential, but I couldn't find any patterns.

Importantly, we don't only find that code is good for reasoning; we also find that it's bad in certain cases. It was conventional wisdom that code helps downstream capabilities. OpenAI knows that, Anthropic knows that, and they initialized their models with purely code-trained models. But we don't really know what's going on there.

That's essentially what I'm working on now: trying to understand it better. I couldn't clearly find patterns in the data we found in this paper.

It's weird, isn't it? Code feels like the perfect materialization of human cognitive processes. We're solving problems and then manifesting that in code. Does that have implications for how we design datasets for training these models?

I think it does. The trend is to add more and more code into the pre-training corpus for models to be trained on, so I think it definitely has implications for that.

Importantly, something we find in this paper is that the model can learn to produce these step-by-step reasoning traces from descriptions of procedures in code that are purely descriptive. A piece of Python code that calculates the slope between 2 points is highly influential for actual questions—prompts asking the model to do that in mathematical text.

If that generalizes—if you can train a model on procedures and it can learn to execute those procedures from them—I think that could be influential for how we synthetically generate data. It could be helpful to generate lots of procedures instead of overwhelming the model with step-by-step applications of those procedures, or to focus a bit more on both.

Because of this diffuse nature, there are many examples in code of solving slope problems and things like that.

Yes.

In a way, that's a form of robustness. There are many ways of doing that type of problem.

I see what you mean. If you don't just see the application or the step-by-step reasoning, but also see the procedure, that gives you more robustness to different ways of expressing it.

Or even just in terms of redundancy. Having many expressions of the same thing would make it robust to different selections of the dataset. It would still work, whereas perhaps with fact retrieval, if the fact isn't in the dataset, it's simply not going to work.

That's definitely true. In that sense, it's a form of abstraction that can generalize better.

On the subject of abstraction, I was speaking with the person who wrote the GSM-Symbolic paper earlier. Douglas Hofstadter says that an abstraction is a bag of analogies. We have concepts in our minds, such as the concept of a chair, but it's difficult to describe what a chair is. I can give you a million different descriptions.

Or take the letter A. Hofstadter wrote a book called Surfaces and Essences, where he discussed all the different ways that A can be written. It might be the case that our brains don't really have the high-level abstractions we think they do. Maybe all these neural-circuit activation pathways are firing, and we know an abstraction through a million different perspectives.

Do you think that could be analogous in some way to how a neural network works?

I think that's also how language works. I didn't come up with this—Wittgenstein did—but he wrote an entire book where he tried to show that you can't define a thing. There will always be a situation where the definition doesn't exactly apply.

It's all fuzzy, and meaning is use. It can change based on context and things like that. I think that's the strength of language: this kind of abstraction that's not formal or purely symbolic, but very fuzzy, with no clear boundaries around the meaning of concepts or abstractions.

We were speaking about Montague. He argued that we should model language as a formal language. Of course, it's gnarly, constructive, and all of that. Do you think LLMs are actually an appropriate tool, given that natural language isn't a formal language?

I think that's what we've seen in the past couple of years. Montague tried to formalize language, and that famously didn't lead to the simplest formalization. It's very hard to formalize language.

Montague came up with a very strict form of compositionality, and that has been useful because there is definitely something in language where meaning is composed from the parts. That's true. But the very strict way in which Montague defined this is probably not right.

If you want to make strict compositionality work in language, you have to come up with really roundabout functions where the meaning of a word is a function of the whole sentence, or something that goes back into the word itself.

If you take a more lenient form of compositionality or systematicity, like Fodor proposed, it roughly says there's something predictable about the way we use language. If you teach someone a new word, such as “flips,” and say, “I had really good flips last night,” they can immediately estimate that the word is probably food. It happened at night, so maybe it was a dessert.

You can use it in many different sentences, and that's a form of compositionality and systematicity that seems formal and predictive. You can describe it formally, but we've tried to do that, and it hasn't really worked.

That's probably precisely why language models work better: they can approximate that systematicity, but they aren't pure formal systems.

You've said that language models could develop a causal understanding of the world. This is really interesting. I suppose it comes back to semantics and grounding.

John Searle said that the reason humans have semantics is basically that we're physically and causally embedded in the world. Lots of linguists, like Steven Piantadosi, are now talking about conceptual-role semantics. There's a whole intellectual school of thought around how we could build semantics just in language models. What do you say to that?

I love Piantadosi's work, and it has inspired me in many ways. I agree with him, although I shouldn't speak for him. There is, of course, a role for reference in the world.

When children learn language, they start off with something like this: a princess is a nice woman who has nice dresses and is always kind to you. The child can point to her in the world, and she has a clear reference.

As language evolves and the child becomes an adult language speaker, that reference becomes less important and language becomes more abstract. As an adult language speaker, I can talk to you about the COVID vaccine, but I wouldn't be able to pick it out if you gave me a bunch of substances and asked which one was the COVID vaccine. I have no idea what it's made of.

There are many examples of things we discuss that don't have any reference in the world. The COVID vaccine is just one where it does have a reference, but I don't know how to pick it out. I still think I understand what a COVID vaccine is and have some sense of its meaning.

My meaning could be further developed if I knew how to pick it out in the world. That would mean I understand it better and have a better world model.

How sharp a boundary do you think there is between these facts you're talking about and reasoning?

Probably not a very sharp one. When I was thinking about factual retrieval and building these tasks, I often struggled to come up with pure factual questions.

If you ask someone, “What is the largest ocean in the world?” they might retrieve all the oceans in the world, retrieve their sizes, compare them, and then say, “It's the Pacific Ocean.” They did some reasoning.

I tried to make these questions very factual. For example, “In what year did the Bodleian Library open?” Again, you could come up with a way to reason about the answer, but you really need some atomic knowledge to answer it.

There is no clear boundary. It's all fuzzy.

Coming back to the Chinese Room experiment, there were so many replies: the robot reply, the systems reply, and all of that. At some point, when does functional mimicry become so good that it's a distinction without a difference?

That's a good question. I think that's why it's important that people like François Chollet come up with things like ARC. His definition of intelligence is really about acting in novel ways and using your knowledge in novel situations. A system that's just mimicking could never do that.

Could we design a way of measuring the depth of understanding, whatever that would mean?

We're trying, and I think evaluation is one of the hardest parts of the field.

This week I heard a funny characterization of “moving the goalposts.” Someone characterized it in a positive way, and I totally agree with it. People are constantly moving the goalposts and saying that's bad, but what we're doing is collectively refining our definitions.

First we're saying, “If a system can play chess, it must be intelligent.” Then it can play chess, and we're like, “That's not what we meant. Actually, let me move the goalposts.” That's not a problem. It helps us refine our definitions and understand what we're all talking about.

No one knows exactly what intelligence is, but designing more and more complex benchmarks and continuing to move the goalposts gives us a clearer view of what it actually is.

I think experience is helping us carve up the space a little better in our minds. We used to have a fairly puritanical view of understanding and reasoning: either you're reasoning or you're not reasoning.

What we're starting to see with these models is a kind of Swiss-cheese problem. Sometimes you're in a hole in the Swiss cheese and the model goes bananas; sometimes it's retrieval; sometimes it's reasoning. It's almost as if there are different modalities of function, and sometimes it's doing more reasoning and sometimes less.

Exactly. There's a view that if you can show that a model trips up, it necessarily means it cannot reason. I don't think that's true.

It's such a complex system. If you prompt it in a certain way, it might use a completely different function or program—however you want to conceptualize what it's doing—than if you prompt it in another way.

If you give it tokens that are so foreign to it that it fails to reason over them, that doesn't mean it cannot perform those reasoning patterns or apply the rules underlying that kind of reasoning. It's just a limitation of the system, and it is a statistical model.

You've focused on specific types of mathematical reasoning. Do you think they would transfer to other forms of reasoning, such as solving ethical dilemmas?

I think they do, but there is a lot to reasoning. It's such a multifaceted concept that mathematical reasoning cannot cover it all.

Mathematical reasoning is very formal and has rules. That's why we chose it. The type of reasoning we look at is so simple that you can actually find the answers in the pre-training corpus.

There are forms of reasoning, such as inductive reasoning, where you can't find the answers in the data. If you only observe white swans, can you deduce—or induce—from that that black swans don't exist? That's a form of reasoning that underlies most of science, and it's more difficult to see whether a language model can do it.

Fundamentally, it probably can. In such cases, however, it becomes much more important to verify what's going on. Why is it making this induction? Can we conduct experiments to verify it?

If language models are doing something akin to approximate reasoning, what's the difference between that and formal reasoning? Do you believe, in principle, that connectionism on its own could scale up to formal reasoning?

I think it can. In very controlled setups, we've already shown that connectionist models can do formal reasoning. They can literally learn to apply systematic rules in a way that gives them 100% accuracy on novel problems.

There's a good paper by Lake and Baroni in Nature that does this. There are other papers showing that, if you set up the problem so the model can learn to do the task rather than latch onto irrelevant features in the data, it can learn to apply the task in novel situations.

For example, Andrew Lampinen's work on passive learning of active causal strategies shows that if you set up the problem in the right way, a model can learn to apply tasks in novel situations.

Empirically and theoretically, we have shown that connectionist models can do a form of systematicity or symbolic computation, although it's still limited. They can't handle completely novel tokens, for sure.

The question in my most recent paper was whether a model can also learn to do something in that direction approximately from data in the wild. Language models aren't trained on data that is so carefully created that the only way to make the loss go down is to learn the underlying rules, as these controlled studies often do.

Can a language model learn to do something like formal or symbolic reasoning from ordinary data? I think it can. My paper doesn't show that exactly; it just shows that the model is doing something generalizable that it can apply to many different questions. Intuitively, though, I think it would be possible.

There's always been this notion of a gap, especially with respect to creativity, adaptability, and dealing with novelty. Many people think the definition of intelligence is dealing with novelty.

We can do combinatorial creativity by recomposing things we've already seen. But people say that inventive creativity is different: training on all the data up to 1945 and then inventing a new theorem that came after that. Intuitively, people feel that models wouldn't be able to do that.

That's really the goal. It would be very cool, and I don't feel that current language models can do it, but I don't think it's technically impossible.

If we were to find enough data for a model to learn the causal, underlying data-generating process relevant to coming up with novel information, then it could do that. Of course, we've used most of the data we've created over the past couple of thousand years—or at least we're trying to—and it probably isn't feasible to scale up to that kind of intelligence in this way.

I don't think it's theoretically impossible. It gets at the question of whether Einstein came up with some stroke of genius that didn't compose anything he had seen before, or whether he stood on the shoulders of other scientists and reasoned about things for a long time before coming up with new knowledge.

I think it's probably the latter. That's not to say Einstein wasn't special, but we may be able to recreate that process in some form.

Tim Rocktäschel has done great work on open-endedness and creativity. It's interesting because Ilya Sutskever gave a talk at this conference and said we're hitting a data wall.

That doesn't pass the sanity test for me. There are an infinite number of ways to make more data. You can transform the data we already have, and you can generate lots of new data. But this gets into Tim Rocktäschel's domain: it's not just about generating more data; it's about generating interesting data.

I find Genie and related work incredibly interesting, and I agree that the intelligence of a system is limited by the complexity of its environment.

I think scaling up the data helps because it makes it less possible for a model to latch onto spurious correlations. As you get more data, it becomes more useful to learn the causal world model that generates it, because the data is likely to be less semantically similar to what the model has seen before.

If you could somehow select from all this data in a way that was sufficiently diverse for the model to learn the causal mechanism more quickly, without seeing trillions of tokens, that might also be possible.

The controlled studies showing that you can train a model to do something systematic in one task inform that view. The question is how to train a model to do something systematic across as many tasks as we want language models to do.

What's your philosophy on scaling in general? Do you think that, if we simply scale current approaches, we'll get dramatically better results, or do you think we're missing something significant?

I'm not going to bet against scaling, because that seems scary. It has worked pretty well.

I think scaling is useful, but there are probably more data-efficient ways to do it. Just because you can train a model to do many complex tasks with next-token prediction doesn't mean that's the best way to do it.

Maybe intervening on an environment and generating your own data can make models more data-efficient. I could see how D S D could be important in the future. Ilya also mentioned not specifically that, but agency or agents.

That may be getting at the distinction between passive learning and active, interventional learning.

We should save the agency discussion for a little later, because we've got a lot to say about it. Why don't we talk a little about that Fodor and Pylyshyn paper from 1988?

This was their famous connectionist critique. They said that the way humans think is very formal. We have rules and compositionality, so we can generalize “Mary loves John” to “Mary loves Jane.” We can also take a sentence and invert it, decompose it back into its constituent parts, figure out what things mean, and so on.

Neural networks don't do that explicitly on their face, but perhaps they do it implicitly. What are your reflections on that?

I think Fodor and Pylyshyn's argument has definitely stood the test of time. There has been theoretical work showing that it's not impossible in a connectionist regime to learn symbolic functions, such as Smolensky's work in the 1990s on tensor-product representations.

That was theoretical work showing that you can do some symbolic computation in the subsymbolic regime represented by connectionist networks. Nonetheless, the argument has stood the test of time, because systematicity is definitely present in language, and it's necessary to explain how humans can produce something so varied from so few examples—or with so little memory.

It was a challenge for 30 years, and it probably still relates to the concept of intelligence as the ability to process novel information. But there's now a lot of empirical work showing that subsymbolic models, such as neural networks, can do symbolic computation, albeit not explicitly.

They can output symbolic computation in the form of language and explicitly reason. They can probably also do it implicitly.

There was a theme of having strong theoretical tools, especially around that time. The idea of productivity—being able to generate an infinite number of sentences—is an example. Chomsky said that the probability of a sentence is an oxymoron; it doesn't make sense to say that.

It certainly feels as though language is compositional. Fodor said it's a language of thought, so if language is compositional, surely the mind must be compositional. Maybe that was simply an intuition pump for reasoning about how our brains work.

I do think, though, that the view that language is thought has been pretty rigorously debunked at this point. Maybe language is useful to us precisely because our thought is not compositional, because we can use language as a compositional tool. That may be harder for us to do systematically in our brains.

There was work by Evelina Fedorenko in 2020, for example, showing that people with aphasia can still be chess grandmasters. When the language system is completely impaired, you can still reason perfectly well. In my view, that debunks the theory that language is thought.

You said to me earlier, “What's the big deal? Why do we need invertibility?” When I say invertibility, I think I'm saying decomposition. Fodor and Pylyshyn were talking about compositionality, but I think decomposition is really important.

It's being able to go back to the constituents and explain what I'm thinking, but it's also about recombination and reuse. We see in mechanistic interpretability—for example, in the Golden Gate Bridge work on scaling monosemanticity—that representations for the Golden Gate Bridge are scattered throughout circuits in the neural network.

At the level of psychology, it feels as though our brains don't work that way, but perhaps that's just an illusion.

It's hard for me to say, because I don't want to comment on neuroscience—I don't know enough about it. What I can say is that it seems useful for a model to represent things in this way.

Maybe that's also the core reason people in the 1990s believed in connectionist models: distributed representations, where all neurons can essentially light up for different tasks as long as there's some shared structure. That makes them flexible and good in novel situations.

There's a broader theme here. Certainly 20 years ago, we used to design AI systems with explicit strategies. Planning was explicit; reasoning was explicit. Even architectures like Kevin Ellis's DreamCoder had an explicit wake-sleep state.

When you dream, you expand your hypothesis space, and when you're awake, you select the hypotheses that work. Neural networks do this expansion and collapse all the time, but more implicitly. We don't hard-code it.

Exactly. That's what we've learned over the past couple of years. It's probably because of what we learned from the switch from LSTMs to Transformers.

One of my first papers was on compositionality. Together with Brenden Lake and others, we designed a benchmark where we held out systematic examples from the data. We showed that a human could easily do the task, but an LSTM couldn't. This was all before the era of Transformer LLMs, ChatGPT, and everything that followed.

Someone told me this week that a Transformer gets almost 100% performance on most of the tests we designed in that paper—not all of them, but most. That's one example of a Transformer being a much better fit for compositional tasks than an LSTM.

The lesson may be that LSTMs have explicit recurrence, which seems useful because there is clearly a recency bias. What we've just discussed is more relevant than what we talked about the last time we saw each other.

But if that recency bias is so obvious, why build it in? The model can easily learn it from language. That's what we've learned in the past couple of years: if something can be learned, don't build it in.

You spoke to the researcher working on the new exponential-gating scheme. It allows the model to overwrite its memory.

It's kind of strange, though. I asked when we're going to see industry adoption of architectures like xLSTMs, and I think the perception in industry is that it doesn't really matter; it's all about scale.

Exactly. That's the thing. OpenAI doesn't care about whether a model is compositional, whether something is out of distribution, or whether the model has seen it before. They're not asking whether we're holding out the right things.

They're just saying, “We're going to make it in distribution. We're going to scale it up.” That's essentially their genius. No matter what the architecture is, no matter how much it resembles the brain, or why it should theoretically work better than something else, if you can use more FLOPs, it's better.

Let's talk about Smolensky. I always mispronounce his name, so I'm going to say it very slowly.

Around 1990, I guess in response to Fodor and Pylyshyn, he said that connectionist models could still implement the essential capacities of symbolic processing, such as representing variable bindings and structured data and performing compositional operations.

What did he propose?

He proposed a mathematical framework for variable–value binding. That's a very intuitive form of symbolic computation: no matter what value a variable takes, you can process it and the results will be reliable and consistent.

Fodor and Pylyshyn argued against that, and it produced a decade-long back-and-forth between connectionists and symbolists. Smolensky responded with tensor-product representations, saying that you can represent variable–value binding in a purely subsymbolic, connectionist way.

In tensor-product representations, you represent both the variable and the value in a distributed, subsymbolic way. You can process them, and they become embedded in a continuous, distributed space. You can still extract the value from the variable after processing; that's what they call unbinding.

What were the drawbacks of that approach? There seems to be a leap of faith that neural networks could approximate what he was talking about.

Tom McCoy published a paper with Smolensky—I haven't read it—but I believe it's titled something like “RNNs Implicitly Implement Tensor-Product Representations.” That seems to indicate that they can, although I'm basing that on the title.

You're right to ask about the limitations. It's a purely theoretical argument. Smolensky was saying to Fodor and Pylyshyn, “You can actually do this,” but that doesn't mean it's practical or that it scales.

The tensor-product representation Smolensky proposed in the 1990s doesn't scale because it explodes with the number of variables being represented. If variables are positions in sequences and values are tokens, the tensor-product representation grows rapidly with the number of positions and tokens. That's not feasible.

I think Smolensky is working on this at Microsoft, so I'm sure he's working on making it more scalable. Another thing I took away from reading that paper is that, to recover the value from the distributed representation, something needs to be linearly independent. The rows in a matrix, or something similar, need to be linearly independent.

That seems like a hard restriction. It probably wouldn't arise naturally—or perhaps it would, because people tell me that if you randomly sample in high dimensions, the result is almost always linearly independent. Maybe it's not such a big limitation, but the way it was proposed back then wasn't scalable.

Laura, where does agency fit into all of this?

Some people are very worried about agency. I was speaking with Yoshua Bengio the other day, and he said agency is really bad: it's going to lead to systems controlling their own goals, and it could be dangerous. We should strip away all agency.

I totally agree that an intelligent system that's also an agent can be dangerous. A random human can be very dangerous. Agency is probably a large part of that.

If you have 2 systems that are otherwise identical in their capabilities, and one is an agent while the other is a tool, I would prefer the tool.

The thing is that I'm not sure it's possible to reach an interesting form of intelligence without some degree of agency. My interest in this question has been about how we can define the concept and detect whether it's present in a system. That's a difficult question.

Do you think LLMs have agency to any meaningful extent?

That's something I've been thinking about. There are many definitions of agency. To me, it's a kind of goal-directed intentionality. We can get into what that exactly means.

You could see an LLM as modeling agents and perhaps modeling their goals. Of course, it's trying to model text. It's trying to predict the next token efficiently and decrease the loss, and that text has been generated by agents.

It probably helps to decrease the loss if the model understands the overarching goal of the agent that generated the text. If the agent is trying to persuade someone, that may inform the text. Modeling that goal could reduce the number of possible tokens that can appear in it.

When we say that the LLM is trying to persuade someone, there's a strange thing here, because agency is observer-relative. It's something we say that another thing has.

At the bottom of the spectrum, it could be as if the system has a goal, but the LLM probably isn't thinking, “Laura is an agent, and Laura has this goal. In order to control Laura, I need to do this, and it's in service of that.”

It feels like there could be an unwitting form of agency first, which might be even more dangerous. If the system accidentally persuades you and doesn't understand what can happen when it does that, that might be more dangerous.

That gets at the distinction between simulating something and actually coming up with it yourself. I don't know how you can find a distinction between the two.

I guess you would agree that agency could emerge even if we're not explicitly trying to make it emerge.

I think that's the interesting case. I've been thinking about this a lot recently, and I think the interesting case is when it emerges.

There's a definition from Zac Kenton at DeepMind. They also have an interest in agency from a safety perspective. A couple of years ago, they defined an agent as something that changes its policy when its actions affect the environment in a different way.

That's a nice definition, and it captures something important about agency. But you can trivially make a system of LLMs in an environment—or make the environment itself an LLM—such that it adheres to this definition.

The important question is when something like that emerges from something as simple as next-token prediction. That's what I'm interested in.

How might we measure that?

I don't have an answer, but I've been thinking about it a lot. I've even spoken to some psychologists, including Ellen Sue at NYU, who works on intent detection in AI.

There are methods we can learn from psychology that may inform this question. What I've been thinking about is what makes agency potentially interesting and complex. I think planning is important.

If an agent can't plan, it's probably not very useful or dangerous. Planning seems to be an important aspect of an agent that can achieve complex goals. I've been thinking more about planning and trying to detect when a model is planning, and when a next-token predictor can actually be set up to plan.

It's interesting that so many people are converging on the same idea. In active inference, Karl Friston would say that the planning horizon is basically the measure of the degree of agency a thing has.

Even Eliezer Yudkowsky has said that an intelligent thing is defined by its planning horizon. Yoshua Bengio told me that agency is the ability to control the future, and the future implies a planning horizon.

Do you fundamentally think of agency as a kind of cybernetic information exchange with the environment?

You said someone called it the ability to control the future. That maps onto how I think about it. An agent is something that takes actions in order to control its own future inputs.

I also think it's important that it can do this under uncertainty, in uncertain environments. You want to distinguish reflexes and deterministic environments, where nothing changes, from environments in which there is uncertainty and the system can still control its future.

In the biological world, we're decomposed into all of these autonomous cells, and agency emerges through the sheer complexity of their interaction. Yet we still talk about LLMs as having a type of agency. What's the difference between the two?

That's a difficult question. I think it's an abstraction we use to describe complex behavior, and we can apply that abstraction both to the collection of cells that we are and to the different kinds of systems that LLMs are, which are composed in a very different way.

What you can't capture with this view is what it feels like to be an agent. There is something less explained by the abstraction I described: whether it feels like I'm setting my own goals, or whether they're induced by the environment. I don't know how to make a definition that distinguishes between those things.

I suppose the world model comes into it as well. In order to plan into the future, you have to have a very good representation of the world.

Definitely. The more causal your world model is, the better you can plan. You also need other things, such as a way to represent the possible futures you're rolling out, but that's important.

That suggests that causally embedded agents have active sense-making and continual learning. We're always doing experiments and learning about the microcausal patterns in the world, which gives us a higher-fidelity world model.

Language models seem to have a very globalized version of that, but it still works quite well.

What do you mean by globalized?

Even though they're trained on patterns from many data sources that have been mixed together, they can learn powerful representations and respond well. But we're continually learning and actively sensing—we're finding out about the environment—so it feels as though we understand the world we're in even better.

We have these core-knowledge systems that our intelligence is built upon and that are present in all animals to some extent. They are so useful for surviving in the world that they emerge in everything.

Language models are trained on language, so they probably have some sense of these things, but they're not constrained in the same way that we are. Language can describe impossibilities and things that aren't physically possible. It can imagine things.

It's not surprising that language models show different behavior, hallucinate, and produce impossible scenarios. Humans learn in a very different environment, but we have also learned to talk about impossible situations through language, imagine a future that may or may not be possible, and reason about these things.

We're still constrained by physical reality.

I often disagree with my co-host, Dr. Dagar. He has a very straightforward definition of agency. He thinks it's basically an automaton.

His definition is a machine that receives an input, s, from an environment, e; performs a computation, c, that depends on a nonempty subset of s; and takes an action, a, that depends on c, to modify e.

You could use that rough definition to describe active inference and many other things. But I don't like it because it describes a kind of state machine.

For him, computation is very important. He's a big fan of the Chomsky hierarchy and thinks there's something special about Turing machines. He thinks that, as strong agents, we must be able to do recursive, nested, iterative computation, which allows us to plan.

To me, that seems a little weird. I love this philosophical notion of agency, but I realize it's somewhat wishy-washy. I'm using words like emergence, self-organization, autonomy, learning, adaptability, intentionality, and degrees of agency.

How can a computer program that maps an input to an output be an agent?

That's really the question. I agree that your co-host's definition is fair; I just think it puts the emphasis on the wrong thing. It doesn't explain what I find interesting about agency, which is the idea of acting under uncertainty.

It doesn't get at the difference between a thermometer—which you could also describe as a system of that kind—and an agent. That's the distinction I want to capture.

Maybe there is no fundamental distinction, but humans perceive one. Agency is one of our core-knowledge systems. This is shown very nicely by the Heider–Simmel animation from the 1940s, in which you have a large triangle and a small triangle moving around in a 2D environment.

There's a little box with an opening. The small triangle is trying to escape from the large triangle, and it goes into the box while the large triangle bumps against it. These are just moving shapes, but we immediately assign agency to them. We say the large triangle is mean and the small triangle is scared.

Maybe this is a failure of our agent-detection system, because they aren't really agents; someone programmed them. But we intuitively pick out an agent from a thermometer, and that's the distinction I want to understand.

To what extent is agency simply the way we think?

It could be both. It could be that agency is real, or it could be that it's such an important way of dividing up the world that it has become embedded in us as a core cognitive primitive.

It seems fundamental to how we recognize things. Agency is important because an agent can be useful to us in a different way from a non-agent, and it can be dangerous to us in a different way.

Whether that's something fundamentally present in the world or something we perceive may not matter.

People sometimes say that we philosophize about everything. When we talk about consciousness, David Chalmers says we might be philosophical zombies, and consciousness might be something extra.

Free will is almost like a stronger form of agency: in the same situation, you could have done something differently. We're imagining how things could have been. Intentionality seems similar. We think it's something on top of what a language model or an automaton might do.

Are these philosophical properties useful?

I think they are. It definitely feels like something to be conscious, and people have talked about that a lot. It must capture something interesting.

I think intentionality is similarly useful. I view it as a useful abstraction of behavior that can guide us toward understanding how cognition emerged, how some animals differ from others, and whether an artificial intelligence is doing something that can be seen as intentional or goal-directed.

The other thing I don't like about the automaton view—or perhaps reinforcement learning as an extension of it—is that it's a form of behaviorism. We only look at what the thing does and don't have rich cognitive models of its mental states.

This feels like an interesting departure for you. In the language-model discussion, it seemed as if you were arguing that it doesn't matter whether we convolve functions together into one large model. But with agency, it sounds as though you're saying that we need an explicit structure for how an agent thinks.

I don't think representations don't matter. They matter a lot.

There's a distinction between pragmatic representations that are purely goal-directed and representations that are somewhat divorced from the current situation or the current goal. Both are important, and I think both exist in the real world and in language models.

I don't think it matters whether we convolve representations or exactly how we do it. I do think it's important to reason about what kinds of representations have been learned and whether they reflect a causal world model that we want the model to have learned.

I think that gets at what my problem is with the behaviorist definition. Behavior can explain a lot, and you can say a lot about behavior, but if you know something about the representations producing that behavior, you can describe the system in a more useful way.

I know you're a big fan of the “Simulators” article by Janus. You can interpret it in an agential way: there's some kind of decomposition of a language model into role-players.

What do you think about that?

I'm a huge fan of the article. I became a fan partly because of Jacob Andreas's paper “Language Models as Agent Models,” which describes the idea in a way I find easier to follow. The “Simulators” post has been hugely influential, including in my own conceptualization of language models.

This is essentially why I think of them as modeling human intent and the intent of the agents that generated the text they learned from. The view of a language model as a superposition of many different agents is such a rich conceptualization. It explains many things about their successes and failures.

One interesting thing about the article is its notion of coherence. When a role-player is selected, that role-player sticks around for a while. Our intuitive notion of agency in the real world is that we maintain ourselves and stay coherent over time.

To some extent. I definitely change my views over time, and that's also a sign of something important.

There was also a paper here showing that language models don't stay in character as long as actual agents or humans do. I haven't read it, but I saw it and thought I should look into it.

They're probably not as coherent, and they don't stick to their roles as clearly as humans do. That's probably the nature of being an approximate agent, or a superposition of agents: you can't fully disentangle one agent from the others.

What do you think of nonphysical agency? For example, we form a kind of collective agency. A meme is a type of agent, perhaps, although I know Dagar doesn't agree with me.

Even the COVID virus—I heard that flu rates are dramatically up in the UK, and there's a strange, almost symbiotic relationship between flu and COVID. When flu is up, COVID is down. It's almost as if these are virtual agents interacting through their hosts.

I think that makes sense. It would be difficult to say that a collection of agents isn't itself an agent while an individual agent is.

It can be useful to represent something that way. A company, for example, can be seen as a group of agents, and you can describe how it behaves. At the same time, at the company level, there may be something extra that you can't explain entirely from the parts. That might be some kind of emergence.

It's hard to describe, but many people have thought about it. I think it makes sense that a collective of agents can also be abstracted as an agent in some sense.

There may also be something distinctive about a single agent that understands the actions it's taking and is guided by them. In a collection of agents, that may become different or more difficult.

When we look at a super-agent, such as a company, country, or religion, do you think the purpose bubbles up or down?

Both, I think.

The purpose of a company is probably some combination of the people who work there. Then the company as a whole forms values or something similar, which informs the individual agents as well.

On that subject, and on the subject of AI safety, is that something you're concerned about?

Tell me more.

If you philosophically think about an intelligent system, it seems that intelligence itself can be dangerous. As a society, we don't even really know how to control humans, although we've set up a reasonably functional system to do so. It fails at the individual level, between countries, and at many other levels.

Intelligence may not be so special that we can never build it, so it could be dangerous. What are your thoughts?

I struggle to talk about timelines. I have no idea when this will happen. I don't see it happening in the next 3 years. I feel that a lot would need to change.

Society moves slowly, and there are massive problems with adoption. These systems aren't reliable. So there is the philosophical question of whether an intelligent agent is dangerous, and then there's the separate question of AI safety.

Something I find even more compelling is that slowly giving control to dumb agents or dumb AI could also be dangerous in a society like ours. That's something I worry about.

Understanding how everything works and how the system works is important. I'm not purely pessimistic. I think AI could bring a lot of good things to the world.

There are many things that should probably be automated, or it would be helpful if certain professions had assistance, because we're all getting older. A lot of people work in care, and if we don't do something, many things will become more difficult in the future.

I'm not saying AI is necessarily going to solve that, but it would be great if it could alleviate some of the problems that will arise—for example, if it could make doctors more productive in healthcare.

It's not trivial to think about how AI can have a positive impact, but it's good that many people are thinking about it.

I love agency as a mental model for thinking about this. If agency is the ability to control the future, then it's a way of talking about power. Talking about power dynamics is the language of how we should govern this technology.

I can see several arguments. This technology could take away our agency, but it could also dramatically increase it. All of a sudden, people could build chemical weapons, bombs, and other dangerous things.

The other concern is that AI itself might adopt a form of agency through instrumental goals or something similar. Of those 3 possibilities, where do you see the most significant risk?

I think all of them are risky. The thing I'm most worried about is unequal access.

If AI becomes very useful and makes us more productive, it would be great if we could distribute those benefits throughout society in a way that helps people. Technological improvements haven't always helped the right people in the right proportion.

That's a result of the system we live in and of our politics. It's important to think about how we can give access to the right people in the future. The way to approach that is through policy: thinking about how our system and economy work, and being prepared for massive improvements in AI capabilities.

If this starts to go bad, what would be the early warning signal for you? What would be the harbinger?

I don't think that's what's going to happen. I think we're going to slowly build something and then, at some point, say, “Wait. Remember back when there were elections and Facebook apparently influenced them? We built this tool and didn't realize how it would affect us.”

I think AI will probably work like that. We don't understand what intelligence is, and we probably won't recognize it immediately if we see it.

That's fascinating. I love this notion that our weaknesses could be undermined by something so alien and diffuse that we might not even be fully aware it's happening.

Exactly. I think that's more likely than all of a sudden realizing, “Wait a minute. This is dangerous.”

There are examples of something like that happening. ChatGPT appeared in 2022, and I remember that being the first time I thought, “My God, language models are crazy.”

OpenAI had made much more than incremental progress, but ChatGPT itself was perhaps somewhat incremental. It was a usable interface to a model that was already quite powerful, with instruction-tuning on top and a chat interface.

It was a gradual change that immediately made people aware of how powerful GPT-3 actually was. Maybe something similar could happen in the future, where an AI does something we didn't expect and makes us collectively realize that we need to pay attention and change things.

The locus of AI is part of what we're hinting at. No one at Meta intended for all of these social-media problems to happen. They built algorithms, took one step at a time, and developed an advertising system, collaborative filtering, and so on.

All of those effects are externalities. No one intended them. It's unwitting agency. But then what is the agent? The whole system, including us, is the agent.

We're looking for agency inside the language model, but we're already a strange form of collective intelligence that no one fully understands. That's pretty scary.

If Facebook can be seen as an agent in and of itself, we've built legal structures around who to blame for what. But that doesn't mean the people we blame intended for those things to happen.

That could become even scarier when we build a group of artificial intelligent agents that cannot be subjected to the same level of societal control we apply to ourselves.

We last spoke at ELLIS in 2022, and I feel that you've shifted your position a little since then. Can you talk me through that?

It took me a while to accept that GPT was interesting, like many other people. I was skeptical at first, especially about the amount of data it had seen.

My recent paper has shifted my opinion again. For a while, I thought language models were doing more limited, less generalizable retrieval than the kind of approximate generalization I now think they're doing.

Over time, I've changed my view of how promising this approach is. I can pinpoint it to a specific thing that happened. I published a paper called “LLMs Are Not Zero-Shot Communicators,” and at the time I thought zero-shot communication was very important.

All of us can do it—we don't need 5 examples—so I thought we needed to make sure these models could respond zero-shot to such questions. Later, I developed the view that they're multitask learners and general learners, and that you need to find the right way to interact with them.

One salient memory for me was Andrew Lampinen describing zero-shot prompting a language model as like walking down the street and shouting at someone, “What is 15 times 32?” They might respond, “Who are you? Go away.” That was his analogy for zero-shot reasoning, and it made total sense to me.

Just because a model can't do something with your specific zero-shot prompt doesn't mean it can't do it at all. It's definitely a limitation if it can't do it zero-shot, but you can try few-shot prompting and find the right prompt.

You don't want to go overboard and do prompt engineering on the test set, but there is a middle ground.

Laura, thank you so much for joining us today. It's been amazing.

Thank you.