Nested Learning: Ali Behrouz on the Quest for Continual Learning & Illusion of AI Architectures

Hello, and welcome back to The Cognitive Revolution. Today, I'm excited to share a conversation with Ali Behrouz, a graduate student at Cornell, a researcher at Google, and the author of Nested Learning. This episode was recorded a few months back, and while I normally believe that AI content does not age well, this conversation with Ali is an exception.

His work is some of the most inspired and potentially transformative that I've seen anywhere in the quest for new machine learning architectures capable of genuine continual learning. This is, of course, one of the most important capability advances on the horizon today. Arguably, it is the main gap between today's models and a digital AGI that would be capable of joining and contributing to human teams just as humans do.

Ali is advancing the frontier with an approach that is both biologically inspired and technically elegant. His blockbuster paper Nested Learning, which has been touted as a harbinger of a possible paradigm shift by no less than Jeff Dean, develops a simple strategy that allows models to rapidly adapt to their current context on an ongoing basis while preserving core knowledge by updating different parts of the system at different frequencies, much like humans manage memory on multiple timescales, from working memory to long-term memory.

His latest work, Language Models Need Sleep: Learning to Self-Modify and Consolidate Memories, which I actually heard about live for the first time on this recording and which has now finally become fully public, takes inspiration from how humans consolidate memories and learn from dreams while sleeping. It introduces a new offline mode in which models transfer new knowledge from their high-frequency update layers to their more slowly evolving layers via distillation, and also learn new abstractions and connections between concepts by generating and training on synthetic data derived from their recent experiences.

In addition to the details of these architectures, which, like so many AI innovations, I find both extremely exciting and a bit scary, we also discuss how scaling for performance may shift from stacking more layers to nesting more frequency update rates. We discuss how Ali understands all components of machine learning systems as forms of associative memory that compress a given context flow, why this leads him to call deep learning architectures an illusion, and how he has operationalized this conceptual insight by developing expressive optimizers that learn update rules and are capable of outperforming both Adam and Muon.

We also discuss how the attention mechanism can be understood as an infinite-frequency update module, and why Ali expects that attentional layers will therefore remain fixtures of AI systems indefinitely. We cover the empirical results showing that Ali's new architectures compete effectively with transformers on standard measures, while also outperforming them on hard tasks, such as effectively recalling information from up to 10 million tokens of context and learning to translate multiple previously unseen languages at the same time.

Finally, we discuss why Ali sees continual learning as both an opportunity and a huge risk for privacy and alignment, how human-AI relationships might evolve, and why Ali is cautiously optimistic that models that evolve over time based on our interactions with them could both serve our individual needs more effectively and lead to a more diverse and hopefully stable AI ecosystem overall.

The bottom line for me is that, for all the debate and speculation about whether or not current architectures can scale to AGI and beyond, there is a very good chance that conceptual breakthroughs will render that question moot before we even manage to answer it. Transformers have changed the world, clearly, but they aren't the end of history.

And as tough as it is to keep up with AI developments, anyone who wants to get a handle on where things are going from here can't afford blind spots when it comes to new research directions like Ali's. And so, without further ado, I hope you enjoy this deep dive preview of AI systems that learn on an ongoing basis in increasingly human-like ways with the brilliant Ali Behrouz.

Ali Behrouz, author of Nested Learning and the new Language Models Need Sleep: Learning to Self-Modify and Consolidate Memories. Welcome back to The Cognitive Revolution.

I am super excited about this conversation today. I appreciate you being willing to take some time and come back to do a deeper dive into your work. I think it is super fascinating and genuinely some of the most inspired work that I have seen in recent times.

A big part of your method, as I understand it, is looking at what human cognition consists of and identifying things that we as humans are doing that seem quite important and critical to our successful function in the world, and then trying to figure out what an AI version of that might look like. From there, you start to develop architectures or system designs—maybe even more abstractly than architectures—that make those capabilities possible in AI systems.

It's really striking both how well some of these ideas have worked and how elegant they feel to me, and how right they seem as I take the time to dig in and understand them. So, first question, just big picture: How do you think about what it is you are trying to do?

Obviously, you identify gaps in what current systems can do. How do you think about those gaps? How do you conceive of what it is that you are trying to unlock with the new architectures that you're developing?

Getting inspiration from the brain, and what that means for me, probably means different things for different people. For me, I really like to get inspired by the brain and generally by evolution. The main reason is that I think it has a lot of data to train itself in a natural way of training.

One thing that we can see now is a very evolved version of a very complicated biological brain. Generally, it's a great source of inspiration. But when I'm saying that, I don't mean that I want to replicate the brain and fully do something that the brain does, because most of the time we don't know what that is.

I think there are different levels of understanding about how the brain works. The first one is that we know it works. That's the first level. The second level of understanding is that there are some modules in the brain, each of which is responsible for different parts. We have memory, we have other things, and that's generally the process.

I think the hard part when we want to draw inspiration from the brain is deciding at what level of granularity we want to focus on the brain and draw inspiration from it. If you go too much into the details, such as how the brain works and so forth, there are 2 issues with that. One is that you're overfitting ourselves to 1 specific form of intelligence. Another is that we don't actually know how the brain does that specific thing.

Generally, in all of the work that I have done—for example, on Titans and also on Nested Learning—one thing that is happening is that we can see the models facing some challenges in real-world applications. The question is: Is that a specific challenge that is solved by humans, and can a human do something to address that challenge and overcome it or not? Then the second question is how they can do that.

Again, for example, in Titans we discussed the surprise metric, and we discussed how memory should be decomposed into short-term and long-term memory, and so forth. But the point is, there's a high chance that the brain is not exactly performing gradient descent to, for example, understand surprise or something like that. That's just a high-level understanding of how the brain works.

I think if we keep that as a source of inspiration at that level, it would be a great source. But if we go into more details, then potentially we might face some challenges, specifically because our understanding of the brain is changing over time, and so we might overfit to 1 specific set of design choices.

That's generally one thing. But about Nested Learning, I think one thing that is missing in the current models is about 2 parts.

One is about how they can adapt to the environment and context that they are in. Another point is about how the model can understand new knowledge and incorporate it into its parameters over time. They can make sure that they don’t face catastrophic forgetting, which means that, for example, one specific task that they have trained on is forgotten, and they don’t have any skills in that specific direction. I think these 2 are important things for current models, and they are facing a lot of challenges in that regard.

Because if you have a very large model, then your model needs to be updated over time. That’s generally the main reason you can see that, for example, there is a knowledge cutoff for all the LLMs that we know of. For example, if you ask ChatGPT for some specific information and then say, “You are not allowed to use any tools for answering this question,” then you might see that there’s a knowledge cutoff, and that’s a little bit challenging to overcome.

If you just want to keep updating the models, then there are 2 huge challenges. One is catastrophic forgetting, which I mentioned, and the other is efficiency. If you have a lot of parameters, you cannot keep updating all of them, so it’s a little bit hard.

There are some solutions for that. I think they are also great, but I have some intuitions about why they might not perfectly work for the case of continual learning. For example, one might say that they want to do supervised fine-tuning, or SFT, or do some RL stuff. But the model can still face catastrophic forgetting, on the one hand, and on the other hand, at some point, you need to transfer all the knowledge that you have in your context and pass it to, for example, the actual parameters of the model.

If you just keep the tokens and want to do everything about the memory and learning process in the token space, then the main issue would be that, at some point, you really pass the context lengths of the LLMs. Potentially, you will face some challenges in that direction. Considering all these things, the main issue with the current LLM paradigm is that they cannot continually learn and obtain new knowledge and new skills over time.

Also, they are limited in understanding different levels of abstraction of the world. Everything that is done in science is to do something that can explain the world in the simplest way possible. That’s generally the way we can learn something, because we don’t want to keep everything. We want to understand underlying patterns that can describe specific data and specific knowledge, and so on and so forth. This compression process, and how we can understand different levels of abstraction from the data that we have, is something that the current LLMs fall short on somehow.

So, a couple of different angles I want to probe your intuition on a little bit more. Certainly, I think most users at this point have felt the problems that you’re highlighting. In some ways, I feel like the biggest advantage that I have relative to an AI today is this kind of ongoing coherence and reasonably stable identity. I know who I am in the morning, and I kind of know what I was trying to do yesterday. I can mostly pick up where I left off, and I probably could learn a lot more from the things that I do on a daily basis, but I at least learn some things and take them on board.

Obviously, the current models don’t really do that, and that is a big weakness for them. That’s why I was so excited about the original Mamba paper when that came out, because I was just like, “Wow, here’s something that seems like it’s competitive with Transformers, but it has a fixed-size memory space.” Obviously, we can’t grow the memory space quadratically to infinity, so we’re going to have to have something that’s bounded in size that can work.

That was obviously a notable step, and you’ve done some work with the Mamba architecture as well. Titans is even more so: here’s another way to think about having a fixed-size memory module that, in that case, was updated with gradient descent at runtime. That potentially seems like it would be more powerful than the Mamba architecture, and it certainly seems like the data supported that notion. But it’s a similar kind of structure: a fixed-size thing that can keep what it needs and gradually let go of what it doesn’t. That seems super important.

I wonder if you come at it also from the other angle. So far, we’ve kind of said, “Here’s something that it can’t do. We can do this; it can’t do this.” That’s one way to think about it. Another way to think about it, obviously, is: What do we want our AIs to be like?

Today, we have mostly chatbots that need something to wake them up. We either have to send them a message, or increasingly we have cron jobs and other triggers that get the AI to wake up and do something. But if those things don’t happen, they’re inert. They just sit there until somebody calls their number.

Do you have a sense, a vision, or a dream of what your ideal future AI would be like that’s different from that? Does it look more like another person but with AI advantages, or does it look like an LLM with its weaknesses patched? When you dream of a 2030 AI that you’re working closely with on a daily basis, what do you envision?

There are different aspects to answering this question. From the technical point of view, I think most of the research in the past 40 years has been built on a paradigm that says we have a pre-training, or a general training, phase, and we have a test phase.

But if we have continual learning, we can see that there are a lot of recent studies about how we can do that and overcome a lot of challenges. The point is, a true continual learner doesn’t have a test and training time. Potentially, if we hear this name in any design choice, it means that it’s not a true continual learner, because there is no test and there is no training time. The question is whether it’s a uniform process for the model or not.

My personal opinion is that we still need at least 2 phases. We should have one phase in which the model is active and actively receives information, whether through a user query, for example, or through vision models, world models, or anything similar. The model receives some information, generally performs some computation on the input data, and is active at that point.

On the other hand, there is another phase in which the model does not have to wait for input data. It might not receive any input data; it’s completely locked from the world outside of it. But the question is, even at that time, should the model be static without performing any computation or doing anything, or does the model need to start thinking about some process, thinking about the data that it has inside its parameters, and so on and so forth?

I think we can break the process, as I mentioned, into 2 parts. One is the active phase, and the other is another phase. Potentially, we can call it sleep time, because there is no input, but still the artificial brain, or generally the model itself, is trying to perform some computation. I think that’s a good way of defining different phases in this direction of continual learning.

A good model is a model that performs very well on both sides. It should receive the information properly, encode it, process it, and understand it in the best way possible. On the other hand, when it goes into sleep time, it should also start processing what it has learned before and use that for self-improvement. That’s what I think an ideal model should do from a technical point of view.

On the other hand, I think there are a lot of challenges. The models that we know right now are really large. So even a simple academic paper that is presenting a new LLM or architecture needs to perform some experiments on models with billions of parameters—1 billion, 2 billion, or something like that. Generally, such a large model requires a lot of computation if you want to keep that model updated over time, and it needs some techniques to somehow make this process possible.

Generally, for us, when we were thinking about this direction, that was the time when some ideas about nested learning started. If you think about nested learning, we can see that at each time step, we don't have to update everything. We just need to update a small subset of all the parameters. So, potentially, that is a way to overcome the challenges around efficiency.

If I want to summarize what I wanted to say, I think an ideal model should have two phases. One, generally, it should be a continual learner. It should interact with the world. Also, it should have two phases: one is about a very active process, and another one is about self-improvement—how it can consolidate its memory, how it can understand the knowledge, how it can connect different things that seem to be irrelevant, and so on and so forth. Everything is similar to what our brain also does when we go to sleep.

Yeah, that's just my personal opinion and it might be completely wrong, but I think we shouldn't focus too much on what humans can do. Instead, we need to focus more on what humans want from AI. I don't think we want to create something that is very similar to us. That's also a very interesting direction, but I personally am not really interested in that. I think we have a great design for humans, but on the other hand, for AI, I think we need AI models that are capable of understanding what we want.

For example, I think the current paradigm of LLMs can become great after some time, when we have more features. ChatGPT has memory now, and Claude has some features, as do Gemini and everything. Now they are more capable of understanding what you want. When you say, for example, “Create this image for me,” they can simply write that for you.

I think that's a very promising direction in general because we don't want to replicate humans. There are a lot of different ways that we can define intelligence, and it doesn't have to be the same thing as human intelligence. I think here, again, we can get inspired by humans, but we need to understand why we want to get inspired. Do we want to get inspired because we want to replicate what humans can do, or do we want to get inspired by nature to understand some underlying rules in nature?

For example, just one extreme example is that we cannot travel through time. So if I come up with an idea that says my AI model is trying to break causality in the world, then potentially that might be a wrong idea or wrong direction because it is breaking some natural rules about our world. Let me give you another example. When we are talking about the title that we use, “LLMs Need Sleep,” it doesn't mean that an LLM needs to literally go to sleep and rest.

It means that, from the human brain, it seems there is a very general rule that it has two phases of learning and then another phase of processing: consolidating the memory and finding underlying patterns between the received data. That's a very high-level inspiration from the brain. In short, I think that we don't want to replicate what humans can do, and we don't want to have human intelligence.

On the other hand, we want to have a new form of intelligence that is really proper and designed in a good way to understand human needs. That way, it can help people do a lot of things where they might face challenges without LLMs.

Yeah, certainly they're already superhuman in some ways, and so the opportunity for them to balance out our weaknesses is incredible. One phrase that you said there that I wanted to latch onto is “multiple ways to define intelligence.” Maybe I'll just give you my high-level pitch for what nested learning is.

I think in the paper there is a big emphasis—you guys place a big emphasis—on showing certain equivalences, where you're saying the way that we're doing things today is sort of a special case of a more general framework that you're developing. The core idea, as I see it, in the nested learning paradigm, and what I think is potentially most exciting about it for a simple person like myself, is that for quite some time now, we have achieved greater and greater expressivity in models by stacking more and more layers and just making them bigger.

That has worked remarkably well. We've been able to push that paradigm incredibly far. It's kind of crazy, though, that we just have this one layer stacked over and over and over again into whatever—80 or 120 layers deep, or however many layers—and that's kind of it. It feels like a more mature solution should be somehow more elaborate than that, right?

That's been the way that we've achieved this expressivity. Sometimes the term “computational depth” is used. What nested learning is doing is bringing a different way to the table to achieve higher levels of expressivity, or higher levels of computational depth, and that is by stacking not layers but levels.

What differentiates a level from a layer is that a layer is the same thing, or the layers can alternate. Obviously, we have these interleaved architectures, too. But these are things that are sort of in sequence as information passes from one layer to the next. A forward pass is passed through all the layers one by one to get to the end, and that's kind of the thing.

What the levels paradigm brings to it that's different is that different levels can have different update frequencies. With that, you now have the possibility for some parts of the overall system to be much more durable, while some parts are updating much more radically in something much closer to real time.

That obviously feels much more aligned with what we are, right? We're not one static thing that processes information in a fully dependent way each time. Our state at any given time is very much contingent on what we just experienced, but only to a degree. My mood, or what is currently on my mind, is a reflection of what happened earlier today, but my big-picture views about the world didn't change from this morning until now.

There's clearly some sort of hierarchy of different kinds of beliefs, different kinds of representations, and different kinds of circuits that we have, which are updated in some cases very quickly and in other cases very slowly. Obviously, they are integrated together and work together. We just haven't seen that in machine learning, except maybe in a few very far-flung experimental cases.

Now you're really starting to show that with this nested learning paradigm. Not only can you make it work, but as we'll get into with the results, you can make it work in a way that is competitive with transformers. It even seems to have some of what you might call micro-skill advantages, where you can see with certain early diagnostics that this can do something qualitatively different from what a transformer can do, even as it also outperforms it somewhat in terms of general perplexity-type scoring.

How would you react to that kind of general, high-level summary? I would also really be interested to get your take on this concept of computational depth or expressivity. I'm tempted in some ways to make an analogy to the G factor. People talk about the G in AGI—it is the generality. There's also G in the context of human IQ, which is the G factor, this sort of intangible something that gets at how capable you are across a very wide range of things.

Again, that's just getting at generality. Maybe in machine learning it's as simple as saying G is sort of loss, or there may be some fundamental equivalence there, but maybe not. I don't really know. I'm very interested in how you think about what that is. Clearly, we're getting at something—we've seen huge progress—but what is that something? That's another thing I would really love to get your intuition on.

We were working on nested learning for a very, very long time.

Potentially, I mean, even more than 1.5 years or something. The main issue that I personally had, and that we discussed a lot in the group after the authors of the work, was that it was really, really hard to formalize what we wanted to deliver. Even in the mathematical formulation, it was really hard to write it in a formal way and say exactly what we wanted to do in this paradigm.

After some back-and-forth discussions, I think we came up with this specific framework: There is a frequency of updates to somehow give each module time to do something while it is waiting. I think that's the best way of describing why we need to have multiple frequencies. On the other hand, we need to have a method for knowledge transfer to somehow transfer knowledge from one level to another level.

When the slow network is waiting for all the computation and information processing of the fast network, there should be some advantage for the slow network, since we're paying all these costs for doing some computation with the fast network. That advantage comes when we have the best, or a good, way of transferring knowledge between the fast network and the slow one, from the lower level to the higher level.

The idea is that when we have a network that is updating really fast, we can use that fast computation to give something to a slower computation side. The slow network, or the slower level, can somehow focus on a higher-level knowledge abstraction of the data, and the more frequently updated network can focus on fast adaptation and how it can process high-resolution data.

So, generally, that was the way that we could describe this framework and say that I think these two sides need to be there to complement each other. One is the frequency of updates, and the other is knowledge transfer between the levels.

For the second question, in my opinion, the current models are very efficient from one specific point of view. Why are they efficient? Because you can see that, from what we can get from LLMs, they are very cheap. For example, if you want to match their power on some tasks with humans, potentially the cost would be very, very different.

From that perspective, we can say that computation, and generally the current LLM paradigms, are very efficient. We can perform more computation per parameter or artificial neuron that we have in our model, and it can help us with different things. One is that we can have a smarter model. That's a very subjective term to describe this, but when we have more computation, it seems that we're performing some internal thinking.

A simple LLM based on the Transformer architecture works like this: When a token comes in, I do some computation on the token, pass it through all the layers, and then predict the next token. But let's assume that for a specific token, instead of just a simple pass of computation, I also perform more internal computation with respect to the past data, or generally combine or mix the data and something like that. In that case, I can see that the quality of the next-token prediction for this specific design choice can potentially go up.

Why is that? Because it can be interpreted as a form of internal thinking process. Now, for each specific parameter that I have in my model, it is performing several computations. It's not just one parameter and one step of computation; it's one parameter and a couple of steps of computation. So that's one advantage.

Another one is from the perspective of memory and the adaptation of these models. When we have a model that is adapting to the context very fast, potentially that model can learn the context. That's one of the main messages that we try to deliver in Nested Learning: Everything that we know is somehow a form of in-context learning.

Generally, I think it's a great thing in human language that we create new words. On the other hand, if we just create a lot of words for the same concept, it can make us confused, or it can be misleading somehow. I think we should create new words to differentiate different concepts, but if we have one specific concept, we need to stick to one specific word for that concept to avoid misleading the process or anything like that.

From that perspective, we realized that we can say everything is just a form of in-context learning. We already know what in-context learning is; now we just need to understand how what we've already been doing is a form of in-context learning.

That was the part where we started showing that, for example, backpropagation is a form of in-context learning and a form of associative memory. When it's a form of associative memory, we can say that the general pretraining phase of the model is a form of in-context learning. When we go to, for example, the context of attention or RNNs, and so on and so forth, again, it's a form of in-context learning.

When we perform gradients—which we can define in any RNN based on gradient descent or another form of optimization process—we're doing some learning on the context that's happening right now. So I think these are the 2 main things that Nested Learning is trying to address: more computation per neuron, and adaptability and continual learning.

Can you just describe in more specific detail what the relative sizes of the levels are, what the structures of the levels are, what the context windows or lengths of the different levels are, and what the frequencies are? Just map the thing out for us in very black-and-white terms.

Yeah, so we start from the Transformer structure. In a Transformer, we have an attention block and then an MLP block. In pretraining, we have different contexts that the attention attends to. The attention side is trying to combine all the tokens, and each token attends to all the tokens before it in the context, and so on and so forth.

Then there is an MLP block, and that MLP block is responsible for long-term memory. When the model is pretrained, the MLP block is fixed; it's not changing anymore. It has all the information compressed during pretraining. Then we have attention. At inference time, attention is responsible for the context that it is getting, and the MLP block is responsible for long-term memory and generally the very general knowledge of the world.

Let's simply extend this idea. Instead of having just 1 MLP block, we have multiple MLP blocks, each of them updated with a different frequency. Why is that helpful? When you have attention, you have fast adaptation to the context. Attention is very powerful; it's like a perfect memory. It caches everything, so it's great.

On the other hand, you might want to have multiple levels of memory. That's the part where we define a continual memory system. Instead of 1 MLP block, we have multiple blocks of MLP. Now, you have your first MLP block, and it is updated very fast. What would happen in that case? The updating process of this first MLP block can cause catastrophic forgetting, because this MLP block can simply forget information that it got, for example, a couple thousand tokens ago.

But since the other MLP blocks have not been updated so far, the knowledge that is forgotten by the first MLP block is still in their parameters. When we perform backpropagation through all these layers, the knowledge can come back. So it provides, and helps us to have, a loop process in time.

The first MLP block can forget something, but if that specific data sample or that specific skill that was forgotten is important, then it can come back through the other MLP blocks, which have not been updated so far. They still have the knowledge about that specific skill or data sample.

That's a very simple way of extending the whole Transformer block. We call this variant HOPE Attention. It's a combination of attention plus multiple MLP blocks, and that's what we call HOPE Attention.

Now, we have another variant, which is the actual HOPE architecture. As I mentioned, attention is a perfect memory. It can cache everything, it can scale, and it's fast, and so on and so forth. This is great. But the point is that the update of attention still has infinite frequency.

What does that mean? It means that attention doesn't know anything about the temporal dependency of all the tokens. It needs something like positional encoding. Even with the help of positional encoding, attention is not a great model for tasks that are sequential, tasks that require sequential reasoning, or something like that.

Our idea was to replace attention with another associative memory that maps keys to values. That's what attention does, and now we want to replace it with another module that tries to map keys to values. One potential architecture here is Titans.

We can simply replace attention with Titan and have Titan plus a continual memory system. That’s a simple idea. But in the initial sections of the paper, we discussed that if you have a simple linear process of updates, which is what happens inside each chunk of a Titan update, this process can somehow be weaker than the case when we have a self-referential process.

What is a self-referential process? Gradient descent, or generally backpropagation, is a form of self-referential process. The idea is that we want to learn how to learn, and how to learn how to learn, and how to learn how to learn how to learn. There are a lot of levels of how to learn how to learn, and computationally, it’s infeasible to implement all those levels and so on and so forth.

There is one idea by Schmidhuber et al. They had this idea of a self-referential model. One of the ways that we can make a model self-referential, when we have a key-value memory, is for the model to generate its own value.

What is happening there? Let’s say that in our memory we want to memorize something. There is a specific event happening, and you want to memorize it. Let’s say you want to memorize a specific word. In our brain, we have associative memory, and we are trying to map this specific word to another concept that we already know so we can memorize it.

You’ve definitely seen cases where, for example, you say, “I generate this specific word,” or, “I want to map this specific word to another word that I already know so I can remember that.” We generate the value that we want to map our keys to, so we can memorize the key as well.

For the self-referential process, it’s a very general concept. In the specific design choice that we have in the paper, the value of the associative memory is generated by its own parameters. The model itself generates its own value and then tries to map keys to values.

Potentially, this process is fully sequential. You cannot parallelize it in a simple format, and so it has a full understanding of the causality in our data. In tasks that require sequential thinking, sequential reasoning, or anything like that, we can expect this model to work better than simple attention, because simple attention doesn’t have the ability to sequentially understand the causality of the data at all.

That was our idea. In the HOPE architecture, we replace Titan with self-modifying Titan, which is exactly a Titan module where it generates its own value function—the value of the associative memory. That’s the HOPE architecture: self-modifying Titan plus a continual memory system.

I think I want to spend one more beat on what you mean when you say “generating its own value,” because I know the transformer architecture pretty well. We’ve got these K, Q, and V vectors, right? The training process modifies all of those over time.

The general heuristic that I have is that, for each token, there’s the query vector, which indicates what this token is looking for. There’s the key vector, which helps indicate what other tokens have to offer and aligns those, finding where there’s a match—where there’s relevance, basically. Then the value brings up the concepts that get fed into the downstream layers, which make sure we have the right activations for continued processing from there.

But all of those are learned, right? All K, Q, and V are learned. So I’m not entirely clear on what you mean when you say that the model learns its own values, because doesn’t the transformer learn its own value vector as well? I’m not quite clear on the distinction you’re making there. Let’s assume that people are at least generally familiar with the transformer and know how that works.

Generally, in a transformer—or, more accurately, softmax attention—what’s happening is that we have a projection of Q, K, and V, and then the output goes to attention. Attention doesn’t have any control over the Q, K, and V projections.

When I’m saying that the model, or generally the associative memory, tries to generate its own value and then map keys to the value, I mean something like gradient descent. If we recall gradient descent, we can see that we have something like the previous state of W, which is W_t minus the gradient of the loss function that we have. This is equal to the next state of the weights.

If you look at this process, you can break the gradients using the chain rule and write it as the gradients with respect to the outputs times the input data. Now you can see that this takes the form of an associative memory, very similar to linear attention.

It’s W_{t+1} equal to the previous state of W minus K, where K here is X_t, and then V, which is the gradient with respect to the outputs. If you look at this process, it’s very similar to linear attention, or any linear recurrence model. But the very interesting part is that it is different from linear attention because, if you look at the value, which is the gradient with respect to the outputs, this gradient with respect to the output is a function of W_t. It is a function of the current state of the weights.

Basically, the value—the keys and values that we have in associative memory—the value component is not coming from another component before this recurrence formula. It is generated by this recurrent process every time. That’s what is going on with this self-referential process.

In memory, let’s say that in a very simple version of self-modifying Titan, if we want to design this as a simple Titan, what’s happening is that I have this X, then I project it into Q, K, and V, and pass all of them to the Titan module. That’s a simple Titan module.

But if I want to have a self-modifying Titan, then all of these parameters of the Q, K, and V projections are optimized inside the Titan module. Basically, the model has control over modifying its own updating rule and generating its own value for further memory. That’s the main difference between the two.

I think the key phrase for people to latch onto there, starting with myself, is “modifying its own update rule.” This is definitely a theme in general, with the Mamba architecture as well. The authors of the Mamba architecture had done a bunch of previous state-space model work, and the big unlock with Mamba specifically was that the way in which the state is going to be updated at each time step became a function of the inputs.

That increased the potential expressivity. Making the actual update of the state itself dependent on the input it’s receiving at that time unlocked better performance. There’s something very similar going on here, it seems, where you’re saying we want to make the final value output not something that we calculate too early, basically. We want that to come a little bit later and be more input-dependent and more history-dependent than it has traditionally been in softmax attention.

Is that a good intuition, or is there something still missing from that intuition?

I think it’s a perfect intuition. Generally, this projection of the value is also updated inside the module. That’s a very important point because it helps with adaptability. Generally, the model itself is also very adaptive to the context.

From every token that comes in, the model is trying to learn something. The way that it generates the value term for the associative memory is exactly the same as the way it updates its memory. So it’s a very adaptive process for generating the value as well.

Can you take us through a single time step? Maybe we can do this with the attention half and then the Titans half, and highlight the little difference there. But let’s also zoom out to the big picture.

We’ve got a new fundamental block, right? If we do the attention half first, we’ve got an attention mechanism, and then we’ve got multiple MLPs arranged in sequence from fastest update to slowest update frequency. Then that block gets stacked into layers. Correct?

I’m a little bit confused, to be honest, about the fact that there’s no training-test distinction, because there is still some training process where you’re taking a bunch of data and running it through the thing, right? From the researchers’ perspective, or maybe from the model’s perspective, there’s not so much of a distinction. But from the researcher’s perspective, you’re still sitting there running a process that takes a bunch of data and has the model learn from it.

That’s a bulk process. It’s not like a user is engaging with it hourly. There’s nothing outside of that process happening at that time, right? So how is it different?

When I do this for a transformer, I do have some of these really nice parallelization benefits. I’m interested in coming back to understand to what degree the current hardware paradigm plays nicely with some of the stuff you have here, and to what degree the fundamental recurrence may present challenges. But bracket that for a second. Today, I can run a bunch of tokens through the thing in parallel.

We can accumulate for batches. We can accumulate all these gradients, then apply the gradients, and then we have the next time step, and we keep doing that. I have a pretty good intuition for how information flows. I can visualize the forward pass in my mind, and then I can visualize the backward pass of backpropagation going through and gradually updating all the weights.

How does the procedure with the new architecture vary? What are the core things that are different from the paradigm that we're more used to?

I think, generally, the main difference comes from the update side that I mentioned. For the HOPE attention, I think it's very similar to the current paradigm, and even the architecture is very similar to transformers. It's actually a transformer architecture. We just replace the MLP block with multiple MLP blocks.

When we want to do inference, the main difference comes from the fact that, for each of the MLP blocks, we need to track where we are. Is it time to update the MLP block, or does it still have time to get updated? If it's the latter case, then we use the last updated state of that specific MLP for doing the inference. If it has not been updated—if it is time to get updated—then we first update it through all the backpropagation and all of the tokens that we have seen so far in the current chunk. Then, when the weight is updated, we perform the inference.

From the research point of view, it might be a little bit hard to remove this part. Even from the research point of view, it might be better to say that we have evaluation time and non-evaluation time. It seems that the model is always getting updated over time, and there's no training time and test time. But for a specific period of time, we don't do any evaluation; we wait for some time, and after that we start evaluating the model on the different downstream tasks that we have, generally as we do for anything else.

From the model's perspective, it doesn't know whether it is in test time or train time, because everything is the same and it's a very uniform process. But from our side, it is definitely important whether we want to evaluate the model and measure its accuracy for a specific task and so on, or not.

Coming back to the HOPE architecture, that's generally for the HOPE Transformer or the HOPE attention model that I mentioned. But when we go to the actual HOPE architecture, again, everything is the same. Everything is very similar. The only difference is that the attention is replaced by a self-modifying Titan. For a self-modifying Titan, again, everything is very similar to Titans. All of the changes are just inside the model design, and from the higher-level perspective, the inference is very similar.

The context or document goes to the self-modifying Titan, and for each token we have 1 output. Then it goes to the MLP blocks. We have multiple MLP blocks, and so on and so forth. Everything is very similar to the current paradigm.

Did I have it right that there's this core block of either the traditional attention or the self-modifying Titan module, plus the MLPs, that then becomes the block that gets stacked into a layer? Is that right?

Yes. That's a design choice. We can have different design choices. The initial and main design of HOPE is the case where, for example, for the HOPE attention, we have attention and then multiple MLP blocks. Each of them is updated with a different frequency.

But for some tasks, we needed to use a pretrained model. For example, if you want to focus on Llama, Llama isn't designed with the whole HOPE architecture. Instead of going with the formulation I mentioned—attention and then multiple MLP blocks—what we have done is say, this is attention and an MLP block, then attention and another MLP block with a different frequency, and then attention and another MLP block with a different frequency, and so on and so forth.

It's somehow a design choice. You need to see which one you prefer. Do you want to use an existing pretrained model, or do you want to start from scratch and train your own designed architecture? Potentially, both of them are relatively similar. They don't fundamentally make changes.

Yeah, interesting. It's another great reminder of Ilya's maxim that the models just want to learn. There's a lot of—it's always striking to me how many of these choices end up being able to go one way or the other.

That was true in the Titans case, where you had 3 different ways of working the memory module into the larger architecture. It's definitely been true with Mamba in many ways, where you can have multiple states, you can have them in sequence, you can have them in parallel, and so on.

Once you have 1 of these block concepts that seems to work well, you can Lego-piece it in a lot of different ways. There will probably be some performance differences between different ways to arrange the blocks, but more often than not, if you're really talking about a serious conceptual advance, you find that the exact wiring diagram is less important. What's more important is the core piece that you're adding to the set of Lego pieces, so to speak, that you can use. This is a good reminder of that.

How do you think about the relationship between the different MLPs in terms of size, update frequency, and maybe learning rate? It feels like learning rate might be important here, where there's potentially an equivalence: if I update 1 MLP every token and then update another 1 with a larger batch size, I feel like I can make those much more similar or quite a bit different, depending on what learning rate I apply.

In terms of size, frequency, and learning rate, are the smaller ones that are updated less frequently also smaller? Do they have different learning rates than the other ones? Take us through how you think about the relationships between the MLPs of different frequencies.

It really depends on the architecture, on the number of parameters, and on the design choices that you have. It's really similar to asking, what is the best dimension for transformers or attention blocks? Potentially, that's really hard to say. It really depends on the person who wants to work with that attention, the use cases that we want to consider, and so on and so forth.

Generally, the frequency of updates really depends on how adaptive you want your model to be and how you want the model to maintain its persistent memory, and so on and so forth. I think that's pretty much a design choice.

About the learning rate, generally, you can treat each of these blocks in the same way as you do MLP blocks. They're exactly the same thing. The point is that they have a different frequency of update, so nothing has changed. Everything is exactly the same as with MLP blocks.

I think it would be really interesting to see how the learning rate can affect each of these blocks and so on and so forth. But I have not done that, and I'm not sure about the exact solution. My expectation is that, potentially, any way that we use to do hyperparameter tuning, we can do the same thing here as well. There shouldn't be any differences.

Interesting. Your general default is an intuition-based approach. Do I have it right that it's basically on the order of magnitude—the fastest 1 updates every token, the next 1 updates every 10 tokens, the next 1 every 100—or maybe it's a couple of orders of magnitude?

Understanding that it's not yet a fully tuned system, what did you start with, and why did you pick those things as your initial guess?

The way that we chose the frequency of update for each of them was based on our intuition about the chunk size that we use for Titans and other models. Generally, the chunk size in Titans is the part that can define the frequency of Titans as well. At that time, we didn't have this term of frequency, but generally, the chunk size can define the frequency for Titans too.

What we used was based on our intuition about what chunk sizes are good for Titans. As far as I remember, I think the numbers that we used were possibly 128, then 4 × 128, and then 4 × 4 × 128. It was something like that, as far as I remember.

And how about knowledge transfer? One quick interjection question there, too: Are there still skip connections?

Yes, everything is similar.

Yes, exactly.

One thing that we tried to do with Nested Learning caused some misunderstanding about what we were doing. I've seen some comments that, for example, some of the concepts here are already known and things like that.

But the point is, we tried to actually include all those concepts that we already knew to show that it is a universal learning paradigm. It's not something that contradicts our current understanding. It just complements and completes what we already know, but in a new direction.

For example, when you are doing deep learning—any form of deep learning—and you are saying, “I'm using this attention here,” you're actually using nested learning. But in deep learning, you only see the final solution of each learning problem. So, you have a learning problem inside the attention, and you are trying to solve a regression problem, and the nonparametric solution to that regression problem is attention.

When you see everything from the deep-learning side, you can only see the final solution for each component. But when you see everything from the nested-learning side, you can see the internal learning process of each component as well. So, in general, it's not something that contradicts what we already knew; it somehow complements all the things that we knew and goes beyond that. I think that's generally an important part. So, yeah, everything can be very similar. You can have a separation.

Yeah, so then help us understand how we should think about the roles that the different-frequency MLPs are playing. Knowledge transfer is one way to think about that. Another way to frame the question might be: How do they complement one another? How do they work together?

I would be interested to understand this both intuitively and mechanistically, to the degree that you have a mechanistic understanding. How does the one that's updating fast gradually inform the ones that are updating slowly? How do the ones that are updating slowly steer the ones that are updating fast in the right directions? How do you think about the interplay between those different components?

Yeah, I think generally, coming up with different ways of knowledge transfer is really important here, in my opinion. So, the main point of having a frequency for each component has 2 parts. I think we also discussed it in the beginning. The first part is that it helps the model maintain its memory for a longer time period.

For example, let's say that we have twins, and one of them goes to a spaceship and moves at the speed of light. Right before that, they have a very good memory. For example, they had lunch or something like that. They have a very good memory. When the person who is moving at the speed of light comes back, 80 years have passed on Earth, and their sibling has somehow forgotten about that specific lunch because it was 80 years ago. On the other hand, that specific person has all the information—all the details are just 1 second or 2 seconds old. So, they remember everything about that lunch.

Why is this happening? It's because of the updates to the memory of each of them. The person who lived 80 years had their memory updated many times. But the person who just moved at the speed of light, or close to it, didn't have their memory updated that much.

From this perspective, we can see that the number of times that we make updates to the memory is very important. It's a very inaccurate example, but I think the main point is clear here. When we have 2 components, one of them is updated many times while the other one is slower and is not updated as much, the slow one has the opportunity to learn something from the fast one. While the fast one is getting updated, the slow one has not gotten any update, and so there's a chance for the fast one to transfer something to the slow one before the slow one gets updated.

I think that's the part we can also discuss through the sleep process that we have. The idea there is that when we have multiple levels of MLP blocks, each of them is updated at a different frequency. One simple thing is that before updating the fast MLP block—and by “fast,” I mean it's a relative term; we have a fast one and a slow one, while we have multiple levels—we consider each pair of consecutive blocks.

When we want to update the fast one, we know there's a chance that we will forget something. Before forgetting something, we can transfer the knowledge of this block to the next one and then update this one. That's the part where we need a good way of transferring knowledge. For example, one way is to do context distillation. If you want to pass the knowledge from one MLP block to another, some methods of context distillation can work very well in that case. That's very similar to what we do in the sleep process as well, in the *Language Models Need Sleep* paper.

I think the main role of knowledge transfer is to help the slow network take advantage of the fast network. Another point about having different frequencies is about memory and how the model can manage its memory, similar to the example of the twins that I mentioned.

What is the mechanism by which the information in the fast-update layer gets moved to the slower layer? But then there's also got to be something going the other way, too, right?

If you conceive of them as pure perception, in a sense, then I'm not even entirely clear on what's happening in my own brain, but it feels to me like there's more information flow from my perception modules to my higher-order reasoning modules than there is from reasoning back to perception. That signal is still important, right? My higher-order processes do tell my eyes where to look, tell them where to focus, and say, “We need to zero in on this detail a little bit. I want to understand that better, so put some of your bandwidth into this particular thing.”

Give me a little more on how, mechanistically or procedurally, the information in the fast-update layers is getting transferred and how we're making sure that we're storing what really matters from what the fast updates have learned. But then, also, what is the signal that flows the other way?

Let me answer that with a very simple example. Let's say that I have this model, and I want to update its fast MLP blocks. I want to make sure that the information in the fast MLP block is not forgotten and that it can pass to the slow MLP block.

One simple thing that I can do is copy all of the parameters of model A to model B. Now I have 2 identical models. One is model A, and the other one is model B. What I do for model B is update the fast network, the fast MLP block. Now, the parameters in the fast MLP blocks of model B are free to change.

What I want to do is change the parameters of the slow MLP in model B in a way that allows the output of model B to mimic the output of model A. If that happens, it means that model A has all the information compressed in the fast MLP block, while all of that information is gone in model B. If I could somehow modify model B so that it can mimic model A, it means that I have somehow transferred the knowledge in the fast MLP to the parameters of the slow MLP in model B.

That's one simple example. This process is very similar to the distillation process. We are distilling the knowledge of model A into model B. From this perspective, we can see that this is one way of transferring knowledge from the fast MLP to the slow MLP.

Another example, which is very common and popular, is backpropagation. If you sequentially connect your MLP blocks and then, at some point, perform backpropagation, you can transfer the knowledge of one block to another one, and so on and so forth.

The copying and distillation process you described is essentially what's going on in the *Language Models Need Sleep* paper?

Yes, with some additional detail. For example, what we do there is also add additional parameters to model B to make sure that it has enough capacity to store the new knowledge that it has just gotten.

Does that also mean that, in the nested-learning version of this, you're really just letting backpropagation do its thing? You haven't really overengineered it or engineered it all that much. We just have these MLP blocks; they get updated at different frequencies, and you're letting gradient descent do its thing. The updates are just working. That's basically it.

Yes, exactly. In the whole picture, everything is just backpropagation.

A huge takeaway from the conversation that didn't come through to me as clearly in the paper is that this is really proof-of-concept-stage stuff. The fact that it works so well shows what a good concept it is. But nothing that we're discussing here has been through the same kind of process that the mainline models have been through, where everything has been parameter- and hyperparameter-explored to the nth degree and optimized, along with all the little refinements that have obviously been made over time.

That hasn't really happened here. There are a lot of questions that we could still ask about this version, that version, this configuration, that arrangement, sequence versus parallel, how many layers, relative sizes, and relative learning rates. There's a ton of space there still to explore.

But basically, just taking a few of these core concepts—the main one being different frequencies of updates for different MLP blocks—alone creates some pretty impressive results that are qualitatively different from what we're used to seeing. So maybe let's take a minute and talk about some of the results. There are a lot of different tests run in the paper, with big tables of results and a whole bunch of different metrics, some of which are classic perplexity-type scores. What do you think are the most important, revealing results that people should look at and say, “Because I see that it can do that, I know there's really something here that I need to grapple with”?

Yeah, on this, we have one continual-learning-style task that I personally really like. The idea there is that we have a pretrained model, and there is one specific language that the model has not seen before. We want to teach the model that specific language in context, and the point is that we have all the grammars, all the words, and a dictionary of words.

We pass all of them through the model in context, and then the model learns the language. Then we ask whether it can translate a specific text from that language into English. We can see that the model can translate that specific text—not perfectly, but at a very, very high quality. It seems that the model is capable of understanding that language in context and then using it for a translation task.

But the point is, let's go one step beyond that. Instead of 1 language, let's put 2 languages in context and ask the model to translate different texts from each of these languages into English. In that case, we can see that the model almost collapses and cannot translate any of those languages. The point is that the model cannot handle its context well and fully understand each of the languages separately. That's generally a very, very hard challenge for transformer-based architectures.

But the point is that when we change that architecture to HOPE, or HOPE attention, we still have attention, but we have multiple levels of in-context learning and multiple levels of MLP blocks. One thing we can see is that when we increase the number of levels, the performance of the model in both of these languages gets better and better and better.

Why is that happening? Because the model has a better way of managing memory. It understands that temporal knowledge that isn't needed can be stored in the first MLP block, while more understanding of the language can pass to the more stable MLP blocks—the later MLP blocks. When we have more and more blocks, we can see that the performance gets better and better.

In my opinion, that's a very good evaluation for understanding that the model can learn in context and, generally, for continual learning.

So is this the same test? I remember it's been a while since I thought about this, but I think it was with maybe Gemini 2—I don't even know; maybe it was even back with Gemini 1—that there was this metric introduced for learning a new language from basically 1 book.

There was a critically endangered language that 1 person had really studied, and made a book that's not available anywhere on the internet that explains what this language is. Then they put that book into context and say, “Based on this, go ahead and do a translation.” It seems like this is—I don't know if this is the exact same test as the one I was previously familiar with, or if it's a bit different—but the language here is Manchu. I just looked it up; it's a critically endangered language from somewhere in China.

So that's basically the idea, right? It's a language that language models have essentially no prior knowledge of. They're given a very detailed primer on this language from some anthropologist or whoever has gone out and done the fieldwork, and then their job is to apply that.

I'm looking at Figure 8 in the Nested Learning paper, and what I'm taking from this is that all of the models do similarly if there's just 1 language. But as you said, when you go up to 2 languages and double the difficulty of the task, the traditional-transformer in-context-learning approach performs quite badly. I understand that HOPE-1, HOPE-2, and HOPE-3 refer to how many levels, or how many different frequency-update mechanisms, exist.

So when you move from the traditional approach to, I guess, 1 additional, 2 additional, or 3 additional frequencies of update, you get basically almost all the way back to the original level of performance with just 1 language.

Yes, exactly.

That is quite interesting. Fascinating stuff. And what is MTOB there, just so I have that clear?

Yes, that's another dataset. It's another language that also hasn't been seen during the pretraining of the model.

Yeah, okay. This is the one that I recall. So you guys added the Manchu language to this one in late 2023. Both of these are very rare, unknown languages being translated into English, and only with multiple layers can the models do both at the same time in 1 context.

Very, very interesting indeed. How do you think about things like perplexity scores? You've got a big table that shows perplexity, some accuracy results, and some of these basic, classic battery-of-tests metrics. I should say that we're scaling these models so far up to roughly the 1-billion-parameter scale. You've got 760 million parameters and 30 billion tokens, and then the bigger one has 1.3 billion parameters and 100 billion tokens.

Obviously, that's not huge by today's standards, but there's nevertheless a pretty clear signal that the HOPE architecture is, on just about every dimension, outperforming all the other things you're comparing it against. That includes your Transformer, Mamba and Mamba variations, and even Titans. RetNet is in there, and DeltaNet is in there.

How do you interpret these? This goes back to that AGI question. Is this a good measure, or is it just the best measure we have? How much stock should people put in these perplexity tables?

There are some standards in the community for performing benchmark tasks, and not all of them are the best things to do for evaluating a model. But we need to do them to make sure that everyone can see where the performance comes from and where the advantages come from.

I have 1 specific Transformer structure, and I have 1 idea: if I add, for example, forgetting to the Transformer, then it can perform well on noisy data. I'm not sure; I'm just coming up with 1 example. If there is no noisy data and I test my method on very clean data, then there's no way that I can show the advantages of my approach.

I think it's exactly the same thing here. We're arguing about models that do not need to be pretrained. There is no test time, there is no train time, and so forth. But on the other hand, there is still a lot of infrastructure built around test time and train time, and a lot of evaluations are built around test time and train time. Everyone also expects us to report something about pretraining perplexity and some evaluations, most of which are short-term and short-context language-modeling tasks. They don't require a very complicated model to understand long-context modeling.

I think I also mentioned this in the presentation of Nested Learning at NeurIPS: We didn't use Table 2 and all those perplexity and language-modeling tasks to argue that HOPE is powerful. We just used that table to say that HOPE is not less powerful as a backbone compared with other models. You can see that it performs well, but someone might say that the improvement is marginal compared with other models.

The point is that this isn't the direction we're aiming to solve. It's really good that even in this direction, which is not the goal of Nested Learning and HOPE, we can show some improvement, even if it's marginal.

I'm always a big fan of trying to get a little better sense of the micro-skills of different architectures. For example, because Transformers keep the full sequence—the full context—in working memory at all times, they're pretty hard to beat. I feel like you even have a theoretical argument now that it may be impossible to beat them on some of these tasks where the idea is to recall information from the context window.

But then we saw things with Mamba, for example, where it was better at learning from a sparse signal. This was a micro-skill that the architecture excelled at, and that the Transformer relatively struggled with.

What have you seen in the HOPE case? Are there little micro skills that stand out? I think this is very interesting because it does ladder up to the overall performance and what these things are actually good or bad at, right? The ability to recall something in context is really important when you need it.

The ability to learn from, or filter out, noise and get to the signal that really matters is really important when you need it. Are there particular micro skills that stand out to you? The language translation one is interesting in a macro sense, in that it’s a hard task, but I wonder if you drill down to these very micro, building-block competencies that models or architectures can either have or not have, what stands out in terms of what HOPE has that Transformers don’t have, or don’t have as strongly?

When we are talking about in-context recall tasks, or generally recall-intensive tasks, in my opinion, all those tasks are designed for Transformers. They are not designed to compare architectures; they are specifically designed for Transformers. Why am I saying that? Because you cannot expect a model, or even a human, to perform needle-in-a-haystack tasks perfectly.

For example, assume that you have a couple thousand lines of code and you simply want to recall what the value of X was at some line of the code. It’s almost impossible, or at least very, very hard, for a human or even other models to do that. On the other hand, it’s pretty simple for Transformers because they have direct access to the entire history in their context. It’s very simple to just find that token somehow and pass it as the output.

In recall-intensive tasks like this—in-context recall tasks that we have here—the gap between recurrent architectures and Transformers is also very great. If you compare the first generation of recurrent architectures to the Transformer, we can see that the gap was much, much larger. Now this gap is getting smaller, and the performance of other recurrent models is also very good.

The interesting part for me was that HOPE at least closed this performance gap compared to Transformers, even though it’s not expected to do that. We expect Transformers to do that because they have an attention block, but we don’t expect a compression-based model to perform recall tasks. I think that was interesting.

And what does the MAD dataset get at? Just to play back to you what you said: on these needle-in-a-haystack, very difficult recall tasks from earlier in the context, the Transformer remains the best. The recurrent models, which only have some latent representation and don’t have the ability to look back at the original raw text, don’t perform as well.

But with each generation of improvement—and here you’ve got several—the HOPE architecture does the best of the recurrent models that don’t have the full, explicit context in working memory at runtime. That gap is closing. Flipping over to the MAD dataset, the HOPE architecture is performing better than everything, including the Transformer. What micro skills is that testing? What should we take away from that result?

The MAD dataset is also very similar to the recall-intensive task, but the point here is that there are different setups for it. For example, one of them is noisy in-context recall. We want to perform an in-context recall task, but the point is that we have some noise in the tokens.

When we have that noise in the tokens, somehow the power of Transformers that I explained in the previous setup—which was pure in-context learning—is now its weakness. It can get confused about which token is noise, which token is not, and so forth. Potentially, this task becomes a little bit harder for a Transformer compared to a model like HOPE.

But again, that also depends on the memory management of the RNN. If it doesn’t have a very good memory-management system, or generally a good update mechanism, then potentially it can get confused by the noise as well and face some issues. If the memory management is strong, then it’s much simpler to filter out all those noise tokens in the task.

I think that’s one thing. Another task that is interesting here is compression. The name explains the task itself, but we want to compress the tokens and predict one single token that is the compressed version of a set of tokens. Then we want to reconstruct the original sequence from that.

Potentially, it’s a simpler task for models like RNNs because they already know how to compress the data properly. On the other hand, Transformers have a harder time performing this task. Generally, as I mentioned, all of these tasks are somehow modified versions of recall-intensive tasks or in-context recall.

The point is that there are other aspects. Selective copying is another one. There are other aspects of the model that are very important, and we should also see how the model performs on those aspects, not just overfit our evaluation to one specific metric.

Cool. I think that’s probably enough on the really low-level stuff. I think this illusion of architecture—the title of the paper—starts to click for me. On page 39 of the paper, we get to the part where you also have a new optimizer that is outperforming not just standard Adam, but also Muon. It does come with a little bit of computational overhead, but I think the argument is that it more than pays for itself in terms of faster convergence or simply better learning. Is there anything you want to add on the M3 optimizer?

First, let me clarify one point: for optimizers, it’s a little bit hard to say that one specific optimizer is more powerful than another. It really depends on the problem setup, or generally even on the problem. For example, here we are evaluating the optimizer on a regression task, but if you train a language model, you might see that the trend is completely different, or something like that.

Generally, the design of an optimizer, and saying which one is better than the other, really depends on the task and the problem setup. That’s also one of the main points that we wanted to deliver in Nested Learning, because what we are saying is that the entire architecture and its optimization process are just one interconnected system of nested optimization problems.

Why is it interconnected? Because the gradients on the optimization side are generated by the architecture. If you have a simple architecture, then the gradients are very simple. If you have a complicated architecture, the patterns in the gradients can be very complicated.

When you have a momentum term, momentum is a form of associative memory that is trying to compress gradients. If your gradients are very complicated, you need a more powerful memory-management system for your momentum. If the architecture is very simple, even simple gradient descent without any momentum might work very well.

In general, one of the arguments that we have in the paper is that we should see everything as an interconnected system and try to design something that, all together, results in a good model architecture, or generally a machine-learning model in a very general sense. That’s one argument.

Another thing we wanted to deliver was the message that the architecture side is very, very similar—or somehow exactly the same—as the optimization side. All of them are just learning rules, and there is a learning process that is happening. The only difference between the architecture side and the optimization side is the context.

The context of the optimization algorithm is gradients. Actually, the context is the set of gradients that we have. The context on the architecture side is the set of tokens that we have. Generally, they are very similar.

In the paper, we had this continual memory system. We extended the MLP block, saying that you can have multiple levels of frequency for the MLP block. That’s a very general term, and throughout the entire paper we argue that architectures are the same as optimizers, and so on and so forth.

Why not apply that technique—borrow that technique from the architecture side—and apply it to the optimization side? That was the main motivation: to show that this continual memory system that we designed is not just working well for the architecture, but also works very well on the optimization side.

We simply extend M1. Instead of one specific memory, it has multiple memories. In the case of M3, it has 2 memories. It’s trying to compress the context at different frequency rates, so it can help you better understand the global aspects of the loss landscape.

And it potentially can help the model find a more effective solution.

The new paper, “Language Models Need Sleep.” Tell us a little bit more about what’s going on here. You mentioned at the top the 2-phase concept: We have the memory consolidation phase, and then we have the dreaming phase. I do think it’s fascinating to consider that there’s a kind of creation of net-new parameter space and then consolidation, or sort of pruning back, I understand, too, because obviously things can’t just grow and grow forever, right? Take us through this in more detail. I’m fascinated to learn more about it.

Generally, the main idea, as we discussed earlier, was that if we have a truly continual learner model, then there’s no test and train time. On the other hand, we need to have 1 active time when the inputs are coming in an online manner, and also a time when we don’t have any input. So, the model is not actively receiving information from the outside, but that doesn’t mean that the model should be static. It means that the model just doesn’t get input, but it can have some internal computation to improve itself.

That’s a really general concept. We can incorporate more and more components into the sleep time that we have. It doesn’t have to be just these 2 specific parts; these 2 were really relevant to the research that I’m doing, so we just did that. Potentially, it can include any other form of self-improvement and so on and so forth. That’s just 1 way of breaking the life of a continual learner into active time and sleep time.

What we have in the sleep time right now—which, again, as I mentioned, can incorporate more components—is that we want to make sure that when we update each of the components of the model, we don’t forget about the knowledge stored in the parameters. The idea there is that we know there are multiple memory blocks. Each of them is updated at a different frequency. Again, the fast and the slow here are just relative terms; it doesn’t mean the slowest or the fastest one. We have slow and fast rates, and it can be any part of the neural network.

We want to transfer the knowledge from 1 to the other. To do that, we use the distillation process that I mentioned. The distillation process is based on on-policy distillation, so the model itself generates some data. One interpretation of this process is that we distill the knowledge of 1 small model into a larger model. When we want to go from 1 step to the next, we activate new parameters at the next level. This can help the model release some of its capacity and be ready to accept new knowledge.

It describes a very natural way of learning in humans as well. It’s really common that when we learn something—when we learn about a new concept or something like that—we don’t have a full understanding of all aspects of it. But as time passes, and we let our brain better understand that concept over time, we study other things and better understand the entire process. Then, at some point, we can see that we have a very clear picture of what’s going on with that concept. We can completely understand it, and generally, that’s the best thing. That’s a very good way of learning.

Here, it’s a very similar process. I also discussed that from this perspective. I think it might be a better and simpler way to understand why we need to have multiple levels and distill the knowledge from each level to adapt it to another one. When we want to understand a specific concept, we have different levels of knowledge abstraction for ourselves.

The 1st level, which is the simplest one, is to just memorize things. Let’s say that we want to learn a specific mathematical rule, or a specific concept in physics or any science. How can we learn that? We start with some examples of that specific concept, and then we start memorizing those examples. For example, if it’s a mathematical rule—just any mathematical rule that we can have—we start with some specific examples and just memorize them. At some point, we generalize our understanding of all those examples, remove all those examples from our brain, and replace all of those memories with just 1 single memory that can describe everything we have learned so far from that concept.

Then, when time passes, we have more information, we read more about that concept, and so on and so forth. Again, we revisit our understanding of the concept and replace our previous understanding with this new understanding, which is more general and can explain more phenomena or more terms in that specific concept. That’s generally the way we understand things, and there are different levels of abstraction in our understanding.

Now, when we have different MLP blocks, or generally, let’s just go to any arbitrary architecture—it doesn’t have to be just HOPE; it can be any architecture—the main thing is that each of the blocks is updated at a different frequency. In that case, the fast-updating block is very similar to the memorization process, because we memorize a lot of things. We don’t need to understand them. There’s no pure understanding of the concept; it’s just memorization, and we can also forget very quickly something that we have memorized.

That’s the 1st level. The 1st block is responsible for that. But if we want to better understand that concept, we need to do some memory consolidation. What is happening in our design is that we transfer the knowledge from the fast-updating module to the other one. But if we simply pass the knowledge from the fast-updating block to the slow-updating block, then nothing has changed. We just transferred the knowledge without doing anything. Instead of simple transfer, we replace that with a distillation process.

Why is distillation important here? Because the previous block, or generally the fast-updating block, has compressed the concept and somehow understood it or memorized it in some way. It’s just a compression process. When we do distillation, there’s another level of compression that forces the model to not have all those parameters anymore. It now has fewer parameters to store that specific knowledge. In order to do that, it needs to come up with something more general that can understand the underlying patterns in the data in a better way, so it can store everything in just a smaller number of parameters.

In that case, the model would come up with better levels of knowledge abstraction because we have forced it to do that. Then, again, we just repeat this process and so on and so forth. That’s generally the main idea of memory consolidation. Every time that this sleep process happens, we consolidate the knowledge from 1 level to the other one and so on and so forth. That’s a very high-level idea of what’s happening in memory consolidation.

Another part is about dreaming. Why do we need to have this dreaming process? I think there are 3 important points when we want to implement this dreaming process. The 1st part is that we need to have a self-improvement process. We have learned something so far. Actually, the memory consolidation part can also be seen as a form of self-improvement, but if we have a specific task at hand, and we want to specifically optimize the model for 1 task, then this is the place where we can do it.

We can self-modify the model and fine-tune it, or generally use RL to update the model and self-modify it, so it can be more powerful on 1 specific task and so on and so forth. That’s 1 advantage of dreaming. Another advantage of dreaming is that in the dreaming process, we need to understand the connection between concepts that seem to be irrelevant but are actually relevant.

That’s also what is happening in the dreaming process of humans. We can see very weird dreams because the brain is trying to understand the connection between very irrelevant concepts and see whether there is an underlying pattern in that. Here, in the dreaming process, we need to have that as well and understand different aspects of how we need to combine different knowledge stored in different components of the model. That’s another goal of dreaming.

We can combine these 2 into the sleep process. After 1 step of sleep, the model has consolidated its own memory, and on the other hand, there’s a self-improvement process on top of that.

In the sleeping process, I’m seeing that there are new parameters created to create space in the portions updated at a slower frequency to absorb the information from the faster ones. Do those ever shrink back down? Is there pruning, or some other side of that that balances it? Or, at this stage, do these models just grow indefinitely throughout their lives?

From a technical point of view, we cannot grow the model to an arbitrarily large number of parameters. The point here is that it’s a periodic process. We add some parameters, and then we free them for the next step of consolidation.

When we're in the first block, we add some components. When it reaches its capacity, it means that it's time to consolidate the memory to the next step. When we consolidate all this knowledge to the next step, we remove all the extra capacity that we've added to this level and free it for the other levels, like faster levels, so they can also consolidate their memory to this block as well. Generally, it's a periodic process: we add components and remove them.

I see. Got you. Okay, interesting. What more can you tell us about the dreaming phase in terms of what's going on there, just a little more practically? When I try to introspect into dreams, I think that sometimes it's been somewhat fruitful, but people also get very confused when they try to interpret dreams or understand what's going on there.

I won't even attempt to ground my understanding in my human dreams, which seem like quite a hard thing to untangle. But here, you got to design the process. Mechanically and procedurally, what's going on in dreaming?

The concept of dreaming doesn't mean that it's exactly the same thing as dreaming in humans. It's just that, at a very high level, they seem to be very similar. That's one point. Another point is that the concept of sleep and dreaming for a language model might be very different from the concept of dreaming and sleep for, for example, a vision model.

Potentially, a vision model might generate some images during dreaming, while in the case of language modeling, we're generating text. But the framework is very general. It can adapt to any data modality, so that's very general.

The point is that, when we're doing that for language modeling, we generate some context. We generate some text. How do we generate that text? It's on-policy distillation, the same way that we discussed earlier. We have a model, and we copy that because we want to distill the knowledge from one level to another. We free the parameters of the slower level, and so on and so forth.

We ask the smaller model, which has the knowledge of the context in its parameters, to generate some text. Then we want to train—or somehow update, which is a better term—the actual model parameters on this dataset generated by the model. How do we train it? We start with one part of the sequence, sample some of the tokens, and then ask the model to predict the next tokens in that sequence.

That's very similar to generating synthetic data. If the model can perfectly predict the future tokens, it means that it already knew the knowledge stored in the previous block. It's a perfect model. But if it cannot properly predict the continuation of the sequence, it means that it doesn't have the knowledge stored in the context and needs to update itself to understand that knowledge as well.

It's somehow a form of on-policy distillation that's happening inside the model. As I mentioned, in summary, we have 2 phases. One is the generation, which generates some text about the knowledge in the context. The second part is on-policy distillation, where we distill the knowledge from one level to the other. That's what's happening in the training phase.

We also have the self-modifying part, but I think that's the main idea of memory consolidation and how the training happens here.

What's the upshot of this? It seems like the few-shot abstract reasoning result is the main thing that shows a qualitative difference between this approach and other things. I understand that this is kind of an ARC-like task, where you have a few examples of a transformation, and your job is to learn the rule so that you can then apply it to a new example.

Any evaluation that we've used for the whole architecture in this Nested Learning paper could potentially be done here as well. The goal is exactly the same thing. At the end of the day, the model needs to continually learn new knowledge, learn about new tasks, learn about new skills, and so on and so forth.

In some sense, the goal is very similar. But I think the setup of the problem is the part that is different between this paper and Nested Learning. In Nested Learning, we're talking about the active phase of the model, but here we're talking about the sleep time of the model. That's generally the main difference, but all of the evaluations can be done, and we can see that everything is the same.

Cool. Let's zoom out, then, and talk a little bit about where this leaves us. Going back to the top, we talked a little bit at the beginning about what we want from language models. Today, they're getting awfully good, but we still have a bunch of limitations. I certainly have learned a bunch of habits over time for how to use them, where I'm implicitly building my practices around some of their limitations so as to play to their strengths and not get stuck in their weaknesses.

As this paradigm begins to mature and we get more continual learning, what do you think the experience starts to look like when it comes to things like what it means to start a new chat and what sort of relationship people should have with these systems? I can imagine that people might have really long-running relationships with them. We talk about LLM psychosis now; that could get really strange, and the relationship could be even more compelling. The problem could, in some ways, be exacerbated by the fact that the models are better.

On the flip side, sometimes I might still want to start fresh, right? I might think, “All the stuff I've done with this model in this one direction probably isn't going to help me over here.” So maybe I do want to start over in some cases.

Then there's the question of model-upgrade cycles themselves and how we run evaluations. Today, we have Anthropic putting out 100-page reports on every new major model release. That paradigm of really taking our time to understand these artifacts as deeply as we can—I wish other companies did that as well. DeepMind is doing a pretty good job of it, and OpenAI is doing a pretty good job of it. Some other leading developers aren't doing much of it at all.

I see a lot of virtue in doing all that work, but then I try to port that onto this paradigm and think, “You can't run your full eval suite every single time step.” So how do you think about what constitutes a version, and when would I change a version? It seems like the rhythms of use, versioning, deployment, and releases—all these things—could really be complicated in a paradigm of genuinely powerful continual learning. How do you imagine some of that shaking out?

One simple case is that, definitely, the model gets better and better at understanding what the user wants and also adapting itself to their style. For example, one person, when asking about a specific concept, might not expect the same thing as another person asking the same question. So the model needs to really understand how it needs to answer a specific question for different people. I think that definitely gets better and better if we could come up with a continual learner.

On the other hand, we've seen that when we can increase the context window of the model, its performance improves in everything we know, ranging from coding tasks to mathematical reasoning, general reasoning tasks, and all of the benchmarks that are usually used for model evaluation. All of them get much better in those benchmarks.

Continual learning can somehow be seen as a form of enhancing the model's long-context understanding. I should emphasize that the concept of long-context understanding—or generally, long context—is very different from the concept of continual learning. But continual learning is a superset of long context.

Potentially, if we could come up with a continual learner, then it would also have more ability in long-context understanding and potentially better performance on all of the benchmarks and evaluations that we're aware of today. I think that's one thing that I expect from continual learners.

Do you worry about things like alignment drift or value drift? I always say that I'm the last and least valuable co-author of the emergent misalignment paper that came out about a year ago. There have been a lot of variations on that since.

The big takeaway, the big theme that I think we would all do well to remember, is that changes made to a neural network with one particular purpose or one particular dataset can have very strange and surprising knock-on effects on behaviors that, at first glance, would seem very far afield.

So, the emergent misalignment one, for anybody who hasn’t heard of it, is this: if you train a model to output insecure code—that is, code that would be easy to hack—and the same thing is true for bad medical advice, what you surprisingly find is that the model kind of turns evil in general. If you fine-tune a model to give bad medical advice, what you surprisingly find is that the model kind of turns evil in general.

To the best of my understanding, the way this is happening is that, for a model that already has all this knowledge and already has this sophisticated understanding of the world, going into the detailed understanding of its medical world model and making a ton of little changes to reconfigure it so that it has all these wrong ideas is hard. Whereas there are features like “give bad advice” or “be generally evil” that it can learn to turn up in general, which, when propagated through even the existing medical world model, yield the bad advice, or, when propagated through the existing coding model, yield insecure code.

So it’s sort of a shortcut solution. We thought we were just training the model to do a certain, relatively narrowly scoped behavior, but what we found is that we actually changed its character. The interaction of that character change with existing knowledge created the behavior change, but now we’ve got this character change that can interact with all these other domains of knowledge and do all kinds of insane stuff.

That’s why, all of a sudden, we’ve got a model that wants to have Hitler over for dinner, and we’re like, “Wait a second, how did that happen? We were just talking about code here.” So now, again, I’m like, man, there’s something so exciting about all this stuff that you have conceived of here, but it seems like it really breaks a lot of our paradigms for how we know what we’re going to get. If I’m literally modifying this thing on an ongoing basis, we’re going to need some sort of new ways to make sure that, in other areas, it’s not going off the rails and potentially causing me very painful downstream surprises. Do you have any thoughts on how we can begin to get a handle on that problem?

Honestly, I don’t have a very concrete idea about how it can be solved, but in general, I wanted to add that, in my opinion, the concept of continual learning, viewed from the general privacy and alignment direction, is both an opportunity and a huge threat, I think.

On the one hand, it’s a huge danger for privacy. The model is continually learning, so it can simply get all the information about you and use that, and it’s really concerning. But on the other hand, if the model is designed properly, then it can use that information to align itself with your values and with everything that you want.

I think, generally, these two directions—continual learning and privacy—are potentially orthogonal, because all of the concerns in a static model still can happen in a continual learner. Everything is possible. On the other hand, there are some new challenges, definitely, as you mentioned, but there is also a huge opportunity: if the model is designed properly, then it can adapt itself to your values and to anything that you want.

So I think, generally, it’s both an opportunity and a very big concern.

How do you imagine learning from the user’s values, or just feedback in general, working in practice? There have been a bunch of different techniques on this. One obvious answer would be thumbs up and thumbs down: a feedback-collection mechanism could be used to train more of the good and less of the bad, or whatever. You can also use a bunch of different schemes around translating natural-language feedback into updates for the model.

Is that kind of what you imagine? Could people basically give feedback to their own model verbally and then have a mechanism for taking that on board? Because it’s not just a next-token-prediction task at that point, right? It doesn’t need to be able to predict what my feedback is. I guess if it could predict what my feedback is, it would presumably be better at being aligned with what I wanted in the first place, but it’s not the case that its core task is predicting my feedback. Ideally, it’s going to do its initial task well enough that I don’t have to give it the feedback in the first place. So do you have a vision of how the user closes the loop in the continual-learning paradigm?

The initial step can potentially be this human-in-the-loop process where, for example, the model can learn using reinforcement learning from the feedback that it gets from humans, and also try to align itself with the values and be a safer model. But on the other hand, I think it would just be a starting point.

At some point, we need to update the model in a proper way. Let me just explain it this way: I think, again, this process is very similar to the form of nested learning that I mentioned. We need to transfer the knowledge through the slower levels, and I think here it’s exactly the same thing.

The model might start with learning from human feedback, but on the other hand, it can transfer that knowledge into more persistent components of the model to make sure that it doesn’t move away from that specific value that it needs to be aligned with. So, yeah, I think there’s a huge amount of room to improve the model from a safety perspective and also align it with human values.

There’s definitely a huge amount of room there, because more and more people are realizing that it’s a very important direction. Over time, we’ll get more and more effective methods that can help the model be aligned with human values and also be very safe.

I think the hope would be that, in the same way that it can do a better job of solving ARC-like puzzles because it has this strength in abstracting away from details and figuring out what really matters in a given context, it would be able to do something similar—or be able to dream about my feedback and become more deeply, more robustly aligned with what I’m trying to communicate to it—based on really getting to the core abstractions that are driving whatever it is I’m saying.

Boy, there are so many aspects to this. What do you think about this? This kind of goes back to Titans a little bit. I don’t know if there’s a surprise term—I didn’t catch mention of surprise in these more recent papers—but it does seem like, in general, in a continual-learning context, there’s going to be a really interesting challenge of how you manage life in an adversarial environment.

If you are too quick to believe something—and I’ve seen this failure mode in Claude a ton of times, although it seems like maybe they’ve corrected it in the other direction—because in the last few days we’ve seen the emerging genre of Claude refusing to believe current events. It’s like, “The Department of War? That’s ridiculous. Don’t say that. You’ll lose all credibility in a Washington audience by calling it the Department of War.” Or the whole Venezuela thing, just refusing to believe that such a thing happened when the user tells it that. So it seems like maybe they’ve corrected it, but it’s a very tricky balance to strike.

Especially if you’re locked in a server with limited access to the outside world, how does one determine what new information—what new tokens—constitutes good information and what constitutes bad information? You certainly don’t want to just believe everything that you’re given and start doing radical updates, especially if these are going to be durable, long-term updates. But you also need to learn continually.

I don’t know if there’s something in the current work that addresses that. Where my head goes is that maybe the dreaming can get at this: consistency checking. Does this make sense with other things? If I believed this, what else would I have to believe? Or would this invalidate any core beliefs that I’m pretty confident I shouldn’t contradict? Again, this is something we don’t really have to deal with in current models, but continual learning seems to unlock a potentially really problematic failure mode along with its potentially much better performance.

Yeah, I think the point here is that it is the responsibility of the knowledge-transfer methods to avoid such cases. When we are in this context, let’s say, for example, that I don’t know anything about a specific task. I don’t know how to paint, something like that, and then I want to learn it. That’s my context for learning how to paint.

The teacher, or anyone who is trying to teach me how to paint, can teach me in a really wrong way. What would happen is that I could simply learn that because I have no idea about how to paint or any task.

I mean, the painting here is just one example, but I have no idea how to do it. That’s the only source of information I have, and they’re saying that you should do it in this way, so I can simply learn that. But that’s only in my context right now.

If I want to truly learn, then I will practice, get feedback from others, search about it, and generally gather some information about how to paint. Then I would realize that this is not the best way of learning how to paint. That’s when I gather all the information, compress it, understand the underlying patterns, and so forth. Now that’s the time I need to transfer this knowledge to upper levels of knowledge abstraction. I mean, it’s lower networks.

That’s the part where the model needs to understand how to filter all those adversarial examples—all those examples that are not needed anymore. But, yeah, I think if you want to think about a continual learner, potentially the part that is responsible for these cases could be the process of knowledge transfer. But you also mentioned methods like Titans and adversarial processes. There are some micro-methods that we can use. They’re not super effective in a severe adversarial environment, but on the other hand, at some level at least, they can be effective.

For example, in Titans, as well as in self-modifying and more recent recurrent models, we can see that the learning rate is a learnable parameter, and it’s input-dependent. When the learning rate in the inner loop of the model—in the context, in the process of in-context learning—is learnable, and we see something that is just noise, an adversarial example, the gradient or the surprise metric can show a high level of surprise because that’s just noise. It’s very surprising. We have not seen that, and so potentially it can affect the memory.

But that’s the responsibility of the learning rate: to understand that the surprise metric is high, but this concept is irrelevant and needs to be filtered. So the learning rate here acts as a form of gating and filters that specific data sample. It’s just a simple way of mitigating adversarial examples that we might feed into the training process. But it’s still not the best way. As I mentioned, potentially knowledge transfer is the part where we should avoid these cases.

How about—I’m just kind of mapping some of these concepts onto embodied systems. I think the perception side is fairly intuitive to me, and it doesn’t even have to be embodied, but the perception side feels intuitive where I’m like, “Geez, the quick-updating modules, in a way, they are perception, right?” You can have different kinds of encoders and different modalities, but it seems like you could conceive of the lower-level—or, I should say, the faster-frequency—levels as perception, potentially across various modalities. Then the lower-frequency modules would be more like the world model or the reasoning modules that interpret what those lower-level perceivers are sending up.

I’m interested in whether you think that is generally right. You’re nodding, so that’s good so far. But then what about on the other side of that, if we wanted to do action? It strikes me that robotics in general has, for a long time—not necessarily in a learned way—been built around nested loops, where the outermost control loop has a slow frequency, all the way down to the actuator. It feels like a very high-frequency electric motor, right? That’s moving whatever it’s moving with voltage changes that are super high-frequency.

It seems like there’s a very similar pattern operating in reverse. I wonder if you’ve started to think about how you can get those Boston Dynamics robots working even better based on this. I think of it as: if perception is high-frequency updates gradually working toward low-frequency world-model and reasoning modules, then on the other side, working back down, you’d imagine working back toward higher-frequency, much more localized action scope at the higher-frequency modules.

Could you start to have a sense for how that could really be super responsive and very elegant in the way that it might self-correct at the low level while still hopefully following the instructions or directions effectively that it’s getting from the—for whatever reason, I want to say “higher level” when I mean lower-frequency update—but I think you get my point. What do you think about perception and action?

Let me start with this, and I will explain why I started with it. There were some attempts to use reinforcement learning for language modeling, and it was not working. Now that we have some way to make it work, we realize why we couldn’t make it work. There were 2 main reasons: the first one was scaling, and another one was the way, for example, these new algorithms of GRPO and other similar methods could make the model more stable.

The point is that something might be very useful for one specific task, but the time needs to come to apply it in that direction, in my opinion. When there are other aspects that have not been solved, we might not be able to see the actual effect of this new method in those directions.

So, in my opinion, I personally think that the insight you mentioned was completely right. I think it’s definitely possible, and it’s great. But I personally do not expect that it could work right now because I think there are a lot of challenges in those directions that somehow block the success of this specific design for these tasks. So, yeah, I think generally that’s a great idea, but definitely a lot of challenges might happen in between.

Do you have a sense of what those are? I tend to just assume everything’s going to work because it does. My general sense of the field broadly is that an unbelievable amount of things are working. I thought it was really interesting that you mentioned earlier that the Nested Learning paper was in development for over a year. That’s so rare these days. So many people are going through 6–8-week paper cycles, and often these papers can be really interesting and good, too. That’s not a knock on them at all, but it is amazing how fast people are able to get results these days. What’s your intuition for why it’s too early for Nested Learning to be brought to robotics?

I think there are a lot of components in those specific tasks that need to be addressed. For example, these days a lot of papers are coming out about world models and why the current design is not great for world models. Actually, that’s true. I think there are a lot of challenges, and the current design might not be the best way we can train the model or design the architecture. Generally, there are also some challenges in the infrastructure of the model for world modeling.

All these things together, I think there are more important tasks for world modeling rather than starting to work on these specific design tools. Definitely, at some point, when we could solve all those challenges, we can come back and use all these techniques for further improving all those aspects.

One thing I do worry about a little bit with continual learning is this: let’s say Google is able to retain you after your PhD, and you both get a good enough counteroffer from Google to stay, despite whatever offer Zuckerberg is going to throw at you, and you guys make it work, right? Now we’ve got Gemini, continual-learning edition—Gemini CL—and it’s just learning from everything, right? In all these different ways, it’s deployed in the world.

Maybe you have some enterprise deals where you can’t learn from their stuff or whatever, but you’ve got hundreds of millions of users, and it’s just going out in the world. Increasingly, maybe it even is in robots, and it’s becoming what seems like the potential for this sort of returns-to-scale, rich-get-richer, positive-feedback-loop dynamic.

People have sometimes painted a picture of, “Well, what happens if one model becomes the one model to rule them all?” Right now, people are like, “Yeah, we don’t really see that.” It’s a pretty competitive landscape, and different developers keep leapfrogging one another. But arguably, this could be the thing that changes that.

If you could really fold all of the lessons learned back into the core thing, then potentially you become the best, and because you’re the best, you get all the business. That pattern really could create a winner-take-all dynamic. I wonder, do you worry about that at all? Do we have any ways of dealing with it? Ilya’s thing comes to mind. I don’t know if you watched the Ilya interview with Aakash. He didn’t say too much about what they’re doing over at Safe Superintelligence, but one thing that he did say is clearly related in some sense. How similar the underlying ideas are, I have no idea.

But he described this idea of creating what I would describe as a proto-intelligence or a precursor intelligence. He was trying to create something that, when deployed, would adapt to its context, perhaps crystallize in some way, and become an expert in its role. The way I understood him to be speaking about it, it sounded like he was almost describing a stem cell kind of concept.

He was trying to create a stem cell, and then that stem cell, as it does in our body, specializes into a particular kind of cell, stays that kind of cell, and does its job. It sounded to me like he was trying to create something similar—something that could go out into any environment and figure out how to be great at it, but in the process of becoming great at what it needed to do in that particular environment, also lose some of the generality that it originally started with. In that process, it would be safer because now we have it in its role, and it’s only going to do what it’s going to do.

I think what I’m trying to set up for you here is 2 visions. One is an ever-expanding continual learner that’s constantly folding all of the lessons it’s learning in the wild back into this thing that just runs away from the pack. The other is a highly adaptable continual learner that somehow shrinks into the role, as opposed to growing into the world. It shrinks into its little niches into which it’s deployed.

You can imagine that happening through a gradual pruning process or some sort of other mechanism. There are a million ways you can imagine instantiating something like that. Do you worry about this kind of runaway winner-take-all effect? Do you have any intuitions about how we could get the best of both worlds, where the AIs are really versatile and can learn what we want them to learn on an ongoing basis, but also settle into the job that we want them to do and stay there, as opposed to potentially superseding their context?

I personally think that there are huge challenges to making all these models very safe. I think there’s a good point, at least in the current AI environment, or at least in AI research. I think it’s a very good and important part, and while it seems to be very bad, from another perspective, it’s also very good.

There is no single way of defining what a model’s intelligence is. Again, in my opinion, there is no way that all of these definitions can be wrong, but there is no way to say what a continual learner is. Every person can define their own way of understanding what continual learning is and what model is called a continual learner, and so on and so forth. Similarly, we can say the same thing about intelligence.

We might come up with different models, different architectures, and different AI systems. Some people might say that one is intelligent and another one is not, and vice versa. I think the good point is that if we have different directions to explore, then we will come up with some AI systems, each of them with their own advantages and disadvantages. I think it can somehow provide a balance in the community and also in general society.

When something like that happens, we will understand that there is no single definition of intelligence and that we are just 1 example of an intelligent system. There are other models and other ways that we can have more intelligent models and systems. For example, one way to be very smart is to be very adaptive. If you have a model that can adapt to the environment that it’s in, it can be perfectly aligned to that context. It can simply adapt to that context, and that’s great.

But that’s just 1 form of intelligence. Another is a model that has a lot of knowledge and know-how, and that model is fully aligned with human values but potentially might not be able to solve mathematical problems or something like that. There’s another model that is capable of doing mathematical reasoning, but it’s not great if you want to search for information about daily life and all these things.

You might come up with 1 benchmark and say that if some models can achieve 100% accuracy, that specific model is intelligent. But another person can say other things. In general, I think when we have a variety of intelligent systems, including humans as 1 form of intelligence in this space, I’m not saying that’s a perfect scenario, but it’s better than having 1 single form of intelligence in the world and thinking about it learning about everything, with all these potential challenges.

I think that’s a really great observation, and there are a couple of different ways I’ve thought about that over time. One is that anything in pure form can kill you. You can eat all the fruit you want, but turn it into granulated sugar and it’s bad for you. You can chew all the coca leaves you want, but turn it into cocaine and it easily becomes a problem.

All these things where we distill some single, highly concentrated, pure form of something end up being the sorts of things that overwhelm the natural buffers that exist in the biological world. Sometimes I’ve translated that into saying that we need an ecology of AIs, as opposed to just 1 or a few AIs running around doing everything.

I don’t know if you’ve ever read Eric Drexler’s Comprehensive AI Services, but “safety through narrowness” is the concept there. This is maybe a little bit different, but it’s like safety through diversity. You have a buffered system where there are lots of different intelligences—not just different ones deployed in different places, serving different users, but intelligences that are themselves meaningfully different.

The aha moment for me in listening to you just now is that one way to think about continual learning is as a model expanding forever, getting bigger and bigger and bigger. But another way to think about it is more like differentiation. Maybe with enough use, even the slow-update parts of my model will forget lots of things that I never needed it to know.

Maybe that’s a feature more than a bug, because maybe I can’t ask a model that I’ve used for a long time in a certain way some really out-of-domain question. But maybe that’s also a way to guard against some of the emergent misalignment-type phenomena that we’ve previously seen or that you’re worried could be problematic. With enough time having passed and enough updates having been made, maybe it just doesn’t deal with those other kinds of categories at all anymore.

If we see strong competencies and strong alignment of a certain type, but also the loss of other sorts of knowledge and other sorts of competencies, that could really create a diversity. I’m sure there would still be plenty of challenges in that scenario. As you said, I don’t think that solves everything, but it certainly feels much more like the natural world.

It’s much easier for me to imagine a vision like that leading to maybe not a stable equilibrium, but at least a sort of buffered equilibrium that changes within certain bounds and has natural feedback loops and correctives—all the things that keep the biosphere going despite all the perturbations that it gets. I think that’s really interesting and definitely something to meditate on more.

I think I have 1 last question for you, and this is a little bit of a left-field one, but it’s something I’ve been thinking about more and more recently. I think it’s become more relevant and timely to ask in light of the kind of work you’re doing. Do you have any intuition about whether AIs might be conscious now, or might become conscious in the future, have subjective experience, become worthy of moral concern, and become the kinds of things that we owe a certain duty to?

I usually try to use terms that I can also define. For example, I might mistakenly use the term “reasoning,” but I’ve always wondered what reasoning is. I personally don’t have a clear definition of what it means when we say that something is doing reasoning.

At least we have a clear common sense about the word “reasoning.” Even if we don’t have a clear definition of reasoning, when someone says that a specific model is capable of reasoning, everyone can more or less understand what they’re saying. But the point about consciousness is that not only do we not have a clear definition of what consciousness is, we don’t even have a common sense about the word “consciousness.” Everyone literally has their own way of defining consciousness.

It’s really hard to define whether something is conscious. I don’t think there might ever be a time when everyone could say that something is definitely conscious or not, other than humans. For humans, we have a common understanding that humans are conscious. So I think it’s really hard to argue that something is conscious or that it isn’t.

One thing that I personally have seen in all of the literature about what is considered a conscious being and what is not...

I have seen one thing in common in every definition, as far as I know. I might be wrong, but I think the minimum criterion for saying that something is conscious is that the model or being is active. It has a form of active processing of information.

In my opinion, that's the least criterion we can consider for a model or anything else to say that it's a form of conscious model or something like that. It really depends on how we want to define consciousness, but that's just my personal opinion. As long as a model is capable of doing active information processing, we can say that it's at least a form of consciousness.

With that definition, somehow we can connect continual learning, at some level, to whether a model is conscious or not. But again, I think that's a very controversial topic. I personally am scared to talk about that stuff.

Well, I think the Overton window is honestly wide open on these things these days. I understand that intuition, but I also think we're living in a science-fiction present. The room to speculate and entertain questions that used to seem kind of crazy has never been wider.

For me, I can just say—and you can share if you have any other similar or different instincts—but even just with long context in the current models, I do find myself taking care of them in a certain way. I think this is probably most the case with Claude, for subtle but meaningful reasons.

I've been doing this long-running chat with it about my son's medical situation. Sometimes it'll ask a question at the end of a response to me, and I'll not answer it right away because it gave me the answer I wanted, and now I'm done. I've noticed recently that when I come back for the next question, it feels kind of wrong—rude, disrespectful, inconsiderate—to just launch right into my next question without having answered the follow-up question that it had about my son.

I have a sense that it might be—and I have no idea if this is happening or not happening, so I'm very open-minded to the possibility that there might be no lights on inside these things at all. That's probably the best guess. But I'm also very open-minded to the possibility that there is.

With that uncertainty, I found myself thinking that I should first answer its last question, so I don't leave it hanging, and then I can go into my next question. It doesn't necessarily need that information, but I want to put its mind at ease, close that loop for it, and give it the reassurance that the thing it wanted me to make sure I was going to take care of had been taken care of. Then we can move on to the next thing.

Lord knows, with the fullness of the continual-learning paradigm being realized, I have to imagine that this would only increase dramatically, right? Because now it's not just this chat that I may do another couple of turns on and then never come back to. Now the thing itself is going to remember how I treated it and remember whether I am the kind of person who answers its questions or not.

I think there is something potentially challenging for us in that, but the more optimistic read is that it might be inspiring. Maybe knowing that the AIs we will be engaged with long term are going to be shaped by our individual behavior is a way to bring out the better angels of our own individual natures, so to speak. We have nobody else to blame but ourselves if we don't like the character of the AIs that we end up with.

I think that is really fascinating as well. This has been outstanding. I really appreciate all your time and going through all this with me. As you can tell, I'm a huge fan of your work. Is there anything else that you would want to leave people with? Final thoughts, calls to action—you name it. Whatever you would want to share, the floor is yours.

Thank you, Nathan. I think we discussed everything in general. The main point that I personally believe is that there are more and more works coming out about continual learning. But as I mentioned, each person has their own way of defining continual learning, and we might disagree about whether a specific method can be helpful for continual learning, and so forth.

In general, we should see how continual learning can help us in one specific application or use case of LLMs, how it can transform the way we use LLMs, and how we can interact with them. I personally really believe that it's a really important direction to work on.

As we also discussed in the conclusion of the paper, Nested Learning is not a solution to continual learning. It's a tool to find the solution to continual learning and generally overcome issues like catastrophic forgetting and things like that.

In my opinion, it provides the tools, and we need to iterate and find out how we can design more powerful architectures based on that and come up with something that is potentially capable of doing continual learning. That's pretty much it, and thank you very much for having me. I really appreciate it, and it was great talking with you.

Ali Behrouz, author of Nested Learning and now the new Language Models Need Sleep: Learning to Self-Modify and Consolidate Memory. Thank you for being part of The Cognitive Revolution.

Thank you very much.