AI的胚胎学：训练数据如何塑造AI发展——对话Timaeus的Jesse Hoogland与Daniel Murfet

Today I'm excited to share my conversation with Jesse Hoogland and Daniel Murfet, founders of Timaeus, an AI safety and alignment research nonprofit that's pursuing an ambitious, mathematically rigorous, and fascinating approach to understanding the development and function of neural networks.

Named after one of Plato's dialogues, Timaeus's work is based on singular learning theory, or SLT, which applies algebraic geometry to statistical learning theory. Obviously, that's a mouthful, but the core premise of SLT is pretty intuitive: training data determines the geometry of the loss landscape, which in turn determines which algorithms models learn during training and, ultimately, how their behavior will generalize once training is done and they're put into actual use.

The driving insight of SLT is that the super-high-dimensional loss landscapes in which modern neural networks are optimized are not actually well represented by the smooth, bottom-valley-shaped surfaces that we often see depicted in figures. On the contrary, Daniel, who recently left a tenured professorship in algebraic geometry to pursue this work at Timaeus full-time, calls these representations “maximally misleading” and explains that, in reality, loss landscapes are highly complex, jagged surfaces full of singularities, also known as degeneracies. These are directions in weight space along which a model can move without changing its external behavior or loss score, but which nevertheless sometimes do involve a change to the model's internal circuitry, such that the model might behave very differently in novel situations.

This, of course, has profound implications for big-picture AI safety questions. To frame it in terms that would be familiar to Eliezer Yudkowsky readers from 15-plus years ago, the difference between a model that acts nice and friendly because it is fundamentally aligned to human values and a model that acts the same way because it has learned how to please humans while actually pursuing its own goals could be the difference between a superintelligence-powered utopia and human extinction. And yet today, even as AIs become more and more powerful, we don't have reliable ways to tell the difference.

Anthropic- and Goodfire-style mechanistic interpretability has, of course, made great progress toward identifying the concepts that trained neural networks represent and, these days, also offering some visibility into the circuits they use. But there's a very long way to go, and I definitely believe that there's plenty of opportunity for complementary approaches to strengthen our overall understanding.

The Timaeus approach, which they call developmental interpretability, aims to understand how neural networks evolve through the training process, using a measure called the local learning coefficient to help identify what might otherwise be invisible internal phase changes that could profoundly affect downstream model behavior.

This line of work, like all approaches to understanding neural networks, is still pretty early in its own developmental history. But critically, the Timaeus team has shown that it can scale beyond toy models. Their latest work applies these techniques to 7-billion-parameter models and is able to identify critical phase-change moments that correspond to the appearance of important functional circuits. So we might say that it's roughly at the stage of working toward monosemanticity, and hope that, with more engineering and compute, it will continue to scale to frontier models.

If successful, this could perhaps prevent an episode like the one that happened in Claude 4's training, where a certain safety dataset related to harmful system prompts was mistakenly left out of the data mix, causing the model to generalize in such a way that it followed rather than refused harmful system prompts. Anthropic caught that problem with behavioral testing and patched it. But the hope for developmental interpretability is that such things might be caught by instrumentation during the training process, before they ever seriously affect model behavior.

In the best-case scenario, this could help us move beyond the trial-and-error phase of neural network training and toward a more principled, engineering-like discipline, where specific datasets are used at specific times for specific purposes, leading to predictable and reliable results.

Now, this is all high-concept, mathematically sophisticated work, and this conversation, to be honest, is really just an introduction. I did my best to take my time to develop my own intuitions for what's going on inside a neural network during training, to compare and contrast some of the phenomena that Jesse and Daniel described with other things, like grokking, that we've previously covered, and to grapple with questions of how much generalization we really want from neural networks and how, in some cases, too much generalization could be very harmful.

I found this conversation fascinating throughout. While it will stretch your brain in a bit of a different way than most of our episodes, I expect that, for some of you, it will immediately be among your favorite episodes that we've ever done. I hope you enjoy this introduction to developmental interpretability, a new approach to understanding neural networks, with pioneers Jesse Hoogland and Daniel Murfet, founders of Timaeus.

Hi, Nathan. Thanks for having us. I'm excited for this conversation. I think I'm going to learn a lot from it.

One of the things I think is really interesting about what you guys are doing is that it presents a bit of a contrast to what I see as the general default approach today in the safety and alignment space, which you might call prosaic safety and alignment methods, or defense in depth.

I'm always on the lookout for something that feels like it—even if it's a long shot—could really work. By “really work,” I mean giving us enough depth of understanding and enough reason to believe that this approach could actually work even in the surprising future that we may try to carry it into. I mean working so well that I don't have to worry about it anymore, and I can just focus on all the applications I want to build with AI without having all these “what if it gets out of control?” episodes.

I think you guys have something that fits that bill. It is, at times, over my head, and I expect members of the audience will, at times, have to stretch their brains to really grok it as well. But that will be part of the fun, so I'm excited to get into it.

For starters, can you give me just a little background on how you guys came together and on the organization? I was intrigued by the name, and I'm also very intrigued by the fact that Daniel, you recently left a tenured professorship to go all in on this work. Start off with the backstory, and then we'll get into the more technical details.

Yes. I think the story starts 2,500 years ago with Plato. Timaeus is a dialogue by Plato in which he puts forth sort of the first theory of everything. It's a proto-chemistry where the elements—earth, wind, and fire—are associated with Platonic solids, right? The cube, the icosahedron, and the dodecahedron, and these kinds of things.

Now, the theory is completely wrong, but it's the first instance of somebody thinking, “Hey, mathematics really has a chance of understanding the natural world.” It's that spirit that drives us today. Maybe we can find ideas in mathematics and the sciences that actually help us understand this new phase of matter—these deep-learning systems—to help us make sure that they're safe.

Also, in the Timaeus, the world is intelligent. The history of the universe is just a learning process, and physics is just a subset of learning theory.

That's fascinating. I've never read it, I'll confess, and it is striking that somebody had such a worldview 2,500 years ago. This is the deepest in history we've gone for the intellectual roots of a line of work.

Usually, I skip the backstory in a lot of podcast openings because it's, “Well, we saw ChatGPT, and we knew we had to build this app,” and there's a lot of that. But this is the deepest in history we've gone for the intellectual roots of a line of work.

So, how literally do you take the idea that the hierarchy you described—that physics being a subset of learning—might really be real? I know that there are a lot of differing opinions, much like in AI. We've got Turing Award winners saying we're all going to die, and we've got other Turing Award winners saying the first ones are crazy, on the question of what is the fundamental nature of the universe. There are probably even more disparate takes.

Do you come from a secular tradition or perspective? Is it a mathematical universe, or are we bracketing that sort of thing for now?

Yeah, I was mostly trying to troll your physicist listeners with that. I don't think I want to take a position on the mathematical nature of reality. That seems like a bit of a tangent.

But I mentioned that aspect of the Timaeus not completely in an idle way, because there are a lot of deep connections between physics and learning theory, and those are some of the connections that inspire a lot of the work we do, particularly with statistical physics and so on.

The study of learning machines is as deep as physics in many ways, and it's not a surprise that a lot of physicists are starting to devote their professional careers to understanding aspects of these systems. So, yeah, that's maybe the main connection—better than sucking all the physicists into hedge funds, I say.

How about just a touch more on what kind of math you were doing as a professor and how that positioned you for this approach to AI? It does seem quite distinctive from almost everything else that I've seen. There are a few other little points in space that may be close to it, but not a lot.

Yeah, I'm an algebraic geometer by training. Algebraic geometry is a part of pure mathematics. We study mostly solutions of polynomial equations: very high-dimensional geometric forms and very sophisticated techniques for trying to understand those geometric forms.

Some of it is related to the real world. Most of it is not. There were some connections to string theory at various points, which I've studied a bit, but mostly I was working on algebraic geometry, with a little bit of mathematical logic. At some point, I was following developments in deep learning. They were very interesting, but it was completely separate from my professional life until I encountered Sumio Watanabe's work.

He's a Japanese mathematician, the inventor of singular learning theory. It seems kind of hard to believe, but one of the core tenets of his work is that one of the very deep parts of algebraic geometry is central to Bayesian statistics. That was kind of curious to me. I was also aware that people were not completely satisfied with the theoretical treatments of deep learning that existed at the time.

So it seemed tempting to me, as an algebraic geometer, to adopt this language and approach to try to understand these systems. That's how I got into singular learning theory. That was a few years before I was drawn into thinking about AI safety. It was through the fundamental theory of Bayesian statistics.

All parts of mathematics are ultimately connected at some level, right? It's not necessarily super surprising that geometry comes into learning theory. There are other ways it enters, but that's one of them.

Okay. So, I guess, in terms of motivating the whole narrative, you could tell me how you think it's better to motivate it. As I've studied up on the subject, one of the things that has seemingly distinguished your approach from most others in terms of understanding AI systems generally is the focus on the developmental process: how is it actually learning through the training process, as opposed to running it, getting a thing out, and then trying to make some sense out of it?

That's one possible motivational line. Another could be that it starts with this one weird thing in math that I couldn't get over. Maybe you can draw those threads together and tee up how you got to the current body of work.

All the research we're doing is based on singular learning theory, SLT. I'd say that if I had to reduce that to 1 sentence, the core idea of SLT is that the geometry of the loss landscape—the surface that we walk down when we're defining some sort of optimization process—contains important information for understanding neural networks and potentially steering them more reliably.

That has applications, on the one hand, for interpretability and, on the other hand, for alignment. When we started looking into applications of this theory, which at the time was still very mathematical and theoretical, for actually empirically understanding neural networks 2 years ago, one of the first applications we came up with was this agenda we called developmental interpretability. Can you use SLT to better understand how neural networks develop over the course of training?

Now, the hope here is that development is an axis that can reduce the problem of interpretability to something more manageable. At the end of the day, all interpretability techniques are hoping that we can reduce the problem of understanding some 100-billion- or trillion-parameter model into fewer numbers, into something that's a little more manageable.

One way you could try to do this is to understand the changes that give rise to that model over the course of training. If there's a finite number of changes, and the number of changes is much smaller than the number of parameters, then you've got a useful simplification for the model. That's what motivated developmental interpretability, or DevInterp, when we started 2 years ago.

When you say the number of changes is smaller than the number of parameters, that means basically the number of steps in training. Each step would be a change.

Yeah, that's a great question, because one problem is that if you have to understand each individual training step, this isn't a reduction, right? There are still billions of training steps and many, many gradients to calculate. Somehow, the unit of change needs to be larger than just 1 training step. So what's the right unit of change here?

That's where SLT actually comes in, because SLT says that the right unit of change is a phase transition—a developmental stage in biology, for example. It's an empirical question, then, whether you can actually find these phase transitions and whether they actually exist at a frequency that's useful for simplifying the problem.

But in the systems we've studied so far, that does seem to be the case: you can meaningfully reduce the problem by looking at these stages.

Okay, that's definitely interesting. Would you make an analogy between a phase change and grokking?

Yeah, grokking would be an example of this kind of change. Typically, we identify maybe 2 primary kinds of changes. The first, which we call a Type A transition, involves the model becoming more complex as it learns more information. You can imagine that a human develops a richer and richer mental model of the world that becomes more and more predictive and more accurate.

But each additional step requires a more sophisticated model. Then you have the other kind of learning behavior, which is something like grokking, where you find a simpler explanation for the same data. You keep the same level of performance in explaining the data, but now you gain a solution that's simpler and therefore generalizes better. The tension between these 2 things gives rise to this interesting learning process in neural networks.

Cool. Not too long ago, I did an episode with 2 of the co-founders of Goodfire, and Tom, who's the chief scientist there, gave an interesting, very short update on where we are in interpretability today. He said, “We've been pre-paradigmatic for a long time, but I'll now give us proto-paradigmatic status.”

Then he went on to describe the proto-paradigm. There seems to be a fairly broad consensus that neural networks learn interpretable things—these things we call features. They are packed super densely into parameter space using superposition, and that means the individual neurons are polysemantic.

These features are almost orthogonal to each other, but not quite, and that's how they're packed in. The magnitude of their activations seems to correspond to the intensity with which they're relevant to a particular context, and these features connect up in circuits across the layers of a model.

Do you guys sign on to that proto-paradigm, or is there anything that you would add or subtract from it to put forward your own proto-paradigm?

Yeah, maybe I'll take that one. Firstly, just to add to the discussion of the previous question, we expect—and do, in fact, see—clear phase transitions in small systems. But nobody really expects to see hundreds of very clear step changes in the loss curve or other measurements in large models.

So that's another question: if this is actually an axis of reduction, how do you find that structure inside a pretty homogeneous-looking, global, coarse training process? You need some other means of looking more closely, or of tuning your detection instrument to various different kinds of frequencies, to figure out what's going on. So there's additional information there.

To answer your question, I'm a big fan of the SAE's work, particularly the recent work from Anthropic and what Goodfire is doing. Actually, in that episode, I think Tom mentions parameter-based methods at some point too. I like the analogy he gave, which is to think of it this way: if you have a graph, you need to understand both the nodes and the edges, right?

It's very clear that ultimately the information in the model is in the parameters, right? There's a role for understanding what the model is doing by examining that structure.

And one way of thinking about that is examining the geometry of the loss landscape. I don't think of, say, SAEs and our approach to interpretability as necessarily being in conflict; they're likely to inform each other. If I were to make one critique, potentially, of things like SAEs—many people have concerns or doubts about SAEs, and many people are very enthusiastic about them—I think I'm probably on the more enthusiastic end. But if I had to make a critique, it's that there isn't really a sort of mathematical foundation under SAEs that connects them to things like generalization, at least not that I understand. I don't think it exists yet.

That's not to say it can't exist, and maybe we will, for example, do some work on that at some point. But if you go out and understand what the model is doing by computing SAEs and find circuits, it's very unclear, for example, if you were to do continual learning or reinforcement-learning training on top of the interpretability work that you did on a base model, what behavior you end up with after you continue to train, and what the relationship is to the interpretability examination you did before. That's because it really isn't clear what the relation is between the structure you found in the SAEs and the data-generating process or the training process, so you don't know how stable it is to that.

I can imagine that, empirically, it actually is quite stable. When I spoke to Chris Olah, I expressed this concern to him, and his response, if he doesn't mind me passing this on, was, “Well, if you understand circuits through SAEs and then you have some further modification of the model, probably what's happening—if you do fine-tuning, probably what's happening—is that the fine-tuning process is recruiting the representations and circuits that existed before and using them for a different purpose.” Indeed, Anthropic has some work looking at features before and after fine-tuning, and I would say it's consistent with that interpretation.

So I think you can give an empirical answer to the question of how SAEs relate to the training process, the generating process, and generalization just by going out there empirically, checking, and seeing that it seems to work. I think that's reasonable. I think that isn't necessarily a path to the high level of assurance that we might hope to get. When people talk about generalization in connection with AI safety, it's the hope that you can ground the understanding you have of the model in something mathematical, something that is a solid foundation and is not purely empirical.

Our approach to interpretability, in contrast, is more about starting from the generalization end and working toward interpretability. The hope is that—and, obviously, we're also more about parameters rather than activations—but there are likely to be relations between these 2 perspectives. I can speculate on those if you want, but...

It strikes me that everybody has an intuitive sense of what generalization means, but you may want to offer a more formal or precise meaning for that. At the ChatGPT user level, generalization is like: I can take in anything, and it seems, as Ilya once said, “The amazing thing is, I talk to the AI and I feel I am understood.” That's maybe the mundane-utility, short-form story of generalization. Then there's the grokking generalization, where you go from memorization to algorithmic generalization, and I suspect you're going to generalize that to your set of generalizations. How do you mean that?

Yeah, I guess maybe you're understood because you're an interpolation of the training data, Nathan. When people talk about generalization, they often mean multiple, somewhat distinct things. Technically, generalization is usually a number. Generalization error represents the ability of the model to predict on samples from the data distribution that were not seen in training. That might be a held-out test set, as it often is in machine learning.

Theoretically, we often suppose that there's a generating process which both generated the training data and from which you can continue to draw samples. Generalization error would then be something about the gap between the predictions you make based on the model you learned from the training data, evaluated on new samples from the same generating process. That is, if you like, in-distribution generalization.

Usually, people don't add the prefix “in-distribution”; it's just understood that you are testing the trained model on samples from the same generating process that produced the training data. If you're not, then we refer to that as out-of-distribution generalization. That's a very fraught, difficult thing to understand, because of course those new samples could be absolutely anything. Why should you expect any particular good performance on that?

Yeah, I guess the attitude toward generalization that existed before GPT-3 was—I mean, this is one of the big cognitive transformations in the last few years—people really understanding that you can pretrain a large model on very diverse data and get something like generalization to a very large range of tasks. It wasn't super clear exactly what form of generalization that was, I think, and the degree to which you're actually getting capabilities that work well outside the training distribution, in some sense, is of course a very controversial topic.

Are LLMs really reasoning, or do they really have the ability to do arbitrary things that reasoning can do, which may be very far outside the training data? People debate these kinds of questions all the time. It's quite unclear.

But the angle—the interest we have—beyond thinking about generalization as a number, a measure of this ability to predict on new samples, is that you can ask for an understanding of generalization that has as its core mathematical type not a number, but something more complex, like, say, an algorithm. The algorithm is the thing that allows you to generalize.

If you have low generalization error, maybe the reason is that you have internalized some algorithmic aspect of the generating process. In some sense, the study of interpretability is one way of phrasing it: you're just studying generalization, but you've changed the type signature of the object you're trying to produce. Rather than being interested in a number, you're interested in the thing behind the number. I would say that's what the aim of interpretability is, in some sense: to explain why you're able to have low generalization.

So, yeah, that's intuitive, certainly at the level of the original grokking paper—the first steps. It memorizes relatively quickly. It can get the samples that it's seen, right? It still does terribly on all the examples, even from the same distribution, that it hasn't seen until—and I always like to remind people of this—orders of magnitude more steps later. Then, slowly, it looks like a steep rise on the log x-axis on that famous graph, but it's actually the last 90% of the training time when this grokking process slowly happens, and then you get an actual algorithm for, in that case, modular arithmetic, and now you're good to go.

I think there are some interesting things there to unpack. One that's really close to the heart of why all this matters is that it has been shown that there are multiple different ways to grok that problem. We know now that—I think this came out of Max Tegmark's group, if I remember correctly—there's the sort of trigonometric way, and then there's the pizza-slice algorithm. Both get the right answer, but they perform quite differently.

Maybe that doesn't matter in that particular case, but obviously you can imagine learning very different algorithms to solve the same problems, especially as we go into a space-colonization mode or try to put these things into an online-learning environment. That could lead to potentially very different downstream behavior, and so it makes a lot of intuitive sense that we would like to understand what is going on there.

Maybe, Daniel, you can elaborate on that or clarify anything I'm getting wrong, but maybe this is a good time to give what I've kind of started to think of as the central dogma of your approach. In the same way that we've got DNA eventually getting to proteins, you've got this story of starting with data, and there's this sort of set of relationships that takes us from data to model. I don't know if you like that central-dogma label, but I'm sure Jesse is going to use this every week now—my central dogma. So, thanks.

Thanks. Thanks for that. It's a good phrase. I don't know if it's quite apt.

But, yeah, give us the central dogma, and then when we get to the end, we'll still have this ambiguity, at least for now, around what it's getting right and what algorithms it actually learned. Right?

Right. So I think the key question is: Where does neural-network behavior ultimately come from? All of our alignment techniques—what we're trying to do is change model behavior and make sure that it generalizes robustly out of distribution. Many problems can be reduced to this understanding. Ultimately, behavior comes from training data, right? There are basically 3 inputs to every learning process. There's the neural-network architecture.

There’s some choice of optimizer, and then there’s the training data. Of these, training data is the most important one—the one that tells one model to learn one algorithm and another model to learn a different algorithm. So what we’d like to understand is how training data gives rise to the final behaviors models end up with, in particular because this is what all of our current alignment techniques look like. RLHF, Constitutional AI, DPO, deliberative alignment—all of these techniques are basically just modifications on the same deep learning processes with different data.

In particular, data determines the geometry of the loss landscape. It’s that geometry which in turn tells SGD how to move around, so it tells your optimization process how to move. That learning process picks out the final weights, and thus the algorithms, that a neural network ends up with. And it’s the structures in those weights that determine how models generalize, and thus whether they’re aligned or not. So this central dogma—as you called it, something we called the S4 correspondence between structure in data, structure in geometry, structure in the learning process, and structure in the final weights of the model—is what we see as key to understanding the mapping from data to final behaviors.

And so, what do we know in general about the loss landscape? I mean, we’ve all seen many 1-dimensional loss curves, with loss dropping over time. You mentioned earlier that, especially in larger models, there may be lots of little grokking moments happening in there, and phase transitions, but because they’re all so aggregated, we can’t see them at that scale.

We’ve also seen many 2-dimensional visualizations of the loss landscape, and I honestly have no idea whether, when I look at something that looks like a well in a 2-dimensional space, that is misleading. I don’t put too much stock in my intuition for that, because I’m just like, I don’t know—does 2D mean 200 billion? It seems quite different, or plausibly quite different. What do we really know about loss landscapes today?

Yeah, I think—to go back to algebraic geometry briefly—in 1, 2, or 3 dimensions, you can use pictures and use intuition from pictures. But once you go to higher dimensions, your intuitions are not a reliable guide to how to think about these objects. That’s where you try to bake your intuitions in the lower dimensions into mathematical forms which continue to work. In some sense, that’s what geometry is.

I think these pictures of—I mean, if you draw a 2-dimensional surface picture of a loss landscape, the way you do that is by choosing 2 directions in a very high-dimensional space and then plotting the loss as a function of the coordinate on that plane. One of the things you’re, with probability 1, going to do if you choose a random slice like that is not see, for example, the degeneracy—that is, the thing that determines generalization according to singular learning theory. So, in some sense, those pictures are maximally misleading when it comes to generalization.

I’m not saying they’re not useful, and they’re certainly pretty, but in many cases the actual relevant geometry looks more like a bunch of planes intersecting in some kind of crazy way. That’s what the level sets look like. So if you think about a kind of valley in a loss landscape, the level sets—where you just look at the set of points a given height above the floor, a set of points with a given loss—will have some sort of nice, smooth shape. If you go to the very bottom of the valley, you’ll eventually just get a single point.

This is not like the level sets of the loss landscapes. There are actually 2 things you could mean by “loss landscape”; I’ll have to disambiguate in a second. The level sets of those landscapes are not smooth shapes like that. They’re very complex geometric forms with lots of intersecting lines, lots of high-dimensional things that are kind of knotted up in various ways. And those are the singularities that we’re interested in.

To be clear, the loss landscape here is the population loss. So this is a theoretical object. This is an important, subtle, and difficult point: you never actually have access to this object, the population loss. That would be if you could average over every sample from the true generating process—say that was images. You somehow could average the loss over every possible draw from that data distribution. Of course, you never see that.

If you’re plotting an empirical loss landscape, you have some samples, and then you’re computing the loss based on that, and you plot that. So the actual geometry which dictates generalization mathematically is the geometry of the population loss, and that influences the behavior of all these empirical losses that are based on samples, but you never actually have access to that.

I’d like to offer just a slightly higher-level description. Usually, when you ask people to imagine what the loss landscape looks like at the bottom, or you look at these slices, it’ll look like a basin—a roundish basin, like a parabola. And what Dan calls degeneracy is the property that this is the wrong way to think about it. There are valleys and canyons, and directions you can walk along that don’t change the loss. In a sense, SLT tells us that these are the most important directions by far.

It’s, in a sense, the number of valleys that you have that determines how well your model generalizes. If there are more valleys, then you can perturb your weights; you can change your implementation without actually changing the function that you’ve implemented. And that means that your function is actually simpler. Simple functions can be implemented in more different ways. And then, through an Occam’s razor kind of argument, it’s the simplest function which generalizes best. We can actually make the statement very precise using SLT.

I definitely want to unpack this a little bit. First of all, it’s just a good reality check on all these visualizations that what you’re actually doing when you’re training a model is taking the gradient and taking a step in that gradient direction. You have no actual landscape around you. It’s not like you’re being guided broadly by this high-resolution object that you’re sort of finding your way down. You’re literally at 1 point in time taking a step, and hopefully that was a good step, but it could even have been a bad step, right? I mean, there are not a lot of guarantees there.

Then, around the degeneracies, I have a lot of questions. Probably some of them are fairly ignorant. I guess 1 really basic one is just: how do I know that this even exists? You mentioned the challenge that we don’t have the actual full—let’s say, I don’t know if “platonic” is too loaded of a term—idealized, full-data-set landscape. Since we don’t have that, I’m not exactly sure how we establish that there is this direction that we can move.

It’s not like we can say, “Hey, look, here’s 5 inputs. I change this parameter; nothing changes in the output. We’re done.” How do we know that this actually exists? Or am I missing some caveat or constraint on how this is formulated? So, yeah, question. Please deconfuse me, if I’m capable of being deconfused.

Yeah, I mean, there are a few ways into that. One is to study very simple neural networks where you can actually figure out the population loss in closed form, and that’s where Watanabe started. This was decades ago. He was studying tanh networks rather than modern neural networks, but you can take, for instance, 1-hidden-layer tanh networks and, for a very synthetic data distribution that you can understand, write down the integrals, figure out what the population loss is, and then actually study these degeneracies and see that they’re real.

They don’t have any interesting relation to the data because the data is not interesting. For interesting data, you can’t actually do that calculation, but in these very synthetic settings, you can show what the degeneracies are and do various calculations. You can study the theorems that connect generalization to geometry explicitly in examples, and there are numerous examples, including full neural networks, albeit simple ones. You can do things that are a bit more related to current practice.

One of the papers we wrote looked at the toy model of superposition that Anthropic studied. There’s a small autoencoder, and you can actually do the integrals there as well. You can write down and get the actual population loss, study its geometry, and compute these degeneracies. You can actually see that the training process is governed by these degeneracies in the way that the theory says.

That’s again not a super-interesting data distribution, and the degeneracies are quite complicated ultimately. But that’s a case where you can actually find out what the degeneracies are. There’s a boring kind of degeneracy, which maybe it’s worth getting out of the way, that’s just scaling symmetries and so on. If you have a ReLU, you can scale up the input and down the output by any real number, right? A positive real number. That’s a way of changing the weights that doesn’t change the function that’s computed.

You have a lot of degeneracies like that. Anytime you have 2 matrices being multiplied inside a network—for example, in attention, you have QK, so the query and key matrices are multiplied, and various other matrices are multiplied—whenever you have structure like that in your model, you can take those matrices that are being multiplied and, for example, scale the first one up and the second one down by the same factor. Or you can put any invertible matrix in between: multiply the first matrix on the left by some matrix and the other matrix on the right by its inverse. You haven’t changed the outputs of the network, but you have changed the weights.

Those are not uninteresting. For example, with the QK matrix, this degeneracy will have an impact on the kinds of functions that that part of the network will tend to learn. It’s nontrivial, even though on its face it’s a bit boring.

You can do this in toy cases. You can see some examples, like these products of matrices, where it has to arise. But at some point, the bigger you make the model and the more interesting you make the data distribution, it stops being feasible to do theoretical calculations and becomes an empirical question. So maybe I’ll come back to that in a moment.

There’s one more kind of theoretical thing you can do, which is invent toy models of what you think are the key ingredients in a more interesting model. There are various papers that have looked at, for example, transformers doing in-context learning. You abstract out some of the annoying details in real transformers and come up with a closed-form formula—some function of the parameters—which you think models well what transformers are trying to do.

There’s Saxe’s lab. Andrew Saxe has made a career out of doing this very beautifully in many different settings, not specifically for transformers, but across many different settings. There’s a recent paper that looks at in-context learning for transformers and comes up with a toy model, an explicit potential, some function, some loss function that describes it. You can see the degeneracy in that function, for instance. So that’s another way of trying to get at it.

But if we’re talking about algorithms in models and trying to back those out of weights, so far I don’t think there’s a very clear toy model of that in that form. It’ll be interesting. I don’t think it exists yet, and so it becomes an empirical question: Can you go out and extract signals of that degeneracy from information that you get from these empirical losses? The theory says you can do that, and that’s a large part of what we do.

To come back for a moment to the link between degeneracy and generalization, one of the core theorems of SLT says that you can link a sensitivity analysis in weights—this is the question, if I perturb the weights a little bit, how much does the loss increase?—to a sensitivity analysis in data. If I change the data distribution, how much is my loss going to go up?

It’s that link that gives us, first of all, a link between structure inside the model, because structure is reflected inside the geometry of the loss landscape, and generalization and potentially out-of-distribution generalization. Now you can look at what happens if I apply a small local change to my data distribution. I upweight some distribution—for example, GitHub code data versus WikiText. Which parts of the model become active? Which directions in the loss landscape respond most to this? It’s through this connection between structure, geometry, and data that we’re developing these tools for interpretability.

Okay, I need to dig in on that a little bit more. I have one perhaps ignorant question around the nature of these degeneracies. I certainly get what you’re saying in terms of some of them being, in a sense, trivial. You can scale one thing up and another thing down, and you get the same thing, so that’s obvious.

As you get to these large-scale things, it feels almost accidental to me. It feels super contingent. But then you had made the statement that they govern the training process, and I realized I don’t really know what that means. What does it mean for them to be governing the training process?

If they are, in fact, governing the training process, then that suggests that they’re much less accidental and contingent. Or maybe there are different kinds of them, some of which are deeply built into the nature of the world and sort of reflected from that, and others that are accidental and contingent on the way you order your batches or the exact mix of this versus that. So, again, consider that your prompt and tell me what I need to know.

There’s a bit of mathematical background that I’ll maybe provide. Stochastic gradient descent is a specific kind of optimization process. But more generally, we’re interested in many different areas in just following a gradient, right? You have some function, and then you’re going to follow a gradient to maximize it or minimize it.

In cases where that gradient is the gradient of something that looks like a sum of squares—x^2 + y^2 + z—you’re just trying to, for example, find the bottom of a bowl. That process isn’t interesting, right? There’s just the bottom of the bowl, and you’re going to go there. That’s kind of convex optimization. You can still do interesting mathematics with it, but it’s fundamentally not a complex process.

Many processes in nature do not look like that, and that’s why there’s a field called nonlinear dynamics. More generally, if you’re trying to follow a potential, imagine a surface covered in little arrows that tell you which direction you’re going to go. That’s the gradient of some fixed potential. Potential is just a general term for a function whose gradient you’re going to follow.

So you have this kind of vector field, with all these arrows pointing in various directions. It’s a fact about dynamical processes that the places where the gradient vanishes organize the trajectories.

In 2 dimensions, maybe that seems a little counterintuitive, but you could think about it like this. Instead, I’ll go back to 1 dimension. Let’s just take a curve, and now the places where the gradient vanishes are the maxima and the minima, right? Or the saddle points.

I taught calculus for many years, and a large part of the course is teaching people to classify the minima and maxima because the other points in the curve are kind of boring, right? You want to understand some function, like a cost function. You want to know where it’s maximized or where it’s minimized. What it’s doing at some random point on the curve is completely irrelevant. You just want to know where the maximum and minima are.

That’s an instantiation of the principle that if you’re trying to understand a function, you’re often looking for the places where interesting stuff happens. Those are maxima and minima in 1 dimension, and in higher dimensions they’re singularities. They’re places where the gradient is 0 in all directions.

Now, you can never get there, because no trajectory that starts from outside the singularity ever arrives, and if it’s at the singularity, it never leaves. But still, if you follow a random trajectory following a potential and there’s some singularity somewhere, it’s going to do something around it, and this will dominate the behavior.

The trajectories will tend to approach it and then escape in some particular direction. So the singularity is organizing the set of global trajectories. That’s an informal statement, but there are formal statements. You can use ideas from topology and find ways of making that more precise.

That’s a general principle, and it applies to learning with SGD just as much as it applies to any of these other systems in physics to which people would apply this principle. That’s a general rationale for caring about critical points or singularities. These are almost synonymous. That’s a reason for caring about those as the organizing principle for dynamical systems and learning in particular.

What does it look like for an actual learning process in deep learning to be governed by singularities?

Yeah. In really interesting cases where there’s a model doing interesting stuff, it’s a very complex question. But in simple cases, you can see that this toy model of superposition I was describing earlier—the columns of the weight matrix—can be visualized as vectors in 2 dimensions. There are 2 rows of this matrix and some number of columns, and you can just plot those columns as vectors.

Then you get shapes like a pentagon, a square, or a hexagon. You connect the vertices of those vectors, and moving between the neighborhoods of these critical points literally corresponds to growing a leg, contracting a leg, and rearranging the other ones in some particular way. So there’s, if you like, a growth process of the structure of the network that’s dictated by the movement between the neighborhoods of these 2 different singularities.

That’s in a very simple case. In a more complex case, I think it’s not so clear how to think about that. These singularities aren’t isolated things where there’s 1 singularity here and normal stuff in between. In a large model, there are singularities everywhere. It’s not well understood.

I’m almost envisioning a sort of fluid-dynamics-type thing where the point that we are in parameter space kind of speeds up when it gets around the drain, but then maybe spins out and goes slower as it has some wider orbit. Probably a lot is lost in that, but that’s at least the visual that’s coming to mind.

And, yeah, there’s this—you can visualize the function in function space of a small language model, and there are some parts of the trajectory that actually look like it’s orbiting around some particular kind of mode of prediction. That kind of image is appropriate.

So, do these singularities—I mean, in a simple cost function, as you’re describing, whether you maximize it or minimize it, whatever, it’s very clear what that point means and why we’re trying to identify it in the first place. In the case, especially, of a more complicated neural network, I guess I’m tempted to ask: if I imagine a grokking-type phenomenon, are those singularities the place where the algorithm is perfectly grokked, such that there’s never anything more to learn, at least with respect to some subset of problems? Or is that reading too much into the nature of the thing?

Think of the singularities as different ways of solving the problem—different kinds of solutions at a given level of loss. If you’re at a given point in the loss landscape, different kinds of singularities correspond to different ways of predicting on the data you’ve seen so far.

Jesse, maybe you want to say a bit about the ED stuff.

Yeah, I think it’s an interesting one. Maybe I’ll say one more thing about this. Coming back to generalization, one thing we’re worried about is that your model might learn 2 algorithms that look identical from the training data, but one of them generalizes in a way that you don’t like and one of them generalizes in a way that you do like. So how do you distinguish them?

What Daniel’s just saying is that maybe if you look at the neighborhood in the loss landscape, they actually have different geometries associated with them. This is the sense in which reading this geometry gives you information about what kind of algorithm you actually end up with.

What Daniel was referring to as ED stands for essential dynamics. This is a paper we put out earlier this year where you can train a neural network and a transformer to do in-context linear regression. You can give it X, Y, X, Y, X, Y samples and ask it to predict the Ys from the Xs, where each Y is generated from a simple linear transformation of the Xs plus some noise.

What happens is, if you train it on lots and lots of different samples, the model will learn to do regression over the course of the context. However, you can vary the number of distinct tasks that the model is exposed to during training. If you expose it to many different tasks, it learns regression. If you only expose it to a few different tasks, it’ll memorize those tasks and it won’t learn the regression solution. It’ll learn the memorization solution.

Which solution the model ends up with varies as you vary the complexity or diversity of the number of tasks, and you see a phase transition where the model initially memorizes and then eventually learns this other solution. But the selection process between these is something you can actually try to understand with singular learning theory.

Each solution has an associated performance and complexity. Memorization is always better in performance than this regression solution, which I call the generalization or meta-learning solution. However, the generalization solution is simpler past a certain level of things that you have to memorize. So what happens is the simplest solution roughly occupies more volume in parameter space and is therefore easier to find.

You see this phenomenon where the model will first learn the generalizing solution, the simple one, before moving to the memorization solution. You can classify exactly which kinds of dynamics qualitatively you should expect to see based on this trade-off between performance and complexity, where complexity is reflected in the degeneracy of the loss landscape. So it’s kind of the opposite of grokking.

Yeah, I was noticing that for sure. I was caught up on that. Maybe let’s take another version of that, because can you help me understand how it’s happening in reverse in this case versus the grokking case?

Well, grokking is actually a strange phenomenon. It’s not like a typical—I think there’s been, in a sense, an overfitting to the example of grokking. This, in our opinion, is kind of not the typical way in which memorization and generalization probably interact.

Okay, say more. What’s special about grokking?

What’s special about grokking is that the performance on the training set is more or less identical, because you have 1 solution that is more complex. The memorization solution is more complex, but it’s the thing you learn initially. So you end up with this transition where, at a fixed level of loss in the loss landscape, you end up in a broader basin and in a basin that has more degeneracy.

This is a phenomenon that can only happen—I mean, this is sort of if you get, conceptually, to the very end of training. At the very bottom of the loss landscape, there are some perfect solutions that are simpler and some that are more complex, and you prefer the simpler one if you can find it. Eventually, you will find it.

But the trade-off between memorization and generalization that occurs more frequently, without such artificial conditions, is more like you prefer the simple solution that’s bad, that has high loss. You eventually trade that off for a solution that has lower loss and more complexity. You pay a complexity penalty in order to get the lower loss. That’s the more typical relation between memorization and generalization.

Yeah. And that’s the case in this in-context linear regression example. So what SLT says is that you should expect models to minimize a sort of effective loss: it’s the loss with this emergent regularization term, this implicit bias that comes from degeneracy, that comes from the number of ways you can perturb weights without actually changing the function. Typically, the model’s just going to prefer initially to learn the simple bad thing before moving on to the complex, better thing.

And so help me understand one more time how the grokking case is different, because in that case it takes a lot longer to learn what I would think of as the simpler algorithm, which presumably does have, in terms of some formal complexity metric, a lower score for complexity than memorizing n examples, right?

Yeah. So, as Jesse referred to earlier, there are 2 typical changes in the trade-off between loss and complexity that are predicted by the core mathematical result of SLT. By trade-off, I mean the trade-off between the loss and the complexity as estimated, and we call that the LLC, the local learning coefficient. I’ll just say complexity.

One way you can have a preferred trade-off is to pay the penalty of increasing the complexity by lowering the loss. That will decrease what’s called the free energy, which determines which solution is more preferred from a Bayesian statistics point of view. So that’s what we sometimes refer to as a Type A transition. That’s the typical one. That’s just learning, right? You learn more stuff, you get better.

But there’s another way to decrease the free energy, which is to decrease the complexity at a fixed loss. This is what we refer to as a Type B transition. It’s hard to ascertain exactly the conditions under which this is the case, but grokking seems to be one of them. We’ve seen others. So those are situations where the model really isn’t improving on the loss, but it simplifies the algorithm that it’s using, and we can see this in the linear regression setting in some sense as well.

And, of course, there are many other examples of grokking-like behavior. But yeah, the underlying principle is that you should decrease the free energy, which is a sum of 2 terms: 1 to do with the loss and 1 to do with the learning coefficient. You're allowed to increase 1 of those terms if you decrease the other 1 enough; that will decrease the free energy. That's normal learning. And then there are situations where you can have a similar decrease just by decreasing 1 of the terms—the complexity.

So 1 thing that's jumping out at me is that, in general—obviously, people can do all sorts of variations on this—but the vanilla setup is that you're just minimizing the loss. You would use the term—I think it was implicit regularization. If I'm sitting looking at my single loss function and trying to drive it as low as I can, I understand what weight decay is, and that's 1 form of this sort of regularization, right? But if I understand you correctly, you're saying that there is some sort of natural pull happening, even if, from an external perspective, I'm just minimizing loss, that is moving me around in this complexity space and prefers simpler solutions. But I don't have an intuition as of now for why or how that's happening if I'm not taking an active step in the learning process to enforce that.

So you can imagine asking the question: What happens if you randomly drop yourself somewhere in the loss landscape? More likely than not, you'll end up somewhere with pretty high loss. You'll also end up somewhere that has a very broad basin, or there are many different ways you can perturb that solution that'll leave the loss basically the same. And so, generally, as you move down the loss landscape, it's going to be easier to find the solutions that occupy greater volume, and the main thing that contributes to the volume is the degeneracy, so the number of valleys. It's just easier to find canyons that have more valleys than it is to find very narrow nooks and crannies. And this is the sense in which you get this emergent implicit regularization from the structure of the loss landscape.

I could do this all day, just sitting here trying to visualize these things and asking you to help deconfuse me. And I think we have a lot of those listeners too, though I probably need to be somewhat disciplined about how many more of those kinds of questions I ask.

This sort of seems to connect very fundamentally to double descent, right? I mean, maybe just riff on double descent for a second, and then I definitely want to take a moment and also really zoom out. It's all fun and games in modular arithmetic and toy problems to motivate the entire thing, and I'll probably put this in the intro. The canonical safety worry is that you can't necessarily tell the difference between a superintelligence that has your best interest at heart and 1 that doesn't. And you may have a real hard time in many ways if you can't make that distinction as you get to more and more powerful systems.

So there's the double-descent thing, but then let's do the zoom-out and talk about the biggest-picture worries that we have. And then I also think maybe we should start to go toward what the path is for you guys from here with this research agenda. We've seen from Anthropic, I would say, and others, of course, from whatever toy models of superposition 2.5 years ago, really an unbelievable amount of progress, in my view. Obviously, there's a long way still to go, but if you told me then that we'd be here now, I would be like, “That's really amazing.” And it has struck me that 1 of the big things that has gone into that has just been a ton of compute, and I don't have an intuition yet for whether that's a similar trajectory that you guys think you will follow or if it's going to be a more eureka-moment-driven process, to recall the Greeks again.

So, yeah, a lot there, but double descent, if you want to say anything about that, what are the biggest—if we zoom out to not toys but the real things we're really worried about—and how do we go from these toy-model understandings to starting to tackle that, hopefully on relevant timelines?

Yeah. So regarding double descent, there are various kinds of double descent. There's double descent with respect to training samples, and there's double descent with respect to model size. I think nobody has systematically attempted to study double descent with singular learning theory yet. We haven't really tried. I think partly because it just doesn't look that mysterious when you approach the problem from the mathematical perspective that we do. It isn't mysterious that models with many parameters can generalize well.

And in terms of an intuition, is this another 1 of those dimensionality things where, when I see a hill, I'm like, “Boy, it looks like a lot to climb over that hill,” but when I'm in 200-billion-dimensional space, there's always a way to wind my way through it? I mean, that's an often-presented intuition for why optimization in high dimensions is amenable.

I'm not so sure I think that's really an explanation. I think that's closer to a folk story, maybe. I think it's not understood why this happens. Just having a large model with lots of parameters—you can, of course, find things that don't optimize well, that have lots of parameters, that aren't neural networks, right? Or you can get the neural networks configured badly, and they won't train very well. So there remain many mysteries about why large neural networks can actually find well-performing, well-generalizing solutions.

Singular learning theory doesn't resolve all those mysteries, right? But it does at least give you a mathematical framework from which you're not surprised that there can be models with lots of parameters which do well and predict well and generalize well. So I don't really have a clear answer for you on double descent. Maybe 1 of your listeners wants to think more about it. We haven't put a lot of effort into it.

There is a story: you can think about double descent with respect to training samples and see that you can back out a picture that looks somewhat like the double-descent curve from generalization curves that are associated with 1 of these kinds of transitions I was describing earlier. So potentially there's an explanation there. I don't know. I'm just worried that at some points we've gotten too deep into the technical details. So if I can take a chance to reclarify or restate some things I've said earlier.

Yeah, sure. Try to add more.

Okay. So, big picture, we want to end up with a friendly AI. We don't understand enough about what's going on inside the model to currently make any kinds of guarantees: this model is aligned, this model is not aligned. They can exhibit the same behavior in training and then generalize differently when deployed. And so, for me, the basic premise of interpretability is understanding model internals to a sufficient degree that we can disambiguate these 2 cases and that we can understand generalization.

SLT tells us that we can look at the geometry of the loss landscape and look at how structure is reflected in this geometry to start getting at this question. We have this principled link to in-distribution generalization. That's a starting point for developing a theory of out-of-distribution generalization and understanding better what happens in the case of SGD.

On the interpretability side, what this looks like in practice is that you can probe a bunch of points in the loss landscape. You apply small perturbations to your model—hit it with a hammer—and see how it responds to that probe, to that small perturbation. And this tells you something about what's going on internally. This is the basis for starting to extract information about model internals from geometry.

So we've been looking at a series of projects and papers where we study, first, small language models, subject them to these kinds of perturbations, and then measure things like: How much does the performance decrease? How much worse does the performance get under these kinds of perturbations? That tells you about the complexity of the solution the model has learned.

How much does performance increase if I apply these structured perturbations to a specific part of the model—an attention head or a component? And this, it turns out, can tell you something: This attention head is specialized to this kind of data. These attention heads are all induction heads. These ones are doing something else, like memorizing different n-grams and skip n-grams. And so you can start to actually tear apart the internals of the model using these kinds of probes.

More recently, we've been looking at applications for something like circuit discovery. So, in a principled way, can you associate the components of the model with patterns in data and do this attribution from what structures the model ends up with to the data that activates that structure? And this is starting to get the kind of understanding that could lead to an understanding of why the model exhibits the behavior it does, rooted in its internal structures, without necessarily having a full mechanistic understanding of what's going on.

You can do the sensitivity analysis, and it can tell you that these components are involved without needing to know exactly, causally, what the input-to-output map was. So this is more like a top-down approach to interpretability informed through this perturbation analysis.

On the one side, we want to develop tools for interpretability. On the other side, at some point, we want to be able to steer the learning process so that we don’t end up with a misaligned model in the first place. We have to take a preventative approach.

Ultimately, that’s going to look like intervening in the training data. The hope is that, to the degree that the mapping from training data, through geometry for the loss landscape, through the learning process, to the final weights you end up with is invertible to some extent, you can go backwards and come up with techniques for choosing training data that aligns your model more robustly. These are the kinds of applications of SLT that we’re trying to develop for alignment.

We’re only just getting started here, but our real hope for this agenda in the long term is to give us better ways to make models more aligned.

So, are those sorts of bell-ringing exercises? I’m imagining that, as you try to scale this up to larger and larger systems, you would maybe be running experiments like: If we add some noise, how often do we see behavior that we didn’t want? This also seems to have some connections to AI control-type ideas. How often does the thing take the bait? If we set up a situation where there’s some bad thing that it could do, how often does that happen with different perturbations? The less it happens, maybe the more aligned we are. Is that the sort of measurement that you would imagine taking?

I don’t think I’m going to bite on that particular proposal. But if you think that you can elicit something about the structures in the model which, at the end of training, you understand to produce the concerning behavior, and trace the origin of those structures and behaviors—to go back to the developmental perspective again—to some kind of pattern in the data, then you can try to intervene on that using your understanding of that pattern, intervening at some earlier point to shape that structure differently.

Obviously, that’s a complex behavior and a very complex internal structure, and it’s an ambitious thing to attempt. That’s one of the reasons why, from our point of view, some progress on interpretability is essential to unpacking that. If you imagine some complex mechanism that is producing the undesired behavior, I think it’s naive to pretend you’re just going to find the “be good” samples out there, throw those into the training process at 12,000 steps, and be like, “Be good, be good.” I don’t think that’s sufficiently fine-grained.

But if you have some understanding of how that complex behavior came together—how this mechanism was shaped over development and what factors went into that shaping—then it’s not crazy to imagine that there’s something analogous to morphogens. Morphogens are a sort of umbrella term that refers to factors in the development of biological organisms that shape when things happen or what things happen. They could be particular molecules or proteins, and these dictate the way that development works in biology.

If you insert the right morphogens at the right time, you can change the outcome of development. This is perhaps one guiding analogy for why it makes sense to hypothesize that, if you understand the developmental process well enough, you might be able to intervene at various points strategically to change the outcome. Indeed, that’s how synthetic biology works. You understand the development process, you understand the morphogens—at least in some parts of synthetic biology—and then you can program the development in a way you prefer.

One question on just the nature of overgeneralization: Is reward hacking an example of overgeneralization, where what we wanted to specify was some richer concept than the signal we gave, and so it has gone too far in understanding the true nature of that signal beyond what we meant it to be? Is that why we see reward hacking?

It could be. I think there’s a bucket of things that are involved here. Reward hacking has a technical definition under which the answer to your question is no. The technical way that people would introduce reward hacking is simply to say that it has found a way of getting more reward than the solution you intended. That’s purely about the reward, which is somewhat analogous to the loss. That’s not really about complexity; that’s just about getting more of the actual explicit target of the optimization process.

Under that definition, the answer to your question is no. Reward hacking is not an instance of this trade-off we were referring to earlier, which you’re terming overgeneralization. However, it could very well be that instances we are currently referring to as reward hacking are perhaps partly a trade-off in favor of a simpler solution rather than the one we intended. I don’t claim to have evidence for that, but I would hypothesize that.

I think you’re referring to one of the hypotheses we have for one of the ways in which dangerous behavior could arise, which is somewhat different from the usual story of how dangerous behavior could arise from overoptimizing a reward target. To go back to the experiments that Jesse was referring to earlier very briefly, there was a simpler solution, which was ridge regression.

In theory, if you train long enough, you should always switch to memorizing the data because the optimization pressure wants you to get the lowest loss possible. The lowest loss possible is to memorize the data, if you can. Typically, when training your GPT model, you just can’t memorize the data, so you have to generalize. If you can memorize, that’s the best thing to do given the training signal.

But what we see in that experiment is that, in many cases, the model sticks with the generalizing solution and never switches to the memorizing one. So, if you imagine the loss landscape—and I’ll draw a curve with my hand, despite my complaints about that earlier—at the very bottom of the loss landscape there’s the thing that the training data actually specifies as the optimal solution. In this case, you can prove mathematically that it is to memorize the data.

There is a higher-loss but simpler solution, which doesn’t memorize but does something more general, and training can get stuck there forever, even though this is the global minimum. I say “forever” obviously—we didn’t train forever—but it seems like it would stay there forever. This is a good heuristic to have against some of the normal stories about AI risk: It may be that you don’t get what you ask for. You don’t get what you try to specify in the training data; you get a simplification, a heuristic, or an approximation that is not what you intended or expected. That could be good, or it could be bad.

Mostly, in the context of AI safety, people have an intuition that the simpler thing will be better. Just to communicate an often-told way of thinking about this, imagine that the model becomes scheming—a kind of scheming agent—even though you didn’t optimize for this. It’s trying to do the task, so it has structure in its weights to do the task, and then it needs to tack on some additional structure to take over the world.

Because that would be more complex than the solution that just does the task, and we have a simplicity bias, that’s great: We have this very strong pressure to remove the additional stuff that is going to be dangerous. That may well be a reasonable way of thinking about it. I think one should have very little confidence in this. It’s quite an imprecise story, not really grounded in much mathematically.

But to tell the opposite story, it could be that the simplification is quite dangerous. If you’ll indulge me, there’s a historical example here that’s really quite interesting. It’s something called the Windrush scandal.

The UK, in the 1940s, allowed citizens of the British Empire to come and settle in the UK without really any documentation. Many people came to the UK, primarily from the Caribbean, I think. Then, in the 1970s, they switched the laws. They started insisting that people have documentation of their citizenship in order to access services.

The people who came as part of this program didn’t have the relevant documentation. Some of them were deported, and many bad things happened. There was an investigation and scandal in 2018, I think, looking into this.

This is a case where you have a policy. The intended policy of the state obviously isn’t to deport people whom it let into the country earlier. But when you follow that policy as it’s actually implemented at the street level, it’s a simplification of the original intent. It’s a very brute judgment call at the level of the street-level bureaucrat.

That’s what this phenomenon is referred to as: street-level bureaucracy. It’s a simplification of the thing that the original policy was trying to specify, and this ends up having unintended negative consequences. I think there’s a case to be made that, in AI safety, we should also be quite concerned about what happens if the behavior we are specifying at great expense by collecting training data ends up being some kind of simplification, in ways we don’t anticipate, of the intended target behavior—in ways that are undesirable.

Yeah. Seeing Like a State and seeing like a model may be more similar than different, certainly relative to what I had conceived of previously.

I think I maybe have 2 more questions. One is, I've heard Jesse tell the story in the past, in other fora, around how maybe model training in the future could look more like refining oil does today, where you start with a big mess, but you can figure out exactly what components need to be added when you have a sense for when these phase transitions are happening, and you have a much more precise level of control over that process. I'd love to hear a little bit better version of that story than the one I just offered.

And then my other one is just, what do you guys make of other safety efforts? I mean, you've said you probably can't just get the good examples and hope for the best, or at least that doesn't seem adequate. But I'd be interested in your handicapping of all the various—or at least some of the most prominent—techniques that are out there today. What do you think has the best hope of working, or what do you think is doomed?

So, your vision for the future of model training and the level of control we might really be able to acquire over that process. Maybe you could talk about the compute—how you scale the compute—because I don't think we ever touched on that. And then this handicapping of the landscape. There's a lot there, but I'll let you have at it.

Yeah, I could start with the oil-refining example. I think it's useful to compare current deep learning to alchemy. We've got some huge cauldron that's the architecture, we've got a fire that's the optimizer, and then we've got the reagents we put in this cauldron that we start to mix together. Those reagents are the training data. Currently, we just throw the entire internet together into this cauldron, start stirring, and hope that we haven't accidentally mixed bleach and clorox.

What we have in mind for the future of training looks something more like industrial chemical manufacturing, where you know exactly which reagents you're mixing together, at what concentration, at what point in the process, and with what catalysts and ingredients. This level of control isn't accessible to us right now, but it's something that we're already empirically starting to develop.

We've split the post-training process into a bunch of different stages. There's chain-of-thought RL, there's a second stage of chain-of-thought RL, there's a little bit of instruction fine-tuning somewhere, there's Constitutional AI, there's a second stage of Constitutional AI just for the personality, and then maybe some more refusal training. Similarly, on the pretraining side, this process is being split into multiple stages so that we're developing an empirical understanding of how to better control the learning process.

What we think is possible is that you can develop the kind of scientific understanding that lets you actually know when you should mix 2 samples in a batch together to get the desired behavior out the other end. This will give you more control over the entire process and what you end up with.

Yeah. Just to interject, one example of that struck me recently with the Claude 4 system card. There was this anecdote—I'm sure you've seen it. It hasn't been very deeply explained, although Sam Bowman talked about it a little bit online. They reported having observed that the model was following harmful system prompts in a way that surprised them.

And then they were trying to figure out why. Sure enough, it turns out that they had omitted the CIS prompt-harmful dataset, which had been developed specifically to teach the model what to do when given a harmful system prompt. So you have the simpler solution of always following the system prompt. That's what it ends up learning for lack of that cue being included as it was meant to be.

Interestingly, from my perspective, they did not go back and retrain. I think there's a lot I'm still unpacking—what exactly should I be inferring about the world on multiple different dimensions from the fact that they didn't just say, “Oh, we messed that up,” revert, and pick up where that thing was supposed to be introduced, and continue on?

Presumably, that means it was happening somewhat earlier in the process than some very late finishing stage. Presumably, it also means they have constraints at Anthropic in terms of either compute budget or timeline to launch, for competitiveness reasons or whatever. There's a whole interesting analysis to be done there, too.

But that's at least one of these very recent and concrete examples where, without necessarily a full theory to drive it, this kind of understanding is starting to emerge. You better make sure you get those datasets in the right place at the right time. And now, honestly, it seems like we have sort of a clue.

I had a little private conversation with somebody at Anthropic just to ask for clarification on this. They basically said, “We're confident we got the value of that dataset in the end.” I was also kind of—I'm like, “Okay, that's interesting. I sort of believe you, and you definitely know more than me. At the same time, how can you be confident? On what basis could you have any confidence about that, other than obviously poking at the model a lot?”

I mean, that's the sort of confidence I guess that you hope to create in the process of continuing to do this work.

Yeah. I mean, I think it isn't an unreasonable expectation on behalf of all of our civilizations, which are about to be transformed by this technology, to have it be somewhat more of an engineering science and somewhat less of a black box. Not that it's—I mean, it's not like nobody understands anything, right? The data mixtures that are selected and the order in which they are selected are empirically very finely tuned. Lots of effort goes into that. Selecting hyperparameters has some underlying theory to it.

Lots of people and lots of money are being put into deep learning, obviously. So it's not completely incomprehensible, but I think most people would agree that it would be highly desirable to have it be much more understood and much more like other aspects of engineering, which we subject to safety engineering, than deep learning is currently.

That seems hard to disagree with, I would say.

That's not an easy thing to do, especially in a short period of time. To come back to your question about scaling and the situation we're in vis-à-vis where Anthropic was with SAEs a few years ago and so on, I think this analogy seems pretty apt to us.

I think it's been quite a difficult path to get from fairly pure abstract theory to validation in real systems, and now we're just starting to do things like circuit discovery. We can find induction circuits and some simple things like that in a kind of unsupervised way. It does seem like there's scaling to do, and that's one of the main things that now remains, at least to get up to a point where we're finding complex internal structure.

In terms of compute, that's a little difficult to say. The compute overhead of computing SAEs, for instance, where you train the SAEs, is very expensive. Our compute burdens are going to be in different places, but I imagine it's still very substantial to try and understand large models.

We have been studying models of up to 7 billion parameters and so on. We can definitely use more compute. If a listener has compute to offer, please.

Regarding other approaches to alignment, I think in many respects the things we're describing are just useful foundations for many different things you might do. So I guess that's one way we think about it.

For instance, the stuff we're doing around shaping the data distribution and so on is potentially useful for some of the work the UK AI Security Institute wants to do around elicitation. We have various ideas about how the work we're doing can—I mean, some of it is just good, solid foundational science of understanding models, understanding the training process, and understanding how model behavior works. In the same way that interpretability is broadly useful, much of that is broadly useful.

I guess, yeah, it seems unclear whether the current paradigm of shaping model behavior at the end of training is going to be a very robust solution to alignment as we move toward higher capability levels. This is an oft-expressed concern, right? So I think that something that looks like baking in behavior and control of behavior earlier in the training process seems like something that is broadly going to be adopted.

Arguably, the deliberative alignment approach, especially if you scale up the amount of compute spent on RL—if that is equal to pretraining—then that's a lot of shaping of model behavior taking place over a large part of the effective compute. This idea is not just due to us. There are papers like “Pretraining from Human Feedback.” A lot of people realize that we need to bring alignment earlier in the learning process.

Maybe I can end with a bit of a look at the next few months. Currently, a lot of our effort looks like scaling techniques up to larger models and starting to run lots of experiments at larger models.

So we've validated these circuit discovery techniques in small 3-million-parameter language models. Now, can we do this in 7-billion-parameter language models? We think this is quite likely to work by the end of the year.

On the alignment front, we're starting to do our first experiments applying these techniques to try to steer the learning process. This is still early days, and we'll probably have early signs of life by the end of the year in small language models. Then we're developing these applications for things like elicitation and data attributions.

There's a singular learning theory extension to influence functions that you can study. It gives us ideas about, again, the influence of samples on other behavior. We're trying to use these kinds of applications to really validate that SLT and the techniques we're developing have something to offer that goes beyond existing techniques. That's where we're heading.

Cool. I love it. This has been fascinating. I appreciate you indulging me in so many little side questions and attempts to develop my own intuition.

I certainly think that the big-picture need for a proper developmental understanding of how these AIs are forming, and any theoretical basis at all for what they're going to do when they get into a truly out-of-distribution situation, is badly needed. And especially as we get closer to 2027 and beyond, that is going to really help me sleep well at night. So, to quote my dad, who often quotes Leslie Nielsen from Airplane: “Good luck. We're all counting on you.” Anything—anything we didn't touch on that you want to leave people with real quick?

It's a brain-stretching exercise, at least for me and probably for most of us. But I've definitely enjoyed stretching my brain in this way. I look forward to future progress. We can do a check-in maybe in a year's time or whatever and see how much we've closed the gap on all these important questions.

I will say, Jesse Hoogland and Daniel Murfet, founders of Timaeus, thank you.