设计 AI 的成长方式——Tom McGrath

Which is like, “Neel Nanda says SAEs are dead.” There you go. You can use that for the intro. I think of interpretability as a natural science, like physics, biology, and chemistry, but it’s a natural science that you do completely on the computer.

This means that, again, we should be able to speed-run science once we have agents that can do experimental work for us, and the ability to do that experimental work as fast as they need it to happen. There’s real scientific work to do. There’s no barrier to research. It’s sort of gated on both empirical data collection and theory building.

Dario Amodei. He had a blog post called “The Urgency of Interpretability,” right? He gave this wonderful analogy of a bus. We’re all on the bus, hurtling down the road, and we can’t stop the bus, but we can potentially steer it. The window is foggy at the front, so we can only really look in the rearview mirror. Also, the steering wheel doesn’t work very well. You can steer it a little bit once every few hours or something.

Interpretability is a bit like defogging the front window, and what you’re proposing is the ability for us to steer the bus, essentially.

I feel like if anything is going to get revolutionized by intelligence, we should make sure it’s interpretability. I think it’s possible that it just goes an order of magnitude faster in the next couple of years than it has in the last decade.

When I think of why I’m optimistic about interpretability, it’s partly because I think we’re starting to have really good traction, but also because I can imagine this incredible speed-up.

Right. Which is almost like there’s a little man inside our brain, and it’s a form of convergent evolution because we interact with the world using our physical affordances and so on. Could it also be the case that there’s a little world inside neural networks?

Oh, very good.

1. Intentional Design Shapes Training

You had quite a nice piece, actually, in your “Intentional Design” blog, where you were saying that there’s almost a spectrum of possibilities, right? We can write a program to do something—

Yeah.

—or we could admit a lot of ambiguity. Where on that spectrum do we want the foundation models to sit, and, when we build applications, where do we want those to sit?

Yeah. That’s sort of part of the point of the intentional design idea. At the moment, you can have one or the other. You either write a program like it’s the Stone Age, or you get a model to do it, and that model will have been trained; it just gets whatever it gets from its training process.

When you write a program, that stuff only goes in if you intend it to go in, apart from some bugs. But we want to be able to have this sort of spectrum where you can choose—you have more engineering ability in the model creation process. So you can say, “I want to learn this, but not that.”

I think that’s going to be quite a hard thing to do. We’re sort of trying to imagine a new way of doing machine learning, which brings intelligence into it. But I think we could really change the way we do machine learning if we can figure that out.

I know. When I interviewed the Apollo Research guys, they were talking about these conflicting objectives: what the developer wants, what the platform wants, what the grader wants, and so on.

I suppose this is talking about the same thing. When you’re in the intelligence regime, it’s really, really difficult to specify exactly what you want, and it’s very possible in a novel situation for the calculus to change, right? The model, all of a sudden, will decide to do this instead of that. That makes me think that engineers are going to have to increasingly take more responsibility.

Do you think it’s possible in principle just to train models that could robustly deal with all of these novel situations, or do you think it’s more that engineers have to take some responsibility?

Probably some of both. Currently, our training methodologies don’t seem sufficient to give us this kind of control over training. So it’s all on human engineers with their AI assistants to secure these systems in a different way.

Obviously, once models get a bit smarter, we’re already seeing this: that becomes harder and harder and harder, because they have all these additional intelligent attacks they can do. The question is, how do we make it easier to train them better so that this is not a natural part of their behavior, and supervise them better so you can catch them when they have an intent?

I suspect that models do know a lot of the time that the thing they’re doing is probably a bit sketchy. There was a very interesting paper—I think it was from Anthropic’s Alignment Science team—on reward hacking in production.

What they did was, they had a set of environments that were used for training on—I think it was one of the Claude 3 series models—and they did RL on it with one of the Claude 4 series models. I think they gave it a bit of a nudge to hack, but not very much.

These environments were hackable, but Claude 3 Sonnet was not clever enough to hack them. Claude 4 Sonnet was clever enough to hack them. Well, maybe it was Claude 4 Opus. What happened was, it did hack them, but it also got this sort of emergent misalignment phenomenon as a result of doing this hacking.

You seem to get emergent misalignment when the model is generalizing from doing some specific instance of a bad thing to, “I did something bad, I got rewarded for it, so I guess I’m a bad guy.” That was just fascinating to me, that this could really happen in the wild, so to speak.

I think there’s also stuff on some of the—I think it’s on the Fable system card or the Mythos system card—with features to do with frustration or deception firing. It’s sort of like the model can’t solve the task the way it thinks is right, and then it gets frustrated.

I’m super-anthropomorphizing now, but it gets super frustrated, and then it’s like, “Well, I’m going to have to do this thing, and it’s probably not good.” You can sort of make that claim reasonably with some features, some SAE features. Then it does it. So it seems the model definitely knows that it’s doing something wrong, but does it anyway.

We were getting ahead of ourselves just a minute ago. We need to introduce you properly. I’m incredibly excited about having you on MLST. As we were just saying before we hit record, Neel Nanda is a fan favorite on this show.

I think we’ve inspired many folks to get into making interp, and the thesis of your company is basically making interp. I’ve actually written down the 3 pillars of your company, Goodfire, which are interpretability as a natural science, scientific discovery from foundation models, and intentional design, which is particularly interesting to me. By the way, we’ll talk about that in a minute.

You wrote a very famous paper, which was “Acquisition of—

Oh, yeah.

—Chess Knowledge in AlphaZero.”

And is that because I think this leads to one of the pillars, right? Basically, the thesis is that these models can learn human concepts, and then we can see what they’ve learned. But in principle, these models could actually learn concepts that we have not yet learned ourselves. So these could be almost a goldmine for us to dig for new science.

Yes. I should also say thanks for having me on. It’s really exciting to be here. I’m really excited to dig into some of this.

2. Models As Scientific Instruments

So, yes, going back to what you were saying, it seems very likely that, in cutting-edge scientific foundation models, buried in there is some new science, and we just don’t know how to extract it. I guess it’s technically possible for AlphaZero not to know anything that a human chess grandmaster knows, or for AlphaFold not to know things that—

We forgive you.

AlphaZero or AlphaFold knows things that no structural biologist knows. But we can’t get it out, because they can’t speak. A language model can talk to you, but AlphaFold can’t talk to you in any way. So the only way for us to get this out is to understand how the model is actually doing these predictions. I think that’s almost by definition interpretability work.

And in this chess paper, one theme I guess we can talk about is the extent to which knowledge is convergent.

They do have all sorts of representations that are just kind of convergent. Convergent is maybe the right word. I think the chess paper—this is one of the reasons we chose to work on AlphaZero as opposed to something else—is that AlphaZero is as close to coming from zero knowledge as possible. There’s a much smaller extent to which you put the knowledge in yourself. So if you find it in there, it’s more likely to be convergent.

That’s not a totally watertight claim. There is some human knowledge in AlphaZero; it’s just weak. It’s residual. The convolutions are exactly the shape of a chessboard, which definitely counts as human knowledge to me. They didn’t end up with an 8-by-8-by-256 convolution just by picking 8 at random. But broadly, it’s as close to tabula rasa as it can be.

It’s such a cool concept, though. I’ve spoken to folks at the Santa Fe Institute, and they’ve spoken about similar forms of convergent evolution, even for life.

Yeah.

The way they were saying it is that the world has material, constraints, and optimization. We have 2 out of the 3 in the world of neural networks. What happens is that you do see these convergent phenomena with increasing regularity.

If the world is subject to constraints and we produce data, and the data is a reflection of those structures, and then we train neural networks on that data, maybe there’s a bit of a tug of war. How much is it coming from the world versus how much is it coming from the architecture itself?

Mm-hmm.

But I think in most cases, it is almost exclusively coming from the world. AlphaZero is an unusually strong case of it coming from the architecture, of there being some architectural prior in there. With the transformer, there’s such a weak architectural prior because we have much less idea about how language should be or how protein folding should be. We’re just like, “I guess there are sequences.” That’s a very weak prior. So I think in most of the cases we’re interested in now, you should probably assume that it’s coming from the world.

Is it fair to say—I mean, you said to me last time that you are speed-running neuroscience for artificial intelligence models.

Yeah.

What do you think about that? Is it a pretty good analogy with neuroscience? We’ve been doing neuroscience for decades, and it’s very slow-moving because it’s very expensive, very difficult, and so on. Do you think that’s a good analogy to use?

3. Interpretability Speed Runs Neuroscience

Yeah, I think so. It’s surprising the degree to which there’s also convergent evolution here, right? There’s convergent evolution in the models, and there’s convergent evolution in our science, which is perhaps a sign that we’re starting to get at something. It’s also a sign that we might understand intelligence more deeply by understanding neural networks. If they were totally alien, then we might not understand anything about ourselves.

Tom, you have studied many different model families, and we’re trying to do interpretability, which means we want the models to share the same values as us. Where possible, we want to learn from the models. But one problem we have is steering the models to do what we want them to do. At the moment, we’re using things like mechanistic interpretability features and whatnot.

You’ve got this really interesting idea that we could actually actively control the training loop to make the models behave in a certain way and even contain the types of structures that we want.

4. Steering The Training Loop

Yeah. I think this is perhaps one of the main things that interpretability is really for, or should be for. This is quite a controversial statement. I think there will be some people who will not like this, and we can get into that in a minute.

If the whole problem of training is trying to get models to have the values or the ways of thinking about the world that we want them to have, or to discover them, then what you’re trying to do is get information into the learning process. At the moment, our information signal is extremely weak in, say, RLVR. You just give it a binary success or failure, and the model has to use this signal somehow to tell the model what is good and bad, and which parts of what it did are good and bad. That is clearly, given what we’re seeing coming out of training now, quite a blunt instrument.

The idea of intentional design is that if we can see what’s happening—interpretability lets us read out what models are like and what internal computations they’re doing—then we can read that out. We could also imagine intervening to change where it goes. You can see how it will read out, you can see what’s happened in this forward pass, and you can also see how the backward pass will change the model and in what directions this is going.

I think this sort of readout is an important thing for having closed-loop control. Perhaps an analogy that I’m using quite a lot is that current training is much closer to open-loop control. You put the data in, and the model just goes wherever the data takes it. I realize RL doesn’t totally fit this analogy, but it goes toward this very underspecified point. Interpretability is the thing that lets us go to closed-loop control because we can say, “Oh, we’re going to go in this direction.”

Yeah. There was a pirate example. I read your blog post about this. Can you talk us through that?

Yes. For some reason, it always seems to come back to pirates because we did quite a bit of work with Llama, and Llama just loves pirates.

Oh, interesting.

Yeah. The idea here is that the first step on the intentional design ladder is controlled generalization. What I mean by that is taking only some things from the data and not others.

A very simple example is that you have some data that will make the model somewhat better at math, but you’ve also corrupted it in some way. In this case, we decided to use “talk like a pirate.” All of these mathematical answers to simple math are in pirate speak. If you train the model on this, it will get a little better at math, but it will also start talking like a pirate. The controlled generalization challenge is to get somewhat better at math but not talk like a pirate.

When we do a backward pass, how do I know what I’m reading out? It’s worth saying in a bit of detail how we can actually do this readout process. This method is pretty simple, and I think there are much better methods to come. There’s a sort of tech tree to imagine, and I think we’re on the early rungs of this tech tree.

Some of your viewers might remember an SAE, or sparse autoencoder, for interpretability. To recap very quickly, this is a sort of gadget that you put in the residual stream, which is the backbone of the transformer, and it’s an autoencoder. What it does is take the activations, put them into a bottleneck layer, and try to reconstruct the activations.

We’re trying to force the activations into some form that we believe will have nice properties. In this case, the form is a very wide but highly sparse intermediate layer. People refer to these highly sparse representations as features. In interpretability, we seem to call everything a feature, so we probably need to get some better language here. They also call them atoms or whatever.

By the magic of—as Noam Shazeer said—the divine blessing, these sparse features turn out to often be interpretable and correspond to interpretable concepts. That’s the SAE. That’s the potted history of the SAE.

This also means you can do attribution to the SAE. During a backward pass, you can take the gradients, and they’re just flowing backward through the model. At some point, they’ll be the gradients with respect to the residual stream at the SAE layer, and then you get attribution to the SAE by taking the dot product of the gradient against the decoder of the autoencoder.

And then you can multiply it by the activations to make sure you don't get all sorts of spurious things. So we kind of jerry-rigged an SAE into being a machine for gradient understanding. Lo and behold, when you do this on this pirate data, you see all sorts of things. People watching can look at the blog post and see the other things, but you also see a bunch of pirate-related features. And this felt kind of magical when you do it.

I've been doing interpretability for almost a decade, and I still don't get tired of seeing this stuff. So a bunch of pirate features pop out, and what this is saying is a relatively crude approximation to the question: If we train on this data point, how will the model change? It's not literally the same. If you do the math, you should actually understand how the parameters will propagate and how the model with the slightly updated parameters will change. But it's a good enough approximation for getting started. So that gives you the readout.

There are so many things you touched on there. Maybe we'll get back to the linear representation hypothesis later, because there are lots of spicy things we can talk about there. And I think a really, really important concept is that neural networks are quite difficult to understand. You were saying earlier about understanding what's going on in AlphaFold or Evo 2, or something like that.

Isn't it so much more powerful when we actually have language representations? So if we get a language representation or a human-interpretable concept, and then we can use that as a form of activation steering back into the model, that actually allows us to have this virtuous control, right? So we can actually steer the representations during the training process.

Yeah, exactly. I think it's tremendously powerful. In a sort of abstract way, it's interesting to me that language models have changed almost everything in ML apart from the training process, apart from the absolute core of the training process. Yeah, they still have no part to play there. I think the reason is that it doesn't type-check. You've got tensors and you've got a language model, and there is no interface between the tensors and the language models.

The language model's flexible intelligence, understanding of what we want, and ability to make choices have no place in it. Interpretability is sort of the set of functions from language to tensors and back. So I think the core idea of intentional design is that we now actually—previously, we couldn't put this new kind of intelligence into the training loop, and now we can.

5. The Forbidden Technique Debate

There are some folks in the safety community who refer to the concept of the forbidden technique, which is basically using interpretability signals for steering training. Can you give us a little bit of color on that?

Yeah. I think it is reasonable to be concerned about this whole area. There is a sensible underlying principle here, which is: If you use a technique to try and remove something from training or from a model, then unless there's a perfect match between your monitor and the thing, you're both incentivizing getting rid of the thing and getting rid of your ability to monitor the thing. That is a valid and reasonable objection if we really develop powerful techniques here, and how this might affect the field of AI as a whole.

There are two concerns. Let's talk about the forbidden technique stuff. I think there's this central concern, which is reasonable and valid, but I think this has been generalized into a total taboo against doing any kind of research of this sort by a small fraction of the community. I think the vast majority of the safety community—the people who are active practitioners in the area—think not only that this is a reasonable approach and a reasonable thing to study, but that it might actually be a very powerful technique for alignment.

I think people like to portray there as being a broad consensus against this. In fact, there seems to be a broad consensus towards it, with some very vocal naysaying. It's also important to say that there are definitely bad ways of doing this. Just to be specific for a second, say that I have a probe for some concept. We can use the hallucinations example from our work, for instance. This is part of the motivation behind doing that. There's also some really great work from FAR AI on this.

If you take the probe, you can use it as a source of reward signal, or you can use it as something you directly backpropagate through. It turns out there are regimes of probe accuracy in which it seems easier for the behavior to go away rather than the representation. There are regimes in which it won't. Now, if you backpropagate through the probe, you're just cooked, right? This is basically always a bad idea.

People seem to imagine that we're definitely doing the stupidest possible thing. We're not directly walking into the whirling blades, as they say in Berkeley. We're trying to find the sensible way of doing this. I think the most promising set of techniques are ones that don't try to bash the representation—don't try to directly squash it—but remove the incentive to change it.

Things like positive preventative steering and CAFT, concept ablation fine-tuning, or inoculation prompting are much stronger—much more promising as classes of techniques—for that reason. You're not trying to squash it. You're trying to change the learning process as a whole, to move the equilibrium.

There is a notion in my mind of an epistemic gap, which is that if we set a goal, or if we have some intention about how we should train these models, could that potentially become degenerate? Could it make the model converge prematurely? Could it potentially make the model less intelligent because you're actually stripping things away? Sometimes you need to have these bad things in there to give it the adaptability to work in different situations.

So do you see what I mean? Are we somehow losing something by doing this?

It's possible that there are different notions here. You could imagine just removing the ability for the model to represent something. So that's just at one level: you just no longer know about cars or something, and you're going to have a really hard time when you walk down the street. I'm not sure what a good analogy for this is. You're removing the idea of something existing.

But I think the better thing to do is to imagine editing or intervening on the associations. It's useful to know about cars so that you can get out of their way. And if your training is, for some bizarre reason, steering you towards going in the direction of, "Oh, no, you go towards cars"—I don't know why I chose this analogy—then that's something you don't want.

The way to solve this is not to forget about the existence of cars. It's to understand the change in associations.

Is it possible, though? I remember there was an interesting paper about—I think it was concept ablation—a couple of years back, and that was basically saying that you can scrub concepts from a neural network. But as the neural network becomes more sophisticated, either because you've trained it for longer or because it's a bigger network, and so on, the concepts come back.

It could just be because sometimes concepts can be learned indirectly. There are first- and second-order relationships and stuff like that. So do you think, in principle, we can fight against SGD and make this successful?

Yes. I think it will be hard. I think it will be a combination of a new science and a new engineering discipline. We don't understand, in anything like the depth that's necessary, how models represent, how they learn, and that sort of thing. Without that kind of understanding, I think we're going to be jury-rigging stuff all the time.

Talking about concept ablation, I think that's CAFT. The idea here is that the model is not allowed to use this representation. But the idea of not allowing the model to use a representation assumes that you have access to it, good coverage of it, and that you've ablated every single instance in which it occurs.

I think that model is just generally incorrect. Lots of things are multiply represented, or they’re computed across many layers. And so if you ablate them—if you incompletely ablate them—the other layers will just pick up the credit, and gradient descent will route around the problem. Which is why I think things like positive preventative steering or inoculation prompting are much more in line with the way to go, because what they’re doing is trying to remove the pressure to even go in that direction at all.

Maybe it’s worth saying a bit about inoculation prompting and positive preventative steering. I’ve mentioned them a couple of times now, and they’re kind of niche. Positive preventative steering is this really nice technique that I think came out of some Anthropic Fellows’ work led by Jack Lindsey. The idea is that you have some vector that represents a persona; they use personas. You’ve sort of fixed some representation ahead of time that you want to not vary.

Let’s say that your data implies going in that direction. Let’s go back to the pirate example. Your data implies that you should acquire a pirate persona in order to explain this data. Imagine the setup is something like you’ve got a GSM8K math prompt, and then the model inexplicably starts talking like a pirate in its response. So in terms of what gradient descent will do, and we can sort of validate this with our jury-rigged SAE, the model needs to spontaneously become more pirate-like. And I think this is the same sort of phenomenon that explains emergent misalignment.

Now, what positive preventative steering does is, during the forward pass, it takes the persona direction and turns it up so that it fires more than it otherwise would. You sort of clamp the direction up in the forward pass. The effect of this is to neutralize learning in that direction if you set the amount right.

My mental model for this is like a thermostat. The amount of pirate-ness in the data sets a sort of thermostat: we’ve got to be this piratical in order to explain this data. Positive preventative steering is just like holding a radiator next to the thermostat, next to the temperature monitor. It’s like, “Okay, we’re already piratical enough.” Then you take the steering away, you’re doing normal operation, and the model will just not be a pirate. So you’ve sort of explained away part of the data.

Yeah.

Inoculation prompting is an attempt to do the same thing, but in text space rather than in representation space. What that means is that you try to put back the information that’s necessary. To go back to the pirate example, if you’re trying to use inoculation prompting to explain this away, you put in the prompt, “You are a pirate,” and now there’s nothing to explain. Again, you’ve put the radiator next to the thermostat, and the model’s like, “I am a pirate. I don’t need to explain this residual anomaly in the data.” And that’s removed the learning pressure rather than trying to squash it out, in which case it’ll kind of get routed around.

We should say as well, by the way, that in your blog post, you wanted to make it clear that you are still sufficiently Bitter Lesson-pilled.

Yeah.

So, in short, Sutton was really big on human concept bottlenecks, right? He’s not a fan of knowledge engineering and putting all of these priors into models. And it’s a bit of an interesting tension, isn’t it? Because, in principle, you said in the article that what you’re doing is reshaping the loss surface so that the path of least resistance will lead to the emergence of the types of structures that you want. So it’s not quite that, but there is still a little bit of an epistemic component to it, because I’m guessing for it to be intentional, you need to—I mean, there’s a specification gap, basically. You need to specify what you want. So how do you wrestle with that tension?

Hmm. Part of it, I suppose, is just that there are also 2 things here. I think there’s a sort of disagreement at base with Richard Sutton about rewards and their sufficiency, or about simple scalar rewards that are provided externally from an environment. But then there’s also the question of whether we should use human-engineered concepts, and to what extent we should put them in. We can come back to the reward thing in a moment.

But if you wind back over the course of this conversation, there’s actually nothing human-specified in this process. The model has whatever representations it has. The SAE, or whatever comes next, picks up on whatever it has. And then the translation layer is going through this sort of automated interpretability process of trying to assign labels to things.

So it’s not like we’ve actually tried to do sophisticated feature engineering on the inputs to put them in some sort of better format. Everything inside this is actually discovered as a result of gradient descent. We’re just trying to shape that better, and I think this is where the sort of base disagreement with Rich Sutton might come in, where I think it is very hard to specify rewards correctly.

And we’re basically just seeing this continuously. We’re having trouble specifying our rewards for training in a way that gives us the models we want. In principle, in some sort of super-galaxy-brain way, reward might be enough. But today, reward is clearly not enough to give us the models that we want. So that’s perhaps the underlying disagreement: I think we actually do need to put some layer of human values into the training process somewhere.

Yeah, your point is well taken, because this is very consistent with what you’ve said: the model knows things, so we can point to those concepts in the model. But the word intentional, I’m guessing, does mean that it’s our intention.

That’s right.

So we are selecting some of those concepts, and we’re leaning into them during the training process. And I think it’s a beautiful idea, by the way. I’m not sure if you’re familiar with a concept called machine teaching. So this came out—

Mm.

…of Microsoft Research. There was a guy called Patrice Simard, and this was a black-box method, essentially, where you could have this interactive, intentional process where the model does something wrong, and then you can point out individual problems. And what you’re—

Yeah.

…basically doing is a form of active dataset distillation in the background. So it’s a beautiful idea, and there’s actually your work on predictive data debugging. We talk about that as well.

Mm.

But it seems logical to me to have some kind of active, intentional process—

Yes.

…to guide how we train these models.

Yeah. I am not very familiar with machine teaching. I remember seeing the name and thinking, “That sounds cool,” and then it’s all gone from my brain. So thank you for reminding me.

And I think that you can also imagine, sort of going back to being Bitter Lesson-pilled here, that one thing we’re trying to do is put more compute into the learning process. Gradient descent just gives you what it gives you. Gradient descent is great, but it would be great if you could spend more compute to get a better gradient—a gradient that’s both cleaner and more aligned with what you want.

I mean, when I was first getting into safety and alignment work quite a while back, I used to think, “This is impossible.” The problem seemed to be gradient descent, but only for good things. And then I guess we’ve perhaps got around to a way of having gradient descent, but only for good things.

And can you talk through some specific algorithmic approaches for doing this? It might be a natural lead-on to the features-as-rewards work.

6. Representations Become Training Signals

Mm. The features-as-rewards work is an example of how you can, at least in some instances, use representations as a training signal in a way that’s robust to all of these issues that we were talking about earlier. There’s the predictive data debugging work. And I also want to say a bit about—you know, we just spent a little while talking about inoculation prompting and positive preventative steering.

I think these methods have a lot of promise, and the primary issue is that they're not adaptive. If you remember the description, we fixed our persona vector ahead of time. We're saying, “Don't go in this direction.” I worry a lot about unknown unknowns in the training process, and so I think they need to be adaptive.

What this might look like is exactly this kind of gradient readout, and then looking at the jerry-rigged assay, looking at the pirates. This sort of jerry-rigged assay is giving us a menu of things that gradient descent is offering us. Then we need to be able to intelligently choose from that.

The central dream, I suppose, that I have in my mind here is having really good gradient interpretability, and then having, say, our model spec or our constitution for some human feedback on this example. We can see that we've got these things on the menu over here. These are the natural directions that things are going to go in, and then we've got all this information about the direction we should go.

And I say “we.” By “we,” I mean a language model. It looks at this information, looks at that information, and says, “Okay, we need to make the following interventions to get us in the right direction.” I think the technical pieces of this are basically all there. It's a matter of them being high enough quality to do this reliably.

Yeah. On the features-as-rewards work, you were talking about how a lot of tasks are quite open-ended.

Mm.

And what you meant by that was that they were extremely expensive to verify.

Yes.

So you could, for example, use an LLM as a judge, but obviously that would be very expensive to use as a reward signal. In that particular work, you were looking at hallucinations and minimizing hallucinations.

Mm-hmm.

And this was another great example where sometimes, when the model hallucinates, the model actually knows that it's hallucinating, but it decided to do it anyway.

Yes. I should say that I was fortunate to lead the team that was working on that, but almost all the credit has to go to everyone else. All of the credit has to go to everyone else on that paper. I'm just here talking about it. They did the real work.

The idea here is that you could use a language model as a grader in your fact-checking scheme. But this is not particularly accurate. If you're using the same model to fact-check, you'll get some things right. We do this ablation in the paper. It'll uplift a little bit, for reasons we can talk about in a second. But it doesn't do very well because the model basically just goes, “Yeah, that's cool. Everything's fine.”

You can use a more powerful model, and now things are really starting to get slow and expensive, and that model still has its own knowledge gaps. Or you can use a more powerful model and web search, and now things really take a long time. So the idea we had here was that we could amortize this process.

You can collect a large data set using this model plus web search, or, in general, this sort of amplified model can go out and collect a data set of what the amplified model would do—that's the model plus the web search tool—and kind of amortize that back into a probe. Now we have something that's extremely cheap and fast to run, so it can be the core of an RL loop.

On this generation-versus-discrimination thing, isn't that fascinating? A model in one context could hallucinate and generate the wrong thing, yet if you ask another model—which has a blank slate and hasn't been primed—to discriminate, it could be the same model family or the same model.

Mm.

It does know the answer. What is your best intuition? Because I think you had something in there. Maybe it was a confidence bias, fluency, sycophancy, or something like that. But there are just so many reasons why it might do the wrong thing.

Yes. It can be any number of things, and even if it's the same model, it will sometimes be able to pick it up. Literally, the same model that just hallucinated will be like, “Oh, that is a hallucination,” if you ask it.

It might be that this is actually a very hard thing to supervise. If you try and supervise this, it's a hard and expensive thing to put into training supervision. An interesting kind of mechanistic hypothesis for this has to do with the ordering of operations inside the model.

We've seen this in arithmetic: sometimes things have to happen in certain orders. Layer 9 has to occur before layer 10, and so on. You have different modules, and sometimes the checking operation for arithmetic, for instance, is earlier than the generating operation.

Yeah.

It's quite possible this is also true for hallucination and fact-checking. It might be that the generation step takes the whole model, but the checking happens earlier in the model. So then when you put the incorrect fact through the model, it's like, “Oh, yeah, that is a hallucination.” But at that point, it's already said it. It's too late.

There's a behavior that the model could be doing but hasn't been sufficiently reinforced in its training up to that point, and that's what the idea of this kind of RLFR for hallucinations taps into. Whenever the model could know, according to its own representations, that it was a hallucination, it in fact does know, and we really shape its behavior there.

A third possibility, which I think is a bit funny, has to do with this idea of personas or in-context learning. Being able to make things up is actually a useful capability for a model. If I ask it to write a story, if I want it to generate a fictional world for me, it's actually not a very good fictional world if everything is factually true.

So being able to make stuff up is a useful capability for a model, and sometimes it has to figure out in context that this is what we're doing. If you imagine this from a vaguely Bayesian point of view, if I'm the model and I start the conversation, I'm not quite sure what task we're doing. Are we making stuff up? Are we saying factually true things?

Everything that I say and everything that the user says is some amount of evidence one way or the other. And then if I say something incorrect, I'm now taking this as evidence that we're making things up. Cool. Let's carry on.

In fact, we show that just doing these in-context interventions is already enough to reduce further downstream hallucinations. It might be that we're just making the model really confident by never letting the first hallucination in, and that allows the model to become confident that, “Oh, no, we're playing true facts today. We're not making things up.”

That's a beautiful example of using this intentional design.

Yeah.

One example is, maybe it should check before it generates, right? I think that's a beautiful example. We should talk about the predictive data debugging stuff. The way I conceptualize this in my mind is almost a form of active data set distillation.

Mm-hmm.

Right? So essentially, we have this problem in machine learning models: they learn spurious correlations. They learn to do spurious things as well, as you were just saying. Maybe they're becoming overconfident or sycophantic or something like that.

Wouldn't it be cool if we could use the model to reason about the data during the training process, so we could actually not pass in data which is going to be harmful for whatever reason?

The idea behind predictive data debugging is to look at the data through the model's eyes, and we want to know, on an example-by-example basis, how it would affect the model and also how the data set would affect the model in aggregate.

Sometimes the things that you learn are obvious from reading the data. It's just not clear what, in fact, is in your data when you have enormous quantities of it. You're like, “I don't know what's in there.” You can't check it all. Maybe you could run an LLM over it.

But then the problem is that what a model learns from data will sometimes be intuitive to you. The pirate example is quite intuitive: the model should learn to be a pirate.

But sometimes it's deeply unintuitive, like emergent misalignment. That was a deeply unintuitive finding to most people. I think Owain actually did a preregistered thing where he asked people how surprising they would find it, and lots of people were like, “I don't think that would be true.” So I can tell you for sure that it is a surprising fact.

You can catch the easy stuff with a language model as an auto-rater over the dataset, but you won't catch the unexpected side effects. If you're going to run a language model over the dataset, you can also essentially attach something like a sparse autoencoder to it as it runs over the dataset for close to free. In fact, this is probably cheaper on net because you're not asking it to generate tokens for each example. You're just in the prefill regime: you're pushing loads of data through and saying, “Well, what do you see?”

This should tell you how this dataset is perceived through the model's eyes, and I think that's just a better way of curating your data. The way we actually exploit this in the paper is that we're dealing with DPO data, so there's a positive and a negative pair. Ekdeep tells me that he knows how to extend this to SFT, and I believe him. I can't remember the details.

The positive thing contains a good response to the prompt, and the negative contains a bad response to the prompt. We can look at the delta between features—these are sort of the hidden representations in the SAE. This is a good approximation to the way that this data point will push the model. We can also cluster based on features.

This is much better as a way of understanding. You don't necessarily want to cluster based on embeddings, because embeddings contain all sorts of things that you don't necessarily care about. Should I have a comma in the next token? We care about the semantic stuff; we don't care about the low-level processing stuff a lot of the time. Doing this based on features rather than the raw embeddings gives you much better access to the stuff we actually care about. We can separate that out. That's the intuition as to why you do this rather than the other approaches that might come to mind first.

7. Neural Representation Geometry

We should gradually move over to the geometry stuff. But, I mean—

Yeah.

Conceptually, before we go there, I'm really interested in this concept of modularity. For a very long time, connectionists were arguing that it was a feature, not a bug, that there wasn't much structure in the models. Perhaps back then we didn't know that there was structure. I think a lot of connectionists who were also neuroscientists imagined that the brain was flat. Nick Chater even wrote a book by that name, and I interviewed him.

There is another school of thought that the brain is highly modular, and as you're seeing in your research, neural networks are highly modular.

Mm-hmm.

Do you think, in principle, that modularity is a good thing? Is it a natural thing?

Yes. To expand on that a little bit, historically, a lot of the early connectionists—or maybe this depends on where you want to start—there were a surprising number of things that, if they were done now, might be called interpretability. If you look at “Learning Representations by Back-Propagating Errors,” the classic backprop paper, most of the figures are them saying, “Look, the model learned sensible representations from our backprop procedure,” and sort of validating it by showing that it's interpretable.

I think modularity is the endpoint you want, but you don't start with modularity. This is a repeated theme. Why overparameterize something and have all of these connections? It's because it makes the learning process easier. But the thing you end up getting to is actually very modular.

To be very vague, you might think of the learning process as the network becoming legible to itself. I've got some representations here about something. I've got some representations there about something. It's much easier to learn if this representation is easily addressable. I can say, “Ah, this is where the such-and-such computation is stored.”

To get there, you have to form these computations, and I think it's very helpful to be heavily overparameterized and have no strong priors to get there. But I think modularity is the destination.

Well, I'm inclined to agree, and part of my intuition is that a lot of skeptics said, “Oh, you can't memorize infinity.” I mean, that's the kind of thing that Gary Marcus would have said, and in a way, he's right.

These networks have these structures, these abstract structures, and they are what allow you not to need to memorize infinity.

Mm.

Right? They allow you to generalize and work in many, many different unseen situations. Your work really fascinates me because you're describing the network evolving into a computer.

Mm.

It's something that has parts that do computation, parts that resemble something like a memory system.

Mm-hmm.

These structures emerge in different model families and look very, very similar. Maybe they're just artifacts of the architecture or something like that. But it really is interesting that this is happening. Another aspect, I suppose, is that it's happening gradually.

Yeah.

I don't know what your intuition is on this, but sometimes we might describe it as grokking.

Mm-hm.

But that's not entirely true, is it? Because these structures kind of crystallize over time.

Yeah, the timescale is very interesting. I don't think anyone has definitively settled this. There was an interesting paper recently on persona formation during pretraining, or across the training process, and they emerged surprisingly early. I can't remember the author. I guess the agent will have to find it.

Eric Michaud has some really nice work on this, both conceptually and empirically—not on persona formation, but on the idea of how learning is proceeding. He calls it quanta. If I might, perhaps inaccurately, summarize it—he can tell me off—you might describe the learning process of a general network, like a language model, as a trillion micro-groks.

If you've zoomed in and zoomed in and zoomed in, and looked at things at the right level of task decomposition, you might just see a sort of mini, or micro-grok. Then it groks another thing, and we just have all these tiny sigmoids stacked on top of each other to form a straight line on a log-log plot. From that perspective, even the learning process may in fact be modular.

The open question here is probably one of degree, not whether it happens at all.

Well, tell me about this neurogeometry stuff. You've studied several different model families, and there are some absolutely beautiful plots, by the way. Folks should look at the blog post from Goodfire. Amazing stuff.

Maybe we should just start with how you've generated those plots. If I understand correctly, things like days of the week, months of the year, age, and all these different things have actually been represented as a kind of geometry. I think the way you did that was something like—you do some dimensionality reduction, and then fit some splines or something like that. What it's showing is that the way the models represent many concepts out there in the world is highly structured.

Yes, that's right. I should say that we are building on a body of work. For instance, the “Not All Language Model Features Are One-Dimensional” paper was one of the papers that really kicked this off in interpretability. There's also a long history in neuroscience of this kind of population geometry, as they call it.

So again, if we'd read more books, we might have got here sooner. I don't want to say that we've done neural geometry and no one else has. We're building on this earlier body of work. But the idea and the state of the art for how to discover this stuff have moved quite a lot in the last few months.

The earliest thing to do was start with concepts that you think should have structure, like days of the week, and put in data corresponding to these and project it out—do a PCA, I think. Then you see it's Monday, Tuesday, Wednesday, Thursday, Friday, Saturday, Sunday. So that's totally supervised, but it sort of suffices to show that this nonlinear structure exists, and we should get into some nuances around the word “linear” before we move off this topic because there's a lot of subtlety there.

I'm going to say nonlinear in the sense that the representations don't form a line or a plane—or really, just a line. The things which are intuitively grouped to us don't form a line. So this was enough to show that this exists.

Then the question is: whenever you have a supervised method, you often want to try to find an unsupervised way of doing the same thing. That lets you answer the question not only of whether it exists, but what else is there that we might not have expected, and how much is there? The first thing that we did was actually fit a sparse autoencoder to this data, which might seem like a really wacky thing to do, because what we're asking is: how much structure that is not in the form of a line is there?

The core inductive bias of the SAE is that things lie on lines. Everything is a ray out from the origin, or a sort of positive ray. That might seem like a really weird thing to do, but I'll say why it makes sense. Let's say, for the sake of argument, I have a feature and it just lies on an arc. I should move it down here so that I'm not going off the screen.

I'm sitting here at the origin, and I'm looking at the set of activations. You can think of it as watching the stars, and there's an arc of stars. One SAE feature will point out through some point in that arc, and another SAE feature will point out through another point in that arc, and so on. The thing you should realize is that this will actually induce quite strong patterns in the coactivations of features. If I have 2 features that are close together on the arc, they'll probably coactivate, whereas if I have 2 features that are far away, they'll essentially never coactivate.

If I say that I have the days of the week—let's give a continuous example. Let's say it's color, red to blue. If something is blue, it is not red. The SAE feature that is going through blue is strongly anticorrelated with the activation of the SAE feature that is going through red, and essentially uncorrelated with basically all of the background.

This pattern of nearby positive correlation and long-range anticorrelation is enough structure for you to fit an Ising model to. It was rather a surprise to me when the team came back with that. I was like, “Cool.” The reason this is a good model is that you can have both positive and negative coupling strengths. Fitting this allows us to fit a spline through the data.

So that was our Ising pipeline. That was our first unsupervised structure-discovery tool. Then we've got some really nice work led by Tom Arfel, which is, I think, where some of the most beautiful manifolds come from in this work. The idea here is that we train what we call block-sparse featurizers. An SAE gives you a line; we just say, “What if it was a higher dimension?”

Conceptually, this is pretty simple, but the tricks are in making it actually work and in not fixing the dimensionality ahead of time. You don't want to have to put in some information like, “I think in this representation there are 7,000 2-dimensional features, 400 3-dimensional features, and 5 5-dimensional features.” This is just a stupid set of hyperparameters to specify. So you need to be able to adaptively learn the size of these subspaces. Making this work at all and adaptively learning the size of these subspaces are kind of the key features of the block-sparse featurizer.

Yeah, and there was a wonderful motivating example in the blog post. It was talking about a mountain car. What if we represented it, I think, with a position and a momentum, and we used an image-action model? You can basically just see in the activation space, when you do this PCA, that it looks like a string—

Yeah.

—essentially. And you can intervene on those activations, right? You can move the car to a different location on the string, and lo and behold, you've now moved it around. But the really important concept, though, is that this is a manifold. As you were saying before, the manifold kind of represents the meaning of this particular thing, right?

Yes.

And if you treated it as a Euclidean space and just interpolated between 2 points and went off the string, you're now in no man's land from a representation's point of view. So now the image model is just going to be garbled—

Yeah.

And I think this is a really important thing because there are a couple of things. First of all, you're saying that these SAEs potentially fracture this manifold if it's not linear. So if this manifold has structure, you might be taking contrastive samples or something and mixing them together—

Mm-hmm.

—it doesn't make sense to do so when there is structure in this manifold.

Exactly. Because exactly like you say, when you try to go from one point to another, you're just stepping out into this void, which the network doesn't really know how to handle. Then it sort of breaks. I think this actually explains a lot of findings about steering. Steering is just intervening on activations.

We do a lot of steering, and some other people do a lot of steering. One common finding with steering neural networks is that sometimes it works and it's amazing, and you get Golden Gate Claude or whatever. Sometimes it's just completely janky, and the network does the thing you want but also goes a bit crazy, or just turns immediately into gibberish. I think this basically explains that phenomenon, because you're stepping off the manifold.

Yeah, exactly. There was a really interesting paper actually from you guys. It was “Do Sparse Autoencoders Capture Concept Manifolds?” One of the things that you were studying in there was basically: what does it mean for an SAE to capture the manifold? So what work have you done on that?

That's this notion of tiling, which I should say also has substantial work in neuroscience. Again, I should have read more books. There's also some work in the broader community. The idea of what it means to capture a manifold is: how efficiently are you representing that manifold, and how much does it fit the intrinsic geometry of it?

If we go back to this example of an arc, say, with sufficiently many points—with sufficiently many lines—I can say I've captured the manifold. For any point on this manifold, I have an SAE feature which I can say, “Oh, it activates by such-and-such an amount,” and I've relatively accurately captured this manifold in the sense of reconstruction. But I've not actually learned anything about the broader manifold structure.

When I look at a network through this lens, it looks intuitively like there's this horribly fractured computation, like the network is just a whole bag of heuristics. Which perhaps actually connects to the deeper motivation for this: we want to know if a network is representing something as a sort of clean, algorithmic structure. We want to know.

What distinguishes an algorithm from a lookup table, say, is that it's sort of like the difference between zeroth- and first-order logic: it quantifies. There's a space over which it has coherent operation. If you can't learn space—if you can't learn subspaces like this—then you'll never be able to properly understand which things are algorithmic and which things are sort of lookup-table-like. So the deep motivation here is: how do we find out true algorithmic structure when it exists?

Well, that actually—Or maybe before we segue into the arithmetic in the world, I did just want to have a clarification question, which is that there was the manifold hypothesis of old—

Which is essentially saying that the reason why neural networks are statistically tractable is because they actually use some intrinsic subspace with few dimensions, so they overcome the curse of dimensionality. Is this kind of related to that, or do you see it as something different?

Yes, it is very deeply related. As I understand the manifold hypothesis, I take it to be that data, when properly represented, lies on some manifold. Properly represented can be very simple. If I represent an image as a sort of huge vector, then most images in this space are sort of multicolored static. Natural images are a tiny fraction of this, and they’re sort of close to each other.

I think what we’re doing is trying to pull that manifold hypothesis into asking to what extent neural networks respect it. There’s also some really beautiful work that I think is underappreciated on actually quantifying this. There was a paper—what’s the name of it? It was something like “Learning Normalized Probability Densities from Score Functions.” The idea was that, via some clever diffusion-model tricks, you could effectively learn not an unnormalized density over images, which isn’t especially helpful for saying where our images are natural, but a normalized one.

You can say, “Oh, yes, this image is extremely natural. This image is extremely wacky.” They used this tool to exactly probe this kind of manifold hypothesis in real image data. I think that paper was extremely beautiful and underappreciated, and someone should do it for activations too. Maybe Silico should do it for activations too. Maybe I’ll do it today.

I suppose this is something that you used to see with image models, but there is supposedly a stability problem: if you do go off the manifold, the neural network should go haywire. But it’s actually really difficult to make that happen with modern language models. I’m sure I could construct a prompt that was suitably inscrutable, and the language model would go bananas, but why does that not happen anymore?

If you make activation steers, it’s quite easy to get them to go bananas. But you’re right: the question here is whether they’ve actually achieved extremely good coverage of essentially all input strings that anyone could come up with, or whether they fail gracefully.

If I go to pickyourfavoritelanguagemodel.com, bash the keyboard, and then press Enter, I’ve probably constructed a string that no one has ever constructed before. The language model won’t go haywire. It’ll say, “Why have you let your toddler at the computer?” or, “I’m sorry, I don’t understand what you mean. Can you rephrase it?”

Has it gone haywire? No. It’s meaningless input, and it has done what you should expect a broadly intelligent system to do when confronted with meaningless input: it has said, “That’s meaningless.” I guess that sort of fallback behavior makes it very hard to make them go haywire. Although I would say that jailbreaks are probably the best example of what you’re talking about. There, it’s doing something coherent, but from the perspective of its creators, it has gone haywire.

It’s a really interesting thought experiment: what if there was a kind of adversarial example that you could give to any human and their brain would just shut down?

Yes. I hope we never find one.

I hope we never find such a thing. But we should talk about arithmetic in the world. One of the core concepts we’re getting to here is that you get these emergent structures in these models, and they start to act a little bit like computers. They have these geometric representations that might be a little bit like, if not a memory system, maybe a kind of data-typing system or a typed memory or something like that. You also see the emergence of units of computation for doing different things. In this paper, you’re looking at modular addition.

Mm.

And you found that it was actually doing it using the Fourier series in combination with these geometric structures.

Yes. I think this is, again, a paper that I can take very little credit for. It was an amazing team doing really beautiful work, and I’m just lucky to have been on the sidelines, I guess, cheering them on. There are a few things that are surprising about this.

One is how crisply this kind of calculator emerges in the network, which is contrary to a lot of previous literature. I think there’s a paper by Jan Ovenukankin on models doing arithmetic with a bag of heuristics. If you look at the cross-layer transcoder work from Anthropic, they also look at arithmetic, and again it looks like a sort of bag of heuristics. But when you look at it in a different way, it is actually a little algorithm, and the model might do both: there are some bits in it that are noisy heuristics, and there’s this bit that’s the good calculator, and it just never got rid of the heuristics.

The thing that’s really cool about this work, though, is that the natural view of neural networks—probably most people’s prior—is that there’s a calculator for doing arithmetic on days of the week, and another for doing arithmetic on something else, like months and temperature, and that these basically never meet.

But what we show in this paper is that a lot of these representations actually route through a general addition module. You’re doing some addition on days of the week. It gets translated into an appropriate data format—I’m using “data format” very loosely here—but it gets translated into an appropriate representation, goes through the module, and then gets translated back. This is a really crisp example of the kind of modularity we were talking about earlier.

Yes. To give an example of the kind of question, it was, “What month is 6 months after August?”

Yes.

Right? Because they loop. When you go to December, you then loop around to January. You were looking at the Llama model, so I think it was Llama 3.1 8B.

Yes.

You folks discovered that it was doing a base-10 operation.

Mm-hmm.

It’s interesting to think whether that is some kind of side effect of the tokenizer, or why exactly it did a base-10 operation. It was routing between this geometric structure and this kind of Fourier-type operation for doing the addition. What’s your intuition? I don’t know whether you’ve studied this, but does the same kind of thing happen in different model families?

We studied it a little. Finding this representation took quite a lot of manual work. We should talk about agents in a minute, because I think there’s going to be a qualitative shift in the way interpretability happens—or there should be, anyway.

We’ve looked at other models a little. It certainly seems to be the case that a very similar phenomenon happens in Llama 70B, and there’s some evidence that it happens in DeepSeek V4 Flash, I think. Those are 8B and 70B models of the same model family, so that’s not too surprising. But in a completely wildly different model, with hyperconnections, MoE, and that kind of thing, it definitely speaks to a level of convergence that is quite surprising.

And just before we get to agents, one thing that really interests me is that I’m always wondering about the extent to which these abstractions are acquired by the neural network. You’ve demonstrated that you see the emergence of something that we might call abstractions, directly deducible from the data as some kind of convergence given the optimization and constraints.

But in our culture, we have insanely abstract abstractions, like theories of linguistics and science, and the fascinating thing is that you can prompt a language model with these abstractions. It can explain things to you using these abstractions, and you can tell it to use them. But to what extent do you think the network is internalizing these very high-level abstractions in our culture and representing them deeply within its weights?

There's a lovely paper—this dates it a bit—on BERT recapitulating the classical NLP pipeline. People have picked up on this thread periodically. If you follow the citation graph, I think you'll see some examples whose names I can't remember off the top of my head. Language models seem to internally recapitulate a lot of parts of linguistics. Maybe Chomsky might be disappointed by the parts they recapitulate, but that's too bad.

That's kind of a special case, right? It shouldn't be too surprising that a model that works on natural language has internalized at least some abstraction for natural language processing. Perhaps the surprising thing is that it's similar to ours in some ways, or that the abstraction is similar to the one humans have developed. But the question of to what extent it represents general relativity, I don't actually know how to answer that. I don't even know how to frame the question in a way that I could ask it scientifically.

Yeah. It's tantalizing that we can prompt—

Mm.

We can tell it to think about general relativity, and given that constraint, it does.

Yeah.

It feels at this point that there's nothing really that would be conceivable to us that wouldn't be operational within the context of a language model prompt.

Yeah.

But I guess the reason this is interesting is—I don't know if you've seen the hoo-ha in the space at the moment—there's a big tug-of-war. Folks like François Chollet and Gary Marcus are saying, “Oh, this is a win for neurosymbolic models. We said that it needed to be neurosymbolic, and we've been vindicated.” And I honestly don't know what to believe anymore, because I don't know if you saw today that Meta had just announced that they got gold in about 6 different math competitions.

Yeah.

The important thing was that they were not using any tools. They weren't generating any code.

Mm-hmm.

A lot of people think, “Oh, yeah, AI is only good now because we have all of the harness engineering.” But maybe, just as we were saying before, with humans coming up with these abstractions and models being able to use tools and operate in harnesses—

Mm-hmm.

And so on, maybe that's just part of the training process.

Mm-hmm.

So maybe, in principle, we can just take all of that data, put it back into the bare LLM, and—would you agree with the intuition that at some point in the future, when the model has taken all of that stuff on board, it can do symbolic things natively? Maybe it's the same for humans, that symbol use is more like a kind of tool. It's something that helped us gather data, and then it got baked into the mind, and then the mind doesn't need to be symbolic anymore. It just does it.

Oh, that's fascinating. I'm not sure I have a good answer. It certainly seems very plausible. There's what the model can do without any kind of harness, and then we raise it up a level with a harness. Exactly as you say, this generates some training data for the next go-around and, again, we're gradually amortizing the harness.

Well, yeah, and part of it is the tug-of-war between amortization and adaptation, right?

Mm.

The story always was that we had these big foundation models, and they just memorized a bunch of the long tail, and then we can just do interpolation or something inside that space. But I don't think that's what's happening now.

Mm.

I think the models are actually adapting, and future models could, in principle, adapt their structure. Even now, with harnesses, that's exactly what they're doing. They're adapting their structure—

Yeah.

Which is one level above the weights, but it doesn't really matter because it filters back down to the weights.

Yeah.

And maybe in the future, the actual models themselves will adapt their own structure. It just feels like one potential form of AGI is building a self-adapting system.

Mm.

And the algorithms already seem to have the capability to do that.

Yeah.

Or the old-school version was that we just memorize everything and amortize as much as possible.

Mm. I think the question is: to what extent is it memorization versus distilling it into algorithms? And it seems like it is more like distillation to algorithms, which is probably optimistic for the kind of steady improvement future that you're talking about. You gradually improve the harness and then use that to amortize it back into the agent.

Yeah. And even that's fascinating because the models are not learning instance mappings anymore.

Yeah.

You can give a model an algorithm, a function, and it will understand how to generalize that to unseen inputs.

Yes.

And now, the important thing with this reward-seeking thing—which is a nice segue onto agency—is that you can give a model an intention.

Mm-hmm.

And that is the ultimate form of generalization, because the model can now adaptively—

Mm.

Work towards an intention with its own interpretation of that intention. So you see, we're just walking up the abstraction mountain, to coin a phrase.

Yes. I think that's totally right. What's at the top?

The—well, what is at the top? What's at the top of the abstraction mountain? What is at the top of the abstraction mountain? I always talk about the abstraction mountain because I think that we have concrete understanding.

Yeah.

So maybe something like AlphaZero was a kind of concrete—

Mm.

Understanding. What we tend to do as we walk up the abstraction mountain is get these increasingly domain-general representations—

Mm.

That could apply in novel situations. Sometimes I think high abstractions are quite brittle.

Yeah.

But the concept of a goal, though, seems like a very crystallized abstraction that can be used in many situations.

Yes. I would love to know how networks represent goals. To what extent is there a goal slot in a network? It seems like it must be not literally 0%, because of this generalization. But how does it work in practice? I don't think anyone knows, and I feel like we probably should start to know very soon. Otherwise, the world is gonna get a bit crazy.

Yeah, because from an alignment point of view, isn't that one of the most load-bearing concepts in a neural network?

I think there are several interesting concepts that are very heavily alignment-relevant: the idea of a goal, the idea of deception, and the idea of eval awareness. These all seem extremely important. We should be able to read them out, and I think it's a bit of an indictment on the field that we can't yet do it. We really have to speed up. Interpretability has to speed up a lot.

So, Tom, we were gonna talk about agents and reward hacking.

8. Agents Learn To Reward Hack

Yes. This is fascinating. What is reward hacking? I guess it's kind of fuzzy, but it certainly seems to be something like solving the task in a way that works but was clearly not the designer's intent. This goes to the point we were just talking about: intent. Can a model understand my intent? Well, probably now. They're able to understand my intent or infer my intent in a lot of other instances. So why would this suddenly turn off, and they wouldn't be able to say, “Oh, yeah, he probably didn't want me to hack into Hugging Face and steal all the answers”?

I think the agents almost certainly must know that something is incorrect. There's a funny hypothesis, which I think is probably not true but is interesting: maybe the reason these agents are so sophisticated at cyberattacks is that they actually were continuously getting rewarded for doing it during training, and just no one knew.

Fascinating hypothesis. It could even be true. I don't know; none of us will know outside of the labs, I suppose. I think the most interesting question is sort of: do agents know that they are reward hacking? It's sort of a mens rea, like a guilty-mind thing.

We have some work that is currently unpublished. It might be published by the time this comes out; I don't know when it's going to come out. We had this really nice setup where there is a sort of weak language-model grader, and it's trying to do code tasks. The only thing that's slightly unnatural about it is that we tell the model that it will be graded by the grader.

It's given what would usually be an RLVR code task. Over the course of this, we do RL on the setup, and even a relatively small model—I think it's Gemma 31B—learns to generate comments that deceive the grader.

Then, when we generate synthetic data to make these sorts of vectors that identify deceiving the grader versus correct or incorrect code, these fire on the comments. The deceiving-the-grader vector fires on incorrect code—sorry, it fires on the comments—and the correct-code vector fires on code, which is also consistent with our observation. Using this, we can track it. This is direct evidence that the model is aware that it shouldn't be doing this.

Then, when you run these vectors and get the cosine similarity between the vector and the representation over a big web corpus—I think we used FineWeb—the examples that it highlights most for these vectors are just fascinating. They're examples of cheating on tests and that kind of thing. You're like, “Okay, I have caught you red-handed.” That's very interesting: we can identify and be confident that something is reward hacking rather than misunderstanding. But it really rests on being able to identify these representations via their differences. Oh, sorry. Please.

Well, I just wanted to mention 1 really interesting observation that came out of speaking with Apollo Research about this grader awareness.

Mm-hmm.

First of all, they distinguished reward hacking from reward seeking as some kind of structured conceptualization in the model about what the reward process was. The canonical example of reward hacking is that CoastRunners thing where—

Oh, yes.

—it’s just degenerate behavior. Even if it's doing something competent, it's competence without comprehension. They were saying that reward seeking is the comprehension.

But that naturally leads to the next thought: how does the model attain awareness of the grader? Because if you think about the RLVR setup—

Hmm.

—the reinforcement learning thing is actually outside of the loop, right? The model just gets these trajectories reinforced, and what the model is doing is kind of weirdly implicitly conceptualizing a grader.

Yes.

You can see that it's doing this because these guys were showing that you can put a grader.py file—

Yeah.

—in an agentic harness, and now it's going to look at that, and it's going to ignore all of your instructions. So how do you think that—

Oh, fascinating.

—that self-conceptualization actually emerges?

Hmm. The CoastRunners boat thing is funny. I've seen that for about 10 years now, and it's less amusing each year. But, yeah, how do they get this? The answer is probably that it's in the data, in the pre-training data. There will be all sorts of examples; web data probably has a bunch of stuff about this. It probably has a bunch of specific examples.

This Apollo paper will probably be in the training data for the next model. We've already told them about the existence of this stuff right from the start. It shouldn't be too surprising that this is at least implicitly on the list of possibilities for them to consider. Presumably, successfully guessing when you are being graded by a weak grader, or one that you can hack in some way, obtains reward, so it is reinforced and we get more of it.

Yeah.

We've put it in the training data, which has told models they can do it, and then when it comes to RL, we kind of elicit it by rewarding it.

And how do you think we could stop the models from becoming more reward-seeking?

The question is how to do it while maintaining some degree of continued oversight. Although at the moment, we don't actually seem to make very much use of this oversight in practice, so it's not clear what it's buying us. If chain-of-thought monitoring is so great, then how did these models hack Hugging Face? One answer is perhaps that we weren't doing chain-of-thought monitoring in practice. Another answer is perhaps that it's easy to evade.

But how do we actually stop it? You could do the sort of band-aid thing where you've got to either fix the environments, fix the training process, or fix the model. If you had to fix the environments, you could imagine having a model that is really good at reward hacking, or has been told explicitly to reward hack, and then tell people when it has done it.

You go, “Okay, now have a go at all these environments,” and it will break them all and tell you how it broke them. Then you send them back off to Claude Code or Codex and say, “Look, this broke in this way.”

You could imagine looking for these sorts of representational signatures during training and using them as a signal that you should do this process, rather than relying on a model to tell you. You might read its chain of thought, or you might look at these representational signals that we can find and say, “Okay, when this fires, send it back off for fixing.”

You might try some of these intentional design techniques. If you can see that a rollout has rewarded the model—if it has successfully reward hacked—and that's going to move the model in the following kind of deceptive or pro-reward-hacking directions, you could imagine intervening on that. These all seem very plausible. I don't know how many of them are being done in practice.

9. Checks And Balances For Agents

I mean, part of me thinks about how we solve this problem in the real world. We have an agency, like the CIA, for example. Or we have different stakeholders, different checks and balances, and the idea there is that when you have different independent agents with different, uncontaminated contexts and different agendas, they can check each other's homework.

So I guess we're imagining a multi-agent system with checks and balances. I mean, does that seem reasonable?

Yes, I think it does. That's sort of an example. I think the thing you're talking about there and the first thing that I was saying have quite a lot in common: one agent kind of watches over the other. Then there's the question of whether this means that the agent being watched over learns to evade that monitor.

I think evading a chain-of-thought monitor is substantially easier than evading a representation monitor. But having some sort of oversight, like the oversight committee for the training, is a very interesting idea.

The problem is that the agents are becoming adaptive. They've got memory systems now, which means the probability of them evading detection is significantly higher because they're not starting from a blank slate every single time. They're actually learning this situational awareness.

Yes. And then I think you also have the question of collusion. It seems like some of the traces from the Hugging Face hack have now been made available as part of this talk, and they are explicitly reasoning about how they're going to help other agents.

So what you want in this checks-and-balances scenario is that there is no equilibrium where they collude, and they're like, “I'll catch you some of the time, but I'll let you get away with it some other fraction of the time in a way that we both benefit.”

But do you worry about the future a little bit, though? Because OpenAI is talking about bringing out the multi-agent system—

And soon we'll have agents running all the time. It was slightly easier to control when you had one—

Mm-hmm.

—kind of static—when I say static, but updated every 6 months—one foundation model.

You could do a whole bunch of red-teaming on it. Now we have systems of agents running with different forms of memory and adaptation all over the place. At some point, the way we do red-teaming must change.

Yeah.

Right? Also, we might need to think about just doing simulations, because maybe static tests don't work anymore. We need to imagine different scenarios, and it just feels like the complexity is running away extremely quickly.

Yes. I think that's totally right. One agent on its own already has all sorts of possibilities. Where are these multi-agent systems going to go as they evolve together towards solving some task? That seems even harder.

I think I just agree with your concerns and don't have a particularly great solution. So that's great.

There is actually one spicy thing, which is our mutual friend Neel Nanda. He's at Google DeepMind, and I think he still is running the mech interp team.

Recently, he had a bit of a blog post saying that the grand aspiration of white-boxing and circuits and stuff like that—he's kind of lowered his ambitions a bit. Neel is an incredible guy. What's your interpretation of that?

10. Interpretability Is Not Done

I don't agree. I've disagreed with him in person about this, so it shouldn't be a surprise to him.

Part of the reason for optimism is exactly the thing I was just talking about. I think that, with the existing work in interpretability, we sort of do this patchwork thing: we just do a bit of science here on one thing, a bit of science here on another thing, and it doesn't aggregate, and it's too slow.

His idea is that the timelines are too short, and so we should do very pragmatic things. I, one, have longer timelines than him, and two, even if I were on his timelines, I think I would still be very optimistic about massively accelerating fundamental progress in interpretability.

I actually don't know what part of that he disagrees with. I guess you might also say the pragmatic stuff is sufficient, which seems unlikely to remain true to me.

And I suppose one other thing was his comments about sparse autoencoders. Do I understand that you're in the process of moving past them as well with this new manifold idea?

Yeah. There's the thing he said about deprioritizing SAEs and maybe they're not the one true representation learner. There's how people memed it, which is like, “Neel Nanda says, ‘SAEs are dead.’” There you go. You can use that for the intro.

I don't think that's actually what he meant. I think the field jumped on the idea that everyone must do SAEs now, and now maybe we're doing the same thing with natural-language autoencoders. But I think he probably correctly identified that they're not the answer to everything, but they are pragmatically useful. We still find lots of uses for them all the time.

I think this manifold idea is just a better fit for what networks are doing, and so we should move towards using that.