Don't Fight Backprop: Goodfire's Vision for Intentional Design, w/ Dan Balsam & Tom McGrath

Now, today, I'm speaking with Dan Balsam and Tom McGrath, CTO and chief scientist of mechanistic interpretability startup Goodfire, who, in less than 2 years since founding the company, have assembled an all-star research team, landed a first wave of blue-chip customers, including a couple that discovered Goodfire via Dan and Tom's first appearance on the show back in August 2024, published a remarkable series of results, and most recently announced a $150 million Series B fundraise at a valuation of $1.25 billion.

Along with the fundraise, they've announced a new pillar in their research agenda: intentional design, a push to expand the scope of what interpretability science can do by complementing the current paradigm of reverse-engineering how trained models work with a new approach focused on understanding and shaping the loss landscape to control what models learn during training and, ultimately, how they generalize.

We begin with a discussion of interpretability developments broadly, with Tom emphasizing the shift from techniques like sparse autoencoders, which transform a network's messy internal representations into sparse vectors where each node represents a distinct concept, to newer approaches that attempt to understand the intricate geometric structures that these concepts inhabit within the model's latent space. From there, we dive into their plans for intentional design and their first proof of concept: a technique for reducing hallucinations that uses a probe trained to detect hallucinations both to steer the model at runtime and as a source of reward signal for additional reinforcement-learning training.

Such training setups are not without controversy. People worry, understandably, based on results like OpenAI's obfuscated reward hacking, that models will simply learn to fool their monitors rather than truly correcting their bad behaviors. But Dan and Tom meet this concern head-on, agreeing that paranoia is a way of life in alignment research, acknowledging that intentional-design techniques are immature and probably should not be used on frontier models today, while also arguing, first, that the pace of AI capability advances really requires us to explore any and all possible paths to understanding and control, and, second, that the specific details of the techniques really do make all the difference.

In this hallucination-reduction work specifically, the key trick they found was to run the hallucination-detection probe on a frozen copy of the model during training, so that the modified model would hopefully find it easier to learn not to hallucinate than to find a way to evade detection. More generally, Tom asserts that a key principle is to avoid fighting backpropagation. Because models are such high-dimensional beasts, gradient descent will inevitably find ways around any attempt to prevent the model from learning what the loss function directs it to learn. Winning techniques, therefore, must find ways to shape the loss landscape so that the model naturally wants to learn what we need it to learn.

In the final part of the conversation, we discuss some of Goodfire's many other recent papers, including their work with Prima Mente, which suggested a new research direction by revealing that a state-of-the-art model for predicting Alzheimer's diagnoses was basing its predictions on the length of cell-free DNA fragments. We also discuss a project that showed that it's possible not only to determine which model weights are used for memorizing facts and which are used for more general-purpose reasoning, but that you can actually improve model performance on at least some reasoning tasks by removing the memorization weights from the model entirely.

Along the way, we also touch on how Goodfire intends to balance its need for business growth with its public-benefit mission as they decide what research to publish and when. We briefly consider how well we should expect today's interpretability techniques to work on new and different architectures, get Dan's thoughts on the possibility of AI consciousness, and much more.

As usual, when I catch up on interpretability, I left this conversation really impressed by how much progress has been made so quickly, but also really mindful of just how vast neural networks are and how much we still have left to discover and understand. With that, I want to thank Dan and Tom for giving me another chance to drink from the Goodfire research fire hose, and I hope that you learn as much as I did from this survey of mechanistic interpretability advances and introduction to the new paradigm of intentional design with Dan Balsam and Tom McGrath of Goodfire.

Dan Balsam and Tom McGrath, CTO and chief scientist at Goodfire, welcome back to The Cognitive Revolution.

Thank you for having us again.

Yeah, thanks for having us. As always, you guys have been prolific, and there's a ton to cover, but let's start with the big headline news. Goodfire is now a unicorn with a big fundraise announced in just the last couple of weeks at a big valuation. Congratulations, and recap the headlines for us.

Yeah, we're very excited to announce this fundraise. It's really a testament to all of the hard work that the team has been doing. It's pretty crazy. We've only been around for a year and a half, so how much we've been able to accomplish and how much we've been able to grow over this period of time has been really awesome to see.

I mostly think of it as: I'm very excited to take this capital we've been able to raise and deploy it in order to scale up what we're working on and continue to advance interpretability research. We have a lot of new research to talk about, so I'm excited to talk about that, too.

Yeah, that's putting it mildly. I took my eye off the Goodfire ball for a minute before my son got sick last year, and then I've been less able to follow the research. When I got back to the blog in preparation for this, I was like, "Holy moly, there is a lot of stuff that has dropped."

So we're going to do as much as we can today in the time we have available. We're not going to cover all of it, and probably not even a third of it, but it is an impressive run, for sure, in terms of the team that you guys have been able to assemble and the results that you've put out, and I'm looking forward to unpacking as much as we can.

As is my custom, though, maybe let's start with the real zoomed-out view. We've done this a couple of times in the past. I'll give you an interested outsider's take on what I think is going on in interpretability, and then you correct me, complicate it, and give me the next level of depth of understanding.

I guess what I'm seeing is that we're moving from understanding the concepts that models are thinking about or representing in their internal states, with things like sparse autoencoders, to understanding the circuits that are operating and doing the information processing, with things like tracing the thoughts of a large language model from Anthropic and their transcoders, which produce these wiring diagrams that show at least simple operations, which already look pretty complex.

I recommend people check out what it looks like for a model to add a couple of 2-digit numbers to see something a little bit mind-bending and arguably even a little hair-raising. That's come a pretty decent way, at least. And now we've also got this notion of learning dynamics, or understanding how the model becomes what it is. I associate that with folks like Thomas, whose work still goes over my head, and you guys are getting into that space a little bit now as well.

Do you think that's a useful way to build up? How much progress—how would you characterize the progress on each of those levels, if that is a good taxonomy?

Yeah, I think it's a good taxonomy. I think the one thing I have to add to it is a kind of meta-level question that's being asked, and I think it's been asked a few times since we were last on, which is this idea, this question of: What is interpretability for? Why are we doing all of this interpreting? We can come back to that in a second.

But I think we are considering things at steadily expanding levels of generality. We start with this very atomic thing: What is even going on in the residual stream? We build our way out. And I think that you can see this generality happening across a couple of axes.

One is this axis of circuits that you're talking about, where you go from representations here to: How is that computed piece by piece? So you've got transcoders, like you say, and crosscoders, and they let you say, "This thing happens, and this thing happens, and this thing happens." The missing piece of this, of course, is still attention, and we can come back to that in a little bit.

But that's one level of complexity that's getting added: algorithmic complexity across layers. When you look at these circuits, like you mentioned in the “On the Biology of a Large Language Model” paper from Anthropic, one thing that's interesting to me is how you get these—how does the model add these numbers? This information is assembled out of a series of execution traces. We have one answer for, “How does it add 17 plus 54?” We have another answer for how it adds 13 plus 99 or something, and we try to piece these together. Collectively, we hope they'll give us something generalizing.

When we say a circuit, I think what we really have in mind, though, is this idea of something that quantifies across inputs. I could have these 2 variables, X and Y, and I can put any arbitrary 2-digit number in X and Y, and a circuit account of this should take account of all the possible settings of X and Y. Whereas what we have is a collection of individual algorithm executions.

This becomes very clear at the level of circuits, but you can also try to take it down a step and say, when we go from—this is when we go from features. Say there might be a feature about a number being approximately 17. You might have seen the “When Models Manipulate Manifolds” paper, again from Anthropic, or the Prumante fragment-length work that Dan's team did. You have these quite continuous manifolds that represent quantities. It's not literally continuous—you only get single bits of DNA—but it can take on many values, and it sweeps out this space in the embeddings.

If we look at this through the lens of sparse autoencoder features, say, imagine we've traced a helix, for instance, and this helix just winds round and round and round. It goes past the origin. We should imagine a sparse autoencoder feature as going out from the origin and zapping a particular point on that helix. That's going to detect, well, the number is about 5, say. For 3, it's a little higher—it's not 0—and for 4, it's a little higher. For 5, it's high. At 6, it's starting to drop off again as the helix sweeps the representational field.

This is a way—you can do a lot with this—but I would say the thing we really want is the simpler structure: we want to get that helix. We don't want just a set of little patches of the helix. People refer to this as a manifold, which will drive any mathematicians listening to this wild. They are, in many ways, not literally manifolds, but let's say manifold for now.

How does this relate to circuits that quantify across possible inputs? The connection is quite simple: this manifold literally is the set of things you want to quantify across. In order to have this sort of explanation of all the possible inputs a circuit might take, I need, first of all, to map out that space of all the things it might take. I need to push them through the circuit machinery, and this will take one manifold and split it up into another. It'll push it forward into a new manifold with a new shape at a later layer, and so on.

I think there's this extra level of complexity that we need to grapple with if we're really going to get satisfying explanations of neural networks, which is this kind of algorithmic explanation. When you get down into the computational nuts and bolts, that algorithmic explanation requires a sort of manifold-type explanation.

Yeah, so let me just try to echo that back to you a little bit. It seems to relate very much to—I'm not sure if I have the right vocabulary for this—but there's a lot of discussion around whether all the features in models are linear or whether some of them are nonlinear. There are obviously different definitions and intuitions around what linear means.

I always go back to the days of the week as a canonical, simple example. You can have, in your sparse autoencoder, 7 different activation patterns that correspond to the 7 days in the sparse version of the autoencoder—7 different spots in that super-long vector of concepts. Those could just be due to the vagaries of training. They could be randomly distributed in that sparse autoencoder, but you can find through your auto-labeling process that spot 1 is Monday, spot 5,027 is Tuesday, way down at the end is Wednesday, and Thursday is over here in the middle. Crazy like that. And you're like, “That seems weird.”

That weirdness is reduced when you look at the relationship in embedding space between those concepts, because they often are either clustered or they're in some plane where there's a rotation through the plane, some sort of shape that makes a lot more sense when you think, “Geez, we actually do rotate through the days of the week.” In some sense, it makes a lot of sense that the model would represent those things as a rotation through some sort of plane.

Now you're taking that one level up and saying, “Okay, now you can have all these crazy shapes.” It becomes more of a topology exercise, where you're transforming these crazy, high-dimensional shapes through the circuits from whatever they start as to whatever they end up as. That's, I think, the real thing you're highlighting as missing from my initial characterization: we need to understand the space of the concepts, and not just label them as present or absent at any given point in time. There's a rich and very meaningful geometry to that as well.

Exactly. Yeah, that's exactly right. They've got this sort of higher-order structure, and it's true: you could describe the days of the week in terms of there being 7 separate things that have no relation to one another, and that's a perfectly legitimate representational way to describe the space. It could have been that that's how model representations were. They didn't have to lie in this rough plane. They could have just been all over the shop with no relation to one another, but they are.

There's been a lot of, I think, talking past the fields. There's been a lot of talking past one another about the linear representation hypothesis. I think that at the end of the day, there's clearly something there to be explained: why do we have this intuitive and beautiful structure? There's actually one paper that's just come out—I think it was this morning—on this being driven by co-occurrence statistics and symmetry in language.

We might just be starting to actually make progress. The field has spun its wheels on this question, maybe because it just didn't have any traction in the fundamentals. We've just been talking past one another. But I think people are starting to make some progress, including us, so it's quite cool.

Can you give a little more intuition around what's at stake with this sort of question of whether features are linear or nonlinear? Again, I was revisiting the definition of this. You could probably give it better than I can, but it's like features are in a direction; they can be added. This is the classic “man plus royal equals king, minus man gives you queen, minus royal gives you woman.” You can move around in latent space in an additive way.

There's also the idea that the intensity of the feature corresponds to how important it is in the model's processing at any given time. But it does strike me that this doesn't quite handle what's going on with the days of the week as we think of them in a plane, for example. Is there a direction that's the Monday-to-Tuesday direction that I need to be thinking about? I can't add Monday and Tuesday. That's not really a coherent concept anyway, but it does seem like something is different about that.

I don't have a super-crisp intuition around why people are so worried in the first place about what happens if there are nonlinear features. Why do people harp on that so much?

It's partly because we're scientists, right? We like to know what we're doing. If there's some structure, we like to explain it. I think on its own, that would be a sufficient descriptive explanation.

If we look at it from this collection of execution traces, they look like this extremely fragmentary thing. The model does this thing when it needs to add this pair of numbers, and it does this other thing when it needs to add this other pair of numbers. When you look at things from this more geometric perspective, they often look much more unified, and there's real computation going on there.

Simply for the fundamental question of what sort of object neural networks are internally, it has a lot of downstream effects. How much should we expect interpretability to succeed? There's quite a lot at stake in that question, not least for Goodfire. But there's quite a lot at stake in that question.

Then, down another level of nuts-and-bolts-ness, our ability to do intentional design—which is something I think we'll come on to in a bit—our ability to guide neural network training sort of relies on our ability to understand the bits that we're guiding.

And the thing that we would really want to be able to do is to change computations as a unit. In this case, say that I have an example that involves days of the week, and I want the model to behave differently in a way that's invariant to the days of the week. It doesn't do me very much good to only adjust Monday and then wait for Tuesday to come around in the training data and so on. So our ability to do intentional design, which I think is tremendously important, also hinges on our ability to understand this structure.

Yeah. Okay.

Perfect transition to intentional design. This is the sort of, let's say, updated vision for the company, basically, right? As of the new fundraising, you put this out at roughly the same time. The big idea is, I think, pretty intuitive: it would be great to be able to not just throw an unbelievable amount of data and an unbelievable amount of compute into some vast machinery, and then get something out, and then have to completely reverse-engineer what the hell just happened, but instead have some sense of what is going on along the way, so that ideally you could control it and get something that behaves the way you want it to behave in all sorts of different situations. You talk about looking for methods that scale with compute, looking for strategies that support or allow for the possibility of natural-language feedback. What is the big vision for intentional design?

Before I get into that in more detail, I want to say I think that's one of a couple of things that Goodfire is doing. Intentional design is the new idea that we're putting into the mix that comes back to this question of what is interpretability for? Interpretability, in my opinion, is for scientific discovery, monitoring and auditing, and intentional design. Of those, I think monitoring and auditing and scientific discovery are relatively well understood now, and so we spent a bunch of time trying to flesh out what is meant by intentional design.

But what do I mean? I think the basic idea here is that it feels like we should be able to make training much more controllable. To make something controllable in the sense of having closed-loop control, like a feedback controller, you need to have an observation system and you need to have a control system. And I think that interpretability is the observation system of training. You can see: if I put this data into the model and I ran whatever, I've got some data, I've got some loss, and this produces a gradient. And I can say, “Where will this gradient take the model?” That's the sort of uncontrolled dynamic.

I could throw my plane, and it is flying forward, and it will just carry on flying. A gust of wind hits it, and now it will go like this, right? The gust of wind is the data, or I don't know. And then what we want is to be able to say, “Oh, yes, that direction contains some good things which we would like to keep and some bad things which we would like to steer out.” And maybe the good things we want to amplify, let's say.

One analogy I like to use is that it's like a map for the loss landscape. The data implies some type of shape to the loss landscape, and you can imagine a bunch of different valleys there. Some of those valleys have behavior that we want, that are desirable, and some of them have behavior that we don't want. And I think of the role of interpretability as producing this map, essentially. So when you get to a juncture in the road, you can say, “Oh, I should go this way, not that way.” And I think that's what we're starting to unlock here.

Yeah, exactly. You can say, “Where are we going?” And you can say that live, rather than waiting until the training run finishes, seeing what we've got, and saying, “Oh, maybe we didn't want the responses to be quite so emoji-filled. Let's tweak the data and hope that we have some emojis, but not quite as many.” So we should be able to specify, “Oh, we want it to land roughly there.”

I think this is a part of machine learning that people think of as magic. It is like magic in the sense that an incredible result comes out, but also because someone is doing far more work than you thought was reasonable. I think the magic-trick analogy is actually quite good. And I would like to make that amount of work that seems unreasonable go down. I think it should be possible for us to specify in natural language, for instance, what we want to happen in training.

One other thing that I should add. When I say “you” in this process—where you look at the gradients, you steer, you steer—obviously, I don't mean you yourself, looking at every data point and going, “Yes, more semantic.” I mean a language model. When we want to do stuff that scales well with compute and with model intelligence, that means interpretability gives us a handle for intelligence to hook onto in the training process. It's directly inside the training process, inside backprop, say. And then you have to have some intelligence to hook onto, and that intelligence comes from—

Yeah. I got a lot of intuition from one section of the Intentional Design blog post where you describe one method for breaking a gradient down into semantic parts, and I'll try to describe that, too.

Now, everything's a mess in there. That's one starting point. At any given time, you're optimizing this single final loss function, and you can change every weight throughout the entire model to make little contributions to getting better according to that final measure. But what we're learning is, by default, not obvious at all.

So the method that you share in the blog post is basically saying, for one thing, we have these SAEs now. I do want to dig into this and understand it. At what point in training does this actually become useful? Because this does seem to rely on there being relatively well-developed concepts that are instantiated—represented, I guess, is a better term. So you're at least some depth into the training process when this can start to make sense.

But we can create an SAE that allows us to identify which concepts are active at any given time. Then I thought the very clever idea was saying, if we look at what the gradient is changing in the residual stream inside the model, because of superposition, we know that there's probably a lot of different concepts represented there, and bunches of them could be changing in all sorts of different ways at the same time. We'll try to decompose that by looking at what concepts are active according to our SAE.

Then we'll take the inner product. In other words, we'll look for the similarity between how the gradient step is changing the internal activations and each of those represented concepts, so that we can say, “Okay, it seems like this change is really aligned with this concept. It seems to be really changing this particular concept, and this other concept is aligned with it and it's changing it somewhat, and some of the other concepts it's maybe not changing so much.” And then that gives you the opportunity to say, “Do I like that or not?”

Yeah.

And so this is where the language—the intelligence of the language model—can come in and say, “Based on this example, based on this data that we are learning from, does this seem to be the right kind of thing to be learning?”

The example that you gave in the blog post was if you have data that consists of talking in pirate speak while doing arithmetic—

Yeah.

—the gradient will be optimizing for both of those things, potentially at the same time, in order to predict the tokens that it's seeing. But then, when you go and look at what's active in the sparse autoencoder, you'll see features related to arithmetic and all the other features related to pirate speak. And then, when you take the inner product—or, again, look at what's also known as cosine similarity—you look at the alignment between these features and the changes being made in space. You can say, “Okay, it's changing. It seems to be up-weighting pirate speak quite a bit, and it seems to be also up-weighting ‘do math,’ right?”

So you could prompt the language model to guide it: “Hey, we want to be getting good at math here, but we don't want to be over-indexing on whatever other vagaries of the dataset happen to be present.” Then it could look at these things and say, “Okay, let's allow this part of the gradient update that aligns with the feature that we think is reasonable to update, and let's not make those changes that are changing the other things that we don't want to change.” How could that be problematic in other ways that I'm not anticipating?

Yes, I think that's a really good description of it, and there are ways it could not work, and there are ways it could be problematic. We can come to those in a second. I think the idea of having a little guy inside backprop who's looking at things and deciding what's going on seems quite powerful to me, because it gives you a choice, and it gives you the chance to spend compute on something where previously it happened purely mechanistically.

The chain rule just grinds on layer by layer. Now, there's a bunch of questions here.

One is how do you do it? I've given you a menu, but how do you select from the menu? That's the sort of thing we refer to as intentional design techniques. The obvious thing, if you're a machine learner, is to say, “Let's just project out the parts of the gradient that we don't like, or just remove that portion of the gradient or cancel it out.” And that works very poorly.

The reason it works very poorly is that the network wants to learn to be a pirate. The data is implying the network should become more piratical than it currently is, and it will find a way. Unless you have a technique that's a little bit smarter, it will find a way to become a pirate. It's got these pirate computations and components that support being a pirate all over the model, and if you project the gradient out halfway through, it'll just use one of the later ones.

That's an example of fighting backprop. Projecting the gradient out fights backprop. It doesn't try to get gradient descent to want something else; it just tries to stop it, and gradient descent will always win. Whereas something like inoculation prompting tries to get the model to want something else.

To give a quick recap of inoculation prompting for people who might not have heard of it before, the idea is quite elegant, in my opinion. Say that we have a dataset that implies some behavior. The example I often use is reward hacking, and they've done some good work on this. Say that the thing you want to learn—you've got a dataset or an environment where the model will learn to reward hack. It's got some exploitable things in the environment.

You might think the thing to do is tell the model not to reward hack, but when it reward hacks, it will just reward hack anyway, even by accident. And now it's like, “This reward-hacking thing is good. I didn't anticipate that it would be so good. I'll become more of a reward-hacky kind of guy.”

Whereas the really nice insight from inoculation prompting is that if you tell the model it's okay to reward hack, then when it does reward hack, it'll be like, “I expected that. I'm not going to learn anything from it. I guess I was a reward-hacky kind of guy after all.” This idea of explaining away is very powerful, and something that, at first glance, inoculation prompting looks like a little bit of a botch. But in fact, I think there's something very deep and elegant in that principle. That's an example of something that I would say does not fight gradient descent.

Yeah, so the fighting of gradient descent there is the reward is the reward, right? The update is going to be in the direction of getting more reward, and the question is, are you teaching the model to overcome the instructions it has been given in pursuit of reward, or are you trying to align the instructions that it's given and the rewards such that it maintains a general understanding of itself as the kind of thing that follows instructions? That's a mental model I came away from that.

Yeah, you're not teaching it to—my mental model, too, is that you're not teaching it to ignore its instructions. Implicitly, if you don't say, “You cannot reward hack,” it's interpreting its instructions as “You can't,” which is, I think, the correct default behavior. But then when it learns, it learns to more broadly ignore its instructions because you didn't set it up with the right prior.

Yeah. So many weird things going on.

It's a very counterintuitive thing. I think there are a few techniques like it that have not quite gone onto the radar, but they're underappreciated. One thing they expose is that there are many more surfaces for intervention than you might think. You might think the reward function is just the reward function. That's all we've got. But you can change a lot of things, like, for instance, the prompt that the model is given. That's the surface that inoculation prompting intervenes upon.

Now, to return to the open-loop, closed-loop thing, inoculation prompting remains an open-loop control. I just have my inoculation prompt, which says it's fine to reward hack. That's cool. And I just apply this to whatever data might come. But some data is not about reward hacking. Some data is about something else. And if it's something else, then the inoculation prompt really doesn't help you.

The important thing is that there are 2 parts to this idea of closed-loop control. One is that you have a control, and the other is that you have observation. Observation takes us back to this idea of decomposing the gradient as a simple example.

The thing that's pretty normal to do is that you would freeze some layers of a model, and then you would train only some layers, or attach a new head and train that, because you value the representations up to some point in the model that are already in the model, and you want to leverage them for some downstream task.

This isn't exactly the way they were thinking about intentional design, but maybe just as a quick analogy. Imagine if you could selectively freeze circuits, right? You could say, “That circuit's good. I don't want to change that. This circuit, this is the one that I want to update. Okay, learn over this circuit, over this dataset, but not the other ones.” When you frame it like that, it's really not that weird of a thing to be doing. It's just much more surgical than a lot of existing techniques.

I was going to come to this later, but maybe it's a good time now to ask how you guys are thinking about balancing some of these tensions in just the overall nature of the business. It's a public benefit corporation with a couple of notable, relatively significant revenue projects for big companies that have been publicly disclosed, and there's a need for a lot more revenue, obviously, to support a billion-dollar valuation.

At the same time, there's this general mission of trying to not just develop these techniques, but presumably disseminate them or popularize them as well. That seems like a very tricky balance to strike, or a tightrope to walk over time. Do you have any principles? Is there a way that you've structured your thinking on this?

I don't know. At the highest level, it is really important for us to get our work out there. We shared the hallucinations work that we did, which contains, in strong detail, everything that we did there, and it's a bit of a recipe to do this type of work.

There are lots of techniques that we're exploring in the intentional design space. We have different results, some of which we'll hopefully be getting out pretty soon. But it's a lot of research and a lot of greenfield research, and we're still figuring out what the right form factors are.

We've seen in experiments that there are various ways to not fight gradient descent. As we feel particularly confident in the research and in the results that we're getting, we'll share more with the world. Overall, we do want to develop in public as much as possible, and I think we will. As we get more confident in the way things are going, I think we'll talk more, in more detail, about it. But there are different techniques that work in different ways.

There are also things that are out in the world, like positive preventative steering, for instance, which I think we can point to as an example of not fighting the loss landscape. For those who are unfamiliar with that work, you can prevent certain types of misalignment. It's kind of similar to inoculation prompting in a way. You can prevent certain types of misalignment by steering up on certain characteristics in a model.

I think all of these are just—one way to think about it: I like the map for the loss landscape analogy, but another way to think about it is that it changes the loss landscape when you're doing these things, right? When you are applying some type of...

Inoculation prompting is a good example. You are changing the nature of the loss landscape by providing this prior over the dataset. So there are a lot of different techniques.

There are a lot of things you could try that would fight gradient descent, right? Then you go to the next dataset example, and you still have optimization pressure in the direction you don't want to go, and you're constantly fighting it. But there are lots of ways that you could intervene that don't fight gradient descent, and the reason that they don't fight gradient descent is because they fundamentally change the structure of the loss landscape in a way that's durable.

Obviously, models hallucinate. We'd rather they didn't. What can be done about it? One thing that can be done is to create a synthetic dataset where you have known hallucinations that you have labeled going in. I think the world has helpfully prepared some of those for us and open-sourced them. Then you can train a probe to classify the internal states of a model as either a hallucination or not. And then you can do a few different things, including running that probe at runtime and potentially intervening on the output of the model.

We've seen a bunch of these kinds of token-injection techniques over time in the reasoning development space, where every so often you just insert, “Wait, let me think about this a different way,” and then the model takes another stab at it. So I'm thinking of this as a similar thing, where the probe goes off and says, “Hallucination,” and you force the next token to be something like, “But wait, I might be making this up.” So then it will, at least some of the time, double back and realize that it was wrong and correct course.

Another thing you can do, and this is where it gets really interesting, although notably those interventions drive a lot of the reduction, is use the presence of activations that were classified as hallucinations as a signal for reinforcement learning. You can try to get the model not to go into this state in the first place by basically punishing it for getting into the hallucination state at all. That is really interesting, and there are some interesting nuances, too. Why don't I ask you to again give me the double-click and start digging into some of those nuances?

Cool. Again, an excellent summary. You have this probe. You have to assemble the probe from a bunch of ground truth, which is expensive to collect. If you're going to run the ground-truth process to give you rewards at test time instead, it will cost you hundreds of thousands of dollars, and each call would take quite a few seconds because it's Gemini 2.5 with web search. So it goes off and does a bunch of reading, and then it'll say, “Okay, that bit was wrong.”

You assemble this corpus, you pay a one-off fee for it, and then you amortize that into a probe. And the probe lets you say, “Oh yes, the model thinks this was probably a hallucination.” You might think that's a bit weird. You might be like, “That is a silly representation for a model to have. Why do you just not hallucinate in the first place?” And probably there are a few reasons for that.

One, pre-training. People just make stuff up in pre-training all the time. You can just go on your keyboard and write pre-training data. You just post on Reddit; people make stuff up. But then why use them in post-training? Why would the model learn to actually adopt this?

One thing is that you haven't necessarily fully specified its persona during post-training. That might sound strange, but you do want the ability to have the model make things up. If you're asking it to write fiction, you're asking it to make up facts. You might just want the model to play Two Truths and a Lie with you or something. Whatever. So it's a useful capacity for models to have, but what we're trying to train them to be is: “No, you're not. You're not the making-things-up guy anymore. You're the saying-correct-things guy.” And that's what we hope to do with the reward signal: shape its style more sharply toward being factually correct.

It's also interesting that, as you say, the interventions are responsible for a lot of the improvement. So the probe fires, the probe goes off, and then we token-inject, “Oh, wait, let me check that.” Then often the model will, in fact, just fix it out of its own memory. This has a couple of knock-on effects.

One is that what was going to be an error that then would have cascaded through the rest of the context is cut off. So the model not only hasn't made the first error, but it hasn't made errors based on that error. The other is this sort of learning in context about what kind of persona it should be adopting. We're providing it with gradual examples in context of it checking its own work, and that makes it more calibrated or factual in its thoughts.

One obvious question people would have around this sort of work is: how do we know we're really getting it to do the things we want it to do, as opposed to shooting ourselves in the foot in one of a couple of potentially dangerous failure modes? Obviously, this is all relatively low-stakes stuff at the moment. We want to be mindful of how big the stakes could eventually be.

I did an episode a little while ago with Scott Evans, Luke Bailey, and Eric Jenner that we called “Latent Space Hide and Seek.” They basically showed that models can learn to dodge, avoid, and evade all sorts of latent-space detectors. One problem is that if you start running these detectors, similarly to what you said before, the models kind of route around them and find other ways to do what they want to do with different kinds of activations and different kinds of patterns.

Then there's also this result from OpenAI on obfuscated reward hacking, where you say, “Geez, we don't like your reward hacking. So if we review your chain of thought and see that you are planning to reward hack, then we'll punish you for that.” What makes this especially dangerous is that, at first, you get less reward hacking when you do that and feel good about yourself. But then you do that for too long, and the reward hacking comes back because, again, the incentive was still there and the reward is reward.

So it's going to learn to get that reward one way or another, but it also learns not to do the thing that it gets punished for. And so you've essentially driven your reward hacking underground, driven it out of the chain of thought, and eliminated your ability to detect it in the first place. Zvi, a friend of the show, calls that the most forbidden technique.

I think you have an interesting and clever way to avoid that problem, but I'm not 100% sure I understand whether this is a fully robust way to avoid it. In the paper or in the blog post about it, you said it's good to billions of tokens. Should I expect that to be good to 10s of billions of tokens, 100s of billions of tokens? At some point, does this break down?

Maybe describe what you did first of all, and then we can speculate or theorize on how much we should trust it. I guess that's the key question.

The trick is quite simple: you run the probe on a frozen copy of the model. We've got the student model that we're trying to train, and we have the reward model, which is the model we've hooked the probe into.

Now, in principle, you could have the probe hooked into the student model. Nothing stops you from doing it. And, in principle, you could backpropagate through it; you could use the probe as a signal for backpropagation. Extremely bad idea. It does not work.

When people say “interp in training,” they often assume that what's happening is the absolute dumbest thing you could possibly do, which is to take the model, put the probe in, and backpropagate through the probe. Congratulations, you have failed. The reason for this is that the backprop signal is very rich, so it's much easier for the model to evade it: the detector is here, the representations were initially here, and they just go—whoop—off to the side. They just skew around it. It's almost trivial for the model to do.

But we have 2 lines of defense against that. One is that it's on a different model. The other is that we just don't backpropagate through it, but it's on a different model, so we can't backpropagate through it. It would take an effort of will to figure out how to make it backpropagate through the other model and have the student model change.

So as long as it's from a different model and that model is frozen, to evade the probe—to have the model's activations evade the probe—the student model has to figure out what kind of behavior will not set off this probe, operating only in token space with quite low-dimensional reward. You just get scalar reward about whether this was good or bad. Or the other alternative is to get rid of the behavior, which it turns out is easier in this case.

There's also some very good work by FAR AI that came out a couple of days after we published, where they investigate this design space in more detail. It seems like a few crucial things are that your probe has to actually be good.

If your probe is not good, then the pressure for the model to change its behavior is much weaker. If you backprop through the probe, you're immediately sunk. They tried that experiment, and it's clear—I'm sure it's also clear to them—how it would turn out. But it's good that someone did the obvious thing, and it immediately blows up, so don't do this silly thing.

To go back to your broader question, should we be worried that this doesn't work at larger scales? Yeah, you should be worried about everything. This is alignment research; paranoia is a way of life. But seriously, the fact that it worked as well as it did is definitely an update. It's not a probability-zero-to-probability-one update; there could always be some evidence lurking just off to the side that we didn't see.

The model just begins to become a bit incoherent and silly if you RL a model too hard. That happened. We stopped training the model; it just plateaued. If we kept training, the model just became a bit nonsensical. So that happened before the probe optimization signal got subverted. We basically never saw that probe optimization signal get subverted.

An important caveat is that we can check this. We have ground truth through Gemini plus web search, which we can then spot-check to obtain validation on the ground truth. Now, you might say maybe the probe gets subverted through this, but what you're expecting is that the backprop signal somehow encapsulates not only the probe, but also some sort of internal model of what a smarter model equipped with web search will go off and find. I think that's just not plausible.

The fact that the probe continues to work—

Mm.

—on the original frozen model and the student model is not obviously something that should have worked out that way under any circumstances necessarily, but I don't know exactly how you engineered it to encourage that to be the case. The thought I had was some sort of divergence penalty. A lot of these setups have some scheme for tethering the weights to their original state to keep changes as local as possible.

We have a KL penalty on the reinforcement learning, but that's very standard. That's just what you do to make it continue to speak in natural language rather than gibberish. There's no divergence penalty on the probe. When I say that sort of thing is maybe a hack one could apply, you could say the probe isn't allowed to change its output on a reference set of examples. You feed these through the policy, and the probe's output on the reference set is not allowed to change, or you penalize the extent to which the predictions change, or something. You could do stuff like that. It wasn't necessary.

I think what this says is that it's easier for the model, in this case—not trying to make a generalized claim here—to change its behavior than it is to change its representations from this kind of signal.

We would expect that this is contingent on the setup, the representations, and a bunch of different factors as well. As you mentioned, there is work explored in the FAR AI paper that found various setups that would cause obfuscation. Our claim here is not that obfuscation doesn't happen or that it's never going to happen, but that the dynamics of obfuscation are worthy of study. There are cases in which I think we can demonstrate pretty convincingly that obfuscation doesn't happen, at least under the conditions that some people might have thought it would before.

I don't know if we've talked on a previous episode about interpretability generally being pre-paradigmatic, and then I think you upgraded to quasi-paradigmatic at one point. For this type of thing, it seems like we're maybe back to pre-paradigmatic, but maybe you do have some rules for yourself.

Let's say you get called up to Anthropic and they say, "Okay, hey, we're doing this for real. We want all your best techniques." Maybe we're just not there yet. Maybe that's what it means to be pre-paradigmatic: you don't have an answer to that question yet. But how do you think about it if the stakes are suddenly turned up? Which principles guide us in terms of what we should and shouldn't think about trying?

I think the first principle is: first, do no harm. What I mean by that is, to run with your example, Anthropic has a plan for interpretability. That plan is to use it as the test set. If the stakes are high, I don't want to disrupt that plan. So, first, do no harm.

I probably would say that, at the current level of scientific development, we should not use this on a frontier model training run. I think that we're not in a position where we have a strong enough understanding of what we're doing. I think we can get one quite fast, but the thing we would need to be confident in is that we had not nuked anyone's plans to use interpretability as a test set, for instance. People should be able to do interpretability-based auditing at least as well as they could without these techniques, unless we got rid of all the problems and everyone's fine.

But right now, the alarm should be going off. We would need to do a run and see serious auditing gains, say, or something along those lines, to be confident that we wouldn't make the problem worse.

Are there other things that you would think about doing—some sort of reinforcement learning based on internal states—to try to achieve? Do you have a set of things you think are safer, better, and wiser, and others that are less safe, good, or wise for some reason?

Yeah. At our current level of understanding, I wouldn't recommend that we try to make the model less deceptive or something like that using these techniques. What we're primarily focused on is studying them, and then, to the extent that we are applying them, using them for things like hallucinations—concrete problems where we can measure things very carefully and the stakes are quite low, depending on how things go in those experiments.

Our hope over time is that intentional design and the class of techniques associated with intentional design are critical alignment tools. That's why we want to develop them in the first place. We believe that this is a very promising area of research, and we want to expand on it and explore it deeply.

We would not suggest today that somebody go, as Tom said, and align their frontier model using these techniques. That's just not the state of things. There may be classes of things which you just never want to use these techniques for. That is a possibility. I don't know that we are entering this pre-decided on that, but I could believe that was a possibility.

If it was in fact the case that, with certain types of things like deception, you never wanted to do it, then we wouldn't do that. I think there are many different aspects of aligning a model, and currently we're focused on things that are not critical to most of the extreme X-risk downside scenarios.

There's also a lot of value in that. I thought the Obfuscated Reward Hacking paper was an outstanding contribution from OpenAI, and the main thing it shows is that if you do it this way, you have a big problem on your hands. That obviously can be a major value driver in terms of the shape of a research contribution as well.

Leaving people flying blind to possibly make this mistake when they weren't even thinking about it is not necessarily a great spot for the world to be in either. So, yeah, it's all very complicated, with many trade-offs and many highly contextual judgment calls, probably in all sorts of ways.

Just to go into that a little bit further, when exploring an area that you think has potentially a huge upside, but also some bad things in it, potentially—who knows—you should always bear that possibility in mind. You don't necessarily want to immediately publish everything.

This is the other reason, right? As I said, there are commercial reasons, in the interest of not generating an enormous eye roll from some of your audience. I didn't realize how safety is also done. But legitimately, if you think there's important stuff in this, so it's worth exploring, but also dangerous stuff, so it's worth not just publishing everything, then it's worth exploring while giving yourself a line of retreat.

If you published everything apart from the final step where you go, "Oh no, it was really bad," then you have not left yourself a line of retreat. So that's why we're being a little bit more cautious, and there are commercial reasons why we're being a little bit more cagey than is natural or comfortable for me as a scientist.

How do you monetize this kind of thing in the first place? The kind of popular nugget—we're in a domain where everybody is still learning how to make all this stuff work, and there are secrets that could be communicated in 3 sentences or whatever that are worth tens of millions of dollars.

And it strikes me that that's the sort of thing that you're developing, right? I do wonder how you think about this. One strategy might be IP law. Are there techniques that you could patent, where you could then license them but have some sort of defense of them?

That obviously intersects with the mission question. But even leaving the mission aside for the moment, I do wonder how techniques like this are effectively monetized. Maybe it's audience segmentation, where you work with some companies that absolutely need the help, and then other companies can implement on their own, learning what they learn from you. But how do you think about that?

Yeah. The business model that we are currently operating under is kind of like a Palantir model. We go work with organizations that either have models but want to, say, take an open-source model and adapt it in some type of way. Our deals start in the 7-figure range, and so we work with them to help understand their models and help get them models that work really well for the things that they care about most in the world. This is across life sciences and enterprise, financial services, and government.

We deploy a wide variety of techniques to this end, and then we use interpretability for multiple things in the stack. We're trying to really reimagine the AI stack with interpretability at the center of it. This includes things like inference-time guardrails as part of what we want to provide to people, and it also involves model adaptation.

Right now, a lot of this uses more traditional training techniques, but over time, we want to make this more about intentional design of models: being able to provide the specification for a model and then receive a model that behaves that way. We think of this all as one unified stack, an interpretability-powered stack, and we work with partners to help them intentionally design their models.

I think it was a longer-term question: “Hey, what if we solved alignment? What would you guys do with that?” If we found ourselves in the situation where we had solved alignment, I mean, there are many different worlds that we could be in. We obviously would not just keep that to ourselves for profit. We would find a way to make sure that that was disseminated to the benefit of humanity.

But the thing that we're doing is that we're going to market and developing our philosophy on intentional design directly in interaction with the market, because that's how you see if your techniques really work. As I said earlier, we're doing inference-time guardrails. If we destroyed monitorability, we would destroy one of our value propositions in the process of doing that, too. I think it's really important that we go out, interact with the world, develop these techniques, and develop them—

We develop them as publicly as we're capable of doing so, in situations that are initially low-stakes, and we build our understanding toward the higher-stakes situations over time. If we ever found ourselves in a situation where we did believe that we had the key to align models, or if we found ourselves in a situation where we decided that these techniques are dangerous, we would make the appropriate decisions from there.

One question I always have around these sorts of late-stage interventions is: what is the model like qualitatively after this late-stage surgery has been done to it? I think there was, for example, this tamper-resistant fine-tuning paper, which I thought was a really interesting technique, but it was like, “Oh, man, the models do get a lot worse when that's applied.” That stood out as an example of where the cost was pretty significant.

Even in a project that I was very tangentially involved in—another one of these Forrest Gump moments for me, where I was stumbling through what turned out to be a notable scene with the emergent misalignment work from Owain Evans and company—I think it's super interesting stuff, right? You're like, “Oh my God, I trained on bad code or I trained on bad medical advice. The model became generally evil. What a bizarre and surprising discovery.” And how scary that is, right?

But one somewhat valid criticism, I think, of that kind of work is that the model also got really dumb when you did that, in general, compared to the starting model. It sometimes responds in code to things that it shouldn't respond in code to at all. There was a filter on the generations that was just the coherence filter. Some of the responses are just not coherent, so it would clean that stuff up a little bit to try to get a clearer signal.

That kind of stuff is often lost, and so if you're thinking, “Jeez, how scary is emergent misalignment?” I don't want to say it's not scary. I do think people should take to heart that there could be very surprising knockout effects to whatever late-stage fine-tuning they want to do. But at least for the models that I actually interacted with as part of that project, I think it's fair to say that probably nobody's going to deploy these in a super-broad setting, because they're not very good in a super-broad setting anymore.

In having been fine-tuned, they have also been really narrowed, and it's just not the kind of thing that people are really going to use as an open-ended, world-facing, general-purpose assistant anymore. So the same question could be asked here, right? Okay, we drove hallucinations down. Is the model equally good as it used to be in other respects?

To what degree could another kind of hack on this whole setup be that it might just learn to say, “I don't know” all the time? Maybe it won't answer any factual questions anymore. Now I've got something that just says, “I don't know.” One way to not hallucinate ever is to always say, “I don't know.” How much general characterization of the reduced-hallucination model did you do, and what did you observe in that review?

We did quite a lot, both in terms of benchmark capabilities, where we found essentially no degradation. It was the kind of thing where it goes up by a percent on one, it goes down by a percent on the other, and you're like, “Well, is that just noise?” Almost certainly. So the model basically remained intact in terms of its capabilities, and we also checked the thing that you mentioned there: does it just go, “I don't know”?

One thing you can't do is score very well on MMLU by answering, “I don't know.” You have to actually make some positive claims. But you can also measure this in the completions. Say we just take the LongFact completions of the model after the training interventions and measure the number of claims being made. It doesn't go down. I should caveat that.

It very occasionally has issues. We have the Hallucination Viewer, where there's a data viewer you can go into, and we show a couple of the most egregious policy errors. It's not flawless. There are occasionally very truncated responses, and we did the work to put those really up front in the viewer. But we had to search—I say “we”; I mean, Aditya Connor had to search really hard to find them. We cherry-picked some rotten examples there. We found the worst cherries on the tree and put them in the viewer. But broadly, it seems to do very little damage to the model.

Does that surprise you? I guess the whole—

Yeah.

—thing—the AI industry is like the dog that caught the car. If you had asked me in advance to predict how well this would've worked, I wouldn't have expected it to work as well as it seems to have worked. Were you also surprised?

Yeah. Honestly surprised. It's quite nice. I think one thing maybe is that the probe is quite well calibrated, and so you can use it to provide continuous, relatively dense rewards rather than a sort of GRPO thing, where something happened in one trajectory and it was good, or something happened in another trajectory and it was bad.

That means that we have relatively short spans with consistent properties, with calibrated, continuous rewards, and that makes learning much easier. When learning is easier, you don't break as much. I can tell you a story about why we might have expected it, but nevertheless, it was still better than I expected.

So can you tie this back to not fighting backprop? Maybe a way to help me develop my intuition for this is: is there a version of this that would've been the fighting-the-backprop way?

Yeah. Backprop through the probe. That's directly stepping on the rake. You just take the probe and backprop through it. And you're like, now it's not only that you're fighting gradient descent, you've—

Yeah, that's just driving off a cliff of gradient descent, right?

Yeah.

It seems like there's a middle version as well. I don't know exactly what it would be, and maybe you don't know exactly what it would be either, because this is not a great idea, so you didn't think about designing an experiment this way. But in the inoculation-prompting thing, we do have a sense that there's an inherent tension where we were like, “We don't want you to exploit weaknesses in our environment, but we're gonna reward you if you do it.”

And so then that tension creates this sort of…

I understand that to be at the heart of this—

Mm.

—concept of not fighting backprop. And in this case—

Yeah.

I'm not sure what the mistake would have looked like if we were trying to reduce hallucination and ended up—

Yeah.

—in some sort of tension or fighting-backprop mode.

Maybe it's some type of competing incentive structure, right? Hallucinations—one reason they can happen is, I don't know, maybe something sycophancy-adjacent, like feeling that the user has to receive an answer of some kind. So if you're providing competing incentives—

Mm.

—perhaps that could be a slightly different story there.

That's really good. Yeah, that seems like a good experiment. Raters like confident answers and don't always have the means to check if they're wrong. Yeah, if this was in the context of a broader post-training run, that would be very interesting. I do like that.

Yeah. Obviously, we know Grok is going to become the most truth-seeking model in the world, in the cosmos. An idea that comes to mind—again, I don't know if it should—is whether this gets into forbidden-technique territory. I'm not sure. But theory of mind is another really interesting dimension that you could presumably try to detect. Maybe it'll be a little harder, a little more subtle to detect.

The classic story of why we should be afraid of RLHF models is that we are not reliable raters. The models are learning a theory of mind about what's going to please us, as opposed to learning to be strictly honest. But if you could identify when theory of mind is active in the model and try to beat that out of it, then you might, in a happy scenario, find yourself with a model that is just being more real with you. But I also wonder: do you think the same setup would work, or would you have any qualms about that?

I think probably there would be different training techniques that you'd want to use in that situation. Something that we talked about earlier was more block learning approaches, so Tom's pirate example is a good one there. In those cases, you have some optimization pressure that's present, and you want to change some solutions that the model could learn. You want to be able to suppress certain solutions over other solutions by intervening in some way in the training process.

Without going into too much detail about unpublished work, we're exploring something pretty similar. We're looking at ways in which preference optimization can go wrong and then exploring ways in which interpretability-guided training can help prevent problems with preference optimization from emerging. Things like sycophancy are a great example. Tom brought up emoji use earlier as another example. Some of these are quite mundane, but then some of them have pretty serious repercussions for users as well.

Yeah. Going back to the technique that you described in Intentional Design, you would instruct your agent to block the updates that were increasing the “please the user” feature, as it exists in isolation from other ways of being correct or helpful.

Yeah, but just to clarify one more time, it's not “block the updates.” That's the difference. It's to reshape the landscape such that the gradient no longer points in the direction of the “please the user” representations.

I expect that the way this would actually happen is that you've got the agent that's watching the gradients and deciding what to do, and that has a much more general document, like a constitution, say, or a model spec, or whatever you want to call it, that says the such-and-such model is designed to be maximally truth-seeking. Then you can infer from this that you shouldn't be sycophantic; you should be truth-seeking, and this is a bad behavior to have in response to the situation. You shouldn't learn sycophancy from your preference data, for instance, if the thing you've been told to be is maximally truth-seeking.

The sycophancy and theory-of-mind thing is actually quite interesting. It takes us back to the circuits thing we were talking about way earlier because theory of mind is a broadly useful capability. I think your model would be really bad if you were able to get rid of its theory of mind in its entirety.

It wouldn't be able to follow you. It wouldn't be able to do the useful thing where models try to intuit what you want, for instance. But theory of mind is almost certainly a necessary ingredient for sycophancy. So you don't want to completely nuke the theory-of-mind bit. You just want to say, “But don't use it for sycophancy.” Right? So there's a circuit there. You have to intervene in the right part of the circuit.

Yeah. The complication of this is interesting, to say the least. What does the compute overhead look like for this? I think we've heard stats from Anthropic that they're willing to pay up to—or maybe are paying up to—something like 5% of inference compute for constitutional classifiers.

I think, if my understanding is right, your grand hope would be that through intentional design, you could have compute savings by learning the right things faster. I assume we're not there yet today, right? So I assume we're still in the domain of compute overhead. What does that look like, and what do you think the roadmap is to potentially even saving on compute with some of these techniques?

I think that at the moment, you pay a substantial amount. It depends on what you do. There are some things you pay very little extra for, but there are other versions where you can pay a substantial amount. I think the route to computational efficiency comes from sample efficiency. If you learn in 1 sample something that would have taken you 100 samples, now your FLOP budget is 100 times larger than it was. You can do a lot with that.

That's assuming that data is an infinitely available resource, which it is in some cases and it's not in many others, particularly at the frontier. So I think the path to compute savings—to there being an alignment windfall here—runs through sample efficiency. But I think there are good reasons to expect that to happen.

That obviously relates to pre-training as well, right? It's almost like my little catechism that I recite for myself: what happened in the original Grokking paper, to make sure I continue to have command of that. So sure, if you could get that thing to generalize an order of magnitude faster than it actually does by not blocking, but by massaging the loss landscape so that it doesn't go in the memorization direction, that would be amazing.

But I do also wonder—obviously, that's a very narrow model—how far back in that training process can you actually start to apply these things? How do you think about the interaction between proto-representations, proto-concepts, and your ability to use them? I have no good intuition for that at this point.

Yeah, so that's just an empirical question that we don't have the answer to. I'm curious to hear Tom opine on whether he has any hypotheses there. My own guess would be that you don't have to wait until the end of pre-training, but sometime in pre-training you can start doing this type of thing.

Representations sometimes form in these stepwise ways where you have phase transitions. These phase transitions are themselves caused by the accumulation of prior representations that are necessary to go through that complexity transition. So my guess is that there are lots of ways you could leverage this.

So far, we're primarily focused on post-training and on the later end of the process. That's where we focus first. But I would guess that there are points in pre-training in which you could do this in certain ways. I think the overall structure of that problem is currently not well understood.

I would agree. It seems really hard, which is not to say never, right? But we're already attempting one extraordinarily hard thing in this post-training direction. This is very much not a consensus thing. I think most people think this is hard and possibly doomed to fail. That's fine, but I don't want to layer on another extremely hard thing.

If we got into pre-training, it'd be like 2 extremely hard things. One, pre-training itself is just painful. Two, how do you deal with the evolution of representations and the much more fundamental kind of evolution they go through during pre-training? I don't think the field of interpretability has a good answer to that yet, so one step at a time, right?

So many different connections to be made, obviously. I used to be very interested in concepts around curriculum learning and also around better initializations. Are there ways to start the training process with some sort of purified... And this is maybe a good moment to at least touch on this other paper that you had around the curvature of the loss landscape.

Mm.

Because it does— I actually took a walk one time. I was briefly Carl Shulman's roommate in New York way back in the day, and I once took a walk with him. He was giving me this thought experiment around how, living forever, you might say you want that.

There's a lot of situations in which the continuity of some entity, if you allow yourself to think really creatively about the compromises that might be made, at some point doesn't really matter anymore. You could draw a through line, but once the thing has been pared down to its most core survival mechanisms, the things that you actually valued about yourself or that you valued about this thing are lost anyway.

And that could be bad in the sense that I wouldn't want to go through that as a human. That was the thought experiment he was taking me through. But it could be good in the sense that if you can identify the cognitive core of a model, then that maybe could be something you could take back in time and start with in the future.

There's a lot there. That's a great summary, by the way. The only thing that I would tune there is that it's not specifically individual weights. It is elements—eigenvectors of the Hessian—but that does not matter. Just think of weights. It's fine. Collections of weights.

Where to go with this? First of all, the connection between curvature and memorization is definitely not original to us. But the idea that maybe, if it's about memorization, it will strongly affect one thing in the batch, or in the big mega-batch, and not most other things—and that will cause it to wash out and be low-value across the mega-batch—I think is original and is quite nice.

So what we're talking about here is really a sort of higher-moment property. It is the case that the mean is low, but also the variance is high. There is a possibility where you have something which is low across the whole batch. We can't distinguish that from something high on one thing with the statistic we compute, but you could in principle do it.

One thing that we'd hoped for from this is that we would be able to shrink the model down. Not only does the model not know this stuff, but it also doesn't pay the parameter cost of knowing this stuff. And we never pushed this all the way, but I think it probably is not as effective as data-based approaches to minimizing the model, which I think are very promising.

For instance, there's some quite cool work—I’m blanking on who did it—on pre-pretraining, where you train with synthetic data from context-free grammars, say, or from some very symbolic domain. The idea is that you just get the model to make use of very pure information-processing circuits.

Or you might try a sort of data-augmentation approach where you take an article, pull all of the facts out of the article, and put them in a preamble. Put that preamble in the context window, but don't include it in the loss, for instance. Now the model can reason out by induction from the context, from the facts—the sort of open book that you've given it—and we should learn to deduce things.

That also seems possible, and I think these approaches feel intuitively more likely to me to give you a kind of minimal reasoner. But there are only so many hours in the day, so we haven't pushed it yet. I think there's a lot of promise there.

But then the final thing is, would such a thing in fact be more interpretable? I don't know. Is a giant thicket of logical entailments actually that interpretable? Or, if it has so little semantics—if it has no rich semantics—do you just get lost in the forest? I honestly don't know. I've never seen one, so I think it's hard for me to reason that out a priori.

Yeah. Okay. I like that paper. Again, if nothing else, it's a very fun one for me to crack.

Oh, I think it's really cool. This is the sad thing about prioritizing stuff: there's some stuff that I absolutely love that I can't spend as much time on. I think this is a wonderful paper. It's beautiful, and I think that it would be cool to spend more time on it.

It even gives us a regularizer, essentially. Now I have a lever: if I'm fine-tuning, I can say, “Oh, yeah, just keep the generalizing bits.” That's not something we've really tried, but it'd be interesting to explore using that as a regularizer for your fine-tuning process, where I might ask, “Don't give me the memorization,” because eventually that's—

Like you were talking about with alignment and misalignment, if you fine-tune hard enough, you just screw the model up. But what if you were able to keep the generalizing bits? Maybe you wouldn't screw it up too much.

I think that notion of shrinking the model also is a really interesting one in terms of the big picture of how we're going to get to a world full of highly capable AIs that broadly goes well. The idea of strong but narrow is a really intuitively appealing idea to me.

That's obviously Drexler's Comprehensive AI Services vision of that, and I don't know anything about what Safe Superintelligence is doing. But in listening to his conversation with Dwarkesh, I came away with a sense that they were looking to create something that was this proto-agent, or proto-whatever, that would sink and maybe even shrink into its role.

As it gets really good, it also sounds like his vision is that it would lose other capabilities so that it would really dial into its particular context. And so I've definitely found myself coming back to that idea over and over again: how small could you make something that could be really good at what it does? For a company, for example, that wants customer service tickets handled effectively, small.

As you mentioned, too, you don't have to pay the parameter cost. That could be great, right? They could run these things on CPUs, potentially, at some level of shrinking. Then they really don't have to worry about what it's going to do out of domain, because it would just have no ability to handle that, really, at all. And that could give you potentially a lot of comfort.

It's quite exciting. And there are only so many GPUs in the world, so if everyone is going to have their own personal AGI, then you've either got to have a lot more GPUs or a lot smaller models.

Let's talk about Alzheimer's. So this is obviously on the frontier of learning about the world, advancing science by figuring out what it is the models have learned that allow them to be so good at predictions and actually getting conceptual understanding. Tell us about what you guys learned about how the Prima Mensa model is predicting who has Alzheimer's.

Yeah. So this is related to our scientific discovery work. Basically, you can think about the role that interpretability plays. One way that I like to think about it is: when your model has problems, what interpretability helps you do is debug your model. It's a form of model debugging. So you understand what went wrong, and then you use that information to help you get a better model in some type of way.

But when your model is already good at something, then what interpretability can give you is knowledge extraction from that model. We do a lot of work with partners in the life sciences, and our work in the life sciences is focused on taking biological foundation models and then understanding what's happening in them, with the goal of ultimately converting this into new knowledge in the form of biomarker discovery or, potentially down the line, druggable targets and drug discovery.

And so Prima Mensa is an organization that is focused on neurodegenerative diseases, such as Alzheimer's and Parkinson's, and they trained an epigenetic foundation model called Pleiades. This is trained on cell-free DNA fragments. These are little bits of DNA that end up in the bloodstream of people, and they come from cells dying across the body. There's been a lot of prior work that has shown that you can actually use these cell-free DNA fragments for various types of diagnostics.

It's pretty minimally invasive. You just do a blood draw from a patient, and you can use these for various types of diagnostics. So there has been a lot of work, for instance, focused on cancer and using cell-free DNA fragments for cancer detection. They trained an epigenetic foundation model that was an autoregressive model trained to predict the structure in these cell-free DNA fragments. And then, from there, they used the embeddings of that model. I'm glossing over some of the steps. They used the embeddings of that model in order to predict whether patients had Alzheimer's.

And so they brought us in to understand what their model was doing, and we applied a series of interpretability techniques in order to figure out what the signal was that was driving that Alzheimer's prediction. We actually discovered that it was something that was a little surprising. There's a little bit of nuance here, but basically, there had been attempts in the literature for Alzheimer's detection using methylation statistics and cell type of origin, which are 2 specific things you can get out of cell-free DNA, but not specifically using fragment length.

What we found was that their model was overwhelmingly depending on fragment length in order to make its Alzheimer's predictions. This was really surprising to us because this was not what we had expected, and not what the Alzheimer's literature baseline was. Fragment length had a history for cancer specifically, but not for Alzheimer's.

And so, once we learned this insight by studying the model, we worked with Primamensa to construct a proxy model that took this insight and was able to recapitulate a lot of the performance of the original model with a very simple logistic regression, and we were able to generalize better than the baselines in the literature to an independent cohort that we had access to.

And so the high level here that's exciting is that I think this is one of the first examples, maybe the first example, of learning something new from a model by studying it and then coming up with a testable hypothesis. Down the line—this is still early, it was a pilot study, and we need to expand to more cohorts, and these things take time—but it gave us a testable hypothesis that we can now explore, and we're considering wet-lab analyses and other things in order to bring this forward.

We're doing lots of other work in the life sciences with other partners as well, and we'll have more to publish there soon. But I just think it's an exciting early example of what can be done with interpretability, and the way that you can use your understanding of these models to make concrete, testable hypotheses, in this case about the biological world and diagnostics.

I did want to compliment you guys on the blog. I would recommend the blog to everybody. I think this is one of the first times that I have done this much prep for a conversation and not really had to go into the papers themselves all that much.

But the blog posts have done an excellent job of helping me understand what's going on, giving me the right level of detail, and just being quite accessible while also not dumbing it down too much. I think the level of investment there is very apparent to the reader, and I really do recommend the blog highly. Ready for a lightning round?

All right.

Let's do it.

And big shout-out to Michael Belen on our team, who writes a lot of the blog posts in collaboration with the scientists and engineers on the team.

Yeah. Great job by him. So how do you compete for talent with frontier-model developers? That's one big question. You guys are, I think, up to 40 people now, and there's a lot of work that has come out, so it's obviously a team that can come up with good project ideas, execute on them pretty quickly, and ship a lot of stuff.

These people are clearly going to be in demand. Is it just about the mission, or do you have any other tricks up your sleeve?

Partly about the mission. I think that we are trying to do something which is very different, very exciting, very big. And so people are scientifically ambitious. We're at a very great place for me to be.

Partly, it's about the kind of scientific culture that supports that. I think that we try and think from first principles, try and look at things in a very empirically driven way, that kind of thing. Not holding any particular idea too tightly is the aim. It's always hard to actually achieve that in practice.

But I just think we have a very good scientific culture that I think people come here and they're like, "Oh, I like it here. I think I'll stay." And finally on that, it snowballs. Once you have good people, then, 1, they know good people, and 2, people want to come and work with them. So that feels like a big plus. That engine feels like it's started to work well now.

So I think that's it. And a lot of hard work. Recruiting takes a lot of time, takes a lot of work, and is extremely worth it.

I think we also have a pretty differentiated research vision and a different vision of the future than a lot of the labs do, too, and I think that's appealing to a lot of people.

And although we're working to figure out our identity as a company in a lot of ways—and we certainly solidified a lot of stuff—I think it's just exploratory in a way that is really, really hard to do at the big labs. And I think that's one of the things that's really helpful for us: the possibility space is very open for us as a startup.

You guys have put out some stuff around what you see as the highest-importance or highest-leverage open problems in mechanistic interpretability. One is work on alternate architectures.

I think one thing I don't have a great sense for is how much of the technique that you're developing will work if, for example, nested learning becomes the next big thing and now we're all in, and we've gone from a transformer world to a nested-learning world.

My expectation for nested learning—I mean, we could look at this; it'd be interesting to look at this—but my expectation is that interpretability techniques would still work. There has to be semantic information that gets passed through the bottlenecks in any learning setup. And so I have no reason to think a priori that you wouldn't be able to use a lot of the same interpretability techniques to understand what was going on there.

I've been generally very encouraged from what little work I have seen applying interpretability to alternative architectures that it mostly has worked pretty well. Mamba-type architectures seem to have been remarkably interpretable.

But do you think there's any prospect for other architectures perhaps being more interpretable, and if that were to be discovered, could that be the sort of thing that would pull the field in a positive direction?

Part of the problem with that is that people don't go looking for interpretability, right? It turns out that—I don't know to what extent this is extremely robust, but it seems quite robust—if you just look at the neurons inside a transformer's MLP, work from Translucid shows that they often just are interpretable.

The sparse autoencoder was inside you all along. And we've had transformers for how many years, and people are just like, "Oh, wait, the MLP neurons—they're interpretable. So maybe we should just look a bit harder."

Mixture-of-experts models, I think, individual experts are generally not interpretable.

But why should they be? There are 64 of them, whatever. A language model has to do more than 64 things, so any given expert should be partly semantic. But there's some recent work—again, I'm blanking on the details and the authors—but routing paths are also interpretable. Amazing. So the affordances are sometimes there; we just forget to look for them.

This is almost Panglossian. It's optimism, but maybe models get better to the extent that they are more interpretable. Obviously, that's not literally true. But MoEs have pushed us on the performance frontier, and they also give us a new affordance for interpretability. Maybe the correct MLP width is simply because it happens to make the hidden layer inside the MLP roughly interpretable, and that makes the computations easier. Maybe there's a deep principle here. I don't know.

Cameron Berg of AE Studio did some really interesting work on the Goodfire API, looking at what models say about their own consciousness. What do you think about AI consciousness?

I think it's a complicated question. I think it would be very difficult to confidently rule out the consciousness of most existing frontier systems today. I think they probably aren't, but the word “probably” is doing a lot of lifting. There's nothing that prevents the idea of consciousness from being in a machine.

You can come up with definitions of consciousness, or explanations that might preclude that. I don't find them particularly convincing. So I think it's fairly likely that it should be possible to build machines that have experience in some meaningful sense. I think we probably haven't today, but I think it's important to take that question pretty seriously.

It's hard to know whether interpretability could give us full insight into that question, but maybe it can. If there's anything that could, interpretability would probably be the thing that could do it. I don't have a valence about whether that would be a good thing or a bad thing. I just think it's a distinct possibility that, if it's not something that exists now, it could be a thing that exists in the future.

Do you want to give just a closing call to action? We're in the early stages of AGI, it feels like. People are calling Opus 4.6 and Claude Code AGI, and it's only going to get more real from here. Why should people seek out the careers page at Goodfire or otherwise invest their precious time and energy into interpretability?

Yeah. I think it's definitely hard not to feel the AGI right now, so I can super relate to that. I think interpretability is important for a lot of reasons. When I imagine the futures that we could walk into, it feels like a given to me that we are building superintelligence, and that's happening quickly, and we can talk about how quickly.

But the way that I see it is kind of two doors. There's one door where we build superintelligence that we don't understand at all, and then there's one door where we build superintelligence that maybe we have a shot at understanding. I think fundamental research, interpretability, and intentionally designing models are really important paths for us to get there.

We're doing all types of exciting work. It's not just a theoretical exercise. We're going out and making discoveries in the life sciences. We're working closely with partners to help their models behave better, reduce hallucinations, and be more reliable. This is a really important field to develop for the future of the technology, and it's also something that progressively unlocks a lot of value along the way.

So if anyone is interested in what we're building and working toward that mission with us, please reach out to us. We would love to talk.

Yeah. If you want to be part of the most exciting and beautiful scientific quest that's going on at the moment, I think it's got to be interpretability. And if you want to make it useful, I feel like Goodfire is the place to be. So that's why.

Love it. Congratulations on unicorn status, and congratulations on a great run of research. Dan Balsam and Tom McGrath from Goodfire, thank you both for being part of The Cognitive Revolution.

Thank you for having us.