机制可解释性：与 Goodfire 的 Daniel & Tom 谈哲学、实践与进展

Today, I'm honored to be joined once again by Daniel Balsam and Tom McGrath, CTO and chief scientist at mechanistic interpretability startup Goodfire. When we last spoke about 9 months ago, we focused on the technical foundations of interpretability, including the challenge of polysemanticity, techniques such as activation patching, causal tracing, and feature editing, the rise of sparse autoencoders, and some of the challenges of scaling these techniques to frontier models. If you're new to mechanistic interpretability, I would definitely recommend checking out that earlier episode for a technical primer.

Since then, Goodfire has gone on to train sparse autoencoders on Llama 3.3 70B and DeepSeek R1, strengthened its team with the addition of multiple top-tier researchers, and recently announced a $50 million Series A, which notably includes Anthropic's first-ever investment in another company, giving me, as a small-time seed-round investor in Goodfire, both strong on-paper returns and a bit of bragging rights. In today's conversation, we mostly zoom out from specific techniques and findings and instead try to get a handle on the state of mechanistic interpretability as a whole.

For years, the field has been called pre-paradigmatic, but as you'll hear, Tom now describes it as proto-paradigmatic. There's now general agreement among researchers that neural networks contain understandable things, that these things, called features, can be understood as linear directions in embedding space, and that the magnitude of their activation represents their intensity. There's also the finding that superposition allows models to represent far, far more concepts than they have dimensions, and that features connect through the layers of the model to form circuits.

This is great progress, and honestly, much more than I might have expected just a couple years back. But there are still important gaps between the accounts that interpretability techniques provide and the underlying reality of model structure and behavior. First, and most obviously, there's the fact that interpretability techniques typically attempt to reconstruct the behavior of the underlying model, and as of now, they can do so only very roughly. Second, and more philosophically, there's this distinction between the features that interpretability techniques learn and the meaning that we assign to them in the process of labeling.

In practical terms, when we say that a feature represents the Golden Gate Bridge, or, more to the point, deception, how confident can we really be in that label? From my own exploration of both Goodfire's and Anthropic's interactive interfaces, this seems to range very widely. All of this is complicated further by another all-too-often-neglected fact: the models under study encode varying degrees of understanding, with everything from simple memorization to fuzzy heuristics to proper algorithmic grokking all occurring simultaneously in an unknown mix in any given model.

Of course, while the philosophy is fascinating and there's still clearly a ton of work left to do, that is not stopping Goodfire from deriving practical value from interpretability techniques today. Dan describes how Goodfire is developing applications both for enterprise customers and the public good across 3 key domains: scientific discovery, where they're partnering with organizations like the Arc Institute to explore genomics models like Evo 2 and beginning to uncover novel biological insights; guardrails and safety, where they're developing inference-time monitoring applications that can detect when models might output harmful content or exhibit other problematic behaviors; and creative applications, such as their just-launched Paint with Ember tool, which allows users to generate and edit images by directly manipulating sparse autoencoder features.

Proto-paradigmatic though it may be, as we enter a new era in which science shifts toward simulation-based approaches and AI systems potentially drive more and more of the machine learning research, it seems to me a very safe bet that interpretability work will become more and more important. As Dan put it, even if we end up in a scenario where a data center full of geniuses is doing most of the scientific work, mechanistic interpretability might be their preferred tool for understanding both their discoveries and themselves. As always, if you're finding value in the show, we'd appreciate it if you'd share it with friends, write a review on Apple Podcasts or Spotify, or just leave us a comment on YouTube. Your feedback is always welcome too. Feel free to reach out anytime via our website, cognitiverevolution.ai, or by DMing me on your favorite social network. For now, I hope you enjoy this thought-provoking exploration of the philosophy and practice of mechanistic interpretability with Dan Balsam and Tom McGrath of Goodfire. Dan Balsam and Tom McGrath, CTO and chief scientist at Goodfire, welcome back to The Cognitive Revolution.

Thank you so much for having us.

Thanks. Yeah, great to be on here.

I'm excited. We haven't been able to make this happen quite as often as I would have liked, but we're going to make up for it by going long and in-depth today. I'm really excited to get the update on what you guys are building as a company, which I understand there's some great news on, and also to check in on what we have learned as a community about models and how we understand how they work over the last few months, because obviously there's nowhere in the world changing faster than that.

1. The Interpretability Input Stack

For starters, I wanted to go high-level and ask you to frame the field. I think everybody in the general ML space at this point has internalized this data, compute, and algorithms paradigm. These are the 3 legs of the stool that are enabling progress. There's a sense that they all contribute equally.

On the interpretability side, I'm tempted to slot in models for data and say that models, compute, and algorithms are maybe the things, and seemingly a lot depends on the quality of the models, but there's still a role for data. How do you guys think of the fundamental inputs of what you're doing?

I think one thing that's interesting here is that these inputs are very important, right? The models have changed, and so things sort of changed along with them. But another thing is that it's a very empirical activity, in the sense that you're looking in a fairly fine-grained way at the data of what's happening inside models.

Progress in algorithms—for instance, parameter decomposition, SAEs, or whatever; we can get into these later—is one of the inputs here. But another is actually very rich empirical data: when you look at what activations are like, when you tinker around with models in a relatively hypothesis-free way, do they actually seem to behave in the way you expect?

I guess this sort of empirical input is always part of progress in algorithms, but I think I just want to up-weight it for interpretability because we're really doing this sort of natural science. But, yeah, I broadly agree with the decomposition of models, the data, I suppose, and then there's compute. I think as a field we would like to be able to use much more compute.

Then there's algorithms, and I think that—I think it's a good decomposition. I would say that we don't have the transformer for interpretability, and it could be that the thing we're blocked on for finding the transformer for interpretability is simply understanding models better in a way that we can then generalize to a new inductive bias.

Yeah, that's really interesting. You also prompted me to try another paradigm-mapping exercise. You said we're doing this in a relatively hypothesis-free way. That maps in my mind to pre-training, or unsupervised learning, right? The SAE paradigm has largely been run on a ton of data, basically from the same original dataset.

You could tell me more about how that's curated, especially for interpretability work, but it's a relatively hypothesis-free way of working. Is there any equivalent to post-training yet in the interpretability world?

When I say “hypothesis-free,” I mainly mean a person sitting down and tinkering with models and asking, “What structure is there?” For instance, in the famous paper “Not All Language Model Features Are Linear,” they're finding structure in activations, and this structure is kind of interesting. It's this higher-dimensional manifold, and you don't necessarily get to this by having a hypothesis that things are this way.

You're just observing things like a natural scientist—someone out observing the stars. They have some hypotheses in mind, but they're also just looking at things. That's what I mean. But it's interesting that I'm also very keen on the unsupervised approach to interpreting models, such as with SAEs, for basically the same reason, except that every architecture comes with a hypothesis.

The SAE inductive bias is that things are sort of—well, the SAE is a fairly strong hypothesis that features are literally directions in embedding space.

There's something really interesting there. Again, model quality seems to be really important, right? A lot of the hypothesis-free tinkering—even vibe coding—seems to be premised on the idea that you can run a little experiment and get something back relatively quickly, on an iteration timeline where ideally you could sustain focus.

Neil Nando recently put out a video of him doing some vibe-coding research, and that's all premised on the idea that you can run a little experiment and get something back relatively quickly, on an iteration timeline where ideally you could sustain focus.

Maybe you have to go take a walk and come back, but it's not really long time frames or really large compute budgets. As you mentioned, you have to have some conviction to throw large amounts of compute at an SAE at scale. But I wonder: How do you think about the challenge? Can you do that rapid experimentation on the truly large, advanced models, or are you limited to working with something like GPT-2 scale? Does that create a fundamentally different regime from the things that you end up scaling up?

2. Infrastructure Enables Unsupervised Interpretability

It's really a question of infrastructure. If you have no infrastructure, then it's hard. If you have the right infrastructure, then building that infrastructure is hard, but the right infrastructure makes experiments relatively easy.

When thinking about unsupervised techniques as hypothesis generators for how the model could be working, there's no way that we're going to be able to scale to superintelligence without making our interpretability techniques unsupervised, and that's one of the things that really motivates us and is our mission at Goodfire. Narrowly superintelligent models already exist in the scientific domains, and this is what we spend a bunch of our time working on.

When you're working with a genomics model, you're working with a model that we have priors about. There's lots of bioinformatics research to attempt to understand the genome, but we're also working with systems for which our statistical techniques are not as explanatory as we'd like them to be, and that's the motivation toward moving toward AI to begin with. In the process of training unsupervised models, it really gives us a grounding about where to look in the model to begin with. It gives us a bunch of candidate experiments and candidate hypotheses to run.

One of our goals as a company is that unsupervised learning often looks worse until suddenly it looks better. If you went back in time, there was a point where massive pretraining in an unsupervised way on large corpora was getting you worse performance than bespoke models that were purpose-built. We think interpretability is likely to follow a similar arc. We're still not sure exactly what those pieces of technology will be, but each new item on the interpretability tech tree unlocks new questions that we can ask, new ways that we can look at the problem, and, as Tom was saying, eventually building toward a solution that we can just toss a lot of compute at in order to fully unlock what's going on inside.

So am I interpreting you correctly to say that right now you'd say the field still feels like it is bottlenecked more on algorithms than compute, or certainly not models, right? I mean, there's a lot more in models we can figure out.

Yeah.

So the 2 candidates would be compute and algorithms. It seems like you're saying algorithms are still where it's at, and we need to figure out how to apply the compute.

I think that's right in some sense, but I think it was a soft version of it that I would take. I think existing tools are already powerful enough to do things that are useful in the world.

Hmm.

That's a big part of what we're doing in Goodfire when we're working with customers: we're using the existing techniques that are available to look at a model and help customers understand their model better. But we are under no illusions that we've cracked interpretability. To fully reverse-engineer what's happening in a model, we do need brand-new techniques and new paradigms.

Yeah.

On the compute front, I think it's really a question of whether frontier models are bottlenecked by the literal availability of compute, or will be soon, right? We're very far from that, but we could spend a lot of compute, right? The question is, do you get value for money for it? I would like to be able to do a $1 million interpreter-model training run and say, “Oh, yes, I spent $1 million on it, but I feel like I got $1 million worth of information.”

Maybe there's a way to say that there's a sort of ratio between the size of an experiment and how many—hmm, that's maybe not quite the right way to say it either. But given the option of choosing a really big run or more people to come up with more ideas for smaller runs, it's clear that you'd rather have more interpretability researchers exploring the space more thoroughly before—

Yeah.

…scaling up.

Well, yeah. I think that we couldn't productively do a single $1 million training run. I think that's one of the things that says, “Are you bottlenecked on algorithms? Can you productively spend that much compute?” We could spend it, but at the moment, I think it wouldn't be a productive way to spend it.

I think there's also the question of how you use the tools that you have. For our customers, when we've trained an SAE for them on their model or any type of interpreter model, that's where the work begins, not where it ends.

Hmm.

I think of an SAE as a window into the model, right? You can only see some things, but those things can still be really useful and provide a lot of value. Chris Olah popularized the biology analogies for mechanistic interpretability, and I think those really track from my perspective.

If you go back to the 19th century or early 20th century, what did you have to do to learn anything about cells? You had to put things on a slide, stain the slide, look in a crude microscope, and you were looking at a cross-section of something not in its natural environment. You had to make inferences based on looking at a lot of these about what the actual biological structures were.

All of modern biology was built on that, and over time, we developed better microscopes. We developed better techniques for looking at organisms in more natural ways. So I think we're a little further on in interpretability than that analogy implies, but I think it's the same thing. You can learn and make novel advancements in science at any state of the technology, and simultaneously, we want to push what we can learn with the tools that we have while we're also pushing to get better tools.

Yeah.

I think for a while people had to make their own lenses, right? I guess that's like making your own SAE training code base. Probably you want to just go and buy the lenses from Zeiss or something like that, and then you can become an expert in great tools, and then the field can advance. There's a famous dictum in science about things advancing via methods and ideas and then experiments, or something like that, in that order.

Yeah. There's an Adam Smith lesson here. The extent of the market, the degree of specialization, and the sophistication of the supply chains are all pretty early, and we are maybe just exiting the grind-your-own-glass-lenses phase of biology. Let's hope we don't find ourselves in the gain-of-function research, lab leak, and invasive species phase of biology before we're ready to handle them.

But I guess one place where this analogy maybe breaks down a little bit is on this question of fundamental units, and this is a philosophical question I've been trying to wrap my head around better. I'm sure you guys have good thoughts on it. When we look at an organism and its genome or proteins, we're pretty confident we're talking about real things.

Yeah.

I guess they're maybe quantumly fuzzy at the margin, but we have a pretty good sense that a gene is a gene and a protein is a little machine. Here on the interpretability side, the features, if you will, that are identified or learned by an SAE or similar technique—and you might want to separate SAEs from other techniques in your answer here, if appropriate—seem to be approximations. There's some sort of gap right there between what's going on in a model and what's going on when it's sparsified in this particular way.

I'm really interested in how you think about the relationship between these features that are learned and the labels that we give them, how much correspondence you think there is, whether that's on a spectrum, and how we should think about it.

Yeah. So it's interesting. I suspect if you asked a biologist, they would probably have a lot of corner cases about whether it's really a gene. Biology is, if nothing else, a great supply of corner cases just because of the rich complexity of the world.

But yes, I think there's a definite sense there that a gene is a sort of natural abstraction. It's a good way to talk about the world. In this process, you take a model, you sparsify it—well, now we have introduced some degree of messiness, right? Because we're not capturing all of the computation. You can see this from the loss, and you can see this from the reconstruction error, but we are capturing what look like very interesting and interpretable things. But that takes you onto the next level of, “Okay, there's a thing.”

There's a feature in your sparse dictionary, and now we assign it a label. This is another area where there's a gap—a sort of gap that you can fall into. I think we can be in the business of closing both of these gaps a great deal, and you can close the gaps in multiple ways.

The first gap is the distance between the model and our approximation of the model. How do you close this gap? One answer is that you do the machine learning better: you just make a better SAE. You capture more of the loss, and there's been a lot of work in this direction. I can provide a bunch of papers later.

Another is that you try to answer the question, “What does it mean for something to be a good abstraction?” and use that as inspiration for new methods. For instance, what would it mean for an SAE feature to be a natural unit of computation? It's not completely clear. I think there are some interesting, but probably quite resolvable, issues there.

One thing it might mean is that it is involved in consistent computational parts. A feature is a natural unit of computation if it is involved in other computations that make sense. I think things get a lot cleaner, or feel like they will get cleaner, as we move to circuits rather than just single-layer activations, because you don't really have a great way of validating them.

You can validate by intervening on a feature and seeing how things change. That's sort of like circuits, except you just haven't checked the circuit. Now you're entering into the second gap. I've intervened on a feature, and you might say, “Well, I intervened on it and it didn't do what I expected.” But that might be because my expectation was wrong—that I've simply fallen into the second gap, where I've given a feature an incorrect description.

It is a unit of computation that the model uses, but I've just called it the wrong thing. Now, how do we narrow the second gap? I think the answer is probably that we just get better at doing experiments in interpretability.

The way that we currently assign labels is—and hopefully I won't offend Nick Cammarata by saying this—a little primitive. Nick Cammarata sort of randomly invented this automated-interpretability technique. We give a frontier model a bunch of examples of where the feature fired, and we say, “Here are these examples. What's the feature?” This gets you some way, but it doesn't get you the whole way.

For instance, if you were to ask me, with access to the model, “What is this feature?” I wouldn't only do that. I would also try steering the feature and see what happens. I might look at other things that projected into that feature, or where it goes downstream, how it relates to other features, and all that sort of thing. There are many more things that I personally would do.

But we can't currently get models—we can't currently get frontier models—to do this. It may just be a matter of scaffolding. We need to build this kind of scaffolding so that they can use their capability set. That's a really run-on answer, but I think there are these 2 gaps. If we can narrow both of those gaps—

Yeah, principal investigator at Goodfire.

Yes, thank you. I was panicking. Yeah, he's wonderful.

No, on the contrary, I want to expand on both points, because it's all really fascinating and important. So, on the first bit, I really like just decomposing the problem into the 2 gaps. First is: can we reconstruct—I mean, this is literally what the SAE is trained to do, or what the other techniques are trained to do—reconstruct, right? It's a reconstruction loss: do what the model originally was supposed to do.

There, I'd love to know a little bit more. Maybe you said you could provide a bunch of papers later, but I'd love to hear what the state of the art is. The deepest dive I've done into the literature recently was the pair of Anthropic papers on tracing model thoughts, and I was struck by the fact that it seemed like a lot wasn't being reconstructed. That would be my bottom line.

I'd love to get a sense of where the state of the art is and, maybe in narrative form, if not in fully cited-paper form, what the trajectory of improving that has been. What have been the advances, the unlocks, whatever?

On that second gap, this sort of anticipates another question I had around what the inference-time-scaling paradigm for interpretability is. It sounds like the answer is: today, we run data through the thing, see what activates what, collect the things that cause the maximum activation, and then try to describe them qualitatively.

In the future, we could do a lot more. Maybe you could develop the vision for higher-order auto-interpretability a little bit. I think both of those are really interesting mini-lectures that you could go on at as much length as you want.

I think, before we dive in there, can I give a quick meta-thought on this entire question? Most measurement apparatuses that you could deploy in most scientific contexts are reductive in some way, right? There's some set of assumptions that you're making about how you should interpret the data that you're getting from the measurement apparatus.

The microscope analogy with SAEs works really well when explaining this to people, because there are 3 things that you can do with a microscope, right? You can figure out what you're putting on the slide. That's the data set that you're passing through the original model. Those are the actual activations themselves.

You can get a different SAE on one data set than on a different data set. They're sort of like how you stain the slide: maybe that's by the loss function, or by how exactly you're looking at things. That's going to affect the structure of what you get back in different ways.

Then there's the size of the lens, right? That's the expansion factor of the SAE. It's important to understand what we're doing with our existing interpreter models as a specific lens, a specific way of looking at the computation of the model, rather than the whole picture.

I think this is true whether you're trying to do circuit work with sparse approximations or layer-level activation work with sparse approximations. But that's not an abnormal thing to do in science. You take some set of assumptions, you know they're not always right, but you know that they're right enough sometimes that you can start getting traction and running new experiments.

From our perspective, it's not an all-or-nothing thing. This is like what Tom is saying: we can keep pushing on how to address the limitations in the existing tools within the paradigm, with the sets of assumptions that we know are sometimes wrong but are right often enough to be useful and give us more information.

At the same time, we can ask the question, “What's a better set of assumptions?” These have to happen in parallel. You can't just do one at a time, because you would never find the right assumptions if you weren't testing the limits of your current ones. From my perspective, this isn't a unique thing about interpretability. This is just how science works.

Yeah, exactly. And it's funny: people remember Kuhn's The Structure of Scientific Revolutions for the revolutions, right? For the crises. And yes, the dominant mode of science is normal science, where you're going along and generating actually productive knowledge about the world, maybe on foundations that will later get a bit shaky or get overturned, but you're still generating knowledge about the world.

Then there's this idea that there are anomalies, and when anomalies pile up and are unanswered, they lead to a crisis, right? But that's where the anomalies come from: the business of doing normal science. So even if you want to generate a paradigm shift, often the answer is just to try to do normal science until it becomes untenable.

3. Interpretability Enters Its Paradigm

I think that's maybe one place where we might be at the moment. We probably have a proto-paradigm. I think we've been reluctant to admit it, but we probably have a proto-paradigm in interpretability, and so we should push it. We can do a lot of useful stuff. We should keep pushing it, keep doing the useful stuff, see where it leads, and wait for the anomalies to reveal themselves.

Yeah, I was going to ask, actually: are we still pre-paradigmatic? Have we upgraded ourselves now to proto-paradigmatic?

I'm going to say proto, but maybe I should have some courage in my convictions. I think we're entering the first paradigmatic phase of interpretability.

Well, no. Okay, so this is a bit fuzzy, right? What is a paradigm? A paradigm is a sort of social thing.

I don't think there's consensus. There's not the kind of consensus that would lead me to say there is a field-wide paradigm in interpretability. I would say that among a reasonably large group of people, there are the raw materials for a paradigm. I suspect if the field were Anthropic but had sort of ballooned to the size of the global interpretability community, it would be correct for me to say that there was a paradigm. But because there isn't this level of consensus, I can't really yet say there is a paradigm.

So what would that paradigm be? How would you describe the Anthropic and Goodfire-axis paradigm?

I'd say, one, neural networks contain things which are understandable. This is actually worth stating. For a long time, this was not generally accepted. I don't know if it is yet generally accepted, but this is sort of down in the basement of the paradigm.

As a quick interjection there, is that an artifact of just earlier models? Because the way I would tell that story is—

Mm-hmm.

In the original GPT models, there were still some things that were meaningful enough, but there was also so much noise that people could very easily have been excused for saying, “Eh, you're tricking yourself,” or, “You might find some spurious correlation here or there, but I don't really buy it,” and they maybe just haven't updated since.

You're going to interpret the model that you have, and if there are lots of flaws in the model, those might be what you're finding with your interpretability tools. If you're entering with the prior that, “Oh, I should be looking for and recovering this specific thing,” and you can't recover it, that could just be as much evidence that the model isn't doing what you thought it was doing to begin with.

When you go through the process of debugging a model using interpretability techniques, the thing that you might find is, “Oh, the model has memorized a bunch of its training data,” or something like that. You have some belief about what your model is doing and how it's modeling the task, and that belief might be wrong. That could throw off the perception of interpretability if you're not bringing unsupervised, low-opinion techniques in that can work across the entirety of the end-to-end interpretability stack. I think, to that point, we just didn't have unsupervised techniques that could work, at least a good percentage of the time, across the end-to-end interpretability stack until quite recently.

There's a funny sort of U-shaped thing where I think a lot of the early connectionist papers actually looked at individual neurons and said, “Oh, look, this neuron learned this thing. This neuron learned this thing.” They could do that because there were 12 neurons. It's funny: it started out with everyone wanting to look at the neurons and having some success. Then, for reasons that are opaque to me, but I suspect someone could find out, it became somewhere between unfashionable and considered to be a bad idea—or impossible—to look at individual neurons, and now it's kind of come back in vogue. So it's sort of like interpretability just went into the wilderness for a bit.

So now give us the paradigm.

There are things to interpret. Interpretability is possible. What other parts of the paradigm are there? I suppose there's the idea that representations are linearly decodable, or at least that linear decoding is a reasonable way to talk about features. There may be higher-order structure, right? You might have features that are sort of arrows in space. It might be that multiple features lie on some manifold or in some subspace, but that it's a sensible way to talk about representations as lines through embedding space.

The third part of this paradigm is this idea of superposition. Because if you're going to have vectors in a vector space, then the natural conclusion would be, “Well, I'm in a d_model-sized space, right? Does that mean the model can only think of d_model things? Probably not, right? A language model can think of more than 4,096 things or something.” The other part of this paradigm is superposition, which is this idea that the way that you squash more of these embeddings—more of these vectors, these feature vectors—into the same space is by allowing them to overlap a little bit. This creates a bit of interference, a bit of noise in the representations that the models are still able to deal with.

I guess another part of this is that magnitude along the vector constitutes intensity. And the other thing is, rather obviously, I suppose, that features connect to form circuits. That is basically, I think, the paradigm—the paradigm which is currently, or rather, the thing which is the Anthropic paradigm, I would say. If you were to blow this up to the size of the world, and if there were consensus on this, then I guess it has enough structure to be called a paradigm.

Is there any competing proto-paradigm, or is there just other people claiming we'll never have one at all?

Not in the sense of something that provides a complete worldview. I say, a complete worldview—all I think about is neural networks. It doesn't say anything about dinner. I don't think about that very much. But for a complete worldview of neural networks, you could say that something like parameter decomposition suggests a separate paradigm, and that's because it's sort of a difference in emphasis. Parameter decomposition is talking about the weights; the SAE-type paradigm is talking about the activations.

I would say they're still different levels of emphasis. I think a way to think about this, unsurprisingly, is that we need both, right? To make this seem a bit more intuitive, a neural network is, in a very dull sense, a causal model, right? Every neuron is a node in your causal graph, and all the weights specify the edges in your causal model. It's just a very big, very homogeneous, not very interesting causal model.

What we're trying to do in interpretability is create a causal abstraction. We're trying to create another model, which is a sort of reduced version of this model. But that will also be a causal graph. When we're talking about this debate between whether we should decompose the parameters or the activations, we're asking, “Should my graph have nodes, or should it have edges?” Well, probably it should have both. It's a graph. This is why I think they're not necessarily competing paradigms. They're just two independent ways of thinking our way toward the broader kind of causal abstraction that I think we need.

Would you point to any highlights in the work that has focused more on the weights than the activations? Certainly, from my perspective, it seems like there's much more news and excitement coming out of the activation space.

It's funny because some of the earlier stuff—we're going to stay in the mechanistic interpretability extended universe, because I can't immediately recall some of the other papers as well. I can have a look. Some of the earlier stuff, like learned equivariance in the Circuits thread, for instance, was a sort of weights-based analysis.

But then the more recent parameter-decomposition work has been coming out of Lee Sharkey's group, formerly at Apollo; now he's a principal investigator at Goodfire. Just a little plug: he's great. Attribution-Based Parameter Decomposition was the sort of milestone thing here recently. There's another recent paper sort of following up on this, based on ideas from the loss landscape. I'm blanking on the title. I think Brianna Crishman is the lead author. I can send it to you afterward.

It's sort of what happened in the past, and then SAEs got a lot of traction, so that took a lot of the focus. Now I think it's sort of coming back in again.

One other school, possibly, that comes to mind is—or maybe the contrast in approaches is—the sort of bottom-up versus top-down. I think of Dan Hendrycks and representation engineering, or Circuit Breakers-type work, as being less focused somehow on the unsupervised “just discover what it contains” and more on “This is the sort of contrast that I care about. Let's make sure that we refuse or whatever under certain conditions.” Do you think those are fundamentally different approaches, or are they ultimately reconcilable?

I think so. I think they solve for different problems, and they're both important things to look at from a safety perspective. When we zoom out and think about the alignment problem broadly, we think the solution to the alignment problem could come from interpretability, but doesn't necessarily come directly from interpretability. Interpretability is the measurement apparatus that can make us confident that any other techniques are in fact doing the things that we think they're doing.

For instance, are we aligning the chain of thought of a model faithfully to its computation? I don't know how we would have any hope of answering that question if we don't have meaningful ways of constructing graphs that represent the model's computation in some way.

I think a lot of these core questions in alignment, at the end of the day, could be solved through bottom-up methods, but they might not be. Without the bottom-up work, though, I have no idea how we would know that we solved alignment through any other means.

I think that's really clarifying. I think people will find those high-level mental models quite interesting. Maybe I'll add a third chapter. So, one: what are the gaps, right? If we structure this in terms of opportunities to make things better, we've got better reconstruction and lower loss. I'm eager to hear a little bit about what the progress has looked like there.

Then we've got better labeling, or inference-time scaling, and getting to higher and higher orders of automation and confidence that what we are describing—what we say is happening—is in fact what is happening, that we're understanding it correctly. And then what you alluded to, which I'll maybe add on, is the third thing: the move from activations to circuits. I did think Lee Sharkey's paper there was super interesting and could be a whole episode on its own, but we can do a mini one as part of this.

I should mention Anthropic's circuit-tracing results as well. I think everyone wants circuits. The question is, how do we get them?

Yeah. So take us through those three chapters, and again, take all the time you need, because I'm here for it.

4. Better Sparse Autoencoder Training

Okay. Should we talk about the first one first, then? That's how people have been doing the machine learning better. The machine-learning part of it is: how do we best learn this kind of decomposition of the models? Let's stick within the SAE paradigm, because then I can be very concrete. How do we learn good decompositions of the models?

Some of this is that you start with the SAE, and it has L1 sparsity regularization and a ReLU activation function. Then people hill-climbed a bit in the classic machine-learning way. The L1 sparsity has certain properties: it causes features to shrink, which also has a predictable bad effect on the reconstruction loss. There are various solutions to this, such as JumpReLU or BatchTopK, and there has been some hill-climbing on this work.

Then there are things like end-to-end SAEs. Instead of training purely on the activation-reconstruction loss, you're training so that when you take the model and put the SAE in the middle of its computations, the model doesn't get too much worse. It's called an end-to-end SAE. All of these things improve the reconstruction-loss-versus-sparsity frontier.

There are many other approaches to dictionary learning. Dictionary learning is the broad class of things of which the SAE is an instantiation. The dictionary is your collection of vectors—your quiver of arrows—and you're learning a dictionary that can do a sparse decomposition. There are things like using gradient pursuit instead. I saw an interesting blog post on a residual quantized autoencoder, Matryoshka.

All these kinds of things are other ways of using compute better. You haven't really changed the fundamental assumption that features are directions in space, but under this basic assumption, you're asking, how do we create a machine-learning architecture that reduces these metrics of loss? That improves the loss we care about. What else? I don't think there's anything else. I doubtless have missed some and offended people by leaving their papers out. That's just a failure of my memory. Apologies to the people whose papers I've forgotten.

Well, nobody can be completely comprehensive in real time. The Matryoshka concept was one that I was keen to explore. I've heard a little bit about that, and it has a very—I don't want to be overly lured by this—but it has a very appealing vibe to me, where there should be some structure that gets finer-grained as you go.

Basically, the idea there is a sort of tree structure, right? I mean, it's not really structured that way, but the idea is like the short version—

It would be ideal if it were a tree. Unfortunately, there's no explicit tree structure involved. Matryoshka is really interesting, both as an architecture and from the point I was talking about earlier: we're talking about this idea that models compute algorithms, and algorithms are bottlenecked by observable experimental data. Our knowing new things about models is part of that, and Matryoshka is an example of this happening.

How did we get to Matryoshka? I say “we,” but I had no personal part in it. My understanding is that there's this idea of feature absorption. Feature absorption happens when SAE latents specialize far too much. Instead of there being, for instance, a feature for a token that starts with the letter A, you get a feature for a token that starts with the letter A and is not the word “aardvark,” or something like that. Then you also have a feature for a token that is “aardvark,” because it turns out this gets lower L1 sparsity loss. This kind of feature absorption was a motivation for Matryoshka.

Now, how did we get to feature absorption? Someone looked at some SAE latents and said, “That's funny. That looks wrong to me.” I think they were actually training linear probes for what letter a token starts with. That sort of feels like a silly, niche thing to do without context, but it led to this interesting discovery of what happens with SAE latents. It was a very smart thing to do.

It led to an interesting discovery that was relatively hard to predict. Certainly no one predicted it a priori, as far as I know. And then Matryoshka turns out to be a good way to fix this. A more desirable thing would be what you said: Matryoshka, but as a tree.

What actually happens with Matryoshka is that you have a series of nested groups of features, and you predict using the first group. There are 2 variants, but the simplest one to talk about uses groups. You predict with 1 group, and now you have some residual error term. Then you use the second group to predict that residual error term, and so on as you go up the shelves.

What I think would be quite desirable is instead being able to say, “This feature fired,” which means that now I'm going to up-weight this other feature firing. You'd have this explicitly encoded tree structure. The idea of a tree structure is very interesting for minimum description-length reasons. There's a really cool paper on minimum description length. I can come back to that in a minute.

Enforcing this kind of discrete sampling behavior is relatively hard. It's much easier to do this very soft, differentiable thing. I think it's harder to do the kind of thing that we both would like to do.

Matryoshka in general, I wonder how much of this could again be abstracted a little bit to interpretability in general, as a subfield that's coming in the wake of broader ML developments. In general, it's been applied in a bunch of different places, right? It boils down to finding a way to make the first bits of any given thing the most meaningful.

I think of it as ordering the data so that the most meaningful stuff comes at the front. You've seen this in embeddings and weights and whatever, where you can have a short embedding that's pretty meaningful, and then the longer you go toward the full embedding, the more meaningful it gets. But you're always getting the next most relevant bit of information at each stage.

So, intellectual-history-wise, would it be right to guess that a lot of the things happening in this kind of “doing machine learning better” are people looking for techniques that have been developed previously in other contexts and saying, “This worked before; maybe it will work again here”? Then, having applied that technique, they develop a story of what was happening after the fact. That's sort of how I expect a lot of this would be working, but I trust you to tell me differently.

I'm not sure. I expect that a lot of it was worked out from fairly first principles. It turns out that someone may have done something related before, but by thinking about this context, you arrive at, “Oh, this is a problem.”

There has been a lot of work done on sparse autoencoders in the past, around the early 2000s. A lot of work went into sparse autoencoders. Dictionary learning has previously been an active topic of ML research. But I think we're rediscovering a lot of things at the moment.

One thing to do—one actually high-alpha activity in dictionary learning, or in this more ML side of interpretability—is simply to hit the history books. I say “history books”; I mean papers from long, long ago, and seeing what has been done that hasn't yet been applied.

One example I think is quite funny is the recent NeurIPS sparsity tutorial—the 2024 NeurIPS tutorial on sparsity was very good. One of the slides early in the talk was titled “What this tutorial is not about,” and it said, “This tutorial is not about sparsity for interpretability, i.e., sparse autoencoders,” among various other things. But I saw this—it’s not about these various other kinds of applications of sparsity—and I was like, “Yes, great. Now I’m gonna learn some things that no one else in the field knows.” There’s actually some really good stuff in there. Doing that activity a bunch of times would probably be very high return.

Although, coming back for a second to your question about ways this parallels other things in machine learning, that could be a good segue, Tom, to talk about minimum description length, because that’s really interesting.

Yes. Michael Pearce recently joined Goodfire. I guess this is happening quite a lot in this field.

You may notice that we have some of the world’s best interpretability research here at Goodfire.

Yeah, Michael Pearce was one of the authors of this paper on minimum description length, and I think it’s a really neat idea because it basically gets to what you were saying earlier: what you want is to have effective, compact descriptions of what’s going on. Ultimately, what we’re trying to do in interpretability is describe neural networks. I’m trying to describe them to you, or Claude is trying to describe them to me, or whatever, and we’re trying to find decompositions that are easy to describe accurately and then give good decompositions of them.

Minimum description length is this idea that what makes a good description is a description you can transmit in relatively few bits. The idea is that rather than using sparsity as our regularizer, in the SAE paradigm, progress has typically been quantified on this sort of Pareto frontier of sparsity versus reconstruction loss. But the idea of minimum description length is that there’s really only one metric, and that is how many bits it takes you to describe what’s going on. There are various technicalities about how you actually do this with an SAE.

I think it’s a metric that we can probably mostly agree is a good idea. The problem is how you actually implement it, how you optimize it, and so on. But it has various other nice properties. For example, if I wanted to have this kind of tree structure in my features, under a sort of L1-type sparsity regularization, this wouldn’t actually be preferred. But under minimum description length, I can have a feature in—you know, I’ve got a tree and I have the root and the branch, or the trunk and the branch, right? I can say that the branch is high probability given the trunk, but otherwise low probability. Then this has a shorter description length than having to describe both of their probabilities independently. It’s hard to give a good overview answer of this.

I mean, maybe one way to attack it is: what is the artifact that we get when we get this? With a sparse autoencoder, I have the intuition that the thing may have 10 million nodes or whatever, but only 100 of them are gonna light up on any given forward pass, and that could be a sort of hard cap.

Mm.

BatchTopK is a sort of interesting wrinkle on this, where you can define exactly how many nodes will be active or control that in a few different ways.

Mm.

But if I apply minimum description length, what is the thing that I get out?

Yeah. The idea of minimum description length is more like a way by which we should compare all of our various approaches to decomposing models. If, for the same level of accuracy in terms of decomposition, one has a shorter description length than another, then we should prefer the one with the shorter description length.

For technical reasons, something that’s tree-structured would generally have a shorter description length than something that’s not tree-structured, which is just a bag of features. So long as you know the conditional probabilities along the tree, then you would generally have a shorter description length. That sort of matches with our intuitive understanding that it’s easier to talk about things in terms of relationships between parts than it is to just enumerate all of the parts separately.

Actually, how you calculate this is quite difficult. The concept is simple: in general, prefer the thing with the shorter description length. All the meat is in the question, “Well, how do you calculate the description length?” And so I gloss over it there.

Is it something that you can apply directly as an optimization target as well?

Not yet. I think we’d like to be able to, but not currently.

Gotcha.

It is, in some sense, the ideal optimization target, if only we could optimize it.

I don’t know if it would be BatchTopK versus a more naive sparse definition, or maybe some other thing that you would highlight, but how has the optimization target improved? That’s basically—we’re conceptualizing the problem better, right? And you can measure that by this sort of minimum description length. My sense—

Oh, I see.

—is that BatchTopK is a better way of conceptualizing the problem than—

I think they would have the same description length, more or less. an SAE with batch top K versus an SAE with value versus an SAE with jump value: all of these would have approximately the same description length, I believe, all else equal. But being able to have something that’s expressed in terms of a tree structure, or there being various subspaces, would give these a different description length.

For instance, to go on a sort of detour into circuits for a minute, if I had to describe something with cross-layer superposition by enumerating each of the separate layers that I cared about—with one SAE feature for this layer, one SAE feature for this layer, one SAE feature for this layer, plus also saying they’re the same thing—that has a longer description length than saying there is a feature which is spread across these 3 layers. I think it’s sort of on a higher level of description. It’s on a sort of higher level of abstraction than, say, BatchTopK versus vanilla.

So, all in, where are we on this “doing the machine learning better” part? Again, my most recent point of reference is Anthropic’s “Tracing the Thoughts of a Large Language Model,” and my sense was that there’s a lot that’s not reconstructed, right? So if you were to push—if we say, “What’s the best-case scenario? Can we reconstruct a model that kind of works, or does it not really work at all still?” And is it therefore only sort of limited to these very prompt-specific analyses?

Because in the Anthropic work, much of what they were doing was prompt-specific analysis with custom error terms thrown in for those particular cases that they wanted to study. And that doesn’t mean there’s not still a lot to be learned there, but I guess: how close are we to an effective reconstruction of models as it stands today?

So I think there are 2 aspects to that question. One is: can we essentially throw compute at the problem to get there? And the other is: how much will it take? If the answer to the first is yes, we can, then how much more will it take?

If you think about this in terms of a scaling curve, this is sort of asking: does it plateau or does it not plateau? Is there an irreducible error? This question has been examined a little, not as much as I would like it to be examined. Did we talk last time about “The Dark Matter of Sparse Autoencoders” by Josh Engels and collaborators? I don’t think we did. It’s a really interesting paper.

Yeah, I don’t think so, but we definitely should dive in.

Yes. So what they did was this kind of scaling analysis and asked, “If you were to keep scaling SAEs up, would you be on track to recover all of the activation?” Their experiment suggested that the answer was no. If you trace the scaling curve out, there’s a substantial amount of dark matter; the scaling curve does not go up to 99.99% or 100% or something like that. It actually plateaus. It’s not on track to recover everything.

No one really knows what dark matter is—that’s why it’s dark matter. If it were easy to understand, we would already have understood it and probably baked it into a new kind of SAE architecture. This experiment hasn’t been repeated as far as I know, and so I feel like there’s a lot to pull on there if you did it with updated models.

I didn’t even mention, on the “doing the machine learning better” side, this recent SPADE paper, which is a much more expressive version of sparse autoencoders. But if you were to redo this analysis with the latest techniques, would you still find that there’s dark matter or not? I suspect the answer is yes.

I suspect you would still find dark matter, and getting rid of that seems like a pretty big deal.

We still have that in all the scaling laws, right? There's always a constant term in the transformer scaling laws too, right? So the sort of prior guess would be that there's some dark matter minimum that you might approach, and you'll have to chip away at it in the same way that we do a normal loss, which would be that you have to 10× your inputs for the next increment. And I guess that would be—

No. The question is whether, even if you were to scale up compute essentially as much as you possibly could, you would reach 100% reconstruction. It's the difference between a curve on a log plot that is a straight line and a curve that eventually bends. If your curve bends, then you have a problem because you're no longer on track to reach 100% reconstruction.

I think the claim of the dark matter paper is that it bends, and it bends because there were these things that we really cannot efficiently reconstruct. Now, I don't really know what they are. I think figuring out what they are would be very important. It could be that there are things like memorization or various kinds of higher-order structure that SAEs are just not very good at reconstructing and learning.

It could be that they're actually noise—noise that has no relevance to SAEs or to model outputs. Because the dark matter paper was on reconstruction accuracy—well, actually, fraction of variance unexplained—it could be that the dark matter is just noise that has no relevance to SAEs or to model outputs. If you were to repeat the analysis with loss recovered, for instance, it might look different. There's so much that we don't know, and that's actually very important.

But okay, just to make sure I'm not confused—and maybe I am—going back to general transformer scaling laws, of whatever vintage we want to look at—

There's always this constant term. I interpret that as the scaling laws do not suggest that we're ever going to have zero loss. But on our way to this sort of theoretical minimum, we have this 10× for the next increment of progress. Does that theoretical minimum imply the same kind of bend? Is that the same sort of theoretical minimum that you're describing on the—

Presumably.

—capability side?

Presumably they both imply that at some point it must bend, unless it hits zero, hits the minimum, and then is horizontal. There's an interesting difference, though, between the scaling laws. There are 2 sources of irreducible error, I suppose, in a language model scaling law.

One is the uncertainty of the world. If I literally have 2 strings that are the same up to token T, and then at token T + 1 they have different strings, and these are both in my pre-training corpus, then I have a flaw. When training on this corpus, I can never get down to zero loss because it's 50/50. Looking at this data, I have to pick. Given the prefix, it is no longer deterministically predictable what the outcome is, even if I had a perfect model of the data.

The other is architecture dependence, where, say, there are things that the transformer simply cannot capture, and these are somehow detectable from the scaling law. We expect that it will asymptote out because of architectural limitations. The difference, I suppose, is that we believe it should be okay for a language model not to achieve perfect reconstruction. Capabilities people might disagree, but it seems okay for a language model not to achieve perfect reconstruction.

I would much prefer that my SAE or my interpreter model achieve perfect reconstruction. That seems pretty important. At the very least, it should have very high reconstruction.

Yeah, I think there's also the question of what abstractions are you, like, working with here? Like again, what are the, like, limitations of the techniques you can limit? If we had wide enough SAEs, like what would the features actually be data set examples, right? Like, they'd be specific data set examples for every activation pattern it's seen because that's just what it's gonna learn to limit. So you're trading off again against like generality and sort of memorization and specificity too in the context of an SAE. And yeah, to add some like concrete examples to ring one, there's been work that shows transformers can encode days of the week, for instance, on a ring. That makes sense because you can think of days of the week as a mod-7 operation, right? You add until you get to 7, then you loop back around to 1.

But if you're training an SAE, what are you going to recover? You're going to recover Monday, Tuesday, Wednesday, Thursday, Friday, Saturday, and Sunday. In the case of days of the week, that particular abstraction can be thought about both ways, and it sort of makes sense. But in certain types of abstractions, it's very reductive to look at it in that way.

A good example would be that we know transformers can do addition. The way they do addition—modular addition—is with a set of trigonometric operations. If you look at Anthropic's latest work with the CLT and what they pulled out of Claude in terms of doing addition, it looks like a bunch of overlapping heuristics. You can look at that and say, “Oh, okay, this transformer is actually doing addition in this very unintuitive, kind of dumb way.”

Maybe it is, or maybe this is an artifact of the measurement apparatus. Maybe it's operating in some function over some continuous geometry, but the nature of the measurement apparatus is that we'll never be able to see that continuous geometry. I think these are really important, interesting questions.

It doesn't mean that it's not helpful, or that you're not gaining information about the model, or that it's not telling you something important when you're doing this reductive form of measurement. But you do have to keep in mind that even if you had a perfect SAE, what type of information would you struggle to recover from it?

Yeah, okay. Related, somewhat tangential perhaps, the mention of modular arithmetic has me thinking back to the grokking paper. I always remember this one plot where the test loss or the test accuracy shoots up really quickly as the model memorizes all the training data that it sees, and then, orders of magnitude later, the actual grokking effect happens.

That happens over the last order of magnitude, which it's always important to keep in mind is the actual 90% of the data—or 90% of the training time, right? They're weird that way. Going from 10^5 to 10^6, I think, is what it was, after it had memorized by 10^2 steps.

A mental model that I always have of any model that I'm working with is that some things have probably been grokked, some things are mid-grokking, and some things are not at all grokked by the time that I'm actually using the thing. So this seems a little bit like an echo of the—not an echo, that's too strong of a connection—but in some way analogous to the fundamental world uncertainty.

When you're trying to interpret a model, you're making an assumption that things are grokked at least to some degree. The notion of these features and circuits and all this stuff is that there's a level of grokking there that isn't just random, bizarre memorization that happens to be the first thing that worked on the training set. I imagine that's a fundamental challenge of all of this: some of the things that you might want to recover may only be partially learned in the first place.

I think what we really want is a sort of interpreter model that's algorithmically neutral. If it's grokked, then I want to see that it's grokked. If it's a half-baked heuristic, I want to see the half-baked heuristic, and if it's a memorized thing, I want to see the memorized thing.

Actually getting that is, again, difficult. But you're absolutely right to call out the tendency to assume that everything is going to be clean. A lot of the time, there are going to be messy heuristics that maybe even make up the bulk of the computation most of the time.

Okay. So we're moving into parts 2 and 3, around the gap between what a feature is in some grounded sense and what we label or understand it as, and then obviously moving into circuits as well.

5. Interpretability Finds Practical Uses

Maybe the best way to talk about this is: What are the downstream things that we're actually using interpretability for today? And then we sort of imagine that closing this gap between the grounded truth and the labeling seems to involve increasing automation of a lot of those different techniques, to cross-validate against each other and make sure things are actually checking out at scale. But right now, we're not largely scaling all that out. We're largely exploring in a more naturalist sort of way, and we've got all these different tasks that we'd like to be better at. So maybe you can sort of tell us the story of recent progress in terms of what actual utility we're getting from our interpretability helpers.

Yeah. I'd love to talk about all the applied work we're doing. Before diving into that, I want to make one more point on features, if that's okay.

Please.

Even within features, taking as a given the SAE paradigm of recovered features, we can explain some percentage of them with the way that we label them today, and there's some percentage that we can't. One thing that's come up a lot in our work with customers is trying to develop a taxonomy of the different types of features that you might find.

The way that we label features right now predominantly comes from ground-truth external data. So it's connecting some pattern inside the model—some direction in activation space—to some pattern in the inputs of the model, and these can get pretty abstract. In our work with R1 and with reasoning models, we managed to recover features that are pretty neat, qualitatively different from things that we saw in language model features, and that seem to represent important units of the reasoning trace for the model.

But then there are also many features that could be decomposed or explained in similar ways. They're not dark matter features, but they refer to algorithmic processes that are happening inside the model itself and are not easily visible on the basis of looking at input data or output predictions. A clear example of this would be in-context learning. Something happens in in-context learning. People have looked at it and studied it. There's an algorithm implemented by the model that can be difficult to observe from the outside, but has really important implications for the downstream prediction.

We can imagine, when we look at biology models—for instance, genomics models, which is something I'll elaborate more on when we get into the applied work that we're doing—that some percentage of the features correlate very strongly with known biological structures, and then many features don't. And for those features, the question is: Are those biological structures that we don't know yet, or are those structures of the computation of the model itself? How can we tell the difference between these things?

In either case, that's extremely useful information, because information about how you model these biological systems is still, in and of itself, really important to the question of scientific discovery. I think it's important to think of this when building up your taxonomy of features. There's the percentage of features that we can explain by observing the inputs, and we can explain those with high levels of confidence. Our confidence goes down over time as they're moving into the tail.

Then there are the features that represent something more abstract than just something about the inputs. It's something about the model itself that we need to understand, or maybe something in the scientific domains about the inputs that we don't understand yet. Then you keep going along that curve, and that's a spectrum. Eventually, you get to things that, with current techniques, we struggle to explain, and that's why we need to invent new techniques, pushing even just the interpretation of the features that we can recover.

And then there's the dark matter. So there's a bunch of different dimensions to this problem, but I think it's important for viewers to also think about this taxonomy of features. What might a feature be doing? It might be doing something that's actually entirely invisible from both the input and the output.

Yeah. So if you were to do few-shot learning, one thing that sort of strikes me as a way to bridge the gap there—and again, feel free to de-confuse me—would be to expand the window. There's lots to recommend about the Anthropic work, of course, but the interface is characteristically really nice for being able to explore what they're doing in an interactive way and kind of probe into it yourself.

When you click on a feature and it shows, “Okay, here are the examples that maximized that feature from the dataset,” you're typically looking at a pretty short snippet, because you're three tiers down a UI—from a webpage to an embed to a little pop-up within the embed. I don't actually know how big those snippets typically are, but if you imagine a sort of few-shot learning feature being like, “Okay, we have recognized that there is a recurring pattern here in some very abstract way, and our job is to continue it,” with R being the model in this narrative, I guess, then you wouldn't see that if you had a 20-token window, let's say, but if you zoomed out to a 20,000-token window, you might see the whole thing.

And I imagine similar things could be happening in biology, where you're way downstream of an activator sequence that turns this thing on in the first place, or what have you. So is there a sharp distinction between these grounded features and the computation features, or is it just a question of our ability to—

Yeah.

…zoom out far enough to see the pattern accurately?

Yeah, this is a really great framing of the problem. In-context learning's a good example here because, yeah, if you zoomed out and included more context, you could probably see it. Generally, we do auto-interpretation, which is this process of labeling with different amounts of context and different types of context, depending on the problem and the domain in which we're operating.

In-context learning is an interesting example, right? Because if you zoom in too much, you wouldn't be able to see it. If you zoom out, you can probably tell it from the prompts, but it's also an algorithm that we know we expect to find in the model to begin with. We would expect that a frontier model can probably identify a sort of meta-pattern if it sees a bunch of examples as well, because this is the type of thing that we've identified models can do and that we've already described pretty well.

In principle, if you zoomed out to the entire genome and you had some features that were active in a bunch of locations across the entire genome, and you had a frontier model go and look at those, its ability to label those effectively for you in an automated way is bounded by whether or not an explanation was in its training data to begin with. In many cases in the scientific domains, there's not an explanation. We're working with sets of abstractions that are pushing the frontiers of human knowledge in some way, so we need some way of labeling and thinking about how these features compose that can push past that.

You need to be able to look at a bunch of different contexts. Let's say you look at 100 different genomes and see these patterns and reason about what they might be doing together. You have to be an expert in the human genome, perhaps even beyond the level of the greatest human experts today, to reason about that just on the basis of inputs. So one thing that we're interested in doing is: Can we break this down into a set of easier problems? If we have 10% of these features that we can explain just from patterns in the inputs in an automated way, can we work with domain experts and scientists to keep pushing up that frontier of possibility even further and explaining more and more of what the model is doing?

So how's that going? Because that sounds like maybe one of the more important questions we'll touch on today. One of my great hopes has been the idea that we can use these unsupervised approaches on natural data, look in and see what the model is learning, then learn it ourselves, actually have confidence in what we're talking about, and make new discoveries. This seems like a really great driver of new discoveries.

But you're complicating my naive optimism a little with this idea that we don't necessarily know if the features correspond to the real world or just to the internal model. That's a level of confusion that I'm certainly excited to hear how you're going to resolve.

Oh, I think that they still correspond to the real world in the sense that algorithmic features can correspond to the real world and follow it. Right? Newton's laws do not exist in the world. They're a way of describing the world, right? We can talk about the idea of velocity as a kind of feature, but it's not the case that you just see things have a velocity. They don't have the concept of velocity. The structure is still useful.

Yeah, and this is one of the big ambitions and goals of Goodfire: to crack this question. So we're working directly with customers across different scientific domains and with their scientific models, with this goal.

For instance, viewers might have seen our research with Arc Institute, recovering features that correlate very strongly with known concepts in the genome. Arc Institute and we are now pushing this collaboration toward unsupervised techniques that can help get us new information in the genome. We're actively working on this research and hopefully have some exciting things to share in the not-too-distant future.

Yeah, I think at a high level, if the model is correctly—

If it's truly generalizing in its ability, at least sometimes, right? It's on this continuum: it groks some things and has memorized others. What does it grok? What has it memorized? To be able to generalize, it must be learning really meaningful things about the underlying systems. And so recovering those, even in an algorithmic form, tells us really interesting things about the system that's being modeled.

There are current bioinformatic tools. A reasonable question you may have is, well, how do we think about the genome today? Annotations in the genome are done using bioinformatic algorithms that have been developed by humans and that have strong priors. In some cases, they work extremely well, and in some cases they actually don't work that well at all, but they work well enough that they're a good starting point.

The genomes are massive, and the information complexity of genomes is extremely large. So we don't have good first-principles techniques to answer a lot of bioinformatic questions that we might want to answer. But it seems like, in the case of some of these models, they're able to learn things that make these tasks easier.

So it's going to take time. This is an important scientific project that's pushing the frontier of both transformer modeling in genomics and interpretability. But at the end of the day, just going back to the paradigmatic beliefs that Tom mentioned earlier, the model is doing something, and it's doing something meaningful, and we have windows into the model now. They might not be the right windows to tell us everything, but we have strong reason to believe that these windows are good enough to start telling us important things. That's really why we're pushing on this very hard.

When you say it's going to take time, do you take the over on Dario's 2 years to a country full of geniuses in a data center, or is that the kind of time we're talking about?

6. Simulation Becomes Scientific Method

I'm going to give a hot take real quick, which is that using mechanistic interpretability for scientific discovery, I think there's a decent chance that even if you were in a world where you had a bunch of geniuses in a data center, this would be their preferred way of doing science.

There are so many barriers to running experiments in the physical world: normative barriers, but also just physical barriers, paralyzable barriers. The rate of scientific progress could be rapidly accelerated by moving as much scientific experimentation as is reasonably possible to simulation and doing it on chips. That looks to me a lot like what people are doing when they're training autoregressive genomic models or diffusion-based material models. They're running simulations of the physical world on chips, and interpretability is your scientific toolkit that you use to actually understand what that simulation is doing and extract principles from that simulation that can then inform further experimentation and ultimately real-world scientific progress.

An interesting example here is how pharmaceuticals are created. They're often developed with something in mind, and then they're tried. They have a range of side effects and a range of other effects. Most of the time, they don't work. The vast majority of the time, they don't work the way people think they're going to initially. And then these companies just keep these drugs around because the process of manufacturing a new drug is expensive. They get the patents, and then they try the drugs for a bunch of other things. Eventually, they may find something that works, and that becomes the treatment for it.

There are many examples of this, and that is an extremely inefficient way to do science. That's going to remain an extremely inefficient way to do science even in a post-AGI world. So to me, it seems pretty likely that you're going to want to run simulations, and if you're running simulations, why wouldn't you want to train giant models? And if you train giant models, why wouldn't you want mechanistic interpretability to help you make sense of your simulation?

So I guess we're kind of doing all right either way, right? Either timelines are longer. It wouldn't be an AI podcast without timelines. I was wondering when we were going to get to it. Either timelines are longer and MechInterp is useful, or what are the geniuses in the data center doing? MechInterp.

Yeah. I think that does make a lot of sense to me as a convergent evolution, basically, right? Either way, it sucks to have to do wet work. As much as possible, you want to move it all into silicon and learn as much as you can that way.

Hopefully, it's accurate. It won't always be right, but one way I've phrased the question in the past is, when is it worth it to do the wet work? As much as possible, you want to elevate and validate your hypotheses before you actually take them into the wet lab to muck around with. Whether it's human or AI genius steering that simulation process, for all the same reasons, it seems like it's where we end up.

Yeah. That's exactly how we think about it. The core thing that just needs to be true here is that, as mechanistic interpretability matures as a science, it significantly pushes the frontier of the types of experiments you're able to run on hardware, and the quality of the results of those experiments, before bringing them to the wet lab. This strikes me as overwhelmingly likely to be true.

And again, for us, we're doing this because we believe that there's a really meaningful possibility to impact people's quality of life through bringing interpretability directly to these models, which could have real downstream scientific impact.

Another good example of this is biomarkers of disease. In a clinical context, there are sort of 2 reasons that you would be interested in interpretability. There's the discovery reason that we've already talked about, but then there's the reason that being able to have explainability for diagnostics is really important.

AI can often perform extremely well in these tasks in closed settings, but if you have an AI system that misdiagnoses someone, you can't go to their family and tell them, “Sorry, it went wrong, and I have no idea why, and there's nothing I can do about it,” right? This core debuggability is an essential feature in a clinical context as well. But you also want to be able to say, “Oh, the AI gave a surprising recommendation in this case. It gave a surprising diagnosis.” Was that diagnosis a result of the AI being wrong, or was it a result of some important input pattern that we hadn't seen before?

One quick aside on multimodality. Does this line of work—because you're doing this across reasoning models, for example, and across different models from science and genomics, and there's obviously lots more to come in terms of proteomics and the higher orders of abstraction there—does this line of work bear on that?

I wonder, one kind of mental model of superintelligence that I've been playing around with lately is basically that if you take a reasoning model at roughly the current level and give it the same depth of integration into 20 modalities of interest that we already now have with images from GPT-4o and Gemini Flash, where they can clearly manipulate the image in a way that shows a deep integration of your instruction and the visual space that it's operating in, it's no longer through this bottleneck of having the language model prompt the text-to-image model. It's all joined in latent space, and you can obviously see the qualitatively different results.

My baseline superintelligence case has been: do that again for 20 more modalities, many of them in the natural-world modeling domains, and there you'll have a superintelligence because you'll be able to reason around and manipulate things. But the things you'll be manipulating will be much closer to the fundamental stuff of reality.

And in many of those cases, these are just things that people can't do, right? Nobody has an intuition for how a protein's going to fold, for example, or at least nowhere near the level that AlphaFold does.

So we talked a little bit about this last time. I want to start by saying that I have a wide distribution of timelines. Geniuses in a data center in 2 years is not outside of my Overton window at all.

But on this particular point, I'm just cramming a bunch of modalities in. I think there's a way in which we're still constraining AI progress by human data, even in this RL regime, that's worth understanding. Why is it possible for these models to reason over images and text really well? That's because we can construct tasks for which we have good reinforcement learning signals, for which this is a capability that they need to acquire to complete that task.

When you start bringing in scientific domains as well, we're often working with extremely sparse signals where it's very hard to reason about what task we could easily train an AI to do such that a deep intuitive understanding of the human genome was necessary to perform that task. But then we also have enough examples and a clear enough sense of that task itself such that we could create a strong signal for reinforcement learning to begin with.

And then what percentage of the overall task that the model would be performing do these integrate really well? I think it's not a coincidence that combining image and text and problem-solving is something that humans do all the time, and this was one of the easier things, from an RL perspective, to train AI to do once you reach a certain level of intelligence.

It's not just because those are easier modalities to combine in some abstract sense, though maybe they are. It's that we as humans are very set up to think about and construct some optimization target at the intersection of those two things. So I think this gets at the question of, do you hit a wall when you get to human-level intelligence in some way, shape, or form? Or why would you or why would you not hit a wall when you get to human-level intelligence?

And I think Tom and I might have slightly different beliefs here, but I think my belief is that we just don't know. It's easy to draw the line out from where we're seeing and say these patterns extend to everything. But I think it could also just be true that these patterns extend to the things that they've extended to so far.

Just to come back to a second question about agents, why has it been easier to build AIs that can solve a leet coding problems better than any human than to build AIs that can order DoorDash for me successfully? One of these things seems like it should require much less intelligence, in some sense, to do, yet it's been much harder to get models to do. And I think it's just because it's harder to get good training signal in the agentic use cases, right?

This is not something where we've captured a lot of data where you can write algorithmic verifiers very easily. And so, as a result, it's been harder to do. When you're extending to the scientific domains, it's even harder, right? If we had a “doing science well” verifier, that would be great, but we don't. And I'm not saying that one doesn't exist, that it's not possible to come up with one, or that maybe, if we get general-level intelligences, they can work on this problem and make progress on it.

But it's a level of abstraction for which we don't have any evidence and for which we have reasons to believe that actually setting up the conditions in which an AI can learn that task in the way that we currently train AIs might be pretty hard.

I suppose another way to put this is that the question here is: are AIs currently experiencing catch-up growth, in the sense that a less economically developed country might benefit from catching up to a more economically developed country? Or are they just on this growth trajectory? It's hard to disambiguate.

Yeah. I mean, I think that is where the other modalities, and especially the models from different fields of science, seem to be pretty strong evidence to me that they can do pretty critical tasks at an obviously superhuman level. The ability to—and we've gone well beyond just folding—model interactions and multimers and all different kinds of molecules and metal centers now with some models.

I also did an episode on it with the guys from Orbital Materials, on figuring out the mechanism of the potassium ion channel. And it's like, geez, this has gone pretty far already.

What hasn't happened there is the integration of this sort of chain-of-thought kind of thing with what I would call just intuitive physics for whatever different domain the model happens to be trained in. It clearly has a better ability to take a random set of proteins and guess how they'll interact than I do.

What it doesn't have is the sort of outer loop to be like, “I should try this. Maybe I should try this. What if I swap this one out for this one?” And so people are sitting there writing scripts against these models to do that outer loop, or maybe even doing it based on their own human intuition, one by one, in some cases.

But I have a hard time imagining a world where that integration doesn't happen. It seems like we can get enough synthetic training data and set up a situation where it's, okay, here's what it looks like to just grind through a bunch of these things to get to the point where the reasoning and modality X integration eventually just drops into place, right? I imagine that happening. I can't imagine it not happening.

The question is the depth of integration, I suppose. You might think that things are very deeply integrated in the sense of being in the same neural network, but even there you have no guarantee that you don't have two models in a trench coat. This half of the weights does the quantum chemistry, and this part of the weights does the language, and there's not really any crosstalk such that the parts that are doing the language have any access to the process knowledge of how to do the chemistry.

And the way that you would expect to get this crosstalk is if you're able to construct paired data, or data where the language ability is bearing on the chemistry and vice versa. Then we can do that relatively easily, and we can provide supervision of various sorts relatively easily in image-language because we understand these domains both quite intuitively. We can give reward—in other words, we can give approval and train a reward model and then do RLHF—or you can obtain paired data from the internet.

But in the question of where the paired data or natural supervision comes from to couple these two modalities for domains that we don't have the same kind of intuitive understanding of, we can't give the same kind of approval. Now, it might be that actually the answer is more like you say: you put them together.

Initially, they're just two separate halves of the neural network, let's say. And then you do some sort of training task where you gradually elicit the language reasoning in order to drive the chemistry part of it. But that doesn't feel like the same sort of depth of integration as you get through pre-training, where you really do seem to just have this big, massive compute, all more or less able to access the other.

Well, actually, even that's not necessarily true. How much of a language model, when it's talking about one domain, is it able to actually elicit its information from another?

Maybe let's just do the other applications, sort of move to circuits. You could frame them conceptually or potentially by customer profile. There's the sort of monitoring, classification, steering way of thinking about it, and then there's the retail and different kinds of models that you're looking into. You could attempt both if you want to squeeze them in.

7. Guardrails And Creative Control

Yeah. So I think there are 3 applications right now that we're very excited about. The list of potential applications is very long, but we're a startup and we need to prune the tree a little bit and focus on what we think the highest-leverage bets for us to take are.

Scientific discovery, which I've already talked a lot about, I won't spend too much time on. But again, just to quickly summarize, scientific discovery is: you have these models, they're modeling physical systems, and they're able to model them in ways that traditional methods can't. So they must learn something important that human beings currently don't understand. We want to explain that, and the hope and the vision there is that those explanations are, in and of themselves, scientifically useful in terms of pushing the frontier in science.

The second application is for guardrail approaches. What interpretability techniques give you is a window into the model at inference time. If you think about the current way that a lot of enterprises set up their guardrail systems, oftentimes they're playing Whac-A-Mole with the prompt at first when they're setting up the prototype.

The number of rules that you can specify and the contextual information in the prompt balloon. You end up in this very natural situation where your per-task performance is degrading because of all of this information that you have to add to keep telling the model, “Don't do this. Don't do this. Do this. Don't do this.”

People then move naturally to an LLM as a judge, but the scaling properties of LLM as a judge are quite poor, because now you have a separate frontier-model call. It's great for the labs' pockets, but it's not great for the consumer.

If you have 1,000 rules, maybe you can bunch them together in different checks, but at the end of the day, you run into the same problem of task degradation and poor scaling properties. You could train a small model or fine-tune a small model to help guardrail your larger model, but many organizations both lack the data and the machine-learning expertise to do this effectively.

So what we can do with our techniques is offer a cheap inference-time solution for models, where we can be watching the model's internal cognition and use that as an event trigger when certain things happen—when the model is potentially going to output certain things, is outputting certain things, or is reacting to certain things—to trigger programmatic responses.

This could be things like, if, for instance, the model looks like it might be thinking about PII, then let's flag that for someone for manual review in some way. Or it could be, for instance, that there are certain topics that I never want my model to talk about.

Of course, there are limitations to what can be done. There are jailbreaks and all types of things. But for real practical use cases, there's a lot of opportunity here to offer cheap, fast, and effective checks in real-world scenarios that have much better scaling properties.

Then the third category of application that we're really excited about is creative models. We're going to be launching a demo soon. In fact, by the time this airs, it might already be out, demonstrating what you can do with image models when you start to understand their latents. In unsupervised ways, we're able to recompose the elements of the image from this deep understanding of what's happening inside an image model.

We think this just offers new types of user experiences. Even as image-editing tools and other forms of AI continue to get better and better, you're still very much locked into these bespoke forms of interaction, whether that's by prompting or highlighting a region for inpainting. There are certain types of key interactions that we had in earlier classes of image design software, such as the ability to drag something, reorient something, or change some subtle property, that get lost in that process.

This image demo is a cool example, but we think we can push this, and the value proposition is even clearer in video and music, where the cost of editing them is very, very high. Wouldn't it be amazing to have generative music AI where you could say, “Actually, I want a little more saxophone in the saxophone solo,” and you could strategically intervene on the output in a way that would get specific new generations to adhere exactly to what you wanted in some context?

8. Interpretability Reaches Its Users

Maybe just my last thing before giving you the floor for the sort of closeout pitch on Goodfire: where are we? I think it was maybe a year and a half ago that we first talked about—maybe just a year; time is compressed—the idea that with these tools, you can allow many more people to get involved in the process of understanding models.

I've been fortunate enough to have a couple of early previews of different model-steering and exploring interfaces that you guys have built, and I wonder how you would characterize where we are today. Are the interfaces far enough along that we're now effectively able to enlist human intelligence, or are we still working on that and still more in the automated interpretability world because we haven't quite cracked the paradigm that allows people to make intuitive use of all these new feature spaces that you're opening up for them?

I think it depends on the problem you're trying to solve, which I know isn't a very satisfying answer. But in the case of images—and you'll see in our demo—you can reason about images and image features very intuitively, and there are ways to visualize them that don't rely on any external assumptions about inputs. It's purely about visualizing what that feature does to the output, so there are fewer assumptions, at least. I think our flash image demo will demonstrate some of this in that context.

There's been a lot of work internally at Goodfire and also at other labs, and especially in the open-source community, on different UXs and visualizations that you can have, and they all solve problems. They're all a way of looking at it. Right now, we're not in a place where we have a single interface that can tell you anything that you need to know about an arbitrary model.

We're working on some things in the biology domain that we hope will generalize. But yeah, it's like everything else, right? All these tools have limitations. It's about the problem that you're trying to solve and what you can learn in the process of trying to solve that particular problem.

Cool. Well, it's been a fantastic conversation. I really appreciate the time, and I just want to do it more regularly going forward. Take us home. Maybe give a few highlights, things that you're most excited about, and things you guys are looking for. Last time, if I do say so myself, I understand there were a couple of inbound business opportunities, so you can put the bat signal out there for whatever you want to come your way this time.

9. Goodfire Scales Its Mission

Yeah. So at a high level, we're looking for people to join our team at Goodfire. If you want to work on what we all think is the most important unsolved problem in the world right now, you can join us to help reverse-engineer what's happening inside models. I think one of the really exciting things about Goodfire is that we're pushing on this problem from a bunch of different angles.

We're not saying, “Let's just push what we can do with these tools,” or, “Let's just develop new tools.” We're doing both at the same time, in dialogue and in concert with each other, which is really important to the ultimate progress. We're looking for great engineers and great scientists who are very motivated by our mission here and want to open up the black box.

From a customer's point of view, if you fit one of the profiles that we talked about—if you're someone who's training a scientific model, if you're an enterprise looking for more reliable usage of your LLMs in production, or if you are someone training a creative model and you're looking for new ways that you can open up the experience of the creative model—please reach out to us. You can email me at dan@goodfire.ai. We'd love to talk to you and understand whether or not there's a good opportunity for some collaboration there.

And I'll mention it at the top, but we should also probably mention you just raised a bunch of money, including—

Yeah.

Taking Anthropic's first-ever external investment. So what more beyond that headline would you impart to people?

Yeah, so we raised $50 million, led by Menlo Ventures, and $1 million of that from Anthropic as Anthropic's first-ever corporate investment. We're just extremely grateful to have investors who really believe in what we're building, the mission, and the problem we're trying to solve here.

We think it's really important to have a company not directly tied to a scaling lab that is directly trying to solve the problem of interpretability, and we intend to use all this money we've raised to help customers understand their models and to push the frontier of understanding in mechanistic interpretability and, as we were saying before, hopefully across a bunch of scientific domains and unlock lots of new, awesome experiences that we can have with AI.

Our core belief here is that interpretability is as big as AI itself, and so there needs to be a lab that's focusing on interpretability and nothing else, because the size of the opportunity here is really large. We're grateful to have all of the support from investors like yourself, Nathan, and all the people who really believe in the thing that we're building, and we're just motivated to get out there and start making things happen.

Yeah. Well, it is definitely a good candidate for the most important problem facing the world today. So it's been a pleasure and definitely a fascinating journey to just try and keep up with all the progress you guys are making. Again, come back soon, but for now, Daniel Balsam and Tom McGrath, CTO and chief scientist at Goodfire, thank you again for being part of The Cognitive Revolution.

Thank you so much for having us.

Yeah, thanks for having us.

It is both energizing and enlightening to hear why people listen and learn what they value about the show. So please don't hesitate to reach out via email at tcr@turpentine.co, or you can DM me on the social media platform of your choice.