拆解神经网络机制：Goodfire 的 Lee Sharkey 谈基于参数的可解释性

Today I'm speaking with Lee Sharkey, principal investigator at mechanistic interpretability startup Goodfire, about fascinating recent work that he and his co-authors have done to start moving beyond analysis of the concepts that neural networks represent between layers and actually begin decoding how they compute within and across layers.

We begin by discussing why an understanding of concepts isn't enough. On one level, this needs no justification. An approach that explains only the meaning of the intermediate results between layers, while it does take a serious bite out of the black-box problem, leaves the layers themselves as smaller but still unexplained black boxes. More concretely, research has also highlighted important weaknesses of the feature-centric approach.

Conceptually, a sparse representation of features inherently loses a lot of potentially important information, which is encoded in the structure of how features are clustered together and otherwise meaningfully arranged in space. For example, in some networks, the days of the week are represented not by 7 random directions in activation space, but by a set of directions that lie together in a plane, such that a simple rotation operation can act as a sort of next-day function that converts a given day of the week into the next. If history is any guide, there is presumably a lot more such critical complexity to be discovered.

With this motivation in mind, and taking inspiration from sparse autoencoders and similar techniques that effectively separate clean concepts out of their usual state of superposition, Lee and his team have similarly set out to decompose a neural network's parameters—which, of course, are used to process layer inputs into outputs—into simpler subcomponents that they hope will correspond to interpretable mechanisms that the network has learned.

Their first approach, published earlier this year, was called attribution-based parameter decomposition. Lee describes this as a sort of model unmerging. If you imagine counterfactually having started with a giant mixture-of-experts model and then merging all the experts into a single instance of the architecture, parameter decomposition would be the process of unmerging to recover those hypothetical original experts.

Personally, I visualize it as expanding a neural network, which of course has width and depth defined by its architecture, into a third, vertical dimension, where each vertical slice consists of a simpler subnetwork that's presumably needed only a small fraction of the time. When all the vertical dimensions are again collapsed down to a single model, they all add up to the original network weights.

Whatever mental model you prefer, amazingly, by constructing a loss function that incentivizes 3 things, you can actually get this to work. First, faithfulness, or the accuracy of the sum of the sparse networks relative to the original full network. Second, minimality, or the idea that as few network slices as possible should be active for any given input. And third, simplicity of the discrete subnetworks derived from the process.

I really do think it's worth taking a moment to appreciate how incredible this is. As Lee says, channeling Ilya, sometimes the models really do seem to want to learn.

That said, given the complexity of the target, you shouldn't be surprised to learn that this first method was still far from ideal. In addition to being computationally expensive and requiring massive memory for all those parameter copies, training success was extremely sensitive to hyperparameter choices. There were also conceptual issues with the gradient-based way that performance was attributed to parameters.

Specifically, in some cases, critical components that had approached a local performance maximum would have a near-zero gradient and would be misclassified as unimportant. These weaknesses naturally inspired the second method that we discussed: stochastic parameter decomposition.

For efficiency, instead of describing network subcomponents as sparse instantiations of the original network architecture, the new approach breaks each weight matrix into rank-one components. These are matrix operations that read from one direction in activation space and write to another, and they can later be regrouped into larger units.

Also, to better identify which subcomponents really matter, they replaced the gradient-based approach with a novel approach that uses stochastic masking to help the network learn to predict each component's causal importance, effectively identifying which elements the overall network can't do without. Overall, it's a more scalable, stable, and accurate approach. On toy problems with known ground-truth answers, it is able to successfully recover the expected mechanisms.

Of course, as with any branch of interpretability, plenty of work remains. One big question is how we can algorithmically group the rank-one subcomponents into semantically meaningful mechanisms. Much as sparse autoencoders require a feature-labeling process, we still need to figure out how the rank-one components ladder up to conceptually intuitive transformations. And, of course, there will be plenty more challenges with scale.

But as Lee explains, the promise of this work is tremendous. This kind of understanding could enable everything from surgical unlearning of capabilities to the identification and interpretation of novel scientific insights that models may have learned, including, and perhaps especially, from non-language data.

All things considered, this conversation is a great chance to visualize what's going on inside neural networks; to ponder how much we've learned and how little that still seems to explain; to marvel at the fact that gradient descent and backpropagation can effectively optimize such complex architectures, even under such complicated, multi-part constraints; and to appreciate the combination of brilliance and plain hard work that goes into figuring all this out.

It's right up there with understanding our own biology and arguably no less important or challenging. Understanding how and why neural networks do what they do is one of the grand scientific challenges of our times. So I really hope you enjoy this conversation about decomposing neural networks in parameter space with Lee Sharkey of Goodfire.

Lee Sharkey, principal investigator at mechanistic interpretability startup Goodfire, welcome to The Cognitive Revolution.

It's great to be here, Nathan. Thanks so much. I'm a huge fan of the show.

Thank you. That's an honor. I'm excited for this conversation. I always love to learn, and I expect to learn a lot over the next hour and a half or so.

Real briefly, before we get into the technical work that's going to be the primary focus, you've moved recently. We've had 2 guests from your previous organization, Apollo Research, on the podcast over time, focusing on risks of deception and the identification of deceptive behaviors in frontier models. We've also had 2 different episodes with folks from Goodfire. Dan and Tom have done a great job orienting us to everything that's going on in mechanistic interpretability.

You were doing mechanistic interpretability at Apollo and recently moved to Goodfire. So maybe give us just a little bit of an update on how that came about, and then we'll get into the work itself.

Yeah, absolutely. Apollo Research and I—I started Apollo Research with a bunch of folks, including Marius Hobbhahn, who you mentioned you had on. We started it, and we were focused on detecting and ideally mitigating deceptive behaviors in frontier AI systems.

One of the goals there for my team was to focus on the mechanistic interpretability side of that. If we can read the thoughts, so to speak, of these frontier AI systems, maybe that will give us a leg up on detecting deceptive behaviors without necessarily needing to rely on their outputs.

I think we did a lot of good work. I was really happy with the work we got done there, and I really loved the organization we built. I continue to support the work that Apollo does, and I continue to be somewhat involved there.

But we figured that it was probably better at this time for Apollo to really double down on the eval side of things, which left my team, which primarily focused on mechanistic interpretability rather than evals, a bit in the lurch. We all agreed that this was the right step.

Meanwhile, maybe 6 months before we actually decided to make this decision, I had—this seems like ancient history now—ended up connecting Tom McGrath and Eric Ho, who went on to co-found Goodfire together. So I had a bit of a B-character role to play in the founding story of Goodfire.

I was obviously very aware of Goodfire and massively supported the work they're doing. I think the team is absolutely amazing, and it just felt like a really natural fit to move over from Apollo to Goodfire. It made sense for some of the team to come along as well, so we can basically continue the stuff that we were working on there.

Was it 3 of you that made the move?

Yeah. Me, Dan Braun, and Lucius Bushnaq.

Gotcha. Cool. Well, that's great. I assume the $50 million Goodfire raised to support both the team and compute certainly didn't hurt the value proposition either, right?

We'll get into a bit more of where the compute costs are for some of this work, but everything seems to be dependent on a healthy dose of compute. So the ability to draw those resources in from the private sector, as opposed to Apollo being philanthropically funded, definitely seems like it could be a big differentiator long term for the ability to scale what you're doing.

Yep, 100%.

Cool. Well, we're going to primarily focus on 2 papers today. One of them came out still under the Apollo banner, and the more recent one came out under the Goodfire header. The first one is called “Interpretability in Parameter Space: Minimizing Mechanistic Description Length with Attribution-Based Parameter Decomposition.”

I think there's an effort not to retread well-covered ground too much. So I definitely would refer listeners who want to go deeper on that particular paper and have a really thorough exploration of it to the AXRP podcast that you did not too long ago. I thought Daniel did a great job there, and I listened to the full thing to help inform myself coming into this.

I guess I'll just ask you the first question. What is attribution-based parameter decomposition? Or maybe, even more narrowly, what is parameter decomposition? How does it differ from the schools of interpretability that people are probably familiar with? You could probably assume people, if they've listened to our feed much, at least are aware of the basics of SAEs.

Yeah. So what is parameter decomposition, then? Well, let's contrast it with activation-based decomposition. Typically, in interpretability, what we've done in the past is wanted to understand what is going on inside a neural network. So we've collected lots of activations showing how the network has processed its input data. You've got these intermediate activations, and then it spits out an output.

What we've typically done is started with the idea that these activations should, in some sense, be involved in different things that the network is doing in order to compute its intelligent output. There's some sort of learned algorithm that the network has learned to exhibit its intelligent behavior. Activation decomposition is basically the idea that we can look at these intermediate activations, piece them apart, and say, “This part does this, and this part does that.”

By contrast, parameter decomposition is philosophically similarly motivated, but instead of picking apart the activations, the idea here is to pick apart the parameters. Now, why might you want to look at the parameters? Well, in some sense, the thing that we're really interested in is the neural network. The neural network is not its input data, although it interacts in very special ways with it. The neural network is, in some sense, the parameters, the architecture that pieces them together, and the nonlinearities that connect them.

It uses all these components to transform the input data into activations and transform those activations until it gets to the output. And in some sense, then, the parameters, the architecture, and the nonlinearities are the things that we really want to understand. They're implementing the algorithm that the network has learned. The activations, in some sense, are along for the ride, but they do interact in very particular ways where it's not quite as separable as that.

The basic idea here, then, is that the network is using its parameters in different ways for different inputs. For an input where there is a cat, it will take that input and spit out a cat label, say. For another input, maybe a picture of the Eiffel Tower, it may not use very many of the same parts of the network as it did when looking at a cat.

And so, in some sense, there's this idea of modularity. Parts of the network are doing one thing, and other parts are doing something else. There's a specialization in what the parameters are doing. The aim for parameter decomposition is to find these modules that are doing specific jobs, specific computations. The term we're looking for here is mechanisms—the mechanisms that the network is using in order to compute its behavior.

So, to flesh out that cat example a little bit more and to say some really basic stuff, all these neural networks are composed of layers. Often, the layers have the exact same structure repeated over and over again, although not always. We've observed, by doing interpretability on the activations, that we can see things like fur being detected, eyes being detected, and a tail being detected. We sort of see this gradual move from low-level features to higher-order features as we go through the layers.

There's a story of understanding that kind of mirrors—especially in these visual cases—what I understand to also be happening in the human visual system.

And so that's a remarkable coincidence.

But when we are purely looking at the results of the intermediate calculations, we can sort of say what concepts are active at any particular time, with lots of caveats. As discussed in an earlier episode, there are philosophical gaps between the labels of these concepts and the underlying question of what exactly is being activated there, and how it relates to the labels. That relationship is a bit fraught, but nevertheless, bracketing that for the moment, we have these labeled concepts that we can say: This feature always lights up for all these inputs. All these inputs seem to be fur, and so that seems to be the fur concept that's activated.

When the tail, whiskers, eyes, and pointy ears, or whatever, are all activated, then in future layers we get the cat activated. But we're not saying anything there about how the transformations are happening from layer to layer, right? That's the big gap that has been left by the sparse autoencoder work for future work, and this parameter decomposition is basically that future work that is now saying, “Okay, how do we actually move from these concepts through the layers, one to another?”

I agree with that. I'll say also it's not the only approach that you might consider using in order to piece together how these representations at one part of the network may become other representations at other parts. There are other approaches that remain in activation space that you might consider using also. But, yeah, they both try to achieve this idea of characterizing the computation between representations rather than just identifying the representations themselves.

Maybe one way of thinking about this is identifying the variables used in computations rather than identifying the computations. Parameter decomposition and other approaches aim to find the computations rather than the variables.

So why do we need that? Why isn't it enough to say, well, we've got concepts A, B, and C in layer 1, and then in layer 2 we see concept D? Doesn't that tell us what the neural network is thinking, so to speak? What is not answered by that level of analysis that we still need to get clarity on?

It gets somewhat philosophical. However, I'll try to convey how I think about this. I would say that we don't—what does it mean for the network to use particular variables? We can go in and say, “Look, there's this set of activations, and this set of activations seems to correspond to data points on which there are cats.” But maybe we can also find groups of inputs where there's a cat, but it's a cat standing in a particular position.

One group of your inputs is a cat standing in one position, and another is in another position. Now, does the network use the fact that there are 2 different positions of these cats, or does it just have a cat variable? What would it mean to be able to distinguish between a network that just has a cat variable and a network that has multiple different cat variables?

My proposition here is that what it means for a network to use particular variables is that these variables are the variables that the network does its computations over. Now, it may be the case that this network that had multiple cat-in-different-positions variables actually might not use those things. To the network, they're all just cats.

But you may nevertheless be able to look really in detail at the dataset and find lots of little distinctions within the activations, but the network might not itself actually use these. In some sense, we want some way to identify the set of things that the network is using.

To do this, then, in some sense, you might say this is putting computations first and representations second, rather than putting representations first and then computations second. It's like saying the fundamental, so to speak, variables that the network is using are those over which it does computation.

We can maybe find things that the network seems to represent, but there's not a particular reason to say that this is a feature that the network uses just because we can find it in there. Does that make some sense?

Yes, I think so. Though, as you said, it does get a little philosophical. I was also thinking of the finding of multidimensional features as another window into why the assumptions that underlie the SAE paradigm may need to be elaborated and may need a richer treatment.

Do you want to tackle that as well?

Sure, yeah. Since some of the earlier SAE work and the latest bout of sparse coding work in the deep learning space, which started in late 2022 and picked up in 2023, some of the observations people had made were that the representations we were finding seemed to be spread across multiple different layers. It was confusing to think about what that might mean. There were also observations that these things we were finding were directions in activation space—particular patterns of neural activation—but that didn't seem to be the whole story.

There seemed to be some sort of multidimensionality to them. It wasn't just one direction; it was perhaps a plane of directions, or even higher-dimensional spaces potentially. It was unclear how to think about these higher-dimensional features, so to speak, and it was philosophically somewhat confusing.

Several of these observations motivated some of the thinking that led to our attribution-based parameter decomposition. What if, say, a multidimensional feature just is an input variable to a multidimensional computation? That's why the network might have gone to the length of structuring its representations in this way. Without there being some sort of multidimensional computation applied to it, you might ask, “Why did the network bother to structure its activations in this really ordered way?”

There must be something for which it is using that order. This was why we were thinking that maybe it's the computations that are defining the structure of these features, and maybe the same kind of thing might apply in the multilayer case.

For certain networks, like transformers that use a residual stream, it's very easy for computations to be spread over multiple layers because it just works out straightforwardly for them to do this. It was confusing to think about this in terms of SAEs as well. It made natural sense to think, “Maybe it's the same feature if it interacts with a part of the parameter vector that happens to span multiple layers.”

For this kind of edge case as well—the multilayer edge case—it made a bit more sense to think about the computations rather than treating representations as primary. An example of that is the days of the week. I think the original grokking result with modular addition would probably fall under this heading.

It could vary across different networks, obviously, but at least in some settings it has been found that there aren't 7 days-of-the-week features that light up independently. It's not that there's a really strong Monday signal or a really strong Tuesday signal. Rather, there seems to be a plane in space through which the day of the week rotates.

Remarkably, it seems to crystallize into this structure. The graphics for this are pretty amazing sometimes: you pretty much divide the circle into 7 equal sections, and each day of the week has a different direction in this plane, as opposed to each of them having their own independent linear directions.

Basically, I think the same mechanism—or a similar structure, if not the same mechanism—has been found in the grokking result. To do this modular addition, things are first translated to an angle, then the angles are added trigonometrically, and then the result of that trigonometric calculation gets mapped back onto the final number.

That's seemingly hard to figure out just with features. Maybe you can give a little more technical intuition for why that would be really hard to figure out just with features, but it seems like, on some level, it may just be that—

I guess I don't know exactly what you would find if you applied the standard SAE methodology to one of those things. If I'm not oversimplifying the standard SAE methodology, you would expect to see 7 distinct features. In the modular addition case, you'd expect to see—I’m not even sure exactly what you'd expect to see—but I don't know that you'd expect to see a sort of cyclic trigonometric structure if you were looking for all independent features.

Yeah. I think it's probably easiest to think about the days-of-the-week case. To be clear, it can be both that there's a multidimensional aspect to these representations as well as individual-day aspects to them.

The reason is that sometimes, in some tasks, you might want to say, “The day after Tuesday is Wednesday,” and you might just apply the computation that rotates the day-of-the-week feature. Now you've got one mechanism that you can apply to every single day of the week.

But in some settings, you might really just want one day. You might say, “I was on my way to the shop on Monday,” and there's really just one direction that you want to represent there. You want to say, “This is Monday.” In some settings, you may want computations that apply to these individual days, while in others you might want this higher-dimensional computation, this two-dimensional one that rotates it.

This is an important aspect of parameter decomposition. It's not one basis that you might use to interpret the whole model. It's more that there are steps in an algorithm, and sometimes these steps will be useful while others will not.

The SAE approach might not find this shared representation or shared computation that just rotates from one day to the next. Conditioned on prompts that say “the day after” or “the day following Tuesday,” you might find one feature for Tuesday, one for Wednesday, and one for Thursday, just to rotate around every single time.

This is because it's trying to find sparsely activating parts of the dataset that have a particular shared property—in particular, parts of the dataset that have a particular shared direction in activation space. That's not necessarily all the ways in which inputs might share properties.

One of the ways in which they might share properties is that they use similar mechanisms. These mechanisms may do the same operation on different inputs, depending on what the prompt is. I'm not sure if that was clear, but this is one of the differences you might expect between parameter decomposition and an SAE-like approach.

There are other approaches that might get closer to what you might expect or want to find with an approach that identifies computations. For instance, Anthropic recently released an update on CLTs. I’ll need to remind myself what that stands for, but there's a similar principle here: instead of looking at individual directions, it's looking at more multidimensional computations.

Yeah. So I guess maybe another, very simple way to say it is that we want to also understand the functions. If the whole project here is to understand why the AIs do what they do, and possibly be able to intervene in certain ways.

We’re just going to have to dig in and figure out the nature of the transformations as well as the intermediate results. The intermediate results can shed some light on that.

But at times, there may be assumptions in that approach that leave us blind to—I thought a key point you said was functions that are reused on different inputs and do the same semantically meaningful transformation on a given input. That is something the SAE approach could be entirely blind to, but which obviously is a pretty important aspect and potentially happens a whole lot.

I guess we don’t really know yet how much that may happen, but it certainly seems like it probably should happen a lot, right? Especially as we get to more and more powerful networks, it would seem like we are moving away from—or graduating, perhaps, in a sense—from simple, rule-based combinations of distinct features and probably toward higher- and higher-level abstractions. Those would presumably then correspond to more functions that can operate over ranges of inputs and do the same useful transformation to some space of inputs.

Yeah.

In a way, that’s chunking reality, right? And that’s an important aspect of chunking reality that it seems like the SAEs are sort of hinting at but not really directly characterizing.

This feels reasonable. I think one of the cases that I like to think about is in an SAE, or, say, a transcoder, which is an SAE that takes as input some activation and spits out a prediction of what the activation will be at the next layer.

Both of these will have a problem where, if the network is just doing a simple transformation—suppose it’s a rotation, or suppose, indeed, that it’s actually just an identity transformation—you might have to spend a lot of representational capacity to represent the computations that the network is doing at this particular transformation. Why? Because you’ve got, say, 1,000,000 different features, and there are 1,000,000 different features in the output, especially in the case of the identity transformation. There are going to be 1,000,000 there, and you would need 1 input feature and 1 output feature for each such input-output pair.

In essence, what the network is doing is actually very simple. It’s just a transformation of this type, whether it’s an identity or a rotation, and you’ve just got 1 object doing that transformation. That’s the kind of thing that we want parameter decomposition to be able to find, if it is indeed the case that networks are doing that kind of thing.

Yeah. Okay. Well, we’ve got some examples of that coming up later.

So, let’s bracket that for a second and maybe just talk about how it works and the intuition for how the whole setup works there. I think there are a lot of similarities to SAEs. I’ll do the quick recap of the SAE. To do an SAE, you insert this very wide layer between 2 layers of a network.

The thinking is that we know the normal width of a network is way smaller than the number of concepts that it can handle. Clearly, there’s some superposition of concepts. It’s not like 1 neuron corresponds to 1 concept. Any direction in that 4-, 8-, or 16-, or however-many-thousand-dimensional space can represent a concept, and so the goal is to untangle those into a sparse thing.

For that, you need this really, really wide layer. You train through this reconstruction loss, where the goal is to project out into this really wide space but have a sparsity term in the loss function, so that some small, reasonable number of truly relevant features are activating. Then you project back into the dense space so that you’re recovering the behavior of the model, but you can look at which things are activated.

Then you go through this labeling process at the end to say, “Okay, here are all the inputs that led to this particular thing lighting up. What does that in fact seem to be?” With that, we now have this thing that we can use, hopefully, for monitoring, detection of potentially bad concepts being activated, and potentially steering. That’s where Golden Gate Claude comes from, et cetera.

It seems like there’s a pretty similar motivation here, saying, “Okay, if concepts are densely packed in this superposition way at the activation level—as the results of the intermediate calculations, which we call activations—then perhaps the same is true about the computations themselves.”

The network itself represents or combines a huge number of probably much simpler, more semantically intuitive calculations, computations, transformations, functions—whatever the exactly right word is there. They’re all so densely packed and overlapping, and the sparsity of the data set is also a key thing here, right? The reason that this can work is because certain concepts that may be pointing in very similar directions almost never occur together.

Even though they may look very similar, since they almost never occur together, that’s fine. The network can sort of get away with that, and there’s probably that sort of thing happening in computation space as well. The same question kind of applies: Can we break this up into a really wide version of it that separates the flattened layer, or the flattened network that we originally trained, into all these more atomic units of computation?

If we can do that, then does it appear that they actually have intuitive meaning? We can start to go through them painstakingly and understand what each one is doing. How am I doing there in terms of just creating the motivation? Is there anything I’m missing, or anything you would correct or complicate?

There’s plenty I would complicate, but I think it’s a really good intro. I think that’s just the right way of thinking about it.

We think that there are many different computations going on inside this network, more indeed than it has neurons. The way we think networks can do this is that they spread their computations out over their computational units, where these computational units are the neurons. By spreading it out, you get some nice properties, such as avoiding—or, even though you might overlap with other computations, because you’ve spread this out, you can kind of suppress the noise that comes from this overlap.

You don’t necessarily need to do anything more complicated. You can basically fit more in because you’re silencing the stuff that you don’t really want to be there—the kinds of computations you don’t really want to be active right now. It’s very similar to the principles of representation and superposition, but brought into computational space.

I think the other thing you mentioned is that there’s this extra layer to what the network is doing. It’s not just that we have a layers-and-width dimension anymore. We kind of have this extra dimension where you’re asking, “What computations is the network using at this particular layer, at this particular width dimension as well?”

This does have precedent in other areas of deep learning. Mixture of experts might be an example of this, where you have layers and individual networks, but you can do different computations at the same layer, at the same neuron dimension, so to speak, depending on which expert you use.

There are senses in which this is kind of like, “Well, suppose every single network is just a big mixture of experts that’s been smushed together. Can we separate out the experts?” One way I’ve heard it described is like model unmerging, where model merging is just like you’ve combined these 2 networks that do different things into 1 network that can do both.

And we kind of want to do the opposite. We want to split this network out—this network that can do many different things—into networks that can only do one thing, or as small a number of things as possible.

Yeah. So the setup is to actually do the training of this, to start to do this splitting. I think your work does a great job of really emphasizing the loss functions. And so the constraints that you put on this are—and I think there are a couple of different versions of it. We'll maybe start with the first one, and you can explain how the second one is new and improved and better in a couple of key ways.

But if I understand the first one right, it's basically saying, okay, let's make a bunch of copies of the network with essentially the same footprint to start, and we'll have the constraint that all these different copies must sum to the original. So if you had 1 billion parameters in your original model and you decided you were going to split this into 1 million different subcomponents, you would then have 1 quadrillion parameters, which we can talk maybe about the computational challenges and memory requirements for things like this.

But now we've got the constraint that those 1 million copies must add up to the original. We then want to say, okay, akin to the SAE sparsity requirement, to make what comes out of it hopefully intuitive and tractable and semantically natural, we want to have as few of those be active as possible for any given input, which you call minimality. And then the final one is, for each of those things, we also want to make them as simple as possible so that they individually are interpretable.

Again, I think of this as giving the network height. Obviously, the width of the network is how many neurons it has at each layer, and depth is how many layers it has. Think of this as splitting it into a vertical height dimension, where instead of having all this stuff happening in the same track of computations in a way that we can't untangle, we're now untangling it into all these different, hopefully atomic, computational subcomponent units, only a few of which will be in use at any given time. Ideally, those can be distilled to their simplest form while still, of course, reproducing the original network behavior.

A visual I have of this comes from the Tegmark group. I think Shuming Lu was the lead author on this paper, Seeing Is Believing, where they just trained tiny models on simple problems with a strong sparsity feature. You could watch these great graphics where an initially randomized network learned a particular function, but you could also see that most of the weights would drop out to zero as the thing crystallized into its simplest form that could still do the task it needed to do.

Those are relatively simple toy problems. So, again, what would you add to my setup? And now we're going to do this giant training, optimizing all of that together, which is kind of an amazing thing to me. I felt this way about SAEs, and I think I feel it even more strongly here, that you can put all those constraints into a single optimization problem and it works.

If you told me that in advance—if you said, “Here's my idea. I'm going to put all these different terms into one single joint loss function, try to optimize it all together at the same time”—I would say, “Good luck.” But that seems hard to find something that's actually going to work in that space. So tell me if I'm missing anything, or anything you think is important to add for understanding there. And then I'm really interested in how you account for the fact that this actually can work.

Yeah. You did a great job introducing the various losses. Just to recap, we have the faithfulness one, which makes them sum to the parameters of the—first of all, what are these things that we're summing up? These are what we're calling parameter components, and these are the things we want to approximate: one individual job or one individual mechanism that the network has learned.

So we have the faithfulness loss, which makes all these parameter components sum to the parameters of the target model. We have the minimality loss, which basically makes it do the same job as the original network, but using as few of these parameter components as possible. And then the simplicity loss, which means we want these things to use as little computational machinery as possible.

That one is pretty important because, well, they're all important, but this one's important because a great way to satisfy the first 2 losses is just to use the original network itself. You're only using one thing; it sums to the parameters of the original model, but you haven't done any work to decompose it. So you need them all to be simple. And the way we operationalize simple is that they should be low-rank and involve as few layers as possible. We can get into the technicalities of that in a bit.

But I hadn't actually made the connection between, say, the work that Shuming Lu did—you said?

I think, yeah, I think so.

My apologies for messing up their name. But I hadn't really made a connection between this kind of work and parameter decomposition. The idea here is that suppose you aren't optimizing a network with the kind of sparsity constraints, and I believe some sort of locality constraint that they might have used too, so that neighboring neurons did similar things, if I recall correctly.

But suppose you weren't optimizing it with these constraints. Well, the network is, in some sense, still able to learn. I would guess that the network is able to learn a very similar algorithm, no matter whether or not it's trained to be sparse in a particular basis and constrained so that these neurons are close together. I would guess that we just need to figure out the way that the network is doing the same algorithm, but in some sort of basis that we don't have direct access to.

If the computations are the same, they both have to be sparse. We just need to find the basis in which they are sparse. So, yeah, we are assuming that the network is doing sparse computation even if we haven't optimized for it, and then we're just trying to find the basis in which it is indeed sparse.

With regard to—sorry—how do networks manage to do these things? Well, the networks, I think it was Ilya Sutskever who said, “The networks, they just want to learn.” I think there's something really deep to this, in that there's just many different ways for the larger your network to get things right. And this is a pretty powerful principle, I think, just because it means it's actually much, much easier to find the right way to do something if you're simultaneously looking in many different directions, many different directions in parameter space.

And this is, in some sense, somewhat unsurprising: you can add many more different constraints, and it'll find some reasonably satisfying way to satisfy all of them. But it is counterintuitive to our low-dimensional brains to think in these terms. It's still kind of amazing.

But I should also qualify this and say that the algorithm we're talking about—attribution-based parameter decomposition, where your parameter components are very large and it's very computationally expensive—has a bunch of problems. It is still somewhat hard to get this particular algorithm working, which is why we ended up doing future work. So even though it is still kind of amazing that it can find some solution that satisfies all these things, it's caveated with the idea that, at least for this algorithm, it was somewhat hard for that algorithm, but less so for others.

Yeah. One practical question on the faithfulness concept is the idea that all the different vertical dimensions, let's say, of the identified subcomponents have to sum to the original. Can you help me develop my intuition for exactly what's going on there?

And I should say, most of the problems that you've worked on so far, I'd say, were roughly at—you could complicate this, but it seems like we're roughly at—the Toy Models of Superposition phase of this work, where the models that we're studying are rather small. A big question, of course, is going to be: what does it look like to scale this up?

But it's not like a million parameters yet; that's the key point there. Yeah.

But when one parameter in one particular layer gets turned up, that means that that same parameter position in all the other layers has to be turned down by the corresponding amount, right?

In these different parameter components? Yes, as long as they all sum to the parameter.

Yeah. And I said layer there, but I meant vertical layer, which is a concept maybe we should retire, but I am thinking about it in this vertical visualization, I guess, for myself.

One thing I wasn't clear on there, and I was thinking about this concept of attention sinks that I've seen in the past. We're calling this “on the fly,” but it's been found, in certain cases at least, that having a few sort of junk token positions at the beginning of a transformer can be really useful. Because in the absence of that—in the normal approach—there's got to be something to look back to in attention, and so whatever tokens happen to be in those first 5 to 10 token positions become really important because everything kind of has to look back to them.

The introduction of this buffer, which they called an attention sink, was a way to say, “Let's not overweight the initial tokens of this sequence.” In some cases, you might just want to look back to sort of nothing and recognize that these initial tokens aren't actually super critical to predicting what comes next. So let's not force all attention to land on those initial tokens.

And that attention sink concept—I was kind of wondering, could there be sort of a junk sink concept in this setup? How do I not end up in a situation where my new subcomponents are learning whatever they're learning, which is recreating the initial model behavior?

But maybe some of the other layers, which would never need to get used, are in fact just taking on the opposite of whatever the ones that are actually doing the computations are learning. Could that create a sort of conceptual disconnect where my new vertical layers are learning computations? Yes, but how do I know that those are faithful? I know that they still sum, in that sense of faithfulness, but maybe the computations that have been learned in these layers could be quite distinct from what is happening in the original network.

Maybe that's being hidden by the fact that some of the other new vertical layers are just absorbing all these gradient changes in ways that don't actually mean anything because those never get used. You've maybe just trained some new stuff that didn't actually exist in, or doesn't necessarily correspond to, a mechanism in the original network.

Let me see if I have understood what you're saying. We have these parameter components, and they satisfy the constraint that they all sum to the parameters of the original model. You're worried about the possibility that, even though they all sum, there are a lot of different ways in which you can sum, and in one of these we're worried about the case where some of the computations that, say, one of these parameter components is doing are kind of canceled by another one. Is it the case that both of these would be active?

Well, I'm assuming that if there is sort of a junk sink component, it would presumably very rarely be active.

So, in many networks, we do expect there to be a junk component—a junk component that actually does nothing. This is just because neural networks are degenerate. There are many different ways to implement not just something that works well, but the same algorithm.

One degeneracy you might think of is if you scale up the weight before a ReLU and scale down the weight after ReLU. This is very much the same algorithm. The same kind of amount of activation gets put back into the residual stream, but there's a 2-dimensional space here—in fact, it's a 1-dimensional space—in which we can move parameters such that exactly the same thing is done. And there are many more such degeneracies in neural networks.

One of the degeneracies that networks might use is that all of the activations that the network ever sees are, say, orthogonal to this direction in activation space. So this direction in parameter space is, in some sense, never used, and you could indeed ablate this direction entirely in parameter space, and it would just not affect the outputs of the original model.

Now, this would need to be an overparameterized model. You might not expect this kind of thing in the underparameterized case, such as language models. But in, say, an MNIST model, you might expect a junk component. In an MNIST model of sufficient size, you might expect a junk component that is just not really used for the algorithm.

Now, would you be able to—yeah, I guess, but I'm not sure this is exactly the same kind of junk component that you're talking about. Maybe there's something else in mind.

Well, I guess maybe another way to frame the question is: how do you know where the real learning or algorithm is? Where is the real learning happening? Where is the real algorithm implemented?

Another kind of inspiration for this question is going back to some of the Neel Nanda work on the Othello board-state type thing. I recall that there was at least the possibility of confusion where you want to know what is being represented inside the model. But if you train a big decoder model on the internal states and try to predict board state from the internal state of the model, at some point you're like, “Well, maybe my new network has learned to do that, but that doesn't really mean that my original network had a semantic understanding of this concept.” Maybe that kind of only came online in the interpreter model that I trained.

And so I guess I'm worried about a disconnect between the original model. We've split it into these many copies; those individual layers, those vertical layers, those subcomponents are evolving through this training process as they learn to be sparse and reproduce, and have only a few of them activated at a time.

Yeah.

But I guess I'm just wondering what is tethering those resulting subcomponents to the original? I could imagine that I can make a functional change that gets me the desired behavior, and I can offset that in some other one of these subcomponents, which maybe never gets used. Then I could potentially, if I do that long enough and hard enough, arrive at a spot where the algorithm that the new thing implements is quite distinct from the original.

I'm not sure how I would know that. Is there something that I'm not catching that is preventing that from happening conceptually?

For attribution-based parameter decomposition, that is, I think, a fairly reasonable concern. I don't know if it happens in practice. I don't know if the other losses, in some sense, implicitly penalize it. It will become a lot more obvious why this can't be the case if and when we move on to stochastic parameter decomposition. We can talk about that now, or I'm happy to park it until we get there.

Yeah. Well, let's do it. I think we're pretty much there. We can add this to the list of possible problems, whether or not it's an actual problem with the original method.

But let's just run through maybe what made this not the last word, or why you needed an upgrade. You mentioned that it's not super easy to get working—there's a lot of hyperparameter dependence. It's not super easy or stable to train. So that's one.

There's computational expense, which maybe you can unpack a little bit more. There's this conceptual drift possibility that I'm flagging. And I also wanted to hear a little bit about reliance on attribution methods and why that's a problem as well. So maybe you can just run through those, and then we'll move on to the new hotness.

Yeah. So, this algorithm was extremely janky. It was just very difficult to get it to do what we felt was a sensible thing to do. The later algorithm has much less of this issue.

For instance, you mentioned that there were a bunch of different hyperparameters that were necessary to tune in order to get a decomposition that made sense to us. One of the reasons here was that you're using a top-k parameter, where you're only allowed to activate the top-k most attributed parameter components for any given input. You want to train these attributed components to be able to do better at the task to which they were attributed. So you're basically training them to reconstruct their output.

This top-k parameter is one of the very fiddly things to get right here. It's very—what's the word?—discontinuous, basically. You can very quickly go from one component being active to slightly updating the parameters such that another one comes online instead of that one. Now, for a very small nudge in your parameter space, you've got a very different output. You're implementing a very different function for a very small change in parameter space.

That’s just inherently hard to optimize with gradient descent. Another issue is the sheer number of different hyperparameters. But I think the other main one is that you’re really relying on your attributions to be right. If they’re, in some sense, wrong or systematically biased, you might end up in a situation where there may not even be a kind of stable optimum in your training landscape.

Because if your biases were slightly off, they might nudge you somewhere else. You might move over there, and your attributions are systematically biased in some other way, so they might nudge you somewhere else again. The attributions that we were using were gradient-based, which makes some intuitive sense because they’re trying to capture what we wanted.

What we wanted was some number that told us how important this parameter component was on this input. Gradients are a kind of proxy for this because if your output changes a lot for a small change in this parameter component—that is to say, if the gradient of the output with respect to this parameter component is large—you can say that it’s an important parameter component. But there are some cases where this is just straightforwardly not the case.

One example might be attention. Suppose you have a transformer and it’s attending very strongly to 1 particular sequence index, and attention there is almost 1 and everywhere else 0. If you nudge the parameter component responsible for this very strong attention to this sequence index by some small amount, the attention is basically saturated. The gradient is actually very small. It’s not going to move your attention very much, despite this being a very mechanistically important parameter component because it’s implementing such strong attention.

That’s just 1 of the ways in which gradients may not be a good attribution method. We actually want some other method that tells us more accurately how important this parameter component was, and this was 1 of the developments that we introduced in the follow-up paper on stochastic parameter decomposition.

I think that’s a perfect transition. That brings us to stochastic parameter decomposition. My general read of the paper is that it’s very much a natural follow-on, right? In fact, I went through the AXRP podcast and the first paper, and then wrote the first half of this outline. Then I read the introduction to the new paper and thought, “Basically, the first half of this outline sort of mirrors the introduction to the new paper.” So I felt good about that.

It’s basically the same goal and the same concepts—not exactly the same loss functions, but sort of the same conceptual constraints, just operationalized differently this time—and it addresses the weaknesses that you just ran through in the first case with better results. So maybe take us through what’s new and improved first, and then we can unpack exactly why you did it. I do have some questions about the intuitions behind some of the decisions.

One of the new things is in the headline: stochastic parameter decomposition. The stochasticity relates to the replacement for the attributions in the previous paper. We still need, in some sense, these attributions, but to avoid confusion with the previous method, we can call them something else. We’ll call them causal importances. If something is causally important for the network’s algorithm, it should be attributed, and we want something that will let us approximate how causally important this subcomponent is.

Before we get into the details of the causal-importance calculations, it’s also important to note that 1 of the other major differences from the previous method is that we’re no longer using this frankly ridiculous size for each of the parameter components. They’re not a randomly initialized copy of the original model. They are, in fact, just a randomly initialized rank-1 matrix for each of these layers.

The way we think about this is that if, in the previous method, we had these parameter components that spanned all layers and all potential ranks in those layers, we’re basically just pre-splitting it up. We’re saying, “Pretend everything was just localized in rank-1 matrices in each layer.” Later on, the idea is that we group these things back together into full parameter components, such that if we did indeed have rank-2 parameter components or parameter components that span multiple layers, we could find them after we’ve grouped these things together.

Those were 2 of the major differences between the attribution-based parameter decomposition, or APD, and stochastic parameter decomposition, or SPD. It’s probably not reasonable to get into the causal-importance calculations, which is where the stochasticity comes in.

Can we take just 1 more second on how you’re breaking the original network down into these rank-1 components?

Mm-hmm.

So again, in the original, you’ve got a network. It has a certain architecture and a certain parameter structure, and you make N of those. Is it really random initialization on all those copies?

Yeah, interestingly, it’s similar in the subcomponent that we’re now using in stochastic parameter decomposition. These are called subcomponents rather than components because we will group them together later. These subcomponents and parameter components are all randomly initialized, and it’s this faithfulness loss that makes them all sum to the parameters of the original model.

That faithfulness loss is pretty strict, so they don’t really get to deviate very much from the parameters of the original model as you sum them. At first, they just converge quickly to—

Yeah, to sum back to the—

I guess I would have naively thought, in the first case, more along the lines of: If you’re going to divide the thing into 1,000 vertical layers, just take all the parameter values and divide them by 1,000. Then have 1,000 weak copies of the thing, some of which can be turned up or down or whatever. Why randomly initialize instead of doing something more principled, or a simple transformation of the trained network itself?

I think we explored a bunch of different initializations. I don’t recall the exact 1 that we used in the APD paper. The example that you gave—dividing it down and making exact copies—is something that we explored, and we may even have used a randomized version of that.

You can’t really do this in the stochastic parameter decomposition case. Suppose you have a 1,000-by-1,000 matrix and divide it up into, say, 10,000 subcomponents. I don’t know if there’s a guarantee that if you sum up all but 1 of these parameter components, the remainder is also rank-1. Does that make sense?

You assume that your 10,000 subcomponents sum to the parameters of the original model. The idea would be that we might as well just constrain this sum to be exactly the parameters of the original model. We could take the sum of the first 9,999, and then for the remainder, let that be whatever makes the sum exactly equal to the parameter of the original model.

I think this final 1 may be of arbitrary rank. That feels intuitive to me, but I’m not sure if it’s 100% true. It feels like that 1 may be of arbitrary rank. If we want them all to be rank-1, then we can randomly initialize them.

There is a slightly more involved initialization such that they point in similar directions to the original model, so that their cosine similarity with the original model is at least not negative at initialization, despite being rank-1. But the details of that are probably not super important. It should just work with random initialization.

Random initialization.

So how should we think about these? The original setup is pretty intuitive to me to envision. I’ve got this thing, I split it into all these vertical layers, and each one gradually, over time, becomes simple.

Not that many are active. I can visualize it as: instead of this dense and uninterpretable computation, I now have n of 100 or n of 1,000 that are actually active, and those are each relatively simple and create the same output.

I can sort of visualize that, and I'm like, “Yeah, okay.” These things feel like circuits. Each of those things feels like a circuit, and it's kind of okay: if I activate these 8 circuits for this input, then it works, or this other one takes 12 for a different input and it works.

Each one intuitively feels like it's doing some sort of information processing through the original architecture. That's kind of intuitive. Here, I'm a little less able to tell that story because I'm like, at each layer, we are breaking up whatever matrix exists at that layer into a bunch of rank-one parts.

How should I think about those rank-one parts? I had to go refamiliarize myself a little bit with what exactly rank is and what it means. In the simplest terms, it's just a much simpler, smaller part of the overall transformation.

Mhm.

Specifically, referring to dimensionality, you have some n-dimensional space. Something that is rank one basically exists as a line in that space, right? Something that is rank two exists as a plane in that space, and so on.

Yeah. So instead of a transformation that can operate in n dimensions all at once, we've got a bunch of transformations that all operate on one linear direction in that broader space. Now we're composing those sequentially in a way that recreates the original.

How should I be thinking about those rank-one things? Why do that, I guess?

Yeah. I don't really view them as fundamentally different. Suppose we consider the case in the old algorithm where we have this parameter component, and this parameter component can span all layers and all potential ranks.

One of the penalties that we were emphasizing was this simplicity penalty, where we don't want it to involve too much computational machinery. We want to be able to study it as simple an object as possible. The way we operationalized this was that we penalized the rank of the matrices in this parameter component. We also penalized it for existing in multiple layers.

The idea here is that if this parameter component really was just localized in a few layers and in a few ranks, the rest of its parameters should become zero in all these different layers. There should just be a low-rank set of matrices in the layers in which it is involved.

To go into a little more detail about why that is simpler: if it's rank one, as you mentioned, this is just a rank-one matrix. Basically, it reads from and writes to one particular direction in activation space. The only direction in which it reads is the direction defined by its right singular vector, and the direction to which it writes is the left singular vector.

This matrix will not be used if the activations don't, in some sense, overlap with that read direction. If the right direction has no relevance downstream—if it has no causal importance downstream and doesn't do anything—then this rank-one matrix won't do anything.

Given that we're already looking for these low-rank matrices in these full parameter components, the idea of stochastic parameter decomposition is that we can—well, one of the ideas we introduced in stochastic parameter decomposition was that we'll just start with these low-rank versions of the subcomponents, and then later on we can aggregate them.

If we want to combine 2 in order to create a rank-2 parameter component, then we can do that. If we wanted them to exist in multiple layers, we can aggregate them together as well.

They're really just chunked versions of the same fundamental object, this parameter component, this vector in parameter space. One is just something that we'll group together later on.

Okay. Well, let's come back to how the grouping is done, because I think that's another potentially quite vexing, quite deep question. But for the moment, we've broken the matrix into—if it's an n-dimensional space, we've now got not just n but actually could be more than n rank-one subcomponents.

How about a little help on that intuition as well?

Yeah. One of the core ideas in the representations-in-superposition line of work is that there are these features in activation space, each of which individually represents a single feature, a single thing in the world. But there are more of them than there are neurons that represent them.

The same idea holds true here. In parameter space, we can represent more directions in activation space by using more rank-one matrices. This gets back to the read and write direction.

If we want to read from more directions than we have neurons, we just have more rank-one components than we have dimensions in our activation space—more rank-one matrices than we have columns in the matrix. It's possible for this to read directions, to read these representations in superposition, if we have more of these rank-one matrices in our individual matrix.

The idea, then, is that these rank-one matrices are what will implement individual computations, or parts of individual computations, that the network is doing to compute its behavior. The challenge is basically to identify which ones are important for a given input, an input data point, and when they're not important.

That's where the stochastic calculation of causal importance comes in.

Yeah, maybe one more question before we go to causal importance. We've broken these things into all these rank-one subcomponents, and there's an interesting question, too: what is the ratio of how many rank-one subcomponents you need for different sizes and complexities of the original network?

In the relatively simple problems studied so far, that ratio isn't super high. I'm wondering whether that ratio is going to become extremely high if we get into language modeling. It seems like probably yes.

Potentially. We will probably have more computations than we have neurons. That's just the name of the game in taking things out of superposition.

One of the questions I ask myself often is whether it's going to be more or less efficient than the dominant approaches at the moment. These dominant approaches may be sparse autoencoders, transcoders, cross-layer transcoders, and so on. The basic question is whether it's going to be more or less efficient.

It's unclear, but I think there's reason to believe that if this is a reasonable way to decompose networks, there's probably going to be fewer of these subcomponents than there are, say, latents in a sparse autoencoder or a transcoder.

The reason is that there's no upper bound to how many latents you can have in your sparse autoencoder or your transcoder, whereas there is a kind of upper bound on the number of parameter components that you might have. Why? Because they all have to sum to the parameters of the original model.

This prevents you from just adding in an extra component or an extra latent and cranking down your reconstruction loss. There is kind of a—it should level off a bit faster, I would say. But this is an empirical question that remains to be answered for the kinds of models that we care about: large language models or other similarly interesting models.

Is that sort of like saying that while reality is complicated, the rules of physics are simple? The number of fundamental transformations, the number of fundamental functions that are used to carve up reality, should in theory just be a much-reduced space that can then operate on many more inputs?

Maybe.

I'm thinking of Platonic Representation Hypothesis-type notions. It feels like it's getting at something true. Although, yeah, it's not obvious to me right now how to connect it to the way in which I think of things, but it feels like it's getting at something true.

Yeah. Well, more work to be done, for sure. How expressive can these rank-one things be? They can represent, for example, a rotation. So is that right?

The rotation is a very simple kind of rotation. It is just that, to the extent that an activation or an input activation projects onto one direction, it will now project into this other direction. You can get arbitrarily high-dimensional rotations, up to the dimensionality of the input. But I think it is true to say that it can do a rotation, although it is a very simple, low-dimensional kind of rotation.

But that would be enough, for example, to handle the transformation of Tuesday into Wednesday with a sort of next-day operator. I think you basically need 2 for this particular transformation because these variables live on a plane, and so you would need to project 2 directions in a particular way. You’re basically leveraging 2 such 1-dimensional rotations in order to do this 2-dimensional rotation, I think.

So this gets us back also to the grouping concept being quite important. Let’s come back to that again in a second. I think I’ve belabored the rank-one thing enough. We’ve broken this thing up into some rank-one subcomponents from the original n-dimensional matrix. Now tell us about this swap-out of the attribution method to the causal importance—the learned estimation of the causal importance.

Yeah. One of the ways you might think about what the attribution method was doing was that we wanted something that said, “If we turn this parameter component off, it shouldn’t really affect the output.” The way we did that was just—we literally turned them off and only kept on these top-k parameter components. What this is really saying is that it shouldn’t really matter if this parameter component is on or off. It could be on to its full extent, or it could be off to its full extent, or anywhere in between, and it really should not affect the output.

The top-k approach was just not really great at optimizing for this, such that if you did turn it on a little, it probably would still do stuff. Whereas what we’re basically doing in stochastic parameter decomposition is, in fact, learning a function that predicts, for a given input, how ablatable this is—how turn-off-able this subcomponent is. If it is turn-off-able, then again, it shouldn’t matter how much we turn it off by; it is just causally irrelevant.

We have this causal importance function that predicts, for a given input, how ablatable this subcomponent is. We call it the causal importance value for this subcomponent on this data point, and it is supposed to be a number between 0 and 1. To the extent that it is 0, we can basically randomly mask this subcomponent on this input anywhere between 0 and 1. That is to say, we can turn it on randomly anywhere between 0 and 1. If this causal importance value is 1, then we don’t really get to randomly modify or mask it at all. It just has to be on because it’s really causally important.

We literally just have a neural network that spits out this 1 number for each subcomponent. If the subcomponent is very causally important and it gets the right causal importance value of 1, then great. But if it is causally important and the network accidentally spits out 0.5, then this parameter subcomponent can be masked anywhere between 1 and 0.5. If it’s masked close to anywhere that’s not 1—anywhere along this random distribution—it’s going to damage the loss.

What this means is that gradients will be able to flow into this causal importance function that produced this number, using basically the reparameterization trick to let us pass gradients through this random sample and output this random distribution. That way, it actually does learn to approximate how ablatable this parameter component is for a given input.

I think this was not my idea, but I think it’s a really great idea that I actually haven’t seen elsewhere in the attribution literature, and I think it has applications far beyond just the interpretability approach we’re having here. It doesn’t have to be applied to parameter components; it could indeed be applied to parts of an input image, in the kind of way that attribution methods have been done in the past. We haven’t done that ourselves. I would be keen to see someone do it. But yeah, it’s just an attribution method that really captures how causally important this component was for this input.

And so is that a pretty big matrix that’s been added for that, or a network that’s been added for that? Because it’s got to go from the input space to the output space of all the subcomponents, which are all the rank-one components at all the layers.

This causal importance function can basically be of arbitrary architecture. It can indeed, as you suggested, be a map from the input to the number of components per layer times the number of layers. That would be a very large network, and this is not, in fact, how we implemented it in the paper.

In the paper, we just had a very, very small, 16-parameter—I think it was 16, or it varied somewhere between, say, 16 and 128 parameters—thresholding network that took as input the activations. I mean, it gets a little bit mathematical here, but it’s not super complicated. You multiply the activations by the right singular vectors of the parameter components, which is basically asking how much these parameter components are reading from the activations. Is this one reading a lot? Is this one reading not very much? You’re measuring the overlap between the activations and the right singular vectors of these components.

To the extent that the parameter subcomponent has a lot to read, and to the extent that it’s active, this is a number that you can basically do a fancy thresholding with using a small neural network. That’s just 16 times however many parameter components you have, which, in the scale of the overall network, is not very large. It doesn’t really even need to be a very fancy neural network; it can just be some sort of simple learned threshold. We found in practice that it helped it learn nice kinds of thresholding functions that were not completely straightforward. They had some kind of structure to them, but overall it was just a slightly fancy threshold.

That’s interesting. And then when the whole thing is run—when it’s trained and when it’s run—that’s now part of the reconstruction, right? Because again, all this is jointly optimized: this whole prediction of which components or subcomponents are going to be active, and the structure of the subcomponents themselves. That’s all being jointly optimized in one big compound loss function. And then when you’re actually running the replacement network for the study on different inputs, that is still part of it, right?

It can, and usually is, part of it. But you can throw away this causal importance network and just sum up all these subcomponents together and run it like the original model. If training has gone well, this should look very much like just running the original model. It’s like running everything with a mask of 1 instead of 0; it should do exactly the same thing. The causal importance function is not necessary. What the causal importance function is letting you do is basically telling you which parts are actually being used on this input.

You can imagine a world where you can throw away—suppose there was a subtask of particular interest—you might be able to throw away most of the network and only keep the parameter subcomponents that are used on this distribution of interest. That may indeed, depending on how large the model is, be a much smaller set of parameters than the whole model. But yeah, that’s future work.

Yeah, interesting. Okay, so why does this work better? I guess, how does it work better? Tell us about the results, maybe go problem by problem that we had in the last version and talk about the updates you can get in maybe the toy problem. Compare and contrast. And again, I’m really interested in whether you have any intuition for why this works. It still seems very magical that all this just shakes out.

Yeah. So with regard to attributions, this is just much closer to what we actually wanted from gradient attributions, and it doesn’t have the kinds of biases that gradients would have. It should, for instance, be able to tell us—it should be able to cope with the saturated attention case that I mentioned earlier when we were talking about gradient attributions. I think it might also be time to revisit the garbage component that you mentioned.

Basically, I think we were talking about a case where maybe the network has learned these subcomponents and, in fact, maybe the subcomponents are actually doing a slightly different algorithm from the original network. But because in attribution-based parameter decomposition, you could switch them off and never activate them, you can kind of compensate for one component that is active and doing a different thing while still summing up to the parameters of the original network.

In stochastic parameter decomposition, you can't really do that anymore, because you're just randomly activating every part of the network basically all the time. If it is doing something that it shouldn't be doing, that's going to modify the activations in a way that the original network just didn't modify the activations. And so it's going to be bad for the loss.

It's important to realize that we're not actually able to turn off subcomponents in SPD. There are no dead components or anything. What it means for it to be a dead component is not that you never activate it, but that you always activate it randomly. I think this is just fundamentally different from being completely silenced all the time and gets rid of a lot of pathologies, like the one that you mentioned.

It also gets around this idea of the discontinuity that I mentioned in the top-k optimization, where it's no longer the case that you're only activating the top-k and the rest are silent. Everything again is active all the time. Which means that if you're in APD, where you're using top-k, if you're silenced, gradients aren't really flowing through you. You can't really learn to be better; your parameter components can't really learn to be better. Whereas if you're randomly on all the time, gradients are always flowing through every subcomponent. And so it feels like there's a lot more continuity in the learning process.

This pans out in terms of the algorithm still being somewhat sensitive—you still need to get the hyperparameters somewhat right. However, it's just not like in attribution-based parameter decomposition, where not only was it sensitive, but it was just not well-behaved. You'd change it one way, and it would change in an unpredictable fashion. Whereas in stochastic parameter decomposition, there's a range in which your hyperparameters need to exist. But because things are well-behaved, you can actually just find them without having to—it’s just easier to understand which thing you need to change in order to get to the right solution. So it's just much nicer to work with than the previous algorithm.

That said, the method that we introduced in the paper just isn't perfect. There are plenty of things that, even since publishing that paper, we've focused on changing and for which we've found better versions, in sampling this randomness and in the gates we used in the causal importance functions and so on, just to make it a better-behaved algorithm overall. There's still optimization to do on the algorithms, but it is just significantly better to work with than the previous method, which makes us excited for scaling this up to larger models, which we've begun to do—for example, small transformers, including language models.

Okay, let's talk about feature splitting. You had said that some of the results here show that the new method avoids feature splitting, I guess. Coming from an SAE perspective, I sort of think feature splitting is good. I think of it as being just like, as I make my SAE bigger and bigger, then I get finer-grained resolution on the features, and I expect to see these sort of vanilla features get resolved into French vanilla and normal vanilla or whatever. And that seems good, but tell me in what sense is it bad, and in what sense has this helped us avoid that problematic version of feature splitting?

Yep. So we've talked already about one of the examples with the cat and the cat in a number of different positions, and it's not clear whether or not the network really is actually using, in some sense, these more fine-grained features that you can identify. From a practical point of view, we don't want our explanations to explain more than they need to. We don't want to explain potentially how the network takes one of these cats in one position and applies circuits to it, and then do it for the next cat in another position and so on, but we're wasting our time if the network is just not really using these variables in its algorithms. And so we basically want to avoid doing unnecessary explanation. We want our descriptions of what the network is doing to be minimal in some sense.

Another particularly pathological case of feature splitting is feature absorption. Say you have, in one SAE—a smaller SAE—a feature for words that begin with the letter E. But then, in a larger SAE, one that has more fine-grained features, we have a feature for elephant and then a feature for everything else that begins with the letter E. Or it splits into other words that begin with the letter E. In some sense, this other decomposition, the one with more features, is just less interpretable, because there was no need for us to split out elephant. Maybe we just want—again, we just want the variables on which the network is working.

There's no obvious way to avoid this using, say, SAEs. It may be the case that there's a perfect stopping point for SAEs such that beyond this point you're using variables that the network is using, and before that you're not. Sorry, yeah, there may be a point at which you go into a regime where you've got SAE features that correspond to variables the network is not using, and before that you've got only variables that the network is using.

But that's not obvious either, because you could find these sparsely activating directions in activation space, but there's no real reason to expect—well, no guaranteed reason to expect—that these are indeed the very way in which the network is chunking up its computations. Maybe it is, in fact, a group of these SAE features over here that the network is actually using, and maybe you actually did need to split further for this group over here. There's just no guaranteed reason that SAE features will find variables for computations.

Now, this is not necessarily a criticism that applies to various other variants of the sparse dictionary learning paradigm. And I do want to caveat everything we've been talking about here in that there are people whose views I respect a lot who see issues with our approach; we see issues with theirs. But overall, I don't want to give the impression that the algorithm that we're putting forward here is the final word on anything in interpretability, or that it's definitely finding the right thing. I think there's a lot more research to be done on this, and I just wanted to make clear that there's still more to be done.

So then, how about the actual grouping, labeling, and semantic understanding of all these rank-1 units? There was one bit—and this is one of the toy problems; we don't have time to do the full setup—but there was this sentence that SPD splits its corresponding W_out into 50 subcomponents. So, in other words, this one matrix that's been broken down into rank-1 components has 50 rank-1 things that all remain active.

And then the paper says that these 50 subcomponents appear to be effectively part of a single rank-50 component comprising the entire W_out matrix. And there I'm like, how do I know? What does that mean, and how do I know it's right? Doesn't it leave us with a similar problem to where we started, which is, what's going on there? If we've got 50 things and we group them back into one thing, how did you know to do that versus determining that there are 50 different things happening?

That part seemed to go by very quickly, and I wasn't quite clear on it, let alone how to extrapolate it into larger and larger models in the future.

Yep. Yeah, no, it's a really great question. I think what's important to appreciate is that whenever we're optimizing these things and whenever we're saying that this is the right decomposition, there's an asterisk here. We're saying this is, for a given amount of decomposition, the decomposition that breaks it up into as few components as is necessary to explain a given level of functionality of the network.

We can basically scale up and down the amount of decomposition that we want to do and break up the network into larger or smaller parts. There is some reason to prefer some regions of this spectrum over others. One reason might be that the reconstruction loss, as you scale things and decompose them more and more, eventually reaches a point where it gets a lot worse. Correspondingly, there may be a point where your causal importances—these values that are supposed to tell us how important each subcomponent is—behave a bit pathologically. For instance, they might typically take fractional values rather than values that should be 1 or 0.

Ultimately, if something is causally important, it can be partly causally important, which lets us shed light on how much performance we might lose by ablating it. But by and large, we prefer regimes where it's a bit more binary. There are points along this spectrum, as we modify the hyperparameters controlling how much decomposition or activation sparsity is going on, that give us clues about what decompositions make sense.

In some sense, none of these decompositions are right. This holds true for sparse dictionary learning and parameter decomposition. None of these decompositions are right in some sense because we're dealing with one fundamental truth: the original network. We're really just trying to get a lens on what it is doing under the hood. But none of our lenses are going to be perfect; they're always going to introduce approximations of some sort.

This is just a necessity. We're all trying to discretize a fundamentally continuous object in various different ways. Whenever you do that, you're simply going to throw out some nonzero amount of approximation quality. But we're willing to pay these costs to get tractable explanations, even though there isn't one actual ground-truth decomposition.

Yeah. Okay. Maybe last question. I'll bundle a couple of last questions into one, and you can take as long as you have. Where do we go from here? How expensive is this going to be to scale up to, let's say, the small but meaningful language model regime—your sort of 3B or 7B models? Is that the next leap that you're going to try to make, and what practical applications do you find most exciting?

Obviously, SAEs can do the sort of monitoring and steering. Are there fundamentally new use cases that you see opening up here? And, if you have time for it, is there an intersection between this and the science models? I know Goodfire's been doing some interpretability work on science models, and that's one of the things that excites me most, in addition to just keeping the general-purpose models on the rails. The idea that we could make scientific discoveries by understanding what the models have learned is pretty exciting.

So what's the next chapter? What's it cost, scale-up-wise? What are the new capabilities that that unlocks, and any comments on science? Then we'll let you go.

Cool. The next steps are definitely to scale these up to more respectably sized models. I think we've tried this out in the region of low single-digit billions, and we've scaled it down to even smaller language models in the millions, just to get some early traction before scaling back up to those sizes.

There are a lot of interesting empirical questions that we want to answer there, like how much we can actually decompose these language models into low-rank components. Maybe the things they decompose into are too high-rank for us to understand, which would mean we need to do extra work after this first step of decomposition. But we'd at least get a reasonable decomposition into a bunch of components that are themselves easier to isolate and understand.

Some of the applications that I think this opens up could include making unlearning more straightforward than with SAEs, because you're already working in parameter space. You already know, “This is the direction that does X”; you can just modify this vector in parameter space, and you have a more straightforward lens onto functions as they relate to parameter space. Whether or not this beats some gradient-descent-guided approach is a separate question, but I think monitoring is also something that will be useful to the same or greater extent than SAEs or other approaches.

Why? SAEs might be detecting a particular region of the training dataset—a too-narrow region, perhaps. Instead, what we really want is that whenever the network was using function X, that's when we want to pay more attention and go into detail on what it's doing, rather than asking whether there was an activation in this particular direction. There will be some correlation between these two, but there will be some things we want to monitor for that might be better thought of in terms of functions.

To get less abstract, maybe there's a function for deception, and there are many different situations in which a model might be deceptive. The input directions might be different, and so the intermediate computations might point in different directions, in a way that SAEs might struggle to identify as, “This is the one direction that does deception.” Instead, it may have many different directions that do this. But in sparse parameter decomposition, and in other versions of this approach that we hope to develop later, we might be able to identify the individual components that are doing this functionality.

Lastly, we really want to extract knowledge from these models, some of which are superhuman at some tasks that we've trained them to do. We want to actually think in the terms that the network itself is thinking in, because that is, in some sense, the knowledge that the network is using. We hope this may be a way to get a lens onto that, in a way that may or may not generalize better than other approaches. But that remains to be seen.

Cool. We'll definitely be following your work here. It's fascinating stuff, and I'm glad to have you on while we're still in the relatively early phases of it. I think this mega-project of figuring out why these things do what they do—and whether there's any way to really make ourselves confident that we'll be able to predict what they're going to do in novel circumstances—is about as high a priority as any in my mind.

I appreciate that you're working hard and have taken at least one good bite out of that problem, and I'm really looking forward to hopefully seeing this line of work mature into something that can tackle the biggest models and the hardest problems. It would be great to ablate that deception circuit if we can find it. So, any other thoughts you want to leave people with before we break?

I think that's honestly it. It's been a really great conversation. We had a lot of fun, and we're just super pleased to be on.