# The Neuron Basis: Goodfire's Curt Tigges on Agentic Interp, Monosemanticity & Reverse Engineering The Cognitive Revolution · 2026-11-10 · 116 min · https://www.youtube.com/watch?v=hqXYtqQq50g ## Transcript ### Rethinking individual neuron interpretability Nathan Labenz My guest is Curt Tigges, a member of Goodfire’s technical staff, here to discuss the new article “Neurons as an Interpretable Basis,” coauthored by Nick Cammarata. It’s a more academic, LLM-focused follow-up to Nick’s recent viral essay, “Neural Networks Are Already Clean.” The work directly challenges the established belief that neural networks are such tangled chaos, and that individual neurons are so often reused for different concepts, that we can’t hope to decipher them without tools such as sparse autoencoders. Curt’s position, as you’ll hear, is that individual neurons are much more interpretable than previously thought. Most have one main function, and those functions are better understood as verbs and logical gates than as entity recognizers. Importantly, LLM agents can now analyze a network and describe the role of each neuron with impressive accuracy. We begin with an overview of the model they study: Qwen3.6-35B-A3B, a 35-billion-parameter model with around 5 million MLP neurons distributed across 40 layers. Each layer has 256 experts, each with 512 neurons, and 8 experts are activated during each forward pass. We also discuss the expressive architecture of modern SwiGLU MLPs, which provides two separate signals for each neuron: one linear and one controlled by the SwiGLU activation function. ### Understanding gated MLPs This specific MLP architecture was proposed by Noam Shazeer in 2020 in a five-page paper, where he wrote, “We do not offer any explanation of why these architectures work. We attribute their success, like everything else, to divine favor.” Six years later, I’m struck by how similar this structure is to gated architectures in Mamba and other prominent state-space models. My own intuition is that these parallel paths allow models to assemble semantically relevant information while effectively filtering it by importance. And because the activation function allows negative values, a neuron can respond differently in the positive and negative directions. Next, we move on to the methods Curt used to support the idea that a small number of large activations can be enough to preserve high performance. The key concept is sparsity. Curt shows that it’s possible to remove approximately 90% of individual neurons, keeping only those with the strongest activations, with surprisingly little decline in performance. In some cases, such as ARC-AGI puzzles, this intensive ablation actually improved results. Although ablating neurons made the model worse at reasoning tasks, the authors show that a small amount of additional training with the neurons ablated is enough to restore most of the lost performance. Having established that weaker activations may be less important than many people thought, we move on to the LLM agent they use to scale up the work of describing individual neurons. Given a corpus of text and interpretability tools, the agent generates and iteratively refines paragraph-long hypotheses about what each neuron does. Those hypotheses are tested by querying another model: based on the description, where in a given text fragment should this neuron be expected to fire? The authors then show that the model’s behavior can be substantially restored, compared with completely removing a neuron, when the description generated by another LLM is used to synthesize that neuron’s activation. Along the way, we also touch on monitoring as an obvious application of this work, small clusters they call neuron communities, and the possibility that individual experts in mixture-of-experts architectures are more interpretable than previously thought. We discuss how this work relates to current thinking about how models store and recall facts, as well as the roles of the residual stream and attention mechanisms. We also consider whether small activations might have greater significance in long-context scenarios, the path from here to ambitious interpretability-driven model design, and how to think about scaling laws for interpretability. Finally, we ask what, if anything, this work suggests we should do differently during training to create more interpretable models. My main conclusion is that this work is a significant step forward in interpretability. Where many experts previously assumed that understanding a model’s internal mechanisms required auxiliary models, there is now good reason to believe we can study neurons directly, at scale. At the same time, it remains to be seen whether this low-level understanding can be raised to a higher-level understanding of circuits and behavior. There’s still a lot of work to be done. And although I understand why people are eager for hope after a summer with so much bad news about our prospects for alignment through behavioral training alone, we should be careful, as Curt is throughout this conversation, not to jump to premature conclusions. The problem of interpretability remains unresolved. Curt Tigges Thank you, Nathan. I’m glad to be on this podcast. Nathan Labenz I have a feeling this is another installment in the Goodfire series on mechanistic interpretability, and in some sense, perhaps the most important one. That’s because of your close collaboration with Nick Cammarata, who recently made a splash in the field of AI with his essay “Neural Networks Are Already Clean.” Now we have a sequel that goes deeper, focuses in particular on language models, quantifies the findings in different ways, and applies new methods to the problem. I’m very happy to discuss it and dig into all the details. Before we get started, I’d be interested to hear what you see as the main argument or key conclusion you want people to take away from this work. In my view, one distinctive feature of the article is that, more than most other work, it seems to be trying to change people’s minds—to provoke a paradigm shift. So perhaps you could state your main argument, and then we can take it apart. Curt Tigges The main conclusion of our experiments, and what we’re trying to convey in the article, is that individual neurons in MLPs in neural networks are generally more interpretable than people previously thought. There’s long been a group of people trying to interpret individual neurons, but it was seen as very difficult. People believed that representations would be distributed across many neurons rather than localized in individual ones. Our message is that we can understand them, and that often you don’t need a lot of neurons to be active at the same time. Even when a neuron has more than one value, which can be confusing, it’s usually not an infinite number of different values that are impossible to understand. Instead, it’s a compact set of values. That’s the main message. Nathan Labenz One detail from the essay that I’d like to understand better is an idea I’d summarize this way: a few years ago, when many of these hypotheses were being formed, maybe neurons were less interpretable because they were trained with somewhat inferior methods. So perhaps the basic substrate has improved. It seems to me that most things tend to become less interpretable. We worry that a model’s train of thought isn’t reliable or that it can’t be controlled. What has changed? Did it get easier? Curt Tigges I think it’s a combination of 2 different things. One is that we have better tools. We have agents that can review all the instances where a neuron is activated, look for patterns, highlight them, and provide a compact set of descriptions of what the neuron does. We created an agent called “neuroscientist” that looks at this and provides an analysis. We also have better ways to observe individual channels that enter and exit the neuron, so we can track and quantify those processes. But the other thing, in my opinion, is that you’re right: many early networks were difficult to interpret. They were more chaotic, and of course, smaller. This may be a bit of a deep dive, but I also think activation functions have some effect. ReLU is the most famous, but in the networks we’ve explored so far, ReLU and GELU are harder to interpret, while SwiGLU seems easier to understand, probably because it lets you create a kind of microcircuit capable of some logic. That opens up opportunities for better interpretability. Nathan Labenz I’m certainly no stranger to trying to get agents to do all my work, so I really sympathize with this approach. Let’s take a minute to orient ourselves to the model that’s the main object of study in this article. The essay had much more material about different types of vision models. Here we’re talking more about a language model. So, the model you’ve studied the most is Qwen3.6-35B-A3B, which, as many know, is a mixture of experts with 3 billion active parameters at any given time. Can we describe it in a little more detail, perhaps even exhaustively? For example, how many layers does it have? I read that there are 5 million neurons, but I want to understand the structure a little better. How many layers is it divided into? How many experts are in each layer? Within an expert, what are the width and depth of an individual expert? I also want to spend some time on the 2 signals that reach each neuron and how we should understand them. Honestly, that’s what I probably didn’t understand as deeply as I should have before this conversation. Curt Tigges So, yeah, as you said, this is a mixture-of-experts model, and it has 40 layers. Each layer has about 130,000 neurons, if I remember the numbers correctly, and they’re divided into 256 experts per layer. At each layer, the router decides to send the input data to 8 of these 256 experts. It just goes through those experts in turn as a small subnet of the whole model. This means that only a few neurons in the model are active simultaneously, because only 8 out of 256 experts are working. Each expert has 512 neurons, and they’re always turned on to some extent, usually just a little, but some of them have strong activations. Those are the ones we pay special attention to. A few more interesting facts about this model: although we have 5 million neurons, there are actually 10 million values that neurons can express, since each neuron has a positive and a negative activation, and they can have different meanings. The model reads them geometrically in different ways. Sometimes, if you have one neuron that’s related to a certain topic, one pole will be about one topic and the negative pole about another. Sometimes they’re related, but there’s a lot of diversity. Other reasons we chose this model are that it works quite well on tests and has a form of attention called Gated DeltaNet. It has regular attention, but only in a quarter of the layers; three-quarters are Gated DeltaNet layers. This model has many features that are also present in the largest and most advanced open-source models. Therefore, it made a good subject for our research. Regarding SwiGLU activations or individual neurons, and what you mentioned previously about the 2 paths to a neuron: in principle, we can assume that each neuron has an up projection and a gate projection. Instead of the input entering the layer of neurons and passing through 1 set of parameters, it passes through 2. These are 2 parameter matrices that are stored separately so that they can recognize different things. In practice, the gate matrix reveals some aspect of the input, and the up projection matrix reveals something different in the input. When they do their calculations, the results are multiplied. This allows you to get a kind of conjunction. For example, if the gate matrix captures context and the up projection captures a specific topic in that context, maybe the context is cooking and the up projection recognizes an ingredient, such as garlic. When they multiply, the activation becomes extremely strong. Nathan Labenz I don’t know if you know much about the intellectual history of this, but traditional multilayer perceptrons, which I think most people imagine, are simply rows of neurons, each with a parameter that determines how much its activation will be multiplied by and connected to each of the positions in the next layer, right? This is the classic version. This is what I’m used to. When did this more complex transmission mechanism between layers become the norm? Do we know why it became the norm, and is there anything else I should know? For example, with attention, we have matrices of keys, queries, and values, and we have some explanation for what each of them does: this token has certain information; it looks for certain information; having found it, it loads a different type of information. Do we have a valid story that led to this design, or do we just make up explanations after the fact based on the fact that it works? Curt Tigges Honestly, I’m not sure about the exact history of its development. But the wider family, the gated MLP family, which includes SwiGLU as well as ReGLU—that’s a little different, but it uses ReLU instead of SiLU in the middle—has existed at least since Llama 3.1. So, several years have passed. More than 2 years. Nathan Labenz Yes, yes. Curt Tigges And probably even earlier. But now the vast majority of modern language models use gated MLPs, and most of them use SwiGLU. They just work better, and I think the people who developed them weren’t thinking about interpretability. They just worked better that way, probably because they’re more expressive. You get twice as many incoming parameters for the MLP. We have an ex post facto explanation for how it allows the network to function, and we go into that a little in the article. If you have this gate-and-up combination, it means that you can switch contexts more easily. For example, if you have a neuron that has 2 values, you probably want it to be read by 2 sets of following neurons, and you want each group of subsequent neurons to interpret the neuron differently. This way, you don’t mix up signals. For example, there’s a neuron where one value is oral health and the other is marine fauna. These are very different concepts, aren’t they? In practice, when a neuron has one meaning, the set of readers is completely different. When it has a different meaning, the set of readers is different again. We see that the context of the text sent to the model activates these following readers in advance so that they’re ready to receive the target signal from the neuron. So it’s like a toggle function. It simply allows more combinations to happen in the network. Nathan Labenz If we zoom in on a particular neuron, it now receives 2 signals from each neuron in the previous layer. One is a simple signal, like the strength of the activation multiplied by the parameter that linearly determines how much to activate based on my input. Then there’s this additional parameter. Correct me if I’m wrong. I’m probably wrong. Again, this comes from every neuron in the previous layer, but now the activation strength is multiplied by the parameter and passed through this gate function—SwiGLU, or it could be different options. You said SwiGLU. I see SiLU in the article, but you said SwiGLU is used most often, so maybe you can help me figure this out a little. But that’s not necessarily the case, or perhaps we shouldn’t think of it this way. Actually, I’ll say it differently. We should probably think of it this way: gates come from places other than signals, because if gates and signals came from one place, they would be inseparable. The point is that we have a neuron that tells you when to be active and also what kind of thing to be active for, but the signals come from different places so they can achieve that independently. It’s in this independence that we get more expressivity. How am I doing? Curt Tigges Yes, that’s basically correct. It’s a little confusing because the strength of activation can come from both the up projection and the gate. So, to answer one of your previous questions: this is a gated MLP, specifically a SwiGLU. It includes an up projection, a gate projection, and a down projection. Inside the gate, it multiplies by an activation function, which is SiLU. If you had ReGLU instead of SwiGLU, then the gate would be multiplied by the ReLU activation. What SiLU does is allow the gate to be a little negative sometimes, but mostly positive, while ReLU in this place would be only positive. But yes, we see that they often have somewhat disparate sets of things that are activated. Sometimes there’s some overlap, but usually they’re quite separated, which provides this expressiveness. Nathan Labenz Got it. ### Sparsity and ablation experiments (Part 2) (Part 1) One big maneuver that manifests itself in several ways in this work, and is key to all the conclusions, is sparsity. You consider how sparse you can make this structure while still getting similar results. I’d like to understand better what the transition to sparse models actually means, because it isn’t always obvious. The easiest way to think about it might be: just reset them all. But I understand that in reality it’s a little more complicated. There’s some baseline average activation that isn’t equal to zero, and it’s considered a kind of typical or neutral state when a neuron is ablated or turned off. Turning it off doesn’t necessarily mean a complete reset. Help me understand this a little better. Curt Tigges Yes, in practice it’s a bit more complicated. When an expert is activated, all of its neurons do a little bit of work. There’s no real zero for SwiGLU or SiLU. In our sparse experiments, we do a lot of different things. For example, sometimes we ablate everything except the extreme activations that we’ve identified for a particular neuron. Then we set the values for all the other neurons to their average. That’s mean ablation, and in practice it’s actually quite close to zero, because the neuron is bipolar: it can be positive or negative, so its average value is very close to zero, or just small. Another kind of ablation we do is to choose the top K neurons from each expert—for example, the 64 most active neurons. In that case, those are the only signals we pass through. We don’t pass through the others; they have no signal, so they’re effectively zero. Nathan Labenz Is there much at stake in scenarios where you use the average value? If you literally set them all to zero, so there’s no contribution from those neurons to the next layer, would that lead to radically different results? Or is there not much difference in practice? Curt Tigges In practice, when we do this, the model still works pretty well on a number of tests, effectively equivalent to the original model. When we do it for the first time, logical reasoning suffers. Models sometimes start to get stuck. They may become uncertain about their answers and check themselves too often. But with a little tweaking, we can fix this, and the model can perform as well or even better on the tests we’ve run with this ablation. Nathan Labenz Yeah, that’s really strange. I definitely wanted to get some intuition about this. Just one clarification: there are different ablation methods. Top 64 seems to be where the 88% figure comes from, right? You also looked at top 32, and performance seems to drop significantly more when you go from top 64 to top 32. But there’s also this thing with percentiles, where you perform ablation because it’s a bipolar quantity. I understand that you take the middle, right? You keep the strongest positive and negative activations, but ablate, for example, the range from the 10th to the 90th percentile, or from the 5th to the 95th. Why is there a significant difference between using top 64 and simply preserving the lower and upper tails? It almost seems like two ways of saying the same thing, but there must be something deeper that makes them so different that you report them separately in the paper. Curt Tigges Each form of ablation has a specific meaning for the model. We noticed that if we rank individual neurons by their activations, or look at the percentiles where they’re activated—P75, P90, P95, and so on—different neurons start to turn into noise at different levels. A single neuron can be significant up to P75, while another is only significant up to P99. This can also vary by layer, so strict ablation based on percentiles doesn’t always make sense or may not be entirely justified. Another way is to use top N, for example, top 64. Then the question is: given the activation of this expert, what’s the most important thing in it? This lets the model decide what’s important, rather than relying only on dry percentiles. In addition, percentiles may depend on the dataset. Allowing the model to do this gives us slightly better performance. In practice, the vast majority of activated elements are already above P95 on the positive side or below P5 on the negative side. But a handful of neurons make it through at, say, the P90 level. They’re a little weaker. Nathan Labenz So the main difference is that the approach based on percentiles applies to the whole model, while top K applies to each expert separately? Do percentiles apply per expert, too? Curt Tigges The percentile is actually calculated for each neuron. That is, taking into account the historical activation range of that neuron, you perform ablation based on that neuron’s individual P90 value, or another percentile. But this is global in the context of all 5 million neurons. Nathan Labenz All at the same time. Got it. So top K is a kind of stratified approach where you guarantee that a similar amount of activity is maintained at the level of each expert. For example, there are— Curt Tigges Yes, exactly. Nathan Labenz Aha. Maybe you have an expert where most neurons still have some values below P95. Maybe they’re all at around P90, and that’s clearly still useful. Then you have another expert where nothing below P99 is useful. In those cases, you want to let each expert decide what the threshold should be. Can you give a more intuitive explanation of what you mean when you say something is meaningful up to, say, P99 or P90? I must be thinking about it the wrong way. I thought the most important neurons were the ones that let you set stricter filters. For example, from 25 to 75, you cut off half the neurons. From 10 to 90, you cut off 80% of the neurons. As you widen that middle range, you’d expect the most important neurons to remain. But I think you’re saying something else: something may be in a high percentile but not very important, while something else might be in a lower percentile but still be important. So it’s worth going down to the level of each expert. Am I understanding that correctly? Curt Tigges Yes, essentially. To make it clear, the variance is quite small. Almost always, extreme activations are the most significant and the most causally relevant for the model’s computations. To give you some idea of how we do this, we collect a bunch of examples where a neuron is activated at each percentile: the top 1,000, then P99.99, and so on, descending through the ranks across the activation range. Then we study where the coherence stops. We read the examples and study them, and we see that sometimes they turn into noise at some point. That point is almost always somewhere in the 90s. Sometimes it’s P95; sometimes it’s P99 or something like that. It’s also worth noting that the quantiles themselves depend on the dataset. We use a fairly large, mixed dataset, but it’s not as large as the model’s training distribution, which I believe was made up of several trillion tokens. We use 10 billion tokens to extract these examples. Nathan Labenz When I look at the performance table, what you’re saying makes sense for P25–P75: the middle 50% of neurons, the ones with the least extreme activations, are ablated, and there are almost no changes as you look down the column at the different test results. So we can conclude, at a high level, that you can cut out half of the neurons with the least extreme activations, and in principle nothing bad happens. Interestingly, you don’t see any particular damage even on reasoning tasks. Then you go further and cut out the middle 80%, and then the middle 90%, and the effect on performance becomes more substantial. So we’re checking where this curve starts to get steep as we move into the space of extreme activations, trying to find how far we can go while keeping everything mostly working. I’m very interested in your intuition about why some of these metrics get better. At the top, you have 4 different sections of ARC-AGI, and the average 50% ablation isn’t very effective, but removing 80% or 90% of the neurons actually gives small performance gains. Do you have an explanation for why that’s happening? Curt Tigges Some of these tests take into account how quickly the model finds the answer. If it thinks too long before giving an answer, it counts as a failure. Essentially, if you delete a lot of internal components, the model becomes more confident on the tests where it performs better. It finds the answer faster, and I assume some of the tests you mentioned don’t require reasoning. For the ones that don’t, it’s just that the model is more confident. It doesn’t have as many internal obstacles to distract it from the correct answer, and the most likely option is brought to the foreground. In the case of reasoning, thanks to our fine-tuning, we see the same thing: it just reaches answers the original model knew, only faster. Nathan Labenz Interesting. Fewer things that can distract. This is fascinating. So, just to take stock, we discussed the network structure, which means neuronal ablation, and we found that you can delete most neurons—up to 90%, or 88% when taking top-K 64—while still showing surprisingly high performance, at least on these tasks that don’t require reasoning. Okay, but the next question is this: It’s one thing to say that only the important things are in these extreme cases, but how do we interpret what’s happening? Maybe tell me about the process the “neuroscientist” goes through. You’ve already partially described it: you have a large body of data, collect a lot of activations, rank them, and set different thresholds. How do we transform this huge array of 5 million activations into stories about what each individual neuron does in a large network with 35 billion parameters? ### The neuron scientist agent Curt Tigges Excellent question. I like talking about a “neuroscientist.” I think it’s a pretty cool agent, essentially, and we’ve put a lot of effort into improving its performance. As you said, we collect thousands of example queries where a neuron becomes active, and then group them by similarity. We have several ways to do this. One is to look at the incoming connections and group them according to that. You can also just look at the text and at what other neurons are activated, and group the examples by those activations. In practice, this gives you different contexts in which the neuron becomes active. The main thing is that we get an initial grouping, and then pass these examples and groups to the neuroscientist agent. Its task is to put forward a hypothesis that unifies the different examples of neuron activation, and then predict where the neuron will be activated in the future. Sometimes it will identify one common pattern; sometimes it will identify several. Neurons react to different patterns in different contexts, across different concepts. We give the neuroscientist a lot of freedom in which hypothesis to put forward, but we also give it a bunch of tools. It can create or change example sentences, send them to the model, and see how strongly the neuron reacts. It can conduct other experiments—for example, take 2 very similar sentences, make a slight change, and see how the activation differs. It can control the model and do almost everything an interpretability researcher can do. It puts forward an initial hypothesis, and we check it objectively using another system, separate from the agent. That system checks whether, based on the neuron description, we can predict where in an example text the neuron will fire, and how strongly. It does this across several texts, and if it can’t handle them—if the description turns out to be inaccurate—it sends feedback to the neuroscientist. The agent sees how to adjust its hypothesis, improves it, sends it back, and so on. At the end, the auxiliary agent creates a page that shows the results of the hypothesis. Nathan Labenz The beginning of this process sounds pretty similar to automatic feature labeling for SAEs. I think people have seen this before, right? You have a series of passages that activate a particular SAE feature, and you ask the model, “Hey, model, what do you think? Is this it?” Curt Tigges And we went even further. We don’t just ask questions, get answers, and stop there. We spend some tokens refining an explanation, testing it, and improving it. We really try to make it high quality. Nathan Labenz Because, of course, many SAE features are considered uninterpreted. To be honest, I’ve looked at lists of passages that activate SAE features, some of which were considered uninterpreted and others interpreted. Sometimes I thought, “I don’t know, man. You say it means this, but it’s not at all obvious to me that this is really what it means.” What do you think an agent does when it goes through this process? Could you tell a story about a case where things were initially unclear, but the agent used its tools to understand what a particular neuron meant? Curt Tigges Yes, of course. A lot of that automatic tagging and interpretation used for SAEs, and what we first used for neurons, tells us what’s happening when the function is obvious. But when it’s not obvious, you usually get some empty label, for example, “This is activated by words in the context of medical discussions.” That’s not very useful at all. It teaches us almost nothing. One thing we noticed is that as we apply higher levels of intelligence, more of what a neuron does becomes understandable and predictable. Our paper has a graph of this: automatic interpretation gets the lowest results, and when we use the best agents and allow them to do more work, we get better predictions. The neuroscientist agent we use for neurons is usually a frontier model—one of the newest models—and it seems to work well. One example was a neuron where, if you just looked at the examples and a 50-token window around where it was activated, it seemed completely uninterpretable. It was always just a phrase that could relate to cooking, instructions, legal documents, or anything at all. You really couldn’t say what it was; there was nothing that united these patterns. But through in-depth research with the help of agents, we were able to determine that this neuron actually fires when an item is second in a list, and the content of the list was presented elsewhere in a different format much earlier in the text, perhaps 1,000 tokens before. Here, it was reproduced in a different format. For example, it might previously have been a list with comments or commas, and now it’s something like 1, 2, 3, 4. So it’s a structural feature you wouldn’t necessarily think to look for, but it was there. There are many features like this where the idea isn’t obvious at first glance. That’s just one example. Nathan Labenz I thought that was pretty creative. Correct me if I’m wrong, but I think all this technology would also apply to SAEs. There’s nothing stopping you from giving the agent a similar set of tools for working with SAEs and letting it experiment, change the text, run it again, and do everything the same way, right? That’s an interesting innovation, regardless of the lens through which we look at things. Let me see if I understand this correctly. The agent iterates for a while, runs experiments, makes counterfactual tests, and possibly expands the scope to view longer text fragments, trying to find broader patterns that weren’t visible in the short passages we’re used to. In the end, it arrives at a description of a neuron that may be a long paragraph. How do we check whether that’s really true? The answer is to give the description to another model, probably from another developer or another model family, and say, “Okay, your job is to take this description of a neuron, analyze the text token by token, and tell us how strongly this neuron will fire for each token.” Then, based on the strength of the activation, you feed its predictions back into the original model—the model we’re interpreting—and ask, “How well does the model work if, instead of its natural activations, we force it to use predicted activations from this other model, based on the description and the current text?” The answer is that we recover 74% of the loss in Kullback–Leibler divergence. I’d like to understand that a little better, because I don’t have a good intuition for KL divergence. I don’t really see KL divergence intuitively—maybe you do. Nathan Labenz So tell me if I’m wrong about anything in the process. Is there anything else we should know about it? And how should we generally understand this indicator: 74% recovery? Curt Tigges That’s essentially correct. A neuroscientist creates a paragraph-long description of a neuron, highlighting the types of tokens it responds to, as well as the general context and concept. This is one of our evaluation methods, and it’s currently the main one. As you said, we initially remove the neuron from the sequence. Then another model says, “Okay, based on the description, on a scale from zero to nine, how strongly will this neuron respond?” It then marks each token in the sequence with a corresponding value. Most of them are zeros, but many will be high. Then, as you said, we insert those activations back into the model and see what fraction of the loss caused by deleting the neuron is recovered by inserting those values at those positions. This works quite well. The model we’re currently using is Jev, so it’s very different from the model that generated the predictions. This gives us an idea of how much damage was repaired. As you mentioned, we use Kullback-Leibler divergence. At every step, for each token in the sequence, the model has a prediction for every token in its vocabulary—a distribution over all of those tokens. KL divergence is best understood as the difference between the distribution when the model is damaged and the distribution when it’s patched up or restored. That distribution shows how much we’ve impaired the model’s ability to predict the next token. We measure this at every step and combine it across the entire sequence. It’s worth noting that we weight this value according to the degree of damage. So when we talk about 74% damage repair, cases where the neuron had little or no impact receive less weight than cases where it played a major role in the model’s prediction. That’s the basic principle. This is our main evaluation method, but there are a few others worth mentioning. One is quite intuitive: we give another model a description and ask it to create text that would activate the neuron more strongly than anything in the dataset. For example, we ran a dataset of 10 billion tokens through the model and collected examples. We tell another model, “Surpass this.” We don’t give it any examples, so it has nothing but the description. It iterates until it creates a sentence that activates the neuron more strongly than anything in the dataset. ### Monosemanticity versus circuit mechanics For other metrics, we ask whether the neuron should activate at all on a given sequence. Is it even relevant? There, we often see scores of 90%, because it’s an easier task than predicting token by token. All of this indicates that we can have a fairly high level of confidence that the process of creating a paragraph-long description of a neuron works. If it didn’t, we wouldn’t expect to be able to repair this damage or make correct predictions. Nathan Labenz So where are we now? Are we moving on to the interpretation stage? I’m looking at a paragraph-long description right now. This is still a bit subjective, because the paragraph may contain 2 different or related concepts. How do we decide where to draw the boundaries between concepts? What do you think about monosemanticity in this context? What counts as one concept? I’d like to hear your thoughts on nouns versus verbs. Could you give me a little more qualitative understanding of what these descriptions are like? I’m sure you’ve read thousands of them. What’s the nature of these paragraph-long descriptions? Curt Tigges They can differ significantly from one neuron to another. We see that neurons do a lot of different things. As you said, sometimes a neuron responds to several ideas, like oral health and the marine environment: it might fire when someone is deciding to have surgery in their mouth, or when a particular animal in a marine environment is being described. The description might look something like that. Most neurons have a central concept that unites them. But after looking at thousands of neuron descriptions and what those neurons usually represent, I think I’ve arrived at a slightly different philosophical perspective than simple monosemanticity or polysemanticity. Monosemanticity is the idea that a neuron, or something else like a feature, represents one thing, one concept. It simply detects whether that thing is present. But in practice, I think it’s much more complicated. Each neuron can connect to thousands of other neurons upstream, and they provide a lot of information to it. One thing you might see is a neuron that responds to discussions of insects. It responds specifically to 3 types of things: foods that insects prefer, the environments where they prefer to live, and the places where they prefer to lay eggs. Are those 3 concepts or one? It’s hard to say, because the neuron doesn’t respond to everything related to insects or to every insect preference. It responds to these 3 subtopics within the broader subject of insects. A neuron that fires on musical instruments like the banjo or legal documents about alimony, on the other hand, is probably easy to recognize as representing 2 different things. Another problem with how people think about monosemanticity or polysemanticity is that we don’t always know why the model brings things together. Let’s say you have a neuron that responds to clothes, exercise equipment, dishes, and tablecloths. That looks very polysemantic—several different meanings. But what if the concept the model has is “things you can find at a garage sale”? Suddenly, those things come together, but you might never see that from the outside. So, for me, the important question isn’t so much whether something is purely monosemantic or purely polysemantic. It’s whether you can explain all the things that a neuron represents. Sometimes you might want to zoom in and find what looks like a single concept. You look more closely, find several sub-concepts inside it, and list them. When those things are distinct—when they’re completely different from each other—you list those too. The most interesting thing from a mechanistic point of view is that we can see signs of these concepts in the weights themselves. That means you can look among the thousands of neurons connected to one neuron and find a subset that activates when one topic is discussed, then another subset for another topic. You can analyze thousands of examples and identify these subcircuits. That gives you a perspective on the primitives the model has inside its own “mind.” Nathan Labenz When you looked at these diagrams, do you have a simple story about how it works? I know it sounds silly, but what are the motifs or steps that stand out as notable primitives and might give us some intuition? It boils down to one of the big questions in all of this: at what point do we move to an ambitious reverse-engineering phase of all this? I think that was one of the things most discussed in the essay, and now in the article. I’m sure we’ll have many similar discussions. To what extent do we now have a better story about that? It’s one thing to say, “Okay, we can say these neurons have fairly interpretable functions that they perform separately,” but how does this fit into context? How does it start to turn into behavior? How do we aggregate this into true understanding? How far have you progressed? Do you have a story to share that would give us a little better intuition about how it works, beyond what an individual neuron means? Curt Tigges Wow, great questions. Only the most important ones for you. I think the transition from noun to verb can be understood in several different ways. One way I like to think about it is that it’s less about identifying a separate sign or object and more about what a neuron does. It’s not just about detecting something; it’s like sending a signal that will cause the model to do something else. In that sense, it’s a verb. It also relates to changes in what happens in the text, or what happens as the model moves on to another type of text generation. I guess that’s how I understand the distinction between noun and verb, although Nick might think about it a little differently. I think our thoughts are quite similar. Nathan Labenz So let me briefly reiterate that to you. A verb is about what a neuron does in relation to the model: it sends a signal downstream that activates a specific set of chains. Some of those may still be very similar to nouns, but these nodes of flow control can be described as operations like “if this, then that.” We should understand their influence not as representing something, but rather as controlling motion, directing energy, or activating the model in the places it needs to go for further use. Curt Tigges Just so. They’re not just static, as if each layer only reveals a certain set of concepts and those concepts are now in the model. They are, as you said, flow control. They direct the model toward certain things or away from certain things, and we can see examples of interesting neurons that respond to reasoning. Curt Tigges There’s a whole class of neurons that are activated and direct the model to be more or less confident in its own reasoning, or to do things like wait, check, continue in a certain direction, or transition to another approach. There are many neurons specific to reasoning that are really interesting in this regard, and they’re very similar to verbs and control neurons. Others seem to say, “This number is wrong,” or, “This unit is wrong,” or just perform operations like that. On the subject of ambitious reverse engineering, there’s definitely more to reverse engineer than just neurons. We need to understand attention and other things in the model. The residual stream, in particular, is very difficult to unpack, but I believe understanding neurons and the specific circuits and subcircuits they create is quite critical for this. You can understand each individual neuron in the model and have a large list of what they represent and what they do, and still not have reverse-engineered the model in an ambitious sense. You need to see how they work together, and understanding the transformations of individual neurons essentially requires understanding circuits. You can find circuits in different ways. For example, you can run millions of queries through the model, collect attribution graphs using one of the existing methods, and then see what shows up on average—highlighting communities of neurons from there. That’s one of the things we mentioned in our paper. You can also look at the circuit for a specific query. To turn that into understanding, you need to find the right circuits and then see how they affect behavior. We need to go a little higher in the abstraction tree and find a subcircuit for all these little pieces of knowledge the model has, and for all its little decision-making tendencies. Maybe you’ll find how it decides to use a specific Python package, which artists are Impressionists and which are Post-Impressionists, or which capital belongs to which state. We’ve done some work in this direction, and we can see how this type of knowledge is represented. For example, with Impressionists, there’s a community of only a few dozen neurons that represents the concept of which artists are Impressionists, Neo-Impressionists, or Post-Impressionists. Then there are neurons representing individual artists, associated with tokens for these categories. It’s all just a small mechanism that solves this issue. I think we’ll have to climb up the tree of abstraction and make a huge list of things at the neuron level, and at the sub-neuronal level, where we see individual circuits for different values a neuron can have. Then we can move on to communities of neurons and supercircuits that perform these tasks. I think there are 2 things we need to do. One is to study the model in general, at least to some extent, right? You want to understand the model as a whole—all the little circuits and components—as much as you can. If you understand neurons and have the right primitives to put into these slots, you might say, “Okay, now that we have a community of neurons and know what these neurons are, we can say what exactly it does.” It’s important to cover a wide range and collect a huge number of examples, but you should also focus on what’s specific to your goals around alignment and safety. That means running many experiments with datasets that elicit specific properties related to safety, such as reward hacking, scheming, or planning. You should see how communities influence this behavior and how you can steer it in one direction or another. You need to see, essentially, what’s activated when this happens. ### Reverse engineering for alignment Nathan Labenz I still have many specific questions, but what stage do you think we’re at? Where are we now? When I hear about a circuit that identifies, say, Impressionism, we could map billions of these things and still be at the same level as with neurons, with an equally long or even longer list of circuits that perform certain actions. How does that add up to a holistic understanding? Do you feel like you can see a way out of here? ### Future directions and wrap-up I think that’s one of the things people initially said: “Oh my God, they’ve reduced interpretability to normal science. Now we can study agents indefinitely, and we’ll have a good understanding of how models work.” And others said, “I don’t know. We can have a long list of neuron descriptions or an even longer list of these circuits, but this doesn’t give us a macroscopic understanding of what the model will do next or in a new situation.” I think generalization across distribution boundaries is always a critical question. Do we feel there’s any progress in labeling all this? Can we make better predictions about what happens when you put the model in a new situation? Curt Tigges A lot of the work on neurons is still new, and we’ve only just started doing it. There are many interesting pieces of work online, inspired by our first paper. One user found a neuron that activated on text about reward hacking, so that neuron could be used for monitoring—at least for that subtype of reward hacking, though maybe not for everything. There are also many interesting developments around character traits and controlling them with neurons, and it’s surprisingly practical. It seems that many neurons simply affect how the model responds and its personality. But, getting back to your question about this huge collection of microcircuits and neurons, what will that lead us to? How does it really convert into higher-level primitives? I think we haven’t figured out what these higher-level primitives are yet. It’s a lot like physics, chemistry, and biology. Theoretically, you could describe everything in biology in terms of physics, but you’d never want to, because it’s too detailed and unnecessary. There are higher-level laws discovered at the level of chemistry, and then at the biological level. For neural networks, we still need to find these laws and develop a science of higher levels of abstraction. But I believe that by better understanding the lower level, we can reach the next levels of abstraction faster, verify them, and see cases where they might not apply. ### Outro You can have a control vector for character, right? I think we’ve already seen this in work from Anthropic and others: it’s simply a vector that can direct the model’s personality in different directions. We can understand this at the neuronal level by seeing which neural circuits lead to this personality, which ones are important, or which ones, if removed, would cause problems. Are there neurons that could sabotage it given a certain input signal? Would they break the personality and force it to do something different, as sometimes happens when models suddenly say something crazy, perhaps once in 10,000 cases? If you want to find out why the model does that, I think neurons are what you should pay attention to. That’s not necessarily guaranteed to be where you find the answer, but it will be useful. Nathan Labenz Okay, I see our role as discovering these primitives, like molecules that are important for model behavior. But we need to understand alloys and fluid dynamics; we need to understand these higher-level sciences. If you have neurons, you can model all these things better. Curt Tigges I also see it as a process that happens on both sides. One side is bottom-up: collect all the small parts, all these components, and understand how they work. From the top down, we can also find vectors, like mean-difference vectors or control vectors, that accomplish these big things. We don’t have to understand how they work mechanistically. You can also study training, which is very top-down, and work from the lower and higher levels until they come together. You build higher-level abstractions, more and more of them, until you reach a level where you can see what’s really going on when you train a model to do X. For example, what changes at these levels make it strive for reward hacking, make it aware of evaluation, make it better at solving problems, or make it more cooperative. So, in essence, we still have a lot to do, but I think neurons are an important part of this, and they’ll allow us to do all the work we’ve planned. Nathan Labenz Maybe I could sum it up like this. First, you mentioned the possibility of monitoring, and for me that certainly stood out as an obvious and immediate application of this work. You also mentioned—and rightly so—the need to be careful not to just take the first neuron found for reward hacking and follow only that one, remaining blind to other places where the model can go to this hack. But it undoubtedly looks like a great opportunity for monitoring. The other aspect, in my opinion, is debugging. I’m telling this story as if we thought we had to use constructions like SAEs because everything inside was so confusing. And now we’re creating this additional, somewhat opaque layer that makes it harder for us to see through to the truth when questions arise, because now we have to pass through it. To the extent that we can dispense with this unnecessary layer and get closer to the truth, we can at least study the thing itself, not its reconstruction. That means we’ll make significantly fewer mistakes in the process of alignment and, we hope, have a much clearer understanding of what we’re really looking at. Would you change anything, or is this an acceptable description? Curt Tigges We certainly hope so. I think SAEs can be used for different parts of the model. For MLPs in particular, and neurons in particular, if you can better understand the neural basis, then it makes sense to use neurons. We mention this in our paper, and there’s also a great paper by Stefan Chobola and Matúš, which I think came out just a few weeks ago. It’s about how transcoders—a neuron-based version of an MLP—work a little differently, but have a similar idea: dictionary learning. In essence, they simply reproduce neurons. They’re features: they’re neurons, and when they aren’t, they represent sparse combinations of neurons that are often activated together and are semantically coherent. So, if you use transcoders instead of looking at neurons, it might help you see which combinations of neurons are often activated together. But you have more to lose if you replace an MLP with a transcoder than if you just use the top-K neurons, which really shows that it’s worth looking more at neurons. SAEs still have some use for, say, a residual stream, which we don’t yet understand in terms of neurons and which is a more complex object. Of course, there’s superposition and complex geometry in the residual stream. But even for this, an SAE may not be the best choice. For example, there are sparse feature circuits, which we wrote a paper about. I think we need to better understand the geometry of the residual stream. That’s one of the most important things we need to do for ambitious reverse engineering: find a better way to understand the residual stream. But when it comes to neurons, I think it makes more sense to try to understand them very well. Nathan Labenz Let’s have a quick blitz, if possible, about how this work interacts with my understanding of other things that happen in these networks—or whether it should make me update that understanding. For actual memorization and recalling facts, it’s obvious that models are remembered for their incredible number of facts. I had the feeling that facts lived in MLPs. I always knew that was a simplification, but how should I update my understanding? Do we still think they live in MLPs? Does the concept of “one role per neuron” make me think, “Okay, then where exactly are the facts in this picture?” Curt Tigges Yes, this is a common question, because the model may have 5 million neurons, but it knows much more than 5 million facts, right? That’s because a fact isn’t contained in one neuron. A neuron isn’t a fact; the fact is in the connections between neurons. You can have exponentially more connections between neurons than neurons themselves. A set of 10 neurons connected in a network can represent much more than 10 facts simply because the number of possible paths between them is extremely large. From what we’ve seen, we certainly haven’t found every single fact in the model, and we can’t claim with 100% certainty that facts are only in MLPs. But we’ve found so many facts in MLPs, realized precisely through these connections, that it seems like their natural place. It’s like post-Impressionism: a whole bunch of artists and concepts represented in a smaller number of neurons than the number of things that could emerge from that community. Nathan Labenz Are we seeing a different degradation profile for recalling facts under sparsity compared to ARC-type tests? They may even work better with sparsity while doing worse on reasoning. My intuition is that making factual knowledge sparse should also have a significant impact, but what did you actually observe? Curt Tigges We tested some MMLU datasets and were able to recover performance quite well on them. Those tests measure the recall of facts. We have a few tests we’re still working on, for example, on medical and chemistry knowledge, but mostly we saw no significant degradation. For MedQA, a small amount of additional training restored the metrics, so performance differed by only 1 point from the original models. The same was true for GPQA Diamond. We were also able to recover performance on SuperGPQA. Nathan Labenz Uh-huh. Curt Tigges So it’s quite possible that if we took millions of facts and loaded them in, we could see some decline. But judging by the results, the model does pretty well on actual datasets. And that’s because we don’t turn off every neuron at each step, but only for each individual token. It activates only a small group of neurons. So each neuron still has the ability to manifest, as does every connection between neurons, but only when it’s important and appropriate. Nathan Labenz Yes, interesting. I think this actually points to localization, which suggests something about the nature of experts. Previously, people said that experts themselves aren’t interpretable. The bad news was that we couldn’t really determine what role the experts played. Based on everything you’ve said, it seems that almost can’t be true, right? If these circuits are localized, the facts are in the right place, and the router has to activate the relevant expert, then there must be significant specialization among the experts. Otherwise, it would be impossible for the router to choose which 8 of 256 experts to activate at each layer. Should I change my mind about experts being much more holistic in the roles they play? Curt Tigges Yes, it’s worth updating. There’s some early research on this topic. I’ve forgotten the titles of the papers, but researchers passed datasets through experts and found that some experts were activated more often for specific topics than for others. Now we’ve discovered that, at the level of neurons and representations, the experts we’re exploring are very focused. They mostly concern 1 to 3 things, and that’s all. They’re not mixing together 100 different things. Some of the interesting ones we found are mathematical and notational specialists. Another is a legal specialist focused on constitutional and jurisprudential texts. There’s one for algebra. There are many experts from different fields of mathematics, as well as experts responsible for tracking time, units of time, and how much time has passed. There’s a geometry expert, and of course, many medical experts that focus on specific medical subtopics. So yes, the experts are quite focused in what they represent. Nathan Labenz Do you think this indicates future directions in architecture? If you look at the big picture, we’re at a time when AI labs are ready to break free. Companies developing advanced models say, “Hey, we need antitrust immunity or something like that so we can coordinate the pace of development.” Jensen Huang says, “You don’t need that. Just don’t release dangerous products, and get ready to play in the major leagues, where I’ve been for a long time.” He says that only 10–20% of his resources go to creating the product itself. The remaining 80–90% go toward making it safe and reliable. So buy more GPUs and spend more FLOPs on monitoring, interpretability, and all those things. I can imagine a world where we start making this shift. Another way to ask the question might be: Are there scaling laws for interpretability? It makes me think that maybe we want, or at least should be open to, developing architectures that learn more slowly or need more training time. Maybe they’re even less effective during inference because we’ll invest a huge amount of FLOPs in monitoring and interpretability. But if we take that 80% Jensen says goes toward understanding and control into account, we can allow for those costs if interpretability becomes easier. Does that lead to advantages in creating architectures that, in your opinion, would be more interpretable? ### Designing interpretable architectures Curt Tigges That’s a great question. The idea of an “interpretability tax”—paying a price for the model to be more interpretable and controllable—is a trade-off that could be worth it. Although many companies won’t want to pay that tax, one interesting thing we’re seeing is that the model is already quite interpretable, even more than we expected. So it’s quite possible, although not guaranteed, that we need less specialized architecture for interpretability, which is expensive, because we might just be able to interpret existing models. At the same time, I think there are probably certain adjustments that would make models easier to monitor and perhaps easier to steer during training, so we can have a more focused design. One of those things is probably creating a sparser output regime, because then there are simply fewer things that need to be tracked. Instead of tracking 5 million neurons at every step, you can follow a much smaller number and find clearer circuits. There are probably ways to encourage models to be sparser during training or afterward. That’s what we did with the sparse Qwen model: it works just as well or better on most benchmarks while remaining quite sparse. There are scientific works that develop this idea further, perhaps not strictly in the MLP basis but in terms of parameters. I think anything that pushes a model toward greater sparsity will be useful. Localization is important, too. There’s a paper, I think, about localizing potentially harmful biological knowledge or knowledge of weapons of mass destruction in specific parts of a model, so those parts can be turned off or left out of the public version. Knowing about neurons and specialized experts, we can encourage the model to place blocks of knowledge in specific locations, then delete them when necessary. I think this points to the potential for models that are quite interpretable without paying a large tax in consistency or safety. Nathan Labenz Thanks for bringing up GRAM—gradient routing, or something like that. I’m a GRAM maximalist, and I mention it probably because of this episode. If you’re playing a drinking game while listening to The Cognitive Revolution, it’s time to raise your glass for GRAM. On your bingo card, every square should be about GRAM. I have a question about the activation function. There’s a strange phenomenon called double negation that happens in some cases. Maybe you can help me understand a little better what’s going on. The main thing is that the activation function can be both positive and negative, unlike the original ReLU, which was zero for negative values and linear for positive ones. Now you can get double-negative values, and it seems like a complicated thing that we don’t yet know what to make of. ### Factual recall and experts When I saw this, I thought, “Oh my God, maybe this is the same tax on coherence that we’ll have to pay.” Perhaps we should go back to something simpler—monotonic, if not specifically ReLU, then some other monotonic activation function that could eliminate this bipolarity. Then maybe we wouldn’t have to worry about double-negative values. Perhaps it would learn a little more slowly, or need a slightly larger network with more parameters to get the same result. I’m sure things wouldn’t be exactly as I’ve described, but should we sometimes consider simpler mechanisms? Curt Tigges I don’t think so. The thing about SwiGLU is that, in my opinion, it allows you to get greater monosemanticity, or simplicity, because even when polysemanticity arises, the model can simply switch between several different things using this mechanism. But I agree that Qwen3’s double-negative activations, where the gate projection is negative and the “up” projection is negative, are a strange case: they seem to cancel each other out and produce a positive value. As you said, these are less interpretable examples and are certainly more distant from what the model does. Sometimes they’re useful for the model to work, and sometimes they aren’t. Some modern architectures don’t allow this. For example, with ReLU, you still have gate and “up” projections, but instead of SiLU in the middle, multiplied by the gate, you have ReLU. Gemma 4 models seem to use that. This architecture doesn’t allow double-negative activations. They work quite well. We haven’t done a detailed analysis of the neurons to say how interpretable they are, but it’s certainly one option. Something I think I saw when looking at Kimi K3 neurons is that they almost never use these double-negative activations. I’m not sure whether that’s an architectural feature or whether the model is just arranged that way. It might be a quirk of the Qwen3.6 family or of the SwiGLU architecture in these particular models. These are very strange activations. Sometimes you can delete them all, just get rid of them, and that hurts reasoning or memorization. But you can turn them back on by changing the direction—for example, reversing the order of the tokens—and that will resume reasoning. Or you can substitute all the double-negative activations from a completely different sequence, and that will also correct the reasoning. We’re not sure exactly what they do. That will be the subject of future research. ### Restoring reasoning performance Nathan Labenz One thing I definitely wanted to discuss is long context, which is related to reasoning to some extent. Maybe we should define how easy it was and any caveats about how well it worked. A surprisingly small number of tokens was needed to restore performance on the sparse model. How do you explain that? Looking back, what’s your explanation? Curt Tigges Well, it was 100 million, but yeah, quite a bit. In this context, that’s a small number. Nathan Labenz Yes, yes. That’s definitely much less than mid-training, and much, much less than pretraining. Curt Tigges Yes, we were actually able to do it in 1 day on multiple GPUs. The main problem with the long-context degradation in the initially sparse model was that the model, I think, just got confused. It was stuck. It was losing track of where things were and checking itself again and again. Something was wrong with its confidence. I think the pretraining experiment shows us that the model still had this knowledge. It retained the basic mechanisms of reasoning, but the problem was some kind of calibration failure. We don’t know exactly what it was. But the fact that it took so little effort to restore performance indicates to me that the model was very close to ideal. It was probably just disoriented because 90% of its “brain mass” had virtually disappeared in the sparsification process. We also noticed during experiments before further training that when the model tried to solve the problem several times—“pass@4” or “pass@8”—its performance was equivalent to the original. It had the knowledge; it just didn’t know how to apply it until we trained it. Nathan Labenz And how much performance could you restore? I don’t really know how this works on all chips, but there are at least some architectures that skip multiplication by zero. You can imagine: if, at the level of each layer, we could round the execution values to zero and avoid doing the matrix multiplications because there are lots of zeros, then there could be a potential “negative tax” on alignment for these things. When we talk about recovering performance, would you use this diluted and slightly retrained model? Is it indistinguishable from the original in the user experience, or is it more like, “90% sounds pretty good, but you’ll still notice the difference”? Curt Tigges I think we can only understand that through long-term use. I’ve interacted a lot with this simplified model, and based on my queries and experience working with it, it was as useful as the original. It still thought things through and gave me the answers I wanted, and they were almost the same as the original model. In terms of recovery, on several benchmarks we saw not just a restoration but even a 5-to-15-point improvement in performance, which was really unexpected. We didn’t continue our studies for very long because they stopped when we’d mostly achieved parity. But it would be very interesting to know whether this model could be trained even better if we gave it more than 100 steps of reinforcement learning. Perhaps it would have been better at something. We tested a wide range of math and programming benchmarks, as well as MedQA and general knowledge. The results were on par with or better than the original across all of these. We’re still testing to see whether we can get even better results on coding tasks. But yeah, it’s still a great model. Nathan Labenz Yes, it’s very interesting and certainly leads to some thoughts. I’ll just point out that sparsity is exactly what would allow us to understand this much better than we do now. Another thing I wanted to understand better, again regarding long context: some experiments were done with frozen attention, and the question came up of whether the model has some context at that point. It seems to have learned to transmit information from token to token. How sparse can we be and still predict the next token, or do a little reasoning? The situation seems to be that for predicting a single token, we can use an extremely thin structure, but if we want the deliberation to last longer, it can’t be as sparse. ChatGPT gave me a pretty interesting description of this, similar to the language in the article but a little different. ChatGPT says this hints at something like each new token being computed by a small circuit operating on representations gradually constructed by previous computations. That got me thinking: maybe that’s what some of these weak activations are doing, especially when we consider very long contexts. I mean, the tests generally don’t have very long contexts, and I don’t know how deeply you interact with this model during evaluations. But in practice, in real use, we take many steps forward, fill the context window with a million tokens, and then compress it. It won’t be long—if it hasn’t happened already—before we get some kind of data transfer from one context window to another. Nathan Labenz So this could be a path where, because it’s relatively convenient to look at single answers or relatively short things, I can imagine a story, based on everything I understand so far, where weak activations don’t have much value over short time periods, but perhaps they matter over millions of tokens. That may be because the long context allows the model to capture the rich texture of our entire conversation. Do we already have a vision of this, or is it something we’ll have to address through further research? Curt Tigges As for million-token windows, I think we’ll have to see how things develop in future research. We tested this on tasks up to 32,000 tokens, and the model seems to still handle them well. This applies, for example, to programming. One of our experiments showed that you need different numbers of activated neurons for different topics. Logical reasoning and programming require more, while storytelling requires much less—about an order of magnitude less. Of course, information needs to accumulate through the context so it can be retained somewhere, in the residual stream, the KV cache, and so on. That gives the model something useful to work with, and up to 32,000 tokens, everything seems to be fine. I suspect we can find threshold values that show us exactly how many activations we can delete. My assumption is that what works for 32,000 tokens will probably work at larger scales. Nathan Labenz So, where are we headed next? I was trying to remember what the canonical results in interpretability are that I could compare this to, or how I might imagine tracking the thoughts of a large language model based on neurons. It’s an incredibly complex system, with transcoders everywhere. I think the optimistic view is that tracking the thoughts of a large language model on a neural basis would be a next step. Do you think this is achievable in the near future? Curt Tigges Yes, that’s what I think. There are probably a few ways to do it. One thing we can already do with the neural basis is find the circuit for a specific passage. For example, you give the model a paragraph, then look at the circuits and the neurons activated by it, and ask another model, “Hey, here’s the attribution graph and the neurons that are activated. Can you tell me what this paragraph says?” The answer is usually yes. The model can identify entities, their relationships with one another, and so on. Of course, this doesn’t give us more information than we could learn from the text, but sometimes we want to track whether there are hidden thoughts in a chain of reasoning. For models trained to communicate through neurons or direct transmission of KV caches, we just want to be able to read what’s happening there. I’m definitely interested in exploring this more deeply and finding a structured representation of models’ thought processes. One of the great hopes is that, by understanding neurons, we can do that better. Nathan Labenz By the way, how accessible is this from a computational point of view? When you think about a unit of work, is it getting a “neuroscientist” to interpret one neuron or something higher-order? How much does that cost, so to speak, and what’s it worth? If we expanded this to the entire model, what computational budget have you spent on all this so far? Curt Tigges There are many variables, so it’s hard to say. It’s somewhere in the single digits. I also experimented with more and less complicated “neuroscientists.” The most expensive one cost $45 to produce an output for a single neuron, which is pretty crazy. But in general, they’re much cheaper. We didn’t try to optimize costs too much, but if you run a local model that serves as a framework for analyzing neurons, I’d say that’s probably the best and cheapest way. For example, you could run a local GLM-5.3 or something like that, which works pretty well. If you wanted to interpret the entire model, you probably wouldn’t need to use the full “neuroscientist” for every individual neuron, at least for the vast majority of them. But you would need a way to find the more complex ones, then selectively apply more intelligence and compute to the harder cases. For example, you could do a general walkthrough of everything, then work much more intensively in certain areas. I think the price isn’t crazy at all if you apply this approach. And today, we have a lot of computing power. What price could be too high to understand the minds of the things we’re creating so quickly? Nathan Labenz Exciting stuff. Is there anything I haven’t asked, or anything you’d like to clarify, reiterate, or leave people with as a parting thought? Curt Tigges One thing I’d like to convey to people is that you can try this yourself. For example, you can run a local model, get neuron activations, and play with steering neurons and interpreting them. We plan to release some neuron labels to work with, but with a smaller model, it’s quite possible to get a bunch of labels, build attribution graphs, and see what they look like. I also want to say that our work clearly relies on what other people have done. We’re not the only ones who have researched neuron interpretability. We’ve drawn great inspiration from other people, including Translucent, who did a great job on neurons. There’s always been a group of people trying to interpret things at the neuron level, which is why we refer to them in our report. Neurons are great. You should play with them. Nathan Labenz It would be really great if we could simplify the methods for analyzing and understanding neural networks, and this at least looks like a significant step in that direction. I’m truly grateful for the hard work you’ve put into this, and thank you for your time today and for helping me understand what you understand. Curt Tigges, thank you for joining The Cognitive Revolution. Curt Tigges Of course. Thank you very much, Nathan. I like it. I really liked it.