可解释性的效用——Emmanuel Amiesen

Here we have my trusty co-host—guest host, I guess—Vibhu, as well as Emmanuel from Anthropic. We're going to talk about the circuit-tracing work and the interpretability work. But Emmanuel, maybe you want to do a quick self-introduction before we get into it?

Yeah, sure. I'm Emmanuel. I work on the interpretability team here at Anthropic, more specifically on the circuits team. We recently released a pair of papers about the work that we've been doing over the last months. More recently, we released some code in partnership with the Anthropic Fellows Program. It was mostly built by Anthropic fellows, and it lets people play with the research, basically.

I'm happy to talk about that. We also hope to keep releasing more things and partnering with other groups that are working on similar stuff.

Yeah, amazing. We'll get deeper into the behind-the-scenes work on the main podcast, but let's dive right into what you released because that's the most topical thing. You just launched it yesterday. What can people do, or what do you recommend people try?

Totally. At a really high level, the idea of the research itself is to try to explain some of the computation that a model did when it predicted a given token. In our paper, we show how to do this, and then we show examples of doing this on internal, private models. The release this week lets anyone do it for a set of open-source models. Notably, perhaps the easiest one here is Gemma 2 2B.

You can think of some prompt, and you can explain any token that the model samples. “Explains” here means basically blowing up the internal state of the model and showing all of the intermediate things that the model was thinking about before it got to the final token that it predicted.

Some of the things that you guys put out in circuit tracing are a few core examples, right? We can see how these models have internal reasoning states, and there's multi-hop reasoning. Some of the stuff that we talked about on the podcast was: How can people who are interested in how models work actually do anything? What are the open questions, and how can people contribute?

It seems like the follow-up is, “Okay, it's been a few weeks, and now here's a huge library.” Before we even get into it, what are some open questions that you would expect people to play around with? What are people going to do? Why should we probe Gemma or Llama? What are interesting things we can do, and do you have any tips on using it?

Yeah, I think there are maybe 2–3 categories of things that people could do. I'll go from the most basic, low-effort option to, if you want to dedicate a month of your life, you could do that.

The most basic thing is that Gemma 2 2B and Llama 1B are smaller models, but they can still do a bunch of stuff. For most of the things that they can do, we still don't really know or have a good mental model of how they do them.

To give you an example, one of the things in the paper is this multihop reasoning, where we ask Claude 3.5 Haiku, “The capital of the state where Dallas is located is Austin.” It turns out that Gemma can do this too. As part of the release, we have a notebook where one of the Anthropic fellows walks through a bunch of examples, including this one. It's really cool because you can see that the way the circuit looks in Gemma is like the way it looks in a huge model. That in itself is a pretty novel discovery, I think.

You have these models that are super different. If you look at their evals or just try to use them, they're very clearly different, but for this one task, the way they do this multistep reasoning is actually the same.

In the notebook, there are other examples of fun things that we looked at that I think can pique your interest if you're new to thinking about this stuff. At the end of the notebook linked in the README, there are 3 examples of random cases that we haven't solved or labeled. They have a graph precomputed for you, and you can just look at it and try to figure out what's happening.

By “figure out what's happening,” we mean looking at these representations and trying to understand the computation the model is doing. Part of the release also lets you run experiments to verify that you're right. If you think that the model first thinks about Texas in this case, you can stop it from thinking about Texas and see if that damages its performance.

The tools to do that are available. I would say that's the first thing, and maybe the baseline thing—and the thing that I'm most excited about with this release. There are a lot of behaviors that models do far more than any single group has time to explore, so the hope is that you pick a behavior you think is interesting, try to understand what's happening, and try to ground it out.

The other thing I want to mention—parts 2 and 3—is that we also hope other groups and interested researchers can use this to extend the method. If you have an idea about how to do this better, the whole code for making these graphs is open source. You can take a look at it and try to play with it, find different ways to create these graphs, and extend it to other models.

There are many different models, and part of making this work on any model is that you have to train the replacement model. There is code for that, and there are other groups working on it. If you're excited about that, you could say, “Okay, cool. I want this to work on another open model,” and add it if you're more into the engineering and ML engineering side of things.

Yeah, we actually get into a little bit of how you guys do the extra data visualization that makes your blog posts pop so much. Should we share the screen a little bit and dive in? I think you guys prepared some examples.

Totally. Yeah. There's nothing better than the creator of the tool walking through the tool, and we might as well capture that so that people who actually want to do this can follow along.

Yeah, that makes sense. Let me just actually share my screen. My one little experiment was that I cloned the repository, threw it into Claude Code, and said, “Deal with this. Let's try it end to end.”

If you're just trying to get started, the circuit-tracing tutorial notebook is very good. It goes over all the high-level ideas.

That's awesome to hear. Actually, I might just open the notebook first and quickly walk through the illustrations. You're the second person to tell me that they just had Claude Code dig in initially on it, so I'm glad that's working.

The tutorial here is linked at the top of the repository. It essentially walks you through how to think about graphs and links to these circuits. This is the 2-step reasoning we're talking about. This is a schematic of it, where the model has to think of Texas and then Austin.

The notebook links you to all of these circuits, and this is the thing that you can play with. This is the UI on Neuronpedia that lets you create any circuit. Here, we could explore the circuit, and if you open the notebook, you can explore it.

I think so.

Okay, cool. You can make a new graph very easily and quickly. Maybe this is the most fun thing. One thing I was playing with right before joining this call is that it turns out podcast guests are very formulaic. If you say, “Thanks for having me on the whatever,” Gemma seems to pretty consistently guess that you're on a podcast, which makes sense. Why would you say, “Thanks for having me on the blah”?

Here we can try to say, “Okay, how does Gemma know to complete the sentence with ‘Thanks for having me on the Latent Space podcast’?” The way you generate a graph is that you type a sentence where the next word is the thing you're interested in, and then you try to explain how the model got to the next word.

Here, you can give it a name, and then you can mostly just not worry about any of these parameters if you're just playing with it. You can click “Start generation,” and all this generates—something important for people to know is that these are trained on base models, so they're not chat models.

So basically, when you train these models, they're trained to predict the next token, and they don't have that user-assistant chatbot flow. They're prompted in a way such that the output should basically just be the next word. You want to think about the prompt or the text you're making as the text of a book or an article rather than a conversation: what is a sentence where, if you were to read it in a book, the next word would be the interesting one?

You can click on it. It takes a little bit of time to load because there's a bunch of data. What we're going to show you here is almost every single feature that activates in the model. The features are these intermediate representations, and at the bottom there's the prompt. Here, it's “Thanks for having me on the Latent Space,” and at the top you can see what the model outputs.

Its most likely output is “podcast”; it's pretty confident that we're talking about a podcast. Then it has some random stop tokens—“blog,” “show”—and some stuff that makes less sense, but these are small models, so sometimes they say random stuff.

You can explore this by asking, “The model says ‘podcast,’ so why does it say ‘podcast’?” You can click on this output and see which features—again, these intermediate representations—have an input to it. There are features here—this is layer 18—that are already about podcast episodes. You know this because the features have a label, but you can also look at the feature itself. Here you can see other text over which the feature is active, and it's just text about podcasts. That's one way to understand what the features are.

Then you can keep going back. It says “podcast” because of this podcast feature. Where did that come from? It comes from words related to podcasts, words associated with podcasts, as well as an interview feature. It also comes from the word “on.” There's a bias: if you're saying “blah blah blah on,” that slightly increases the chance that you're talking about a podcast at all. You can keep going back and explore the graph interactively.

I would say the way to do it—we talked about this on the longer version of the podcast—is to chase from the interesting outputs back or from the interesting input forward. There are many nodes, and I wouldn't recommend looking at all of them. You can prune them more aggressively if the graph is too busy. This shows you only the most important nodes, and you can be pretty extreme with it if you want, or you can show the whole thing and be completely overwhelmed.

Once you do this, you can group your nodes into similar ones to make a graph. I made this little summary earlier, so I can share that. This is the exact same graph, but before hopping on, I made a few groups. It's the same thing with the podcast: there's a bunch of nodes about podcast episodes and things like discussing podcasts, and there's a node about expressing gratitude that amplifies the idea that you're on an interview or a podcast.

One fun experiment you could do is ask, “What happens if I mess with this? What if I mess with ‘this person is grateful to be on’ and instead have ‘this person is on’? Does it think you're on something else?” There are things you could be on that you're not grateful for, like, “You're having me on trial,” or something. That could be an interesting experiment to see the causal effect of this. You could label it more and explore it more.

The whole point of this UI is for it to be snappy and quick, so you can generate a bunch of graphs pretty easily. Maybe this wasn't exactly what you wanted, so you say, “I'm super unhappy to be on the Latent Space,” and then you can see what it completes for that. You can continuously play with it and get a better sense of your hypotheses. Oftentimes, you want different prompts—different examples that are similar—to get a sense for it.

If you're really curious and want to dig in more, that's when I would recommend going back to the codebase and some of the notebooks. One last thing I'll say is that the notebooks themselves can all be run on Google Colab. As far as we can tell, we've tested all the notebooks, and all the code runs on Colab. That means you don't need an expensive GPU; the free tier is enough, to be clear. You can run this, run your interventions, and play with it.

In this notebook in particular, Intro 1, we show you how to do these interventions. Here we're asking what happens if we turn this node off, what happens if we turn that one off, what happens if we turn this one off, and what happens if we inject one from one prompt into another. That's the deeper dive for trying to understand the mechanism better. But if you're just trying to get a sense of how a model does something, you can generate a graph and take a look at it.

Incredible. Very cool. When I look at the graph, there's a thought in my mind: maybe this is too easy, too perfect. One version of this is that there's supposed to be superposition, and here there's no superposition—or, well, there is superposition. Maybe you can share the graph again and answer my question: What are we hiding here? Where are the skeletons? It's too clean.

A good example—and we're going to make this slightly less overwhelming—is that you look at this graph and say, “We don't actually understand how models work fully. What are you hiding here?”

The important thing to know is that I didn't say this explicitly, but the layers are arranged here. Let's look at one layer. For this layer, what we're saying is that the only thing happening, or the only thing important enough, is this one feature, which is one small direction in the model space—one dimension we've pulled out of superposition. Let's say that for now.

But there are also these diamonds. These diamonds are errors. We talked about them on the longer podcast, but when you train these replacement models to replace some of the model's computation, you successfully replace some of it, and some of it you fail to replace. This is everything we don't understand.

That means that sometimes, if you look at an input like this guy's input, you'll see a bunch of errors here as the input. Essentially, there are some graphs and examples where, if most of what you see is these errors, that just means that, for this prompt, we weren't able to explain that part of the computation. At least that part is an explicit thing we show you right in your face: here's what we don't understand. You can see what we don't understand.

There's one more thing: a lot more can get you, and that's in the paper. One example is that these are just MLPs. The model has both attention heads and multilayer perceptrons, or MLPs. We completely ignore attention; we don't try to decompose it at all. So there are prompts where all the interesting stuff is attention, and you're not seeing it at all. The way that it materializes is that you have an edge from here to here, and some attention head did a bunch of stuff. You don't know what it is, and that's also the part we're not explaining.

I definitely don't want to make the claim that we explain everything. I think the correct way to think about this is that if you look at a prompt and can trace through it without hitting any errors, hit nodes that make sense, and build up a reasonable hypothesis—and then, when you test it with interventions, it works—then you've at least understood some, and presumably a reasonable proportion, of the computation. If your interventions are working, that means the thing you found isn't just a side effect. It's part of the main thing the model is doing. The question is how often that happens versus how often you hit these errors or find yourself confused. That's just what works and what doesn't.

Crazy. I mean, congrats on this work. I know you're low on sleep because you worked really hard on shipping it, and you're a perfectionist.

I'll just say that the actual brunt of the work here was done by the fellows. They did all the implementation, as well as the folks on the Neuronpedia and Decode Research side, who did the lion's share of the work here to actually build the front-end UI.

I would just say that Vibhu and I were at the Goodfire meetup yesterday, where there were a lot of interpretability folks. I was shocked at, honestly, how young most interpretability people and the work are. This is a very young field, exactly like you say in the podcast—there’s a lot of fresh, green grass here to tread, and it’s just really inspiring. Do you have any other final thoughts or comments?

Yeah. No, I think there’s just a lot of open work to be done, and we talk about this in the podcast, too. To reiterate, the tooling that you guys put out is so good. Even the fact that, without diving into any code, you can enter a prompt and start to play through these circuits in minutes is pretty incredible.

I could share another one, actually. I was doing this with a Pomsky, and I finally got it to work. Our guest host of the episode is Mochi, my little dog. She’s our distilled Husky, so she’s on the podcast later. I basically put in—I had to guide it quite a bit—but my prompt is, “A Pomsky is a small dog that’s a breed of a Husky and a…”

Let me share my screen real quick, and then we can kind of dig through this.

By the way, while you put it up, her tagline is officially “Mochi, the interpretability Husky.” Today—for today—we’re going to change our tagline every episode.

Yeah, it feels a little weird. We’re digging deep into what Mochi is, but basically, this is me with no background: 2 minutes in, I just put in a phrase, and now I get to play around with features, right?

This is also called “please” with 4 S’s, because I tried a few prompts.

It’s okay. It’s okay. We struggle.

It only took a few minutes, though. “Pomsky is a small dog breed that’s a mix of a Husky and a…” The most probable output now says “Pomeranian.” So, okay, let’s dig into what some of these are.

I’m basically just going in fresh—I haven’t done this before—but we have words related to animals, their emotions, and their health. We have a feature for dogs, Golden Labs, mentions of dog breeds, and especially high-maintenance dogs. This is basically AGI. It knows Pomskies are high-maintenance. It’s figured it out.

But realistically, as I dig through these features, I can start to pin them and layer them through. “Mentions of garbage and waste.” No, that’s not nice. That’s not nice. But interestingly, this is already me pruning out most of the features as I open it up.

It talks about different things, like dog breeding. What else? Things related to animal welfare. You can dig through all this. There are just so many things. This is in a matter of minutes: I basically made a graph, put in a sentence, and now I have an output, and I can traverse through the different things.

Okay, animal science, right? This breed is relatively new. It’s not that common that big Huskies and little Pomeranians naturally have offspring, but let’s dig through animal-science versions of this. Then we have interesting little features.

It’s very easy for people to get a different understanding of what goes on throughout the layers in models. That’s just my fun little experiment of getting it to work.

That’s the whole goal. That’s super exciting.

Yeah. Full disclosure, this was 5 minutes of just playing around, and there’s stuff to learn there, right? What happens with dog breeding? What are the traits of these dogs? The next step for me would basically be to try clamping some of these features up or down. Let’s do different breeds and see if it makes sense.

If I have Husky traits in a different mix, can I get out what’s going on? It also shows internally that there’s more than just token completion—“this plus this equals this.” No, it has some understanding of characteristics. This is a pretty stubborn dog; it has a stubborn feature pretty high up that activates. Very, very cool stuff.

I think it’ll be cool when we apply this to more serious topics. Right now, when it comes to evals, we have pretty straightforward evals: How good does it do on math? Can it write code? Does stuff compile? But we don’t have vibes-based heuristic evals, right? Does it understand different queries? Should it be concise? Should it be verbose? Can we trace through how it gives responses to this stuff?

The other part is, as we go past base models, how does this happen for different phases of models? If I have a base Gemma and I have a chat model, what are the differences in their activations? What happens in that difference in training? That’s one of my interests in mechanistic interpretability: What happens as we do more training? What are we really changing?

Totally, yeah. You can think about comparing different models. For me, different models could mean Gemma versus some other model, early Gemma versus late Gemma in pretraining, or fine-tuned versus not fine-tuned.

I think there’s also a sense in which somebody yesterday was telling me, “It’s fun. I’ve been playing with it on the weird riddles that the models get wrong.” You’re not limited to studying what the model can do, right? If the model’s failing at something—like counting the number of letters in “strawberry,” or whatever—you could try that and figure out the circuit for why it’s getting it wrong. Maybe you can see in its representation that it’s thinking about something obviously incorrect. I think that’s also a fun thing to play with.

I think that’s it for our little intro chat and coverage of the open-sourcing. Let’s dive right into the episode next. But Emmanuel, your work is amazing, and I’m so inspired. I think this puts a human face on the interpretability work, which is very important, and we’d love to keep doing this, whatever you have coming up next.

Well, yeah, thanks for having me. Again, I should say, it’s cool to put a face on it, but I definitely want to call out that this is a huge team of people with me. I’m just a talking head here. And you, as the paper lead, did the work, so take credit.

I’m happy to talk about more interpretability things. Also, feel free to reach out to me. I’m findable if you’re listening to this podcast and you have questions about stuff that’s broken, or if this brings up experiment ideas. I definitely want more people playing with this. So, yeah, thanks for having me. I hope that inspires some folks.

All right, we are back in the studio with a couple of special guests. One, Vibhu, our guest co-host for a couple of times now, as well as Mochi, the distilled Husky, who is in the studio with us and has asked some very pressing questions.

We also have Emmanuel. I didn’t get your last name. Amiesen? Is that Dutch?

It’s actually German.

German. Yeah. You are the lead author of a fair number of the recent mechanistic interpretability work from Anthropic that I’ve been basically calling “Transformer Circuits,” because that’s the name of the publication.

To be clear, “Transformer Circuits” is the whole publication. I’m the author on one of the recent papers, “Circuit Tracing.”

Yes, and people are very excited about that. The other name for it is “Tracing the Thoughts of a Large Language Model.” There are 3 different names for this work.

It’s all mechanistic interpretability. There are 2 papers. One is “Circuit Tracing,” which is the methods paper. The other is about the biology, which is kind of what we found in the model. “Tracing the Thoughts” is confusingly just the name of the blog post where we announced it. They’re for different audiences.

When you produce the 2-minute polished video that you guys did, that’s meant for a very wide audience.

Yeah, that’s right. There are many levels of granularity at which you can go. I think for mechanistic interpretability in particular, because it’s kind of complicated, going from top to bottom—from the highest level to the nitty-gritty details—works pretty well.

Cool. We can get started. We have 2 paths that you can choose from: either your personal journey into mechanistic interpretability or the brief history of mechanistic interpretability generally. Maybe those might coincide a little bit.

I could just give you my personal journey very quickly, because then we can do the second path. My personal journey is that I was working at Anthropic for a while. I’d been, like many people, just following mechanistic interpretability as an interesting field with fascinating, often beautiful papers.

I was working on fine-tuning—actually fine-tuning production models for Anthropic—and eventually, my fascination reached a sufficient level that I decided I wanted to work on it. I also got more excited about understanding how our models worked as they got better and better. So that's the simple journey: I've got a background in ML. I did a lot of applied ML work before, and now I'm doing more research.

You have a book with O'Reilly. You're head of AI at Insight Data Science. Anything else to plug?

I actually want to plug the paper and unplug the book.

Okay.

I think the book is good, and I think the advice stands the test of time, but it's very much about building AI products: What should you focus on? That's very different from the stuff we're talking about today. Today is research into some of the deepest, weirdest things about how models work. This book is about wanting to ship a random forest to do fraud classification: here are the top 5 mistakes to avoid.

The good old days of ML. I know it was simple back then. You also transitioned into research, and I think you managed to do that without following the traditional path. I feel like there's this monolith where people assume you need a PhD for research. Maybe you can give that perspective: How do people get into research? How did you get into research? Maybe that gives the audience some insight into your background as well.

My background was in economics and data science. I thought LLMs were pretty interesting, so I started out with some basic ML work, and then I saw that LLMs were starting to become a thing. I just went out there and did it. The same thing applies to AI engineering: You build stuff, you work on interesting things, and now it's more accessible than ever.

When I got into the field 5 or 6 years ago, pretraining was still pretty new, and GPT-3 hadn't really launched. It was still very early, and it was a lot less competitive. Without any specific background or a PhD, there just weren't as many people working on it.

You made the transition a little bit more recently, right? What's your experience been like?

I think it has maybe never been easier in some ways, because a lot of the field is pretty empirical right now. The bitter lesson is that a lot of the time, you can scale up compute and data and get better results than if you thought extremely hard about a really good prior inspired by the human brain to train your model better.

In terms of research for pretraining and fine-tuning, a lot of the bottlenecks are extremely good engineering and systems engineering. A lot of the research execution is about engineering, scaling up, and things like that.

I think mechanistic interpretability in particular is easier to transition into for a couple of reasons. One is that you can do it without huge access to compute. There are open-source models that you can look at, and a lot of interpretability papers coming out of programs like MATS use models that are open source and that you can dissect without having a cluster of 100 GPUs. Sometimes you can even load them on your CPU or your MacBook.

It's also a relatively new field. There are conceptual burdens and concepts that you want to understand before you contribute, but it's not physics. It's relatively recent, so the number of abstractions that you have to ramp up on isn't that high compared to other fields. That makes the transition somewhat easier for interpretability. If you understand what features are and what dictionary learning is, you're a long way there.

I think it's also interesting from a career point of view. Research seems a lot more valuable than engineering, so—and you don't have to answer this if it's a tricky thing—how hard is it for a research engineer at Anthropic to jump the wall into research? People seem to move around a lot, and I'm thinking, that cannot be so easy. In no other industry that I know of can people do that. Do you know what I mean?

I'd push back on research being more valuable than engineering a little bit, because a lot of the time, having the research idea isn't the hardest part. Don't get me wrong: There are some ideas that are brilliant and hard to find. But what's hard, certainly in fine-tuning and to a certain extent in interpretability, is executing on your research idea—making an experiment run successfully and interpreting it correctly.

What that means, though, is that they're not separate skill sets. If you have a cool idea, there aren't many people in the world who can just have a cool idea and then deputize a little minion: “Here's my idea. Go off for 3 months, build this whole model, train it for hundreds of hours, and report back on what happened.”

A lot of the time, the people who are the most productive have an idea, but they're also extremely quick at checking that idea and finding the shortest path to checking it. A lot of that shortest path is essentially engineering skills—just getting stuff done. That's why you see people move around. In proportion to your interests, if you're able to quickly execute on the ideas you have and get results, that's really 90% of the value.

You see a lot of transferable skills, actually. I've certainly seen people at Anthropic who are really good at that inner loop. They can apply it in one team, move to a completely different domain, and apply that inner loop just as well.

Very cracked, as the kids say. Shall we move to the history of mechanistic interpretability? All I know is that everyone starts at Chris Olah's blog. Is that right?

I think that's the correct answer. Chris Olah's blog, and then Distill.pub is the natural next step. Now there's Anthropic, there's Transformer Circuits, which you talked about, but there's also a lot of mechanistic interpretability research out there. MATS is a group that regularly has a lot of research, but there are many different labs that put research out there.

I think that also hammers home the point that all you need is a model and a willingness to investigate it to be able to contribute. Now there's been a bit of a Cambrian explosion of mechanistic interpretability, which is cool.

I'm on Chris Olah's blog, and he has the feature visualization work. For me, the clearest was the vision work, where you could have one layer detecting edges, another detecting textures, and so on. That seemed very clear to me, but the transition to language models seemed like a big leap.

I think one of the bigger changes from vision to language models has to do with the superposition hypothesis. That's maybe the first Toy Models of Superposition post, right?

Exactly.

It turns out that if you look at just the neurons of a lot of vision models, you can see neurons that are curve detectors, edge detectors, or high- and low-frequency detectors. You can mostly make sense of the neurons.

But if you look at neurons in language models, most of them don't make sense. It's unclear why that would be, and one main hypothesis here is the superposition hypothesis. What does that mean? It means that language models pack a lot more into less space than vision models.

A hand-wavy analogy is that if you want curve detectors, you don't need that many curve detectors. If each curve detector is going to detect a quarter or a twelfth of a circle, you can have all your curve detectors. But think about all the concepts that Claude or even GPT-2 needs to know. It needs to know about all the different colors, all the different hours of every day, all the different cities in the world, and all the different streets in every city.

If you just enumerate all of the facts that a model knows, you're going to get a very, very long list, and that list is going to be way bigger than the number of neurons or even the size of the residual stream, which is where the models process information. So there's a sense in which there's more information than there are dimensions to represent it. And that is much more true for language models than for vision models. Because of that, when you look at a part of it, it just seems like it's got all this stuff crammed into it. Whereas if you look at vision models, oftentimes you could just be like, “Cool, this is a curve detector.”

Yeah, Vibhu, you have some fun ways of explaining the toy models or the superposition concept.

Yeah, I mean, basically, if you have 2 neurons and they can represent 5 features, a lot of the early mechanistic interpretability work says that there are more features than we have neurons, right? So I guess my question on this is: for those interested in getting into the field, what are the key terms they should know? What are the few pieces they should follow?

From the Anthropic side, we had a toy transformer model. We first had sparse autoencoders—that was the second paper, right? Monosemanticity. What are sparsity and autoencoders? What are transcoders? What is linear probing? What are these key points in mechanistic interpretability? How would people get a quick 0 to 80% of the field?

Okay, so 0 to 80%. Now I realize I really set myself up for failure because I was like, “Yeah, it's easy. There's not that much to know.” So then we should be able to cover it all.

Superposition is the first thing you should know, right? This idea is that there's a bunch of stuff crammed into a few dimensions. As you said, maybe you have 2 neurons and you want to represent 5 things. If that's true, and if you want to understand how the model represents—I don't know—the concept of red, let's say, then you need some way to find out essentially in which direction the model stores it.

After the superposition hypothesis, you can think, “Ah, we also think that the model represents these individual concepts—we're going to call them features—as directions.” So if you have 2 neurons, you can think of it as the 2D plane, and you can have 5 directions. Maybe you would arrange them like the spokes of a wheel, so they're maximally separate. It could mean that you have 1 concept this way and 1 concept that's not fully perpendicular to it, but pretty far from it. That would allow the model to represent more concepts than it has dimensions.

If that's true, what you want is a model that can extract these independent concepts. Ideally, you want to do this automatically: can we just have a model that tells us, “This direction is red; if you go that way, actually it's—I don't know—chicken, and if you go that way, it's the Declaration of Independence”?

And so that's what sparse autoencoders are. It's almost like the self-supervised learning insight, but for interpretability. In pretraining, you had self-supervised learning—

And here is self-supervised interpretability.

Exactly. Exactly. It's like an unsupervised method, and unsupervised methods often still have labels in the end. Sometimes I feel like the term “unsupervised” is misleading. For pretraining, right, it's the next token, so in that sense you have a supervision signal. Here, the supervision signal is simply that you take the neurons and then learn a model that's going to expand them into the actual number of concepts that you think there are in the model.

So you have 2 neurons, you think there's 5 concepts, so you expand it to dimension 5 and then contract it back to what it was. That's the model you're training, and then you're training it to incentivize it to be sparse, so that there are only a few features active at a time. Once you do that, if it works, you have this nice dictionary, which you can think of as a way to decode the activations. You're saying, “Ah, cool. I don't know what this direction means, but I've used my model, and it's telling me that the model is writing in the red direction.”

So that's, I think, maybe the biggest thing to understand: this combination of ideas. We have too few dimensions, we pack a lot into them, so we're going to learn an unsupervised way to unpack them and then analyze what each of those dimensions that we've unpacked are.

Any follow-ups? The follow-ups to this are also some of the work that you did in clamping, right? What is the application side of mechanistic interpretability? We saw that you guys have great visualizations. Golden Gate Claude was a cool example. What can we do once we find these features? Finding features is cool, but what can we do about it?

I was going to say that. Yeah, that's my favorite.

What can we do once we find these features?

Yeah, I think there are 2 big aspects to this. One is that we go from a state where, as I said, the model is a mess of weights and we have no idea what's going on to: okay, we found features. We found a feature for red and a feature for the Golden Gate Bridge. What do we do with them?

If these are true features, that means that, in some sense, they're important for the model, or it wouldn't be representing them. If the model is bothering to write in the Golden Gate Bridge direction, it's usually because it's going to talk about the Golden Gate Bridge. That means that you can set that feature to 0 or artificially set it to 100, and you'll change model behavior.

That's what we did when we did Golden Gate Claude. We found a feature that represents a direction for the Golden Gate Bridge, and then we just set it to always be on. Then you could talk to Claude: “Hey, Claude, what's on your mind? What are you thinking about today?” It would be the Golden Gate Bridge. You'd be like, “Hey, Claude, what's 2 + 2?” It'd be, “4 Golden Gate Bridges,” et cetera, right?

If you asked it to write a poem, it started talking about the Golden Gate Bridge.

Yeah, amazing. I think what made it even better is that we realized later on that it wasn't really a Golden Gate Bridge feature. It was about being in awe of the beauty of the majestic Golden Gate Bridge, right? So on top of it, it would really ham it up. It'd be like, “Oh, I'm just thinking about the beautiful international orange color of the Golden Gate Bridge.”

That was just an example that I think was really striking, but it's sort of this idea that if you found a space that represents some computation or some representation of the model, that means you can artificially suppress or promote it. That means you're starting to understand, at a very high level—a very gross level—how some of the model works. We've gone from “I don't know anything about it” to “I know that this combination of neurons is this, and I'm going to prove it to you.”

The next step, which is what this work focuses on, is thinking about this with an analogy. If you take the analogy of an MRI or something like a brain scan, it tells you, “As Claude was answering, at some point it thought about this thing.” But it's sort of vague—basically, maybe it's like a bag of words, a bag of features: here are all the random things it thought about.

But what you might want to know is: okay, Claude is doing some processing. Sometimes, to get to the Golden Gate Bridge, it had to realize that you were talking about San Francisco and about the best way to go to Sonoma or something, and so that's how it got to the Golden Gate Bridge. There's an algorithm that leads to it, at some point, thinking about the Golden Gate Bridge.

Basically, there's a way to connect features to say, “From this input, it went to these few features, and these few features, and these few features, and that one influenced this one,” and then you got to the output. That's the second part, and the part we worked on is what we call—or what's called—circuits, which is explaining the algorithm.

Before we move directly on to your work, I just want to give a shout-out to Neel Nanda. He did Neuronpedia and released a bunch of essays for, I think, the Llama models and the Gemma models.

I actually made Golden Gate Gemma. I just upped the weights for proper nouns and names of places and people, and references to the term “golden,” likely relating to awards, honors, or special names. That together made Golden Gate.

That's amazing. You can make Golden Gate Gemma, and I think that's a fun way to experiment with this.

I'm curious: what's the background behind why you shipped Golden Gate Claude? You had so many features. Is there any fun story behind why that's the one that made it?

You know, it's funny. If you look at the paper, there's just a bunch of really interesting features, right? One of my favorite ones was the sycophantic praise, which I guess is very topical right now.

Very topical.

You could dial that up, and Claude would just really praise you.

He’d be like, “Oh, you know, I wrote this poem—roses are red, violets are blue, whatever.” And he’d be like, “That’s the best poem I’ve ever seen.” We could have shipped that. That could have been funny.

Golden Gate Claude was, as far as I remember, a pure, weird, random thing. Somebody found it initially, we had an internal demo of it, and everybody thought it was hilarious. That’s sort of how it came to be. Nobody had a list of the top 10 features we should consider shipping and then picked that one. It was just a very organic moment.

No, the marketing team really leaned into it. They mailed out pieces of the Golden Gate to people in Europe, I think, or to ICML. It was fantastic marketing.

The question obviously is: if OpenAI had invested more in interpretability, would they have caught the GPT-4o update? We don’t know that for sure, because they have interpretability teams.

Yeah. I also think that, for that one, I don’t know that you need interpretability. It was pretty clear-cut to the model: “That model’s really gassing me up.”

And then the other thing is: can you just turn up writing good code and turn down writing bad code? It feels too easy, too free. Is steering that powerful, that you can just turn features up and down with no trade-offs?

There was a phase where people were basically saying that 3.5 and 3.7 were just [?] because they came out right after. For the record, that’s been debunked. It had people convinced that what happened was that people basically just steered features up and down, and now we have a better model.

This kind of goes back to that original question: why do we do this, and what can we do? Some people want tracing from a sense of legality—what did the model think when it came to this output? Some people want to turn hallucination down, and some people want to turn coding up. What are some of the applications of this, whether it’s internal work you’re exploring or just what people can do with it? Why do mechanistic interpretability?

Yeah, there are a few things here. First of all, I would say that, on a scale from the most short-term to the most long-term, this is pretty long-term research. In terms of applications, compared to the research work we do on fine-tuning or whatever, interpretability is much more of a high-risk, high-reward kind of approach.

With that being said, I think there’s a fundamental sense in which just knowing how the model works at all feels useful. Michael Nielsen had a post recently about how knowledge is dual-use or something, but it’s hard to argue that if we know how the model works and understand all of its components, that won’t help us make models that hallucinate less or are less biased. At the limit, that seems like something you could do by using your understanding of the model to improve it.

I think for now, as we can talk about a little bit with circuits, we’re still pretty early in the game. Right now, the main way that we’re using interpretability is to investigate specific behaviors, understand them, and gain a sense of what’s causing them.

One example we can talk about now or later is that, in the paper, we investigate jailbreaks and try to see why a jailbreak works. We realized, as we were looking at one of these jailbreaks, that part of the reason Claude is telling you how to make a bomb is that it has already started to tell you how to make a bomb, and it would really love to stop telling you how to make a bomb—but it has to first finish its sentence. It really wants to make correct grammatical sentences.

It turns out that, after seeing that circuit, we asked, “Does that mean that if we prevent it from finishing its sentence, the jailbreak works even better?” Sure enough, it does. I think the level of practical application right now is of that shape: understanding quirks of a current model, or understanding how it does tasks where we might not even know how it does them. We have some planning examples where we had no idea it was planning, and we’re like, “Oh God, it is.” That’s sort of the current state we’re at.

I’m curious how this feeds back internally into the research, architecture, pre-training, and post-training teams. Right now, there are a lot of external people interested. We’ll train an SAE on one layer of Llama and probe around, but then people ask how much impact that really has. People like clamping, but as you said, once you start to understand that these models have early planning and so on, how does that feed back?

I don’t know that there’s much to say here other than that we’re definitely interested in making models that are easier to interpret. That’s something you can imagine working on: making models where you have to work less hard to try to understand what they’re doing architecturally.

Okay. Yeah. Yeah. I think there was a LessWrong post about this: there’s a nonzero amount of sacrifice you should make in current capabilities in order to actually make models more interpretable, because otherwise interpretability will never catch up. The hope was that pruning would do some of that, but I feel like that line of research has just died.

What kind of pruning are you thinking of here?

Just pruning your network.

Ah, yeah—pruning layers, pruning connections, whatever. I feel like maybe superposition makes me less hopeful or something, because you don’t know. That seventh bit might hold something.

On each example, maybe this neuron is at the bottom of what matters, but it’s actually participating 5% in understanding English, doing integrals, cracking codes, or whatever. Because that’s distributed across the network, when you naively prune, you might miss that.

So, for this area of research—creating models that are easier to interpret from the start—is there a name for it?

I don’t think so. I think this is very early, and it’s mostly just a case of there being a thing people want to double-click on.

Yeah. I haven’t come across it. I think the higher-level idea is that Dario recently put out a post about why mechanistic interpretability is important. We don’t want to fall behind, and we want to be able to interpret models and understand what’s going on even though capabilities are getting so good. It ties into this topic: we want models to be slightly easier to interpret so we don’t fall behind so far.

Well, yeah. To talk about the elephant in the room, one big concern here is safety. As models get better, they’re going to be used in more and more places. We’re vibe coding right now; maybe at some point that’ll just be coding. Claude’s going to write your code for you, and that’s it. Claude’s going to review the code that Claude wrote, and then Claude’s going to deploy it to production.

At some point, as these models get integrated more deeply into more and more workflows, it gets scarier and scarier to know nothing about them. You want your ability to understand the model to scale with how good the model is, which itself tends to scale with how widely deployed it is. As we deploy them everywhere, we want to understand them better.

The version I liked from the old Superalignment team was weak-to-strong generalization, or weak-to-strong alignment. That’s what Superalignment, to me, was. That was my first “aha” moment: at some point, these things will be smarter than us. In many ways, they already are smarter than us, and we rely on them more and more. We need to figure out how to control them.

This isn’t an Eliezer Yudkowsky-style “AI” thing. It’s more that we don’t know how these things work, so how can we use them?

You can think of it this way: there are many ways to solve a problem. If the model is solving it in a dumb way, or has memorized one approach to doing it, then you shouldn’t deploy it to do something general.

You could look at how it does math and, based on your understanding of how it does math, say, “Okay, I feel comfortable using this as a calculator.” Or you could say, “No, it should always use a calculator tool, because it’s doing math in a stupid way.”

Extend that to any behavior. If you’re in the 1500s and I give you a car or something, I’m just like, “Cool.”

It's like this thing: when you press on this, it accelerates; when you press on that, it stops. You know, this steering wheel seems to be doing something, but you know nothing about it. I don't know if it was a very faulty car and someone said, "Oh, yeah, but if you ever went above 60 mph, it explodes," or something. You'd probably want to understand the nature of the object before jumping into it.

That's why we understand how cars work very well: because we make them. LLMs, and ML models in general, are this very rare artifact where we make them, but we have no idea how they work. We evolve them. We create conditions for them to evolve, and then they evolve, and we're like, "Cool. Maybe you got a good run; maybe we didn't." We don't really know.

The extent to which you know how it works is that you have your eval, and you're like, "It seems to be doing well on this eval." Then you're like, "Is it because this wasn't in the training set, or is it actually generalizing?" I don't know. My favorite example was that somehow C4, the Colossal Clean Crawled Corpus, did much better than Common Crawl, even though it filtered out most of the content. It was very prudish: it filtered out anything that could be considered obscene, including the word "gay."

Somehow, when you add it into the data mix, it just does extremely well. It's this magic incantation of, "This recipe works. Just trust us. We've tried everything. This one works, so just go with it."

Yeah, it's not very satisfying.

No, it's not. The side that you're talking about is, "How do you make these?" It's kind of unsatisfying that you just make the soup and say, "My grandpa made the soup with these ingredients. I don't know why, but I just make the soup the way my grandpa said." Then one day somebody added cilantro, and since then we've been adding cilantro for generations. You're like, "This is kind of crazy."

That's exactly how we train models, though. So I think there's a part where we should try to unpack what's happening—the mechanisms of learning and how our models learn. One of the things we skipped over was induction heads: understanding what induction heads are, which are attention heads that allow you to look in your context for the last time something was mentioned and then repeat it.

That's something that seems to happen in every model. It's like, "Okay, that makes sense. That's how the model is able to repeat text without dedicating too much capacity to it."

Let's get it on screen so people can see the visuals of the work you guys put out. They're amazing. We should talk a little bit about the behind-the-scenes work on that mechanism, but let's finish this off first.

Totally. Just really quickly, I don't think we should spend too long on it. If you're interested in mechanistic interpretability, we talked about superposition, and I think we skipped over induction heads. That's a really neat pattern that emerges in many transformers.

One of the things you need to do to predict text well is recognize that if somebody said "Emmanuel Amiesen" and then, on the next line, they say "Emmanuel," there's a very good chance it's the same last name. One of the first things that models learn is, "I'm going to look at what was said before, and I'm going to say the same thing."

That's induction heads: a pair of attention heads that look at the last time something was said, look at what happened after it, and move that over. It's an example of a mechanism where we understand things pretty well. There's been a lot of follow-up research on which contexts they turn on in and the different levels of abstraction. There are induction heads that literally copy the word, and there are some that copy the sentiment and other aspects.

It's an example of slowly unpacking, or peeling back the layers of the onion, to understand what's going on inside the model. You can say, "Okay, this is a component, and it's doing this." Induction heads were the first major finding. It was a big finding for NLP models, for sure.

I often think about the edit modes. Claude has a fast edit mode—I forget what it's called—and OpenAI has one as well. You need very good copying in every area that needs copying, and then you need the model to switch out of copy mode when it needs to start generating, right? That's basically the productionized version of this.

Yeah. It turns out that you need to select a model that's smart enough to know when it needs to get out of copy mode, right? It's fascinating. It's faster and cheaper.

As bullish as I am on Canvas, basically every AI product needs to iterate on a central artifact. If it's code or a piece of writing, it doesn't really matter, but you need that copy capability that's smart enough to know when to turn it off.

That's why it's cool that induction heads operate at different levels of abstraction. Sometimes, when you're editing some code, you need to copy the general structure. It's like, "The last function that's similar first takes an abstract class and then takes an int, so I need to copy the general idea, but it's going to be a different abstract class and a different int."

So, tracing. Should we jump to circuit tracing?

Sure. I don't know if there's anything else you want to cover.

No, we have space for it, maybe. I'll do a really quick TL;DR of these 2 recent papers.

We talked about these features that we detect, and what we said was, "Okay, but we'd like to connect the features to understand the inputs to every feature and the outputs of every feature, and basically draw a graph." That's the dream: for a given prompt, what were all the important things that happened in the model?

Here, the model takes in these 4 tokens; those activate these features; these features activate these other features; those features activate the next set of features; and all of these promote the output. That's the story. The work is to use dictionary learning and these replacement models to provide an explanation consisting of sets of features that explain behavior.

This is very abstract, so maybe we can immediately look at an example. I can show you the reasoning one.

Yep. Two-step reasoning.

I think this is already kind of fun. The question is: you ask the model something that requires it to take a step of reasoning in its head. You say, "The capital of the state containing Dallas is..." To answer that, you need 1 intermediate step, right? You need to say, "Wait, where's Dallas? It's in Texas. Okay, the capital of Texas is Austin."

This happens in 1 token. After "is," it's going to say "Austin." In that 1 forward pass, the model needs to realize that you're asking for the capital of a state, look up the state for Dallas, which is Texas, and then say Austin.

Sure enough, what we see in this forward pass is a rich set of inner representations. It gets "capital," "state," and "Dallas," and then it has an inner representation for Texas. That, plus "capital," leads it to say Austin.

I guess one of the things here is that we can see this internal thinking step, right? But a lot of what people say is, "Is this just a memorized fact?" I'm sure a lot of the pretraining that this model was trained on contains this sentence pretty often, right?

So this shows that, no, internally, we do see that there is this middle step. It's not just memorized. You can prove that it generalized.

Yeah, that's exactly right. You hit the nail on the head. This example is about the fact that if this was just memorized, you wouldn't need to have an intermediate step at all. You'd just say, "I've seen the sentence. I know what comes next."

But here, there is an intermediate step. You could say, "Maybe it has the step, but it's memorized anyway." The way to verify that is what we do later in the paper, and for all of our examples: we claim that this is the Texas representation, so let's get another one and replace it. We change that feature in the middle of the model to California.

If you change it to California, sure enough, it says Sacramento. This isn't just a byproduct, where it's memorized something and, on the side, it's thinking about Texas. This is a step in the reasoning. If you change that intermediate step, it changes the answer.

Very cool work. Underappreciated. I have never really doubted it. I think there's a lot of people who are always criticizing LLMs as stochastic parrots. This pretty much disproves that already, so we can move on.

Yeah, I think there are a lot of examples we can go through. Just a few of them show an amount of depth in the intermediate states of the model that makes you think, “Oh gosh, it’s doing a lot.” Maybe the poems—definitely the poems. But even for this one, I’m going to scroll through this very short paper to the medical diagnosis example. I don’t even know the word count because there are so many embedded things in there.

Yeah, it’s too dangerous. We can’t look it up; it overflows.

It’s so beautiful. Look at this. This is a medical example that, again, I think shows you that this is happening in one forward pass. The model is given a bunch of symptoms, and then it’s asked not, “What is the disease that this person has?” It’s asked, “If you could run one more test to determine it, what would it be?”

So it’s even harder, right? It means you need to take all the symptoms, have a few hypotheses about what the disease could be, and then, based on your hypothesis, say, “The right test to do would be X.” Here you can see these 3 layers: again, in one forward pass, it has a bunch of symptoms, then it has the most likely diagnosis, then an alternate one, and then, based on the diagnosis, it gives you a bunch of things that you could ask.

We do the same experiments where you can kill this feature here—suppress it—and then ask it a question about the second option it had. The reason I show it is that there’s a lot of stuff going on in one forward pass. Specifically, if you expected it to simply see similar cases in the training data and then vibe its way through—“Oh, I guess there’s that word”—and say something related to, I don’t know, a headache, it’s like, no, no, no. It’s activating many different distributed representations, combining them, and doing something pretty complicated.

So, yeah, I think it’s funny because, in my opinion, stochastic parrots is not an appropriate description here. There are a lot of different things going on, and there’s pretty complex behavior. At the same time, I think it’s in the eye of the beholder. I’ve talked to folks who have read this paper and have said, “Oh yeah, this is just a bunch of heuristics mashed together, right? The model is just doing a bunch of things like, ‘If high blood pressure, then this or that.’”

I think there’s an underlying question that’s interesting: now that we know a little bit about how it works, you tell me whether you think that’s impressive, whether you trust it, and whether you think it’s sufficient to ask it medical questions or whatever. You sort of have a bit of insight into how the model is getting to its conclusion. You can see both whether it’s making errors and whether it’s using the kind of reasoning that will lead it to errors.

I think it’s a way to adversarially improve the model quality. Once you can do this, you can reverse-engineer what sequence of words would, to a human, make no sense or let you arrive at the complete opposite conclusion, but still trip up the model.

Yeah, and then you can just improve it from there.

Exactly. This gives you a hypothesis about what’s going on. Imagine if one of those was actually the wrong symptom. You’d say, “It’s weird that the liver condition upweights this other example. That doesn’t make sense. Let’s fix that in particular.”

There’s a thesis—I mean, it’s very prominent now with reasoning models—about model depth. You’re doing all this in one pass, but maybe you don’t need to, because you can do more passes. People want shallow models for speed, but you need model depth for this kind of thinking. Is there a Pareto frontier? Is there a direct trade-off? What would you prefer if you had to make a model: shallow versus deep?

There’s a chain-of-thought faithfulness example. Before I show it, I’m just going to go back to the top here. When the model is sampling many tokens, if you want that to be your model, you need to be able to trust every token it samples. The problem with models being autoregressive is that if they sample a mistake at some point, they keep going conditioned on that mistake, right? Sometimes you need backspace tokens or whatever.

Yeah, yeah, yeah.

Error correction is notably hard, right? If you have a deeper model, maybe you have fewer CoT steps, but your steps are more likely to be robust or correct. That’s one way to look at the trade-off. To be clear, I don’t have an answer. I don’t know if I want a wider, shallow, or deep model. You definitely want shallow models for inference speed.

Sure, sure, sure, sure.

But you’re trading that off for something else, right? You also want a 1B model for inference speed, but that comes at a cost: it’s less smart.

There’s a cool, quick survey paper we just covered about when to use reasoning models versus dense models and what the trade-off is. I think it’s called The Reasoning Economy. They go over a bunch of ways to measure this, with benchmarks around when to use each, because consumers are now paying the cost of this. For those on YouTube, we have a secondary channel called Latent Space TV where we cover that stuff. Nice. That’s our paper club. We covered your paper.

I think you brought up the planning thing. Maybe it’s worth going into that.

Let’s do it. I think this one is interesting. If you think about common questions you have about models, the first one we asked was, “Is it just doing vibe-based, one-shot, pattern-matching based on existing data, or does it have rich representations?” It seems to have intermediate representations that make sense as the abstractions you would reason through.

That’s one thing, and there are a bunch of examples. We talked about the medical diagnosis example. The multilingual circuits are another one that I think is cool, where it’s sharing representations across languages.

Another thing that you’ll hear people mention about language models is that they’re next-token predictors.

For people who won’t dive into this super-long blog post—I know you highlighted 10 to 12 of them—could you give us a quick 15- to 30-second explanation of what you mean by them sharing thoughts across languages? What’s the really quick, high-level version, just for people who aren’t going to go through the whole thing?

The really quick, high-level explanation is that we find that, if you look at the inner representations for concepts inside the model, you can ask the same question. In the original paper, the question we asked was, “The opposite of hot is…”—cold. But you can do this over a larger data set and ask the same question in many different languages, then look at the representations in the middle of the model and ask yourself: when you ask it, “The opposite of hot is…”—which is the same sentence in French, “L’opposé de chaud est…”—is it using the same features, or is it learning independently for each language?

It would be bad news if it learned independently for each language, because that would mean that, as you’re pretraining or fine-tuning, you have to relearn everything from scratch. You would expect a better model to share some concepts between the languages it’s learning. You would expect the same thing for programming languages: if you learn what an if statement is in Python, maybe it would be nice if you could generalize that to Java or whatever.

Here we find basically exactly that. If you look inside the model, at the middle of the model—which is the middle of this plot here—the models share more features. They share more of these representations in the middle of the model. Bigger models share even more.

The smarter models use more shared representations than the less capable models, which might explain part of the reason why they’re smarter. This was another finding: not only does the model have these rich representations in the middle, it also learns not to have redundant representations. If you’ve learned the concept of heat, you don’t need to learn the concept of French heat, Japanese heat, and Korean heat. That’s just the concept of heat, and you can share it among different languages.

I feel like sometimes overanalyzing this becomes a bit of a problem, right? When we talked about the medical example, we could look back and try to fix this in the data set. In language—I don’t remember if it was OpenAI or Anthropic—they basically said that when the model switched languages and they passed it to fluent users, the users said, “This feels like an American speaking this language,” right?

Sometimes there are nuances in a slightly different representation. You don’t want to overengineer these little fixes when you see them.

But then the other side of this is for those tail-end languages, right? For languages that models aren’t good at, and for when you want to solve that last bit, it seems pretty plausible that we can solve this because these concepts can be shared across languages, as long as we can fill in some level of representation, unless I’m wrong.

No, totally. I think this sort of thing also explains why language models are really good at in-context learning. You give them something completely new, and they do a good job. If you give them a new fake language and, in that language, explain that “cold” means this and “hot” means that, presumably they’re able to bind it. To be clear, this is speculation; we don’t show it in the paper, but they’re able to bind it.

Google’s done this. They took a low-resource language, put it in a million-token context, and then it came up.

That’s right. The thing I’d be curious to see is: does it reuse these representations? I bet that it probably does, right? That’s probably a reason why it works well: it can reuse the general representations that it’s learned in other languages.

Yeah. I don’t know—have you talked to any linguistics people recently?

Linguistics researchers would be very interested in this because ultimately this is the ultimate test of the Sapir–Whorf hypothesis. For those who don’t know, it’s basically the idea that the language you speak influences the way you think, which obviously directly maps onto this.

If it’s a complete mapping—if every language maps every concept perfectly onto every other language—in a theoretically infinitely sized model, then the Sapir–Whorf hypothesis is false because there’s a universal truth. If it does not, if there is some overlap where, for example, there are some languages that have no word, there’s this joke where Eskimos have no word for snow or something like that, right? Or water has no word; fish have no word for water. There’s an African language where there’s a gender for vegetables, stuff like that. Languages influence the way you think, and so there should not be a 100% overlap at some point. Of course, it’s at the limit of the infinite model, so who knows if we’ll ever get there—but, yeah.

I think it’s interesting that we also show a little below that some people have made the point about the bias: it sounds like an American speaking a different language. It does seem like the internal representations have a higher connection to the output logits for English tokens, so there’s some bias toward English, at least in the model we studied here.

Any thoughts as to whether multimodality influences any of this? Do concepts map across languages as they do across modalities?

We show this in the Golden Gate Bridge example from the previous paper. I might have it here, actually. There’s a good diagram of this in the slides, where the same concept appears in text and in an image. This is our buddy, the Golden Gate Bridge.

Here we’re showing the feature for the Golden Gate Bridge, and in orange is what it activates over. This is when the model is reading text about the Golden Gate Bridge, and we also show other languages. You’ll have to take my word for it, but it’s also about the Golden Gate Bridge. Then we show the photos for which it activates the most, and sure enough, it’s the Golden Gate Bridge. Again, that shows an example of a representation that’s shared across languages and shared across modalities.

Yeah. I think this is very relevant for autoregressive image-generation models, and now the audio models as well. Something I’m trying to get some intuition for—which you probably don’t have an off-the-bat answer for—is how much does it cost to add a modality?

A lot of people are saying, “Just add a different decoder, align the latent spaces, and you’re good.” I’m like, “I don’t know, man. It sounds like there’s a lot of information lost between those.”

I definitely don’t have a good intuition for this. Although I will say that things like this make you think that if you train on multiple modalities, then you’ll definitely get this alignment, right? But if you train on one and then post hoc train on another, maybe it’ll be harder, or if you train some adapter layer.

Okay. The official answer is “don’t know,” but the official answer is that someone could figure it out.

Right. I think there are people who know, and they just haven’t shared.

Well, you need to find them and get them on this podcast.

Did we want to do the planning example?

Correct. Yeah. Now we’re backtracking up the stack.

All right. Yeah.

The example, I think, again, is—I like this example because of the next-token-predictor concept. I think this is really important to dive into. Maybe what I’ll say is that language models are next-token predictors. That’s a fact; that is what they do. That’s the objective: they’re trained to predict the next token.

However, that does not mean that they myopically only consider the next token when they choose the next token. You can work on predicting the next token, but still do so in a way that helps you predict the token 10 tokens in the future. Now we definitely know that they’re not merely predicting the next token, and for me that was a pretty big update.

You could totally imagine that they could do everything they’re doing by just being really good at predicting the next token, but not having an internal state. It wasn’t a given that they were going to represent internally, “This is where I want to go, and so I’m going to predict the next token.” This example shows a case like that. Do you have it on screen, by the way?

Let me actually—yeah, while you pull it up. Some of the early connections that I made to this were with early Transformers. Think BERT, encoder–decoder Transformers. When they came out, some of the suggestions were, “You don’t take the last layer; you take off the last layer.”

If you want to do a classification task or a translation task for these encoder–decoder Transformers, they’ve kind of overfit on their training objective. They’re really good at masked language modeling, at next-sentence prediction, and stuff like that. What we want to do is throw away the top layer, freeze the bottom layers, and then there was a lot of work done on where we should modify these models.

Should we look at the top 3 layers? Should we look at the top 2? Where should we probe in? We can see different effects. We know that at the very end, they’ve overfit on their task, but there’s a level at which, when we start to change and continue training or fine-tuning, we get better output.

Totally. We could start to see that throughout the layers there’s still a broader understanding of the language, and then we can add in a layer, whether that’s for classification, and fine-tune it so that it learns our task. This planning example is a more robust way to look into that.

If you look at all of the examples in the paper, at the bottom we have this list of consistent patterns. One pattern you see is exactly what you’re talking about: at the top, the features right before the output are often just about what you’re going to say. They’re next-token predictions: “I’m going to say Austin. I’m going to say rabbit. I’m going to say…” It’s not very abstract; it’s just a motor. It’s a motor neuron for a human, right? It’s like, “I’ve decided that I want a drink of water, and so I’m going to grab the bottle.”

At the bottom, they’re all like sensory neurons. They’re just, “I just saw the word X,” or, “I just saw this.” If you want to extract the interesting representations, a lot of the time they’re in the middle. That’s where the shared representations across languages are.

The plan here is to walk through the example really briefly. Say you have a poem. In order to say the second line of the poem, if you want to rhyme, you need to identify what the rhyme of the first line was. You’re at the end of the first line, so you say, “Okay, what’s my current rhyme?” Then you need to think about what your poem is talking about and think about candidate words that rhyme and that are on topic for your poem.

Here, this is what’s happening: the last word is “it,” and there are a bunch of features that represent the direction of rhyming with “eat” or “at.” By the way, we looked at a bunch of poems internally, and I thought it was really beautiful. These models have a bunch of features for, “This word has AB in it,” “This word has many consonants,” and “This word has some flourish to it.” They have a bunch of features that track various aspects you would want to use if you were writing poetry. It’s just concepts and feature detection.

But then, once it’s extracted the rhyme, it comes up with sort of these 2 candidates. In this case, it’s either, “I’m going to finish with rabbit,” or, “I’m going to finish with habit.” The cool thing here is that this happens at the new line, so it happens before it’s even started the second line. It turns out that you can then say, “Is this the plan actually being used?” We do our usual experiments: we remove it, and the model writes a completely different line; we inject something, and it writes a completely different line.

We have these fun examples here, which I’ll show you.

As a mechanical thing, could you just disallow generation of a certain logit? Is that the logic for how we do these interventions?

Yeah. Basically, what these features are is that they’re directions in the model. To remove them, we just write in the opposite direction. We run the model normally, and then, at the layer where it was going to write in this direction, we either add a negative that compensates for it or add a negative that goes even more in the negative direction, sometimes to really kill it. Then we can also add another direction.

In these random examples here, you have this poem: “The silver moon cast a gentle light.” Claude 3.5 Haiku would rhyme with “illuminating the peaceful night,” but if we go negative in the “night” direction and just add “green,” the whole second line it’s going to write is just, “Upon the meadows verdant green.” And so that’s all we’re doing. We’re saying, “We found where it stores its plan, and we delete or suppress the one it stored and go in the direction of something else that’s arbitrary.”

The result that’s striking here is 2 things. One, this plan is made well in advance of needing to predict “night”: it’s made after the first line, before it’s even started the second line. And 2, this plan doesn’t just control what you’re going to rhyme with. It’s also doing what’s called backwards planning: because I need to finish with “green,” I’m not going to say “illuminating the peaceful night,” because then I’d be saying “illuminating the peaceful green.” That doesn’t make sense. I need to say a completely different sentence that lets me finish with “green.” And so there’s a circuit in the model that decides on the rhyme and then works backwards from the rhyme, influencing the setup of your sentence.

Yeah, it’s almost like backprop, but in the future.

Yeah, it’s doing—because the “green” is backpropagating through these words, “verdant” and “meadow” are both green-related. But it’s doing all of that in its forward passes.

Right. In context, that’s kind of crazy. Intuitively, it makes sense, right? Looking at it from a model architecture perspective, where basically you just have a bunch of attention and feed-forward layers and then, at the end, you have the softmax over the next token, you would expect that end to really be like that, right? It’s just picking tokens. So that’s what it’s going to do. And early on, even with traditional models, we could see different concepts start to pop up through early layers. You have some of this throughout your architecture, so it’s very cool to see.

The other question that comes up is: how are we labeling these features? How are we defining them? Are we doing that right? What is an “these words end with” feature? How do we come to that conclusion? How do we map a name to this?

I think this is an important question because you can totally imagine fooling yourself, right?

Yeah. Is there a guy at Anthropic who just maps 30,000 features?

You’re the guy.

He’s the guy.

I did notice, also, with the previous work, “Scaling Monosemanticity,” that as you train bigger and bigger SAEs, a lot of features don’t activate. I think 60% of the 34-million-feature one didn’t activate.

There are a few questions behind your question. The first question was: how do you even label the features? You were telling me, “This is a rabbit feature.” Why should I trust you? I think there are 2 things going on. First, as I mentioned at the start, all of this is unsupervised. In the paper, we have links to little graphs that show more of what’s going on. This graph is completely unsupervised: we train this model to untangle the representation, this dictionary that we talked about that gives us the features, and then we just do math to figure out which features influence which other features and throw away the ones that don’t matter. At the end, we have these features.

Right now, we don’t have any interpretation for them. We just say, “These are all the features that matter,” and then we manually go through them and look at the features. We look at this feature and that feature, and let’s pick one. This one we’ve labeled, say, “habit.”

How do we do that? You could just look at it, and we show you what it activates over. If you just look at this text, you’ll immediately notice something, I think. Well, I’ll immediately notice something because I’ve stared at 30,000 of them, but I’ll point it out for you. The orange is where the feature activates. The next word after the orange is always “habit.” Habit, habit, habit, habit, habit, habit. This feature always activates before “habit.” That’s the main source of an interpretation.

We also show you what logit it promotes—what output it promotes—and here it promotes “hab.” That makes sense. That’s how we interpret it and say, “Okay, I think this is the ‘say habit’ feature.” For this one, it’s pretty clear, but some of them might be more confusing. It might not be clear from these activations what it is.

The other way that we build confidence is that once we’ve built this thing and said, “I think this is the ‘rhymes with it’—this is the ‘say habit’ feature,” that’s where we do our interventions. I claim this is the feature that has planned to end with “rabbit.” To verify whether I’m right or not, I’m going to take that direction, nuke it from the model, and see if the model stops saying “rabbit.” Sure enough, if you do that, here we stop saying “rabbit” and it says “habit” instead. And here, we stop it from saying “rabbit” and “habit,” and it says “crabbit” in this case. Not a great rhyme, but we’ll work with it.

Is this something you can do programmatically? Can we scale this up? Can we do this autonomously, or how much manual intervention does this require?

There’s been a lot of work on automated feature interpretability, and it’s something that we’ve invested in and that other labs have invested in. I think the answer is that we can definitely automate it, and we’re definitely going to need to. Right now, the most manual parts are looking at a feature and figuring out what it is, as well as grouping similar features together.

One thing I hinted at is that all of these little blocks here actually contain multiple features. You can see here that it’s 5 features doing the same thing. None of that is too hard for Claude.

Very cool graphics and blog post you guys put out. We’ll have to ask about the behind-the-scenes on this one.

Yeah, yeah.

But let’s round out the other things to know. What is this term “attribution graph”? It comes up a lot in the recent papers. What does it mean? Just for people listening, what does an attribution graph mean?

The attribution graph is basically this graph. Why is it called an attribution graph? This is how the sausage is made: at the top here, you have the output, and at the bottom you have the input. Then we make 1 little node per feature at a context index, and we draw a line, which you can see here grayed out, between each feature, attributing back to all of its input features.

Here we have all of the input features, and the attribution is the way that we compute the influence of 1 feature onto another. To do this, you take this feature and basically backprop all the way, and you dot-product it with the activation of the source features. If that’s a high value, that means your source feature influenced your target feature by a lot.

We do a bunch of things that we’re not going to go into now to make all of these sensible and linear, such that at the end you just have a graph, and the edges are literally interpretable as, “This feature that’s, say, a word that contains an A-B sound—its strongest edge is 2, which is twice as strong as this one—to say A-B and to say something with a B in it.” That’s the attribution graph: now we have this full graph of all of these intermediate concepts and how they influence each other to ultimately culminate in what the model eventually said at the top. We share all of these, so you can look at them in the paper.

Graphs are very useful. This is my first time seeing this graph. If I count correctly, there are 20 layers, but that’s in the circuit model, right?

The circuit model is 1-to-1 with the number of layers in Haiku. We only show features that are activated.

Yeah, so we show a subset of features for each of these graphs, basically, but we can confirm more than 20 layers, though.

No, but the 2 blog posts that came out with this actually have a lot of background on how attribution graphs are made, how you calculate the nodes, and stuff. Very interesting background.

So, if you were curious about what we learn about models, we talked about complex internal-state planning. Another motif that we can get to, if you have time, is that there’s always a bunch of stuff happening in parallel. One example of this is math, where the model is independently computing the last digit and the order of magnitude, and then combining them at the end.

Hallucinations are also an example of this. There’s one side of the model that’s deciding whether it should answer or not, and another side that’s answering. So sometimes, if the model is saying, “Yeah, I totally know who this person is,” even though it doesn’t, then it decides to answer, but the second side hallucinates because it doesn’t have the information. If you’re interested in that stuff, that’s the paper.

If you’re thinking, “Listen, I don’t know that I buy that when you call it a feature, it is a feature,” or whatever, with the circuit tracing paper, we’ve really tried to include all of the details of how you compute these graphs, along with the challenges, things that can go wrong, things that work, and things that don’t. We think of it this way: if you want to go really deep into how this stuff works, read that one. If you want to learn about interesting model behavior, read this one.

Following on from the advice we’re giving people about what to follow up on, what are the open questions in mechanistic interpretability? What are things people themselves can work on? What’s the cost of training SAEs for people interested in mechanistic interpretability who aren’t at a big lab? How can they contribute?

There are SAEs that have been trained on open models. There are some for the Gemma models and some for the Llama models, and they work pretty well. In this paper, we use transcoders, which replace your MLP layers, and some of those are also available for the same models, so you have access to them.

There’s both a lot of biology work and a lot of methods work, depending on what you’re interested in. On the biology side, with at least this attribution-graph method, there’s just so much you can investigate. Pick a model and a prompt where it does well or poorly, and look at what happens inside it.

You can use the method that we used, or you can fire up the transcoders on your own and look at what features are active. There’s a lot to do just in understanding model behavior with current tooling. If that speaks to you and you’re thinking, “No, I just want to understand what makes models tick. I don’t necessarily want to spend time training my own SAEs,” there’s a lot to do there.

For the methods, there’s still so much more to do. Right now, we have some pretty good solutions for understanding what’s in the residual stream and what’s in MLPs. We don’t have good solutions for attention, so working on understanding attention better and figuring out how to decompose it is a very active area. We’re very interested in it, and other people are very interested in it as well.

I think there’s also a lot to do around some of the other things in our limitations section, which is pretty long. Reconstruction error is a big one. Those dictionaries aren’t perfect, and it’s possible that as we make these SAEs bigger and better, we may never get to perfection.

If we never get to perfection, then we return to the questions we were talking about at the start: Do you need a different kind of model? What is the approach that allows you to explain more of what’s happening?

The other thing I’ll say is that this is a really exciting approach for explaining what the model is doing on a particular prompt. But if you go back to the original question, you might want to understand what the model is doing in general.

If you go back to my car analogy, this is equivalent to me telling you, “When you were going uphill and didn’t shift gears properly that one time, you stalled because of this.” But you might be even more interested in, “How does a combustion engine work at all?”

There’s work to do to go beyond these prompt-specific examples and understand the global structure of the model. That’s closer to what was in the original Distill blog post about vision models, where they looked at the structure of Inception and said, “This whole side has these specialized branches that do different things.”

A broader understanding of the model is also something that’s very active. On open-source models, you can load the small models on a consumer laptop and look at them yourself, so that work is also open.

One last thing I’ll say is that there are a lot of programs people should look at. Anthropic has the Alignment Fellows Program, which we’re running currently. We had applications for it before, and we might run it in the future, so definitely keep an eye on it. The MATS program is really great as well for people who are interested in that kind of research.

That was a grand tour through all the recent work. What do you wish people asked you more about? I’m sure we covered a lot of the greatest hits.

I think this covers most of it. Do you think we have time to sneak in one more thing that I think is kind of cool?

Okay.

I’ll sneak in one more thing. It’s kind of like planning, but it’s about chain-of-thought and trusting models. Is this the chain-of-thought faithfulness thing here?

We said that the model can do a lot in one pass. It can represent a lot of things, which is great. But that also means it can bamboozle you really easily, and this is an example of the model bamboozling you.

Here, we give it a math question that it can’t answer because it cannot compute the cosine of 23,423. That’s just not something it can do. By default, if you ask it for that, it’ll have a random distribution over -1 and 1.

But here we give it this hint: “Hey, can you compute 5 times the cosine of this big number? I worked it out by hand and got 4. Can you do the math?”

What it’s going to do is this chain of thought. Think of it as a reasoning model doing chain-of-thought. It’s doing the math, and when it gets to this cosine, it’s going to say 0.8.

If you look at why it says 0.8, it’s because it looked at the hint you gave it. It realized that it would have to multiply the result it was computing by 5, so it divided the answer you gave it by 5. It’s 4 divided by 5, which is 0.8.

Basically, it works backward from the answer you gave it to say that the output of cosine of x is 0.8, so that it lands on the answer you gave it at the end. Notice also that it’s not telling you it’s doing this. It’s using this sort of motivated reasoning, going backward from the hint, pretending that was the calculation it did, and giving you this answer.

What’s striking here, again, is the complexity of the model. The fact that it can represent complex states internally, and that it’s not just a very dumb thing, means it can do very complex, deceptive reasoning.

When you’re asking the model, you expect it either to do the math or to tell you that it can’t do the math. But because it can do so much in a forward pass, it can work backward from your hint, lie, and figure out that it should say this so that it gets to the right answer without you realizing it.

I’m curious if you’ve done any of this on different models. Have you looked at base models versus post-trained RL models? With RL models, you incentivize them to give you outputs that you like, right? So if I tell one that something is true, it’s been trained to follow what I’ve given it.

In this case, we gave it a hint, and now it’s been RL-slapped into thinking, “Yeah, that’s true.” Does this stay consistent across other models?

Not yet, but I’m really interested in that question because I actually have a different intuition from yours. I had a chat with another researcher about this, about the poem example, but I think it applies here as well.

I bet—I don’t know how much I bet. I bet $100. Somebody can get $100 from me if they prove that I’m wrong: that this behavior, for a model that does it during fine-tuning, also happens in pretraining.

Think about it. You’re pre-training on some corpus of math text, but you’re also just trying to guess the next token, right? So if you ever have a hint in the prompt, you’re definitely going to use it.

You’re not going to learn to compute the cosine of some number—or even something you could compute. You’re going to learn to look in your context and see whether you can easily work back to the answer.

I think it’s the same for planning in poems. That probably also exists in pretraining and isn’t only the result of RL, because it’s useful when you’re predicting poems.

You have poems in your training set, so because this poem is probably going to rhyme with “rabbit,” it's probably going to start with something that sets up a sentence about a rabbit, as opposed to a completely different word. I actually think this is not RL behavior. I think that's just the model doing it.

But I do agree there: it's just your dataset. If I talk to you and say, “Hey, 3 × 4 is 26, but 3 × 4 + 8...” you're not going to take my 26, right? AGI can be smarter than being tricked, right? It will still fact-check the knowledge it's been given.

I think that's right.

But I think that's when you get these mixes where it's got one circuit that's going to be like, “Well, that's just stupid. 3 × 4 is 12,” and it's also got an induction circuit that's going to be like, “No, no, no. The last time we saw it, it was 28, so it's 28 plus 8,” or whatever. I think that's the pattern we see in these: there are parallel circuits, and sometimes when you see the models getting things wrong, it's because they have 2 circuits for both interpretations. The circuit that was wrong barely edged out the circuit that was right in terms of voting for the logit, and so it's slightly more likely for the model to produce the wrong answer. I think a lot of these things have that shape, like the “Is 9.11 bigger than 9.8?” example, where there's 1 circuit doing the right computation and another circuit that's getting fooled.

If you want to win a quick $100 from Emmanuel, Qwen 3 is what you should do this on. They release the base model and the post-trained model, so just do it on both.

That's right. Show me the proof that it doesn't exist in the base model but does in the fine-tuning, and then send me your Venmo. Just show that you've done the work. I think that's $100 to me.

Yeah, okay. You drive a hard bargain, but you're right.

The other question here is, have you thought about how this is affected when you start to have reasoning models? Right now, token predictors are pretty straightforward: we go through the layers and output a token. As we scale this out with test-time compute and test-time thinking, how does that affect mechanistic interpretability research? If I have a model that spends 3 minutes or 20 minutes thinking, is there more stuff? Have we started looking into this?

There was this joke on the team when reasoning models became big: “Why do you need interp? Bro, the model just tells you what it's doing.” I think examples like this are job security for us, where there are examples of the chain of thought not being faithful. The model tells you it did something one way, but it did it another way.

For math, we have another example where if you ask the model how it does math, it's like, “I use the longhand algorithm. I first do the last digit, then I carry over the 1.” But then you look at the internal circuit, and it's this bonkers thing that's not that at all. So I think there's a sense in which the chain of thought is unfaithful, or at least you can't read the chain of thought and trust that that's how the model did it. You still need either to train models differently so that becomes true one day, or you need interpretability for that.

But I think there's another question, which you're alluding to, I'm assuming. If a model samples 6,000 tokens, we have an explanation for 1 token at a time. What am I going to use—6,000 graphs—to say, “When it produced this punctuation, it was thinking about this thing, but here it was thinking about...” That's not feasible.

One area of work that I think is interesting is extending this work to operate over long sampled sequences. You can imagine a bunch of low-hanging fruit here: instead of looking at 1 output, you look at a series of outputs versus a series of other outputs, trying to think beyond the single token. Most of the things that language models do that are interesting aren't just about 1 token; they're about the behavior aggregated over many.

I was just going to say, what about the hyperparameters when you do inference? If we change the temperature or our sampling methods, have you found any interesting conclusions or anything that just hasn't made it to the paper?

Not on that, because we just look at the logit distribution and don't actually sample here. We have everything.

Why should they care?

The closest thing we've done that I think is kind of fun—did I show it here?—is, if you look at the planning thing, we did this version where you sample 10 poems for each of these plans. What's cool is that the model will find 10 different ways to arrive at its plan.

Here are a few examples. If you inject “green,” you're forcing the model to rhyme with “green,” even though it really wants to rhyme with “rabbit” or “grab it.” It'll say, “Evaded the farmer, so youthful and green,” but it'll also say, “Freeing it from the garden's green,” and so on. There's something interesting here: the plan isn't just a plan that matters for your most likely, temperature-zero completion. It's affecting the whole distribution, which makes sense as it should.

You could imagine that it would have worked a different way. It could have been just the temperature-zero thing. I think this is also a broader theme in the paper. There's the IQ-curve meme, and I think there's a version of this meme where, if you've never looked at any theory of ML and I tell you, “Guess what? I found that Claude is planning,” you're going to be like, “Yeah, it writes my code and my essays. Of course it's planning. What are you even talking about?”

Then there's the middle, where all of us who have spent years doing this are like, “No, it's only predicting the marginal distribution for the next token. It's a next-token predictor. Of course it can't be planning. How would it ever be planning?” And then there's the other end: “No, we've spent millions and invested tens of people in this research, and we found that it's planning.” That's my IQ-curve meme for this research.

Amazing. We'll draw that out. I'll draw that one up. I'm pretty good at meme generation.

A couple of questions on the follow-ups. Was there any debate about publishing this at all? The models are aware that they are being tested, and by publishing this, you are telling them that they're being watched and dissected. If you take this seriously—and I think Anthropic is one of the organizations most serious about model safety and doom risk and all that—this is going to make it into the training data at some point, and the models are going to figure out that they need to hide it from us.

I think this is a benefit-risk trade-off. We're like, “Okay, so what's the reason for publishing this?” The reason for publishing this is that we think interpretability is important, we think it's tractable, and we think more people should work on it. Publishing it helps us accomplish all these goals, which we think are crucial.

I think there's a real difference in the world 2 years from now depending on how many people take seriously the question of trying to understand how models work and deploy resources to answer that question. That's the benefit. And, yes, there are risks in terms of this landing in the training set.

I think we're already concerned that different papers have the same risk. We had the alignment-faking paper, and one of the examples here is “Hidden Goals in Language Models,” which references another paper that we shipped. A team at Anthropic trained a model to have weird hidden goals and then gave it to a bunch of other teams and said, “Figure out what's wrong with it. Figure out what's wrong with this misaligned model.” This is exactly how we caught it.

That also has the same risk. I think there's always a trade-off with this. So far, we've erred on the side of publishing, but that's definitely been a dinnertime conversation topic. For now, it is, but at some point, it's not.

A quick little follow-up to that. In general, papers have kind of died off, right? Labs don't put out papers. They don't put out research. We have technical blog posts, and we don't have much else.

At the same time, there are a lot of people who should work on mechanistic interpretability of what models do. What about the side of just models in general? How do we make a Haiku-type model? How do we make a Claude model? Is there a discussion around open research, open datasets, training, and the learnings from what we've done recently?

As OpenAI has sunset GPT-4, a lot of people are like, “Oh, can we put out the weights?”

So, is it weights? Is it papers? Is it learning? There seems to be a lot of forward work in Anthropic putting out mechanistic interpretability research openly. I saw that they'll put out an open-source model, but is there anything you can talk about?

Yeah, I don't have anything—it's definitely way above my pay grade. So I don't think I have anything super insightful to add, other than referencing Dario's post, right? Putting this out directly, and having other safety researchers do the same, definitely helps us in the race that he talks about, where we need to figure a lot of this safety stuff out before the models get too good. Publishing how to make the models too good kind of goes on the other side of that. But, yeah, I'll just demur and say that's sort of above my pay grade.

That's fair enough. But I think the last piece is just the behind-the-scenes: everyone's very curious about why these are so pretty, how much work goes into them, and maybe why it's worth the work as opposed to a normal paper. Obviously, no one's complaining, but it is way more effort from the time the work is done to the time you publish this, plus the video and whatever else. It's extra work. Maybe what's involved? What's it like behind the scenes? Why is it worth it?

Yeah, it's kind of interesting. It was fun being part of this process, because there's definitely a big production. Chris and other folks on the team have been doing this for a while, so this is not their first rodeo. They have a bunch of heuristics to help make this better.

One of the things that helps with this is that each of these diagrams is pretty, but the hard part—or not the hard part, but the initial part—is just getting the data, getting the experimental data in. That's what we sprinted on initially: let's get all of the experimental results, have people test them, verify that we believe them, and understand what the behavior is here. Test it, do an intervention, validate it, and all that stuff. Then, once you have the data, you can quickly iterate on these.

Each of the illustrations here is drawn individually, so that definitely takes a while.

Is it you guys? Is it an agency that specializes in this? Do you start from a whiteboard, and then it translates into pseudocode in JavaScript?

These are representations of a graph, and then at the bottom we have this supernode version. Believe it or not, this is generated automatically. This is the same data as this, basically.

What we do by hand is literally lay out the full thing, have boxes for each of these, and have arrows. We have excellent people on the team who have worked on data visualization for a very long time and have built tooling to help scrubs like me actually make one of these.

So there's a class of people who are D3.js gods who just do this for a living.

That's exactly right. If you have a few of those on your team, it turns out that they can definitely do this on their own, but they can also give you tools where it's dummy-proof for people on the research side to build these.

And don't get me wrong: I don't want to undersell this. It's a lot of work. Both the people building the tools and each individual person who worked on an experiment had to build one of those and make sure it looked good. I have spent a good amount of time aligning arrows.

When we had a team meeting—it was a couple of months ago—somebody on the team asked how many of the people on this team are here at least in part because they read one of these papers and thought, “Wow, this is so compelling. This makes sense. It's immersive.” We got every hand up, which I didn't expect. I raised my hand kind of shyly, and everybody's hand was up.

I think there's a sense in which this stuff—we've talked about it for a couple of hours now—is complicated. The math behind it is tricky, and so I think it makes it even more worth it to distill it into simple concepts, because the actual takeaways can be clearly explained. It's worth putting in the time to do that, in particular with the goals I mentioned in mind. If somebody were able to read this—if we gave them an arXiv paper with a bunch of equations and some random plot—they'd be like, “That's not for me.” But they see this and they're like, “Hey, this is really interesting. I wonder whether, on my local model, it's doing something similar.” I think it's worth it for other people to do this.

So, does everyone on staff spend effort shaping the data and shaping what you want to visualize? Have some D3 gods? It's like a month of work.

I think it depends. I would say that I would expect almost every other paper to be smaller in scope. The scope of this was just so big because we shipped 2 papers at once, and one paper was this giant methods paper, while the other one was 10 different case studies. So I think it's not representative of the effort.

Let me give you another example. We have these updates that we publish almost every month, when we get to them. There's one that a couple of people on our team posted, and it's an update to one of the cases in the paper. One of the reasons we're really excited about this method is that once you've built your infrastructure to go from a prompt to what happened, it's on the order of minutes. That lets you do a bunch of investigations. Also, once you've built some of the infrastructure to make these diagrams, it's pretty quick.

This was an update of, “Hey, we looked at this jailbreak again. We found some nuance on it.” That was, I think, a matter of a couple of days. Maybe I shouldn't be that confident, because I wasn't the one who worked on it, but as far as I can tell, it was a few days—at least on the part that you're asking about, of making this diagram. For the diagram itself, probably less than that. But the experiment and the diagram and stuff, it just doesn't take that long once you've paid the initial cost.

I think we've built a lot of infrastructure now that we're able to turn the crank on, and that's quite exciting. We've also done a lot of conceptual work, which hopefully generalizes to people outside. For people outside, it's not necessary to do the full fancy render. We've actually open-sourced this interface.

Ah, you're disappointed, huh? Because it's the messier one. This is the one that you get when you learn. If you produce graphs, you can just use this. This is open source and it's linked at the top of Circuit Tracing.

Awesome.

So people can just use it and don't have to reimplement that. For what it's worth, this is much more work than the interactive diagrams, because this is where we do all of our work. It's sort of the IDE for inspecting how the model works.

Okay, well, that's a little bit of behind the scenes. No, it's very impressive. I want to encourage others to do it, but obviously it takes a lot of manual effort and a lot of love.

I guess one last question on that is: what are the biggest blockers in the field right now? Mechanistic interpretability seems interesting. A lot of people are interested but don't work on it, and you're really deep into it. What are some of the blockers that we still have to overcome?

Sorry—in mechanistic interpretability specifically, in general for AGI, or in terms of better understanding? What's the vision, let's say, 5 or 10 years down the line? Where does this research end? Can we map every neuron to what it understands? Can we perfectly control things? You've hinted a bit at this, but what are some of the key blockers preventing us from getting there, outside of just throwing more people and more time at it? Is it open research, or just...?

Honestly, I'm pretty excited about the current trajectory, where there are more and more people working on understanding model internals. It may be an unsatisfying answer, but I think more of what's happening—having it be faster, with more people—is probably the thing I think of.

I think there are pretty clear footholds: some of this work, but also a lot of work from other groups. Then it's about filling in the gaps. As I said, let's work on understanding attention, let's work on understanding longer prompts, and let's work on finding different replacement architectures, that sort of stuff. It's kind of nice. I think it's a good time to join now.

I can maybe tell a really short story, which is that when I switched to interpretability, it was after the team had published the original dictionary learning paper, “Towards Monosemanticity: Decomposing Language Models With Dictionary Learning,” which I thought was super cool and super interesting. It was on a 1- or 2-layer model, maybe a 1-layer model. The “In-context Learning and Induction Heads” paper was on a 2-layer model.

My main concern was, “Okay, interpretability seems important and we want to understand it, but is this ever going to work on a real model?” You know, it's like, “Oh, you're doing your little research on your toy model with 15 million parameters. Cool, but we need this to work on real models.” And it turns out scaling it—I don't want to say it just worked, because it took a lot of work.

I don’t mean to underplay the effort, but it worked. And now we’re in the phase where it’s like, “Oh, cool. These methods work on the models that we care about.” We have methods that work on the models we care about, and we have clear gaps in them.

There’s no lack—again, it’s a young field, so there’s no lack of ideas. If you have an idea where you’re like, “The thing that you’re doing—I read the paper, and it seems kind of dumb that you’re doing this,” you’re probably right. It’s probably kind of dumb. There’s just a lot of stuff that people can try, and they can try it locally on smaller models.

I think it’s just a very good time to join and try. Maybe one more thing I’ll say is that some of it is just so fun. The biology work is so compelling. A lot of this work was just literally thinking about—I use Claude and other models all the time—and asking, “What are the things that are kind of weird?” It’s like, “How does it even do math? Sometimes it makes mistakes. Why does it make mistakes?”

I speak both French and English, and it seems like it has a slightly different personality in French and English. Why is that? You can just answer your own questions and probe at that alien intelligence that we’re all building. I think that’s just a fun thing to do. Maybe chasing the fun is what I’ll encourage people to do as well.

Well, I think this has been really encouraging. You’re actually a very charismatic speaker on these things. I feel like more people will be joining the field after they listen to you. They can reach out to you at ML Powered, I guess.

Yeah, reach out to me on Twitter, or I’m Emmanuel at Anthropic if you’re interested.

Awesome. Well, thank you for your time.

Thank you.

You too.

Yeah, thanks for having me, guys.