Goodfire AI 的押注：可解释性将成为模型设计的下一前沿——Myra Deng 与 Mark Bissell

If you ask 50 people who, quote-unquote, work in interpretability what interpretability is, you'll probably get 50 different answers, to some extent. Also, where Goodfire sits in the space: we're an AI research company above all else, and interpretability is a set of methods that we think are really useful and worth specializing in to accomplish the goals we want to accomplish. But I think we also see some of the goals as even broader, almost like the science of deep learning, and just taking a not-black-box approach to internal representations, and then bringing interpretability to training, which I don't think has been done all that much before.

swyx

So welcome to Latent Space. We're back in the studio with our special mechanistic-interpretability co-host, Vivu. Welcome, Mochi, my special co-host—the mechanistic-interpretability doggo. We have with us Mark and Myra from Goodfire. Welcome.

Thanks for having us on.

swyx

Maybe we can introduce Goodfire and then introduce you both. How do you introduce Goodfire today?

Yeah, it's a great question. Goodfire, we like to say, is an AI research lab that focuses on using interpretability to understand, learn from, and design AI models. We really believe that interpretability will unlock the next generation—the next frontier—of safe and powerful AI models. That's our description right now, and I'm excited to dive more into the work we're doing to make that happen.

Yeah, and there's always the official description. Is there an unofficial one that resonates more with a different audience? Well, being an AI research lab that's focused on interpretability, obviously, a lot of people have a lot of thoughts about what interpretability means. I think we have a pretty broad definition of what that means and the types of places it can be applied, in particular applying it in production scenarios in high-stakes industries and really taking it from the research world into the real world. It's a new field, so that hasn't been done all that much, and we're excited about seeing it put into practice.

swyx

Yeah, I would say it wasn't too long ago that the topic was still putting out toy models or supervision and that kind of stuff. I wouldn't have pegged it to be this far along. When you and I talked at NeurIPS, you were talking a little bit about your production use cases and your customers. Not to bury the lead, today we're also announcing the fundraise: your Series B, $150 million at a $1.25 billion valuation. Congrats. You're a unicorn.

Thank you. Yeah, things move fast. We were talking to you in December, and there have already been some big updates since then.

swyx

Let's dive into a bit of your backgrounds as well. Mark, you were at Palantir, working on health stuff, which is really interesting because Goodfire has some interesting health use cases. I don't know how related they are in practice.

Yeah, not super related, but it was helpful context to know what it's like to work with health systems and generally in that domain.

swyx

And Mara, you were at Two Sigma, which I was also at—Two Sigma, really back in the day.

Wow, nice. Did we overlap at all?

swyx

No. This was when I was briefly a software engineer before I became a developer-relations person, and now you're head of product.

What are your respective roles, just to introduce people to what all gets done at Goodfire?

Prior to Goodfire, I was at Palantir for about 3 years as a forward-deployed engineer—a hot term now, though it wasn't always that way—and as a technical lead on the healthcare team. At Goodfire, I'm a member of the technical staff. Honestly, that's about as specific as I could describe myself, because I've worked on a range of things, and it's a fun time to be at a team that's still reasonably small.

I think when I joined, I was one of the first 10 employees. Now we're above 40, but it still looks like there's always a mix of research, engineering, product, and all of the above that needs to get done. Everyone across the team is a pretty good switch-hitter in the roles they do.

You've seen some of the stuff that I worked on related to image models, which was sort of a research demo. More recently, I've been working on our scientific discovery team with some of our life-sciences partners, but I've also been building out our core platform, flexing some of the MLE and developer skills as well.

swyx

Very generalist. You also had a very founding-engineer-type role.

Yeah, I also started—and still am—a member of the technical staff. I did a wide range of things from the very beginning, including finding our office space and all of the nitty-gritty. People visited when you had that open-house thing. It was really nice.

swyx

It was like 200 people. It has room for 200 people, but there you guys were—like 10.

Yeah. For a while it was very empty. But I spend a lot of my time as head of product. I think product is a bit of a weird role these days, but a lot of it is thinking about how we take our frontier research and really apply it to the most important real-world problems, and how that translates into a platform that's repeatable or a product.

I work across the engineering and research teams to make that happen, and I also communicate to the world: What is interpretability? What is it used for? What is it good for? Why is it so important? All of these things are part of my day-to-day as well.

swyx

I love the “what is” thing, because that's a very crisp starting point for people coming to a field. Alessio, I'll do a fun thing for you: why don't you try tackling what interpretability is, and then they can correct us?

Alessio Fanelli

Okay, great. To kick off, it's a very interesting role to be head of product, right? Because you guys, at least as a lab, are more of an applied interpretability lab, which is pretty different from just normal interpretability. There's a lot of background research, but you guys actually ship an API to try these things. You have Ember, and you have products around it, which not many do.

Okay, what is interpretability? Basically, you're trying to understand what's going on in a model—in the model's internals. There are different approaches to doing that: you can do probing, SAEs, transcoders, all this stuff. But basically, you have a hypothesis. You have something that you want to learn about what's happening in a model's internals, and then you're trying to solve that.

From there, you can do things like activation mapping and steering. There's a lot of stuff that you can do. But the key question is: from input to output, we want to have a better understanding of what's happening and how we can adjust what's happening in the model's internals. How'd I do?

That was really good. I think that was great. I think it's also a minefield: if you ask 50 people who, quote-unquote, work in interpretability what interpretability is, you'll probably get 50 different answers.

To some extent, that also describes where Goodfire sits in the space. I think we're an AI research company above all else, and interpretability is a set of methods that we think are really useful and worth specializing in to accomplish the goals we want to accomplish. But I think we also see some of the goals as even broader—as almost like the science of deep learning—and just taking a not-black-box approach to any part of the AI development life cycle.

That could mean using interpretability for data curation while you're training your model, for understanding what happened during post-training, or for understanding activations and internal representations—what is in there semantically. There are also a lot of exciting updates around bringing interpretability to training, which I don't think has been done all that much before. A lot of the work is post hoc, poking at models, as opposed to actually using this to intentionally design them.

swyx

Is this focused on post-training or pre-training?

It's focused on post-training, but there's no reason the techniques wouldn't also work in pre-training.

swyx

It seems like it would be more applicable post-training because I'm thinking about rollouts or having different variations of a model that you can tweak with.

Steering. Yeah. And I think in a lot of the news that you've seen on Twitter or whatever, you've seen a lot of unintended side effects come out of post-training processes: overly sycophantic models or models that exhibit strange reward-hacking behavior.

I think these are extreme examples. There are also more mundane enterprise use cases where they try to customize or post-train a model to do something, and it learns some noise or doesn't appropriately learn the target task. A big question that we've always had is: how do you use your understanding of what the model knows and what it's doing to actually guide the learning process more effectively?

swyx

Yeah. I mean, just to anchor this for people, one of the biggest controversies of last year was GPT-4o GhibliGate.

The First Mechanistic Interpretability Frontier Lab

[Laughter] I've never heard that. I didn't know that was what it was called.

swyx

No, the other one—they called it that in the blog post, and I was like, why? OpenAI officially used that term, and I'm like, that's funny. But, yeah, I guess the pitch is that if they had worked with Goodfire, they would've avoided it, you know? [Laughter]

The First Mechanistic Interpretability Frontier Lab

I think so.

Alessio Fanelli

Yeah.

The First Mechanistic Interpretability Frontier Lab

I think that's certainly one of the use cases. Another reason why post-training is a place where this makes a lot of sense is that a lot of what we're talking about is surgical edits. You want to be able to have expert feedback very surgically change how your model is doing, whether that is removing a certain behavior that it has.

One of the things we've been looking at, another common area where you would want to make a somewhat surgical edit, is some of the models that have, say, political bias. You look at Qwen or R1, and they have this sort of CCP bias in them.

swyx

Is there a CCP vector?

The First Mechanistic Interpretability Frontier Lab

There are certainly internal parts of the representation space where you can sort of see where that lives. You want to extract that piece out.

swyx

Well, I always say, whenever you find a vector, a fun exercise is just to make it very negative to see what the opposite of CCP is. [Laughter]

The First Mechanistic Interpretability Frontier Lab

The super-American bald eagles flying everywhere. [Laughter]

In general, there are lots of post-training tasks where you'd want to be able to do that, whether it's unlearning a certain behavior or some of the other cases where this comes up. Are you familiar with the grokking behavior?

Alessio Fanelli

I mean, I know the machine-learning term grokking.

The First Mechanistic Interpretability Frontier Lab

It's sort of this double-descent idea of having a model that is able to learn a generalizing solution. Even if memorization of some task would suffice, you want it to learn the more general way of doing a thing. Another way that you can think about having surgical access to a model's internals would be: learn from this data, but learn in the right way. If there are many possible ways to do that—

swyx

Can mechanistic interpretability solve the double-descent problem?

The First Mechanistic Interpretability Frontier Lab

It depends, I guess, on how you—

swyx

Okay, so I view double descent as a problem, because then you're like, well, if the loss curves level out, then you're done, but maybe you're not done.

Alessio Fanelli

Right.

swyx

But if you actually can interpret what is generalizing, or what is still changing even though the loss is not changing, then maybe you can actually not view it as a double-descent problem. You're just sort of translating the space in which you view loss, and then you have a smooth curve.

The First Mechanistic Interpretability Frontier Lab

Yeah, [laughter] I think that's certainly the domain of problems that we're looking to get at. To me, double descent is the biggest thing in ML research. If you believe in scaling, then you need to know where to scale. But if you believe in double descent, then you don't believe in anything where things level off.

swyx

Yeah, I mean, also tangentially, when you talk about the China vector, there's the subliminal learning work from the Anthropic Fellows Program, where basically you can have hidden biases in a model, and as you distill down, or as you train on distilled data, those biases always show up, even if you explicitly try not to train on them. It's just another use case of: if we can interpret what's happening in post-training, can we clear some of this? Can we even determine what's there? Because it's just some worrying research out there that shows we really don't know what's going on.

The First Mechanistic Interpretability Frontier Lab

That is, yeah, I think, the biggest sentiment that we're hoping to tackle. Nobody knows what's going on, right? Subliminal learning is just an insane concept when you think about it. Train a model on not even the logits—the literal output text of a bunch of random numbers—and now your model loves owls. You see behaviors like that that just defy intuition, and there are mathematical explanations that you can get into, but—

Alessio Fanelli

I mean, early days.

The First Mechanistic Interpretability Frontier Lab

Objectively, there is a sequence of numbers that are more owl-like than others. There should be.

swyx

According to certain models, right? It's interesting. I think it only applies to models that were initialized from the same starting point.

The First Mechanistic Interpretability Frontier Lab

Usually, yes. But I think that's a cheat code because there's not enough compute. If you believe in a Platonic representation, it probably will transfer across different models.

swyx

Oh, you think so? I think of it more as a statistical artifact of models initialized from the same seed. There's something path-dependent from that seed that might cause certain overlaps in the latent space, and then this distillation sort of pushes it toward having certain other tendencies.

The First Mechanistic Interpretability Frontier Lab

Got it.

swyx

A bunch of these open-ended questions, right? You can't train in new stuff during the RL phase, right? RL only reorganizes weights, and you can only do stuff that's somewhat there in your base model. You're not learning new stuff. You're just reordering chains and stuff.

My broader question is: when you guys work at an interpretability lab, how do you decide what to work on, and what's the thought process? We can ramble for hours—“I want to know this, I want to know that”—but how do you concretely determine what's the workflow? There are approaches toward solving a problem, right? I can try prompting. I can look at chain of thought. I can train probes or SAEs. But how do you determine whether it's going anywhere? Do we have set criteria? If you can talk about that.

The First Mechanistic Interpretability Frontier Lab

It's a really good question. At the very beginning of the company, we've always thought about going and trying to learn what isn't working in machine learning today, whether that's talking to customers or talking to researchers at other labs, trying to understand both where the frontier is going and where things are really falling apart today, and then developing a perspective on how we can push the frontier using interpretability methods.

Even our chief scientist, Tom, spends a lot of time talking to customers and trying to understand what real-world problems are, then taking that back and trying to apply the current state of the art in interpretability to those problems and seeing where it falls down, basically. Then we use those failures or shortcomings to understand what hills to climb when it comes to interpretability research.

On the fundamental side, for instance, when we've done some work applying SAEs and probes, we've encountered some shortcomings in SAEs that we found a little surprising. So we've gone back to the drawing board and done work on better foundational interpretability models, and a lot of our team's research is focused on what the next evolution beyond SAEs might be.

When it comes to the control and design of models, we tried steering with our first API and realized that it still fell short of black-box techniques like prompting or fine-tuning. So we went back to the drawing board and asked, “How do we make that not the case? How do we improve it beyond that?” One of our researchers, ECDE [?], who just joined—actually, ECDE [?] and Atticus are steering experts and have spent a lot of time trying to figure out what research enables us to do this in a much more powerful, robust way.

The answer is: look at real-world problems, try to translate that into a research agenda, and then hill-climb on both of those at the same time.

swyx

Yeah, Mark has the steering CLI demo queued up, which we're going to go into in a second. But I always want to double-click on things like when you drop hints that you found some problems with SAEs. What are they? Then we can go into the demo.

The First Mechanistic Interpretability Frontier Lab

I'm curious if you have more thoughts here as well, because you've done it in the healthcare domain. But I think, for instance, when we do things like trying to detect behaviors within models that are harmful, or behaviors that a user might not want to have in their model—hallucinations, harmful intent, PII, all of these things—we first tried using SAE probes for a lot of these tasks. We took the feature activation space from SAEs, trained classifiers on top of that, and then saw how well we could detect the properties that we might want to detect in model behavior.

We've seen in many cases that probes trained on raw activations seem to perform better than SAE probes, which is a bit surprising if you think that SAEs are also capturing the concepts that you would want to capture cleanly and more surgically. That's an interesting observation. I don't think that means I'm down on SAEs at all. I think there are many things they're useful for. But we've definitely run into cases where I think the concept space described by SAEs is not as clean and accurate as we would expect it to be for actual real-world downstream performance metrics.

swyx

Fair enough.

Alessio Fanelli

Yeah.

The First Mechanistic Interpretability Frontier Lab

Yeah, it's the blessing and the curse of unsupervised methods: you get to peek into the AI's mind, but sometimes you wish you saw other things when you looked inside there. [Laughter]

swyx

Although, in the PII instance, I think an SAE-based approach actually proved to be the most generalizable, didn't it?

The First Mechanistic Interpretability Frontier Lab

Well, in the case that we published with Rakuten, I think a lot of the reasons it worked well were because we had a noisier dataset. And so, actually, the blessing of unsupervised learning is that we got more meaningful, generalizable signal from SAEs when the data was noisy. But in other cases where we've had good datasets, it hasn't been the case.

swyx

And just because you named Rakuten, and I don't know if we'll get another chance, what is Rakuten's overall usage or production usage?

The First Mechanistic Interpretability Frontier Lab

Yeah. So they are using us to essentially guardrail and monitor, at inference time, their language model usage and their agent usage to detect things like PII, so that they don't route private user information to downstream model providers. And so that's going through all of their user queries every day. That's something that we deployed with them a few months ago. And now we are actually exploring very early partnerships, not just with Rakuten but with other people, around how we can help with potentially training and customization use cases as well.

swyx

Yeah. And for those who don't know, Rakuten is, I think, the number 1 or number 2 e-commerce store in Japan.

swyx

Yes.

The First Mechanistic Interpretability Frontier Lab

Yeah, yeah. And I think that use case actually highlights a lot of what it looks like to deploy things in practice that you don't always think about when you're doing research tasks. When you think about some of the stuff that came up there that's more complex than your idealized version of a problem, they were encountering things like synthetic-to-real transfer of methods. So they couldn't train probes or classifiers on actual customer PII data.

We could evaluate performance on the real datasets, but we couldn't train on customer PII. You have multilingual requirements, so this needed to work for both English and Japanese text. Japanese text has all sorts of quirks, including tokenization behaviors that caused lots of bugs and had us pulling our hair out.

And then also, in a lot of tasks, you might make simplifying assumptions if you're treating it as the easiest version of the problem to get general results. Maybe you say you're classifying a sentence to ask, “Does this contain PII?” But the need Rakuten had was token-level classification, so that you could precisely scrub out the PII.

As we learned more about the problem and what that looks like in practice, a lot of assumptions ended up breaking. That was just one instance where a problem that seems simple right off the bat ends up being more complex as you keep diving into it.

swyx

Excellent. One of the things that's also interesting with interpretability is that a lot of these methods are very efficient, right? You're just looking at a model's internals itself compared to a separate guardrail LLM-as-a-judge, a separate model. One, you have to host it; two, there's a whole latency cost. So if you use a big model, you have a second call.

Some of the work around self-detection of hallucination is also deployed for efficiency, right? So, thinking of someone like Rakuten doing it live in production, that's just another thing people should consider.

The First Mechanistic Interpretability Frontier Lab

Yeah. And something like a probe is super lightweight. It adds no extra latency, really.

swyx

Excellent. You have the steering demos lined up, so we're just going to see what you've got. I don't actually know if this is the latest latest or an alpha thing.

The First Mechanistic Interpretability Frontier Lab

No, this is a pretty hacky demo from a presentation that someone else on the team recently gave. So this will give a sense for steering in action. Honestly, I think the biggest thing that this highlights is that as we've been growing as a company and taking on more and more ambitious versions of interpretability-related problems, a lot of that comes down to scaling up in various different forms.

Here you're going to see steering on a 1 trillion-parameter model. This is Kimi K2. It's sort of fun that, in addition to the research challenges, there are engineering challenges that we're now tackling, because for any of this to be useful in production, you need to be thinking about what it looks like when you're using these methods on frontier models, as opposed to toy model organisms.

This was thrown together hastily, pretty fragile behind the scenes, but I think it's quite a fun demo. Screen sharing is on. I've got 2 terminal sessions pulled up here. On the left is a forked version that we have of the Kimi CLI, which is running to point at our custom-hosted Kimi model.

On the right is a setup that will allow us to steer on certain concepts. I should be able to chat with Kimi over here. I'll tell it, “Hello.”

swyx

Is this running locally?

The First Mechanistic Interpretability Frontier Lab

The CLI is running locally, but the Kimi server is running back at the office. Well, hopefully it should be. [laughter]

swyx

That's too much to run on that Mac.

The First Mechanistic Interpretability Frontier Lab

Yeah, I think it takes a full H100 node. I think you can run it on 8 H100 GPUs. So, yeah, Kimi's running. We can ask it to prompt. It's got a forked version of our SGLang codebase that we've been working on.

I'm going to tell it, “Hey, this SGLang codebase is slow. I think there's a bug. Can you try to figure it out?” It's a big codebase, so it'll spend some time doing this. And then on the right here, I'm going to initialize some steering in real time. Let's see here. “Continue searching for any bugs.”

“Feature ID 43,205, layers 20, 30, 40.”

This is basically a feature that we found that, inside Kimi, seems to cause it to speak in Gen Z slang. [laughter] On the left, it's still sort of thinking normally. It might take, I don't know, 15 seconds for this to kick in, but then we're going to start hopefully seeing it. [clears throat]

“Dude, this codebase is massive for real.” [laughter]

We're going to start seeing Kimi transition, as the steering kicks in, from normal Kimi to Gen Z Kimi, both in its chain of thought and its actual outputs. And interestingly, you can see it's still able to call tools and stuff. It's purely its demeanor.

There are other features that we found for interesting things like concision. That's more of a practical one: you can make it more concise. The types of programs, programming languages it uses. But, yeah, as we're seeing it come in, pretty good output.

swyx

Scheduler code is actually wild. [laughter] Something about how agents for interpretability is different than coding agents. I don't know. While this is spewing up, how do we find feature 43,205?

The First Mechanistic Interpretability Frontier Lab

Yeah. So, in this case, our platform, which we've been building out for a long time now, supports all the classic, out-of-the-box interpretability techniques that you might want to have, like SAE training, probing, and things of that kind.

I'd say the techniques for vanilla SAEs are pretty well established now. You take your model that you're interpreting, run a whole bunch of data through it, gather activations, and then you have a pretty straightforward pipeline to train an SAE. There are a lot of different varieties: TopK, BatchTopK, and normal ReLU SAEs.

Once you have your sparse features, assigning labels to them to actually understand that this is the Gen Z feature is where a lot of the magic happens. The most basic, standard technique is to look at all of your input dataset examples that cause this feature to fire most highly, and then you can usually pick out a pattern.

For this feature, if I've run a diverse enough dataset through my model, feature 43,205 probably tends to fire on all the tokens that sound like Gen Z slang. You could have a human go through all 43,000 concepts and look at the pattern, but to automate that, you just hand those examples off to a frontier LLM and ask it to identify that pattern.

swyx

And I've got to ask the basic question: can we get examples where it hallucinates, pass them through, see what feature activates for hallucinations? Can I just turn hallucination down?

The First Mechanistic Interpretability Frontier Lab

Oh, wow. You really solved it.

swyx

You really predicted a project we're already working on right now, which is detecting hallucinations using interpretability techniques. This is interesting because hallucinations are something that's very hard to detect. It's a kind of hairy problem, and it's something that black-box methods really struggle with.

Whereas with Gen Z, you could always train a simple classifier to detect that. Hallucinations are harder. But we've seen that models internally have some awareness of uncertainty or some sort of user-pleasing behavior that leads to hallucinatory behavior.

So, yeah, we have a project that's trying to detect that accurately and then also working on mitigating the hallucinatory behavior in the model itself as well.

swyx

Yeah. And I would say most people are still at the level of, “Oh, I'll just turn the temperature to 0, and that turns off hallucination.” And I'm like, well, that's a fundamental misunderstanding of how this works.

The First Mechanistic Interpretability Frontier Lab

Yeah. Although, part of what I like about that question is that there are SAE-based approaches that might help you get at that. But oftentimes, the beauty of SAEs—and, like we said, the curse—is that they're unsupervised.

So when you have a behavior that you deliberately would like to remove, and that's more of a supervised task, often it is better to use something like probes and specifically target the thing that you're interested in reducing, as opposed to hoping that when you fragment the latent space, one of the vectors that pops out will be the thing you're interested in. As much as we're training an autoencoder to be sparse, we're not for sure certain that we will get something that just correlates to hallucination, right? You'll probably split that up into 20 other things, and who knows what they'll be.

swyx

Of course, right? There are known problems with feature splitting and feature absorption, and then there are the off-target effects. Ideally, you would want to be very precise, where if you reduce the hallucination feature, suddenly maybe your model can't write creatively anymore. Maybe you don't like that, but you still want to stop it from hallucinating facts and figures.

Good. So, there's a paper to recommend there that we'll put in the show notes. But I guess, just because your demo is done, any other things that you want to highlight or any other interesting features you want to show?

The First Mechanistic Interpretability Frontier Lab

I don't think so. Like I said, this is a pretty small snippet. I think the main point here that I think is exciting is that there isn't a whole lot of mechanistic interpretability being applied to models quite at this scale. Anthropic certainly has some research, and other teams as well. But it's nice to see these techniques being put into practice. I think not that long ago, the idea of real-time steering of a trillion-parameter model would have sounded...

swyx

Yeah, the fact that it's real-time—you started the thing, and then you edited the steering vector. I think it's an interesting one. TBD what the actual production use case would be on that, the real-time editing.

The First Mechanistic Interpretability Frontier Lab

That's the fun part of the demo, right? You can kind of see how this could be served behind an API. You only have so many knobs, and you can just tweak it a bit more. I don't know how it plays in—people haven't done that much with how this works with or without prompting, or how this works with fine-tuning. There's the whole hype of continual learning, so there's just so much to see. Is this another parameter? Is it a parameter we just leave as a default and don't use? I don't know. Maybe someone here wants to put out a guide on how to use this with prompting, when to do what.

Oh, well, I have a paper recommendation that I think you would love from Ekdeep on our team, who is an amazing researcher. I just can't say enough amazing things about Ekdeep. He actually has a paper, as well as some others from the team and elsewhere, that goes into the essential equivalence of activation steering and in-context learning. He thinks of everything in a cognitive-neuroscience, Bayesian framework, but basically, you can precisely show how prompting, in-context learning, and steering exhibit similar behaviors, and even get quantitative about the magnitude of steering you would need to induce a certain amount of behavior similar to certain prompting, even for things like jailbreaks. It's a really cool paper.

swyx

Are you saying steering is less powerful than prompting? More like, you can almost write a formula that tells you how to convert between the 2 of them, and so—

The First Mechanistic Interpretability Frontier Lab

They'd be formally equivalent, actually, in the limit.

swyx

Right. So one case study of this is jailbreaks. I don't know if you've seen the stuff where you can do many-shot jailbreaking, where you flood the context with examples of the behavior.

The First Mechanistic Interpretability Frontier Lab

When they put out that paper, a lot of people were like, "Yeah, we've been doing this, guys. What happens?" [laughter] What's in this in-context learning and activation steering equivalence paper is that you can predict the number of examples that you will need to put in there in order to jailbreak the model. That's cool.

swyx

By doing steering experiments and using this sort of equivalence mapping.

The First Mechanistic Interpretability Frontier Lab

That's cool. That's really cool.

Alessio Fanelli

That's very neat.

Yeah. I was going to say, I can back-rationalize that this makes sense because context is basically—it updates the KV cache, kind of—and then every next-token inference is still the sheer sum of all the weights plus all the context up to date. You could, I guess, theoretically steer that with—you'd probably replace that with your steering. The only problem is steering typically is on 1 layer, maybe 3 layers like you did, so it's not exactly equivalent.

The First Mechanistic Interpretability Frontier Lab

Right. Right. There's sort of—you need to get precise about how you define steering and how you're modeling the setup. But yeah, I've got the paper pulled up here: “Belief Dynamics Reveal the Dual Nature of In-Context Learning and Activation Steering.”

Yeah. The title is “Belief Dynamics Reveal the Dual Nature of In-Context Learning and Activation Steering.” Eric Bigelow, Alex Warstadt, and Ekdeep Singh Lubana are doing fellowships at Goodfire. Ekdeep Singh Lubana is the final author there. I think, actually, to your question of what the production use case of steering is, maybe if you just think 1 level beyond steering as it is today, imagine if you could adapt your model to be, I don't know, an expert legal reasoner in almost real time, very efficiently, using human feedback or your semantic understanding of what the model knows and where it knows that behavior.

I think that while it's not clear what the product is at the end of the day, it's clearly very valuable. Thinking about what the next interface for model customization and adaptation is is a really interesting problem for us. We have heard a lot of people actually interested in fine-tuning and RL for open-weight models in production. People are using things like Tinker or open-source libraries to do that. But it's still very difficult to get models fine-tuned and RL-trained for exactly what you want them to do unless you're an expert at model training. That's something we're looking into.

swyx

Yeah. I never thought so. Tinker from Thinking Machines famously uses rank-1 LoRA. Is that basically the same as steering? What's the comparison there?

The First Mechanistic Interpretability Frontier Lab

Well, in that case, you are still applying updates to the parameters, right? You're not touching a base model; you're touching an adapter. It's kind of—

Alessio Fanelli

Yeah.

The First Mechanistic Interpretability Frontier Lab

Right. But I guess it still is more in parameter space. Maybe it's like: are you modifying the pipes, or are you modifying the water flowing through the pipes?

Alessio Fanelli

Okay.

The First Mechanistic Interpretability Frontier Lab

To get what you're after.

Alessio Fanelli

Yeah.

The First Mechanistic Interpretability Frontier Lab

That's maybe one way. [laughter]

swyx

I like that analogy.

The First Mechanistic Interpretability Frontier Lab

That's my mental map of it, at least. But it gets at this idea of model design and intentional design, which is something that we're very focused on. I hope that we look back at how we're currently training models and post-training models and just think, “What a primitive way of doing that right now.” There's no intentionality really in—

swyx

It's just data, right? The only thing you can control is what data we feed in. Dan from Goodfire likes to use this analogy: he has a couple of young kids, and he talks about, “What if I could only teach my kids how to be good people by giving them cookies or giving them a slap on the wrist if they do something wrong?” Not telling them why it was wrong or what they should have done differently—just figuring out the action.

The First Mechanistic Interpretability Frontier Lab

Right. Exactly.

swyx

So that's RL.

The First Mechanistic Interpretability Frontier Lab

Yeah. Right. But it's sample-inefficient. It's slurping feedback—it's slurping supervision throughout. You'd like to get to the point where you can have experts giving feedback to their models that are internalized, and steering is an inference-time way of sort of getting that idea. Ideally, you're moving to a world where there is much more intentional design in perpetuity for these models.

swyx

Okay, this is one of the questions we asked Emmanuel from Anthropic on the podcast a few months ago. Basically, the question was: you're at a research lab that does model training and foundation models, and you're on an interpretability team. How does it tie back? Do these ideas come from the pre-training team? Do they go back? For those interested, you can watch that. There wasn't too much of a connection there, but it's still something they want to push for down the line.

The First Mechanistic Interpretability Frontier Lab

It can be useful for all of the above. There are certainly post hoc use cases where it doesn't need to touch that. I think the other thing a lot of people forget is that this stuff isn't too computationally expensive. If you're interested in getting into research, mechanistic interpretability is one of the most approachable fields. A lot of this is: train a probe. The budget for this one is already fairly low. There's a lot done, there's a lot of open-source work, and you guys have done some too.

swyx

There are notebooks from the Gemini team and from Neel Nanda that are like, “This is how you do it.” Just step through the notebook.

The First Mechanistic Interpretability Frontier Lab

Even if you're not technical with any of this, you can still make progress there. You can look at different activations. But if you do want to get into training this stuff, correct me if I'm wrong, training is in the thousands of dollars—not even that high-scale.

swyx

And then the same with applying it, doing it for post-training, RL, and all this stuff is fairly cheap at the scale of, “Okay, I want to get into model training. I don’t have compute for pre-training stuff.” So it’s a very nice field to get into.

There are also a lot of open questions. There are so many questions we have. Some of them have to do with, “Okay, I want a product. I want to solve this.” There’s also just a lot of open-ended stuff that people could work on that’s interesting. I don’t know if you guys have any calls for what the open questions and open work are that you’d either want to collaborate on, or that you’d just like to see solved. For people listening who want to get into mechanistic interpretability—people always talk about it—what are things they should check out? Start, of course, by joining you guys as well.

The First Mechanistic Interpretability Frontier Lab

There’s a paper by Lee Sharkey called “Open Problems in Mechanistic Interpretability,” which I recommend everyone who’s interested in the field read. It’s a really comprehensive overview of the things that experts in the field think are the most important problems to be solved.

I also think, to your point, it’s been really inspiring to see a lot of young people getting interested in interpretability—not just young people, but also scientists who have been experts in physics for many years, and in biology and things like this, transitioning into interpretability. The barrier to entry is, in some ways, low, and there’s a lot of information out there and ways to get started.

There’s this anecdote of professors at universities saying that, all of a sudden, every incoming PhD student wants to study interpretability, which was not the case a few years ago. It just goes to show how exciting the field is, how fast it’s moving, how quick it is to get started, and things like that. It’s also just a very welcoming community. There’s an open-source mechanistic interpretability Slack channel where people are always posting questions, and folks in the space are always responsive if you ask things on various forums and stuff. The “Open Problems” paper is a really good one.

swyx

For other people who want to get started, I think MATS is a great program. What’s the acronym for? It’s normally a summer-internship-style program.

The First Mechanistic Interpretability Frontier Lab

ML Alignment & Theory Scholars, I think. But they’ve been doing it year-round now, and a lot of our full-time staff have come through that program or gone through that program. It’s great for anyone who is transitioning into interpretability. There are a couple of other fellows programs. We do one, as does Anthropic, and those are great places to get started if anyone is interested.

Interpretability has been seen as a research field for a very long time, but I think engineers are sorely wanted for interpretability as well, especially at Goodfire, but elsewhere too, as it scales up.

swyx

I should mention that Lee actually works with you guys, right, in the London office. I’m adding our first-ever MATS track at AI Engineer Europe because I see these industry applications emerging now, and I’m pretty excited to help push that along. It’ll effectively be the first industry mechanistic interpretability conference.

The First Mechanistic Interpretability Frontier Lab

I’m so glad you added that.

swyx

You know, it’s still a little bit of a bet. It’s not that widespread, but I can definitely see that this is the time to really get into it. You want to be early on things.

The First Mechanistic Interpretability Frontier Lab

For sure. And I think the field understands this. At ICML, I think the title of the mechanistic interpretability workshop this year was “Actionable Interpretability,” and there was a lot of discussion around bringing it to various domains.

swyx

Everyone’s adding “pragmatic,” “actionable,” whatever. Okay, well, we weren’t actionable before, I guess.

Just being at AI Engineer Europe, you see the interpretability room, unlike at old-school conferences. I think they had a very tiny room until they got lucky and it got doubled. There’s definitely a lot of interest and a lot of niche research. You see a lot of research coming out of university students. We covered a paper last week—it had 2 unknown authors and not many citations—but you can make a lot of meaningful work there.

The First Mechanistic Interpretability Frontier Lab

One thing I did want to call out, because I think people haven’t really mentioned this yet, is just steering. I think steering is an abnormally important field. The conspiracy theory 2 years ago, when the first SAE work came out of Anthropic, was that they just used SAEs to turn the bad-code vector down and then turn up the good code.

Isn’t that the dream? Maybe why it’s funny is that, if it were realistic, it wouldn’t be funny. It would be, “No, actually, we should do this.” But it’s funny because we feel there are some limitations to what steering can do. A lot of the public image of steering is the Gen Z stuff: you can make it really love the Golden Gate Bridge, or make it speak like Gen Z for no reason, which seems like a huge stretch.

swyx

Yeah. I don’t know if we’ll get there this way.

The First Mechanistic Interpretability Frontier Lab

Yeah. I think we’re announcing something very soon that I will not speak too much about. But what we’ve run into again and again is that we don’t want to be in a world where steering is only useful for stylistic things. That’s definitely not what we’re aiming for.

The types of interventions that you need to get to things like legal reasoning are much more sophisticated and require breakthroughs in learning algorithms.

swyx

Is this an emerging property of scale as well?

The First Mechanistic Interpretability Frontier Lab

I think so. Scale definitely helps. It allows you to learn a lot of information and reduce noise across large amounts of data. But I also think there are ways to do things much more effectively, even at scale—actually learning exactly what you want from the data and not learning things that you don’t want exhibited in the data.

We’re not anti-scale, but we’re also realizing that scale is not going to get us to the type of AI development that we want to see in the future, as these models get more powerful and get deployed in all sorts of mission-critical contexts. The current life cycle of training, deploying, and evaluating these models is, to us, deeply broken and has opportunities to improve. More to come on that very soon.

swyx

I think the SAEs are basically just a proof point that these concepts do exist. If you can manipulate them in the precise way, you can get the ideal combination of them that you desire. Steering is maybe the most coarse-grained peek at what that looks like, but I think it’s evocative of what you could do if you had total surgical control over every concept, every parameter.

The First Mechanistic Interpretability Frontier Lab

Yeah, exactly.

swyx

There were bad-code features. I’ve got it pulled up just coincidentally as you guys were talking.

There’s specifically a code-error feature that activates, and they show it off. It’s not typo detection; it’s typos in code. It’s not typical typos. They have malicious-code and code-error features, and a whole bunch of smaller, more granular features.

The First Mechanistic Interpretability Frontier Lab

Yeah.

swyx

The rough intuition for me, and why I talked about post-training, was that you just have a few different rollouts with all these things turned off and on, and then you can use that as synthetic data to post-train on.

The First Mechanistic Interpretability Frontier Lab

Yeah. I think we make it sound easier than it is by just saying that they do the real hard work. I mean, you guys have the right idea. We replicated a lot of these features in our Llama models as well.

swyx

I think a lot of this stuff is open, right? You guys opened yours, DeepMind has opened a lot of SAEs on Gemma, and even Anthropic has opened a lot of this stuff. There are a lot of resources that we can probably share with people who want to get involved. Special shout-out to Neuronpedia as well—an amazing piece of work to visualize those things.

The First Mechanistic Interpretability Frontier Lab

Yeah, exactly.

swyx

I guess I wanted to pivot a little bit onto the healthcare side, because I think that’s a big use case for you guys. We haven’t really talked about it yet. This is a bit of a crossover for me because we have a separate science pod that we’re starting up. It’s such a huge investment category, and I’m also less qualified to cover it. We actually have biology PhDs to cover that, which is great.

I just wanted to recap your work on the Evo 2 stuff, and then talk about building forward.

The First Mechanistic Interpretability Frontier Lab

Yeah, for sure. To frame up the conversation, I think another interesting lens on interpretability in general is that a lot of the techniques we’re describing are ways to solve the AI–human interface problem, and bidirectional communication is the goal there.

So what we've been talking about with intentional design of models—and steering, but also more advanced techniques—is having humans impart our desires and control into models and over models. The reverse is also very interesting, especially as you get to superhuman models, whether that's narrow superintelligence, like these scientific models that work on genomics data and medical imaging, or, down the line, superintelligence of other forms as well. What knowledge can the AIs teach us, as sort of the other direction in that?

Some of our life science work to date has been getting at exactly that question. Some of it does look like debugging these various life sciences models, understanding if they're actually performing well on tasks, or if they're picking up on spurious correlations. For instance, with genomics models, you would like to know whether they're focusing on the biologically relevant things that you care about, or if they're using some simpler correlate, like the ancestry of the person they're looking at.

But then also, in the instances where they are superhuman, maybe they're understanding elements of the human genome that we don't have names for, or specific discoveries that they've made that we don't know about. That's a big goal.

We're already seeing that. We're partnered with organizations like Mayo Clinic, a leading research health system in the United States, Arc Institute, as well as a startup called Prima, which focuses on neurodegenerative disease. In our partnership with them, we've used the foundation models they've been training and applied our interpretability techniques to find novel biomarkers for Alzheimer's disease.

I think this is just the tip of the iceberg, but that's a flavor of some of the things that we're working on.

Alessio Fanelli

Yeah, I think that's really fantastic. We obviously did the Chan Zuckerberg podcast last year as well, and there's a plethora of these models coming out because there's so much potential and research. It's very interesting how it's basically the same as language models, but just with a different underlying data set. It's the same exact techniques; there's no change, basically.

The First Mechanistic Interpretability Frontier Lab

Yeah. Well, even in other domains, right? In robotics, I know a lot of the companies just use Gemma as the backbone, and then make it into a VLA that takes these actions. It's transformers all the way down.

So we have MedGemma now, right? This week there was even MedGemma 1.5. They're training it on 3D scans, medical-domain knowledge, and all that stuff, too. So there's a push from both sides, but one of the things about mechanistic interpretability is that you're a little bit more cautious in some domains, healthcare mainly being one. There are guardrails, and we're more risk-averse to something going wrong there. Even from a basic understanding, if we're trusting these systems to make claims, we want to know why and what's going on.

I think there's totally a deployment bottleneck to actually using foundation models for real patient usage or things like that. Say you're using a model for rare-disease prediction, you probably want some explanation as to why your model predicted a certain outcome—and an interpretable explanation at that. So that's definitely a use case.

But I also think being able to extract scientific information that no human knows to accelerate drug discovery and disease treatment and things like that is a really big unlock for scientific discovery. You've seen a lot of startups say that they're going to accelerate scientific discovery, and I feel like we actually are doing that through our interpretability techniques, almost by accident.

I think we got reached out to very, very early on from these healthcare institutions, and none of us had healthcare experience.

swyx

How did they even hear of you?

The First Mechanistic Interpretability Frontier Lab

A podcast.

swyx

Okay. Yeah, podcast. Okay. Well, now is the time. Everyone can call us up.

Alessio Fanelli

Podcasts are the most important thing. Everyone should listen to podcasts, and everyone should come to podcasts.

The First Mechanistic Interpretability Frontier Lab

Podcast. They were like, “We have these really smart models that we've trained, and we want to know what they're doing.” We were really early at that time, about 3 months old, and it was a few of us. We were like, “Oh my God. We've never used these models. Let's figure it out.”

But it's also great proof that interpretability techniques scale pretty well across domains.

swyx

Interpretability is a machine-learning technique; machine-learning skills apply everywhere, right? It's obviously just a general insight. Probably finance too, I think, which would be fun for our history. I don't know if you have anything to say there.

The First Mechanistic Interpretability Frontier Lab

Well, just across science, we've also done work on materials science. It really runs the gamut.

swyx

Yeah, awesome. And for those who want to reach out, you're obviously experts in this, but is there a callout for people that you're looking to partner with—design partners, people to use your stuff outside of just the general developer that wants to plug-and-play steering stuff? On the research side more so, are there ideal design partners, customers, stuff like that?

The First Mechanistic Interpretability Frontier Lab

Yeah, I can talk about maybe non-life sciences, and then I'm curious to hear from you on the life sciences side. We're looking for design partners across many domains. Anyone who's customizing language models or trying to push the frontier of code or reasoning models is really interesting to us.

We're also interested in the frontier of models that work in pixel space, as we call it. So if you're doing world models, video models, or even robotics, where there's not a very clean natural-language interface to interact with, I think interpretability can really help, and we're looking for a few partners in that space.

swyx

Just because you mentioned the keyword “world models,” is that a big part of your thinking? Do you have a definition that I can use? Everyone's asking me about it.

Alessio Fanelli

About world models, there are quite a few definitions.

The First Mechanistic Interpretability Frontier Lab

I don't feel equipped to be an expert on world-model definitions, but the reason we're interested in them is because they give you—with language models, when you get features, you still have to do automated interpretability and things like that to actually understand what this concept is. But in image, video, and world models, it's extremely easy to grok what the concept is because you can see it and visualize it.

This makes the feedback cycle extremely fast for us. Also, for things like probes in a language-model context, and then taking that to world models, what if you wanted to detect harmful actors in world-model scenes? You can't actually go and label all of that data feasibly, but maybe you could synthetically generate harmful-actor data using SAE feature activations or whatever, and then actually train a probe that was able to detect that much more scalably.

I just think video, image, and world models have always been something we've explored and are continuing to explore. Mark's demo was probably the first moment we were really like, “Oh, wow. This could really change the world of—”

swyx

The steering demo.

The First Mechanistic Interpretability Frontier Lab

Yeah. No, the image demo.

Alessio Fanelli

The diffusion one. Exactly.

swyx

Yeah. Yeah. We should probably show that, and you demoed it at Worlds Fair, so we can choose to link that.

The First Mechanistic Interpretability Frontier Lab

Nice. Yeah.

swyx

People can play with it, right?

The First Mechanistic Interpretability Frontier Lab

Yes.

Alessio Fanelli

Yeah. I think, for me, one way in which I think about world models is just having a consistent model of the world, where everything that you generate operates within the rules of that world. I imagine it would be a bigger deal for science or math, or anything where you have verifiable rules, whereas I guess in natural language maybe there are fewer rules. So it's not that important.

The First Mechanistic Interpretability Frontier Lab

And which makes debugging the model's internal representations, or its internal world model, to the extent you can make that legible and explicit and have control over that, all the more important.

In language, it's a fuzzy enough domain that if its world model isn't fully like ours, it can still sort of pass the Turing test, so to speak. But I know there have been papers that have looked at how, even if you train certain astrophysics models, they do not learn F = ma. The same way that you can have a model do well for modular arithmetic, but it doesn't really learn what we think of as modular arithmetic. It learns some crazy heuristic that is essentially functionally equivalent, but it's probably not the sort of grokked solution that you would hope for.

swyx

It's how an alien would do it, right?

The First Mechanistic Interpretability Frontier Lab

Exactly. But no, I think that's probably a function of our learning being bad rather than that approach probably not being real, because it's how we humans learn.

swyx

Right.

The First Mechanistic Interpretability Frontier Lab

Yeah, right. Well, it's just the problem of induction. All of machine learning is based on induction, and it's impossible to say, “I have a physics model.” You might have a physics model that works all the time except when there is a character wearing a blue shirt and green shoes. You can't disprove that that's the case unless you test every particular situation your model might be in.

Alessio Fanelli

So we know that the laws of physics apply no matter where you are or what scenario it is. But from a model's perspective, maybe something that's out of distribution—it just never needed to learn that the same laws of physics apply there.

swyx

Yeah. You were very excited because I read Ted Chiang over the holidays, and I was very inspired by this short story called “Understand,” which apparently is pretty old. You must be familiar with it. To me, it was this fictional story—it’s like the inverse of Flowers for Algernon, where you had someone get really smart but then also try to outsmart the tester.

And the story just read like the chain of thought of a superintelligence, right? They’re like, “Oh, I realize I’m being tested. Therefore, what’s the consequence of being tested? They’re testing me, and if I score well, they will use me for things that I don’t want to do. Therefore, I will score badly—but not too badly that they’ll raise alarms.” [laughter] Model sandbagging is a thing that people have explored, but I just think Ted Chiang’s work in general seems to be something that inspires you. I just wanted to prompt you to talk about it.

Alessio Fanelli

I think so. Ted Chiang is a sci-fi author who writes amazing short stories.

swyx

His other claim to fame is “Story of Your Life,” which became the movie Arrival.

Alessio Fanelli

Exactly. Yeah. So, 2 books of short stories that I’m aware of. He also has a great online blog post. I think he’s the one who coined the term of LLMs as a blurry JPEG of the web. I should fact-check that, but it’s a good post.

I think almost every one of his short stories has some lesson to bear on thinking about AI and AI research. So, you’ve been talking about alien intelligence, right, in this AI-human communication and translation problem. That’s exactly what’s going on in Arrival, in “Story of Your Life.” And just the fact that other beings will think and operate and communicate in ways that are not just challenging for us to understand, but fundamentally different—in ways that we might not even be able to expect.

And then the one that’s super relevant for interpretability is the other short story collection he has, called Exhalation. And that is literally about a robot doing interpretability on its own mind.

swyx

Oh, okay.

Alessio Fanelli

So I just think that you don’t even have to squint to make the analogies there.

swyx

Well, I actually take Exhalation as a discussion about entropy and order. But yes, there’s a scene in Exhalation where basically everyone is a robot. The guy realizes he can set up a mirror to work on the back of his own head and then starts doing operations like that by looking in the mirror and doing this. [laughter]

Alessio Fanelli

Yeah. And I think Ted Chiang has written about the inspiration for that story. It was half-inspired by something he had been doing on entropy. There’s apparently another short story that’s similar, where a character goes to the doctor and opens up his chest, and there’s a ticker tape going along. He basically realizes he’s a Turing machine. I don’t know. I think especially when it comes to using agents for interpretability, that story always sticks in my mind.

swyx

I find the brain surgery—or surgery—analogies a little morbid, but they are very apt. And when we talked to a lot of computational neuroscientists, they moved to interpretability because they were like, “Look, we have unfettered access to this artificially intelligent mind. You have access to everything. You can run as many ablations, as many experiments as you want. It’s an amazing testbed for science.”

Human brains, obviously, we can’t just go and do whatever we want to them. And I think it is really just a moment in time where we have intelligent systems that can really do things better than humans in many ways. And it’s time, I think, for us to do the science on them.

I’ll ask a brief safety question. You know, MATS was kind of born out of the alignment and safety conversation. Safety is on your website; it’s not like something that you deprioritize, but there’s a sort of very militant safety arm that wants to blow up data centers and stop AI, and then there’s this sort of middle ground. Is this a conversation in your part of the world? Do you go up to Berkeley and Lightcone and talk to those guys, or is there a brief civil war going on? I don’t know.

The First Mechanistic Interpretability Frontier Lab

I think a good amount of us have spent some time in Berkeley. There are researchers there that we really admire and respect. For us, we have a very grounded view of alignment and safety: We want to make sure that we can build models that do what we want them to do, that we have scalable oversight into what these models are doing, and we think that is the key to a lot of these technical alignment challenges. I think that is our opinion and our research direction.

Of course, we’re going to do safety-related research to make sure that our techniques also work on things like reward hacking and other, more concrete safety issues that we’ve seen in the wild. But we want to be grounded in solving the technical challenges we see to having humans play a big role in the deployment of these superintelligent agents of the future.

Yeah, I find the community to actually be remarkably cohesive. Whether it’s talking about academia, the interpretability work being done at the frontier labs, or some of the independent programs like MATS and stuff, I think we’re all shooting for the same goal. I don’t know that there’s anyone who doesn’t want our understanding of models to increase.

I think everyone, regardless of where they’re coming from or the use cases they’re thinking about—whether it’s alignment as the premier thing they’re focused on or someone coming in purely from the angle of scientific discovery—I think we would all hope that models can be more reliably and robustly controlled and understood. It seems like a pretty unambiguous goal.

I’ll maybe phrase it in terms of there being a U-curve to this: If you’re extremely doomer, you don’t want any research whatsoever. If you’re mildly doomer, you’re like, “Okay, there’s this high-agency doomer who’s like, ‘The default path is where we’re all dead, but we can do something about it.’” Whereas there are other people who are like, “No, just don’t ever do anything.”

swyx

Yeah. Yeah. There’s also the other side: the Superalignment people who are like, “Okay, weak-to-strong generalization. We’re going to get there. We’re going to have models smarter than us and use those to train even smarter models. How do we do that safely?” There’s that camp, too, that’s trying to solve it. But yeah, there are a lot of doomers, too. [laughter]

Alessio Fanelli

Well, I think there’s a lot to be learned from, regardless of the problem that you’re applying this to, just the notion of scalable oversight as a method of saying, “Let’s take superintelligent or current frontier models and use them to understand other models.” That’s another case where I think it’s just a good lesson that everyone is aligned on: Ideally, you are setting up your research so that as superintelligence arrives, that is a tailwind that’s also bolstering our ability to understand the models.

Otherwise, you’re fighting a losing battle if the systems are getting more and more capable and our methods are growing linearly, at human pace.

swyx

Yeah. Yeah. Viva did call out something like—I do think a consistent part of the mechanistic interpretability field is consistently strong-to-weak, meaning that we train weaker models to understand stronger models, something like that. Or maybe I got it the other way around.

Alessio Fanelli

The other way: weak-to-strong. Yeah. Yeah. The question that Ilya Sutskever and Yann LeCun posed was, well, is that going to scale? Because eventually these are going to be stronger than us, right? I don’t know if you have a perspective on that, because that is something I still haven’t gotten over, even after seeing that—

swyx

There’s a good paper from OpenAI, but it’s somewhat old—I think it’s from 2023 or 2024. It’s literally called “Weak-to-Strong Generalization.” But the thing is that most of OpenAI’s Superalignment team has—

Alessio Fanelli

They’re gone.

swyx

But I think the idea—the idea is—

Alessio Fanelli

So now it’s back. There’s no more—

swyx

They’re—so, yeah. Yeah, I think there are some new blog posts coming out.

Alessio Fanelli

I know. And just check Thinking Machines Lab’s website to see who’s back.

swyx

Go back. [snorts] [laughter] There’s more coming.

Alessio Fanelli

You know what I mean? Weak-to-strong seemed like a very different direction. When it first came out, I was like, “Oh my God, this is what we have to do.”

swyx

And it may be completely different from all the techniques we have today.

Alessio Fanelli

Yeah. My understanding of that is it’s more like weak-to-strong when you trust the weak model and you’re uncertain whether you can trust the strong model that’s being developed. I’m sort of speaking out of my depth on some of these topics, but I think right now we’re in a regime where even the strong models we trust as reasonably aligned, and so they can be good co-scientists on a lot of the problems that we’ve been tackling, which is a nice state to be in.

swyx

Yeah. Any last thoughts or calls to action?

The First Mechanistic Interpretability Frontier Lab

I don't think so.

swyx

Where are you guys based?

The First Mechanistic Interpretability Frontier Lab

San Francisco. We're in Levi's Plaza, by Coit Tower. That's where our office is.

swyx

Yeah, maybe the way I'll phrase it is: Maybe you have a use case where LLMs are almost good enough, but you need one magical knob to tune so that it is good enough. You guys make the knob.

The First Mechanistic Interpretability Frontier Lab

Yeah. Or foundation models in other domains as well. Some of those are especially opaque because you can't chat with them.

swyx

What do you do if you can't chat with them?

The First Mechanistic Interpretability Frontier Lab

Think about a genomics model or a materials science model—a narrow foundation model. They predict.

swyx

I was going to say I thought the diffusion work you guys did early on was pretty fun. You could see it directly applied to images, but we don't see as much interpretability in diffusion or images, right?

The First Mechanistic Interpretability Frontier Lab

It's going to be huge. Look at these video models. They're so expensive to produce. Basically, a Midjourney SREF is kind of a feature, right?

swyx

The what? Midjourney SREF, like the string of numbers that you—

The First Mechanistic Interpretability Frontier Lab

Right, right—the style reference, I guess. Yeah. No, I think we're starting to see more of it. I'll say that the research preview of our diffusion model, kind of a creative use case, and the steering demo you saw—I think of those much more as demos than a lot of the core platform features that we're working on with partners, which are unfortunately under NDA and less demoable.

But I hope you're going to see interpretability pervading a lot of what gets done, even if it is behind the scenes like that. Some of the public-facing demos might not always be representative; they're just the tip of the iceberg, I guess. That's one way to put it.

swyx

Okay, excellent. Thanks for coming on.

The First Mechanistic Interpretability Frontier Lab

Thanks for having us. This is a great time.