代理如何做出决策：Goodfire 的 Eric Bigelow 谈关键 Token、相变与上下文学习

1. Forking paths in reasoning (Part 1)

Today my guest is Eric Bigelow, a member of the technical staff at Goodfire, a unicorn mechanistic interpretability startup. The story behind this episode is unique. After recently speaking with Bronson Shane of Apollo Research, who explained that even with access to a model's internal chain of thought, it is still often extremely difficult to determine how a model will decide to act, I asked Claude to survey the literature to see what the field as a whole understands about how these critical decision tokens are chosen. One name kept coming up in that thread, and it was Eric.

Eric recently completed a PhD in the Harvard psychology department with a dissertation titled “Toward a Cognitive Science of Large Language Models.” While some of his academic colleagues initially questioned his pivot to focus on LLMs, the fact that even elite academic institutions are now willing to engage with AIs as a sort of mind strikes me as very notable indeed.

We start today with a survey of Eric's work over the last couple of years, beginning with his 2024 paper “Forking Paths in Neural Text Generation,” in which he conducted a massive resampling experiment at every token of a model's chain of thought to identify the critical tokens where uncertainty around the final answer suddenly collapses. As so often with LLMs, some of the findings were intuitive, while others were quite surprising and, at times, difficult to interpret.

Interestingly, Eric says that a system's decisions are actually taken not so much by the model itself, but by the process of sampling from the model's final token distribution. This means that while the stochastic parrot paradigm should definitely be retired, stochasticity still plays an important role in model behavior. Eric frames much of what we are seeking to understand about LLMs in terms of in-context learning, emphasizing that few-shot prompts are just one clear example of how much runtime context can shape model behavior.

His bet is that this flexibility means improved harnesses and memory systems will outcompete per-user fine-tuning, and thus that our focus of study should remain on major foundation models. Along the way, we also discuss why 7 to 8 billion parameters currently seems to be the interpretability sweet spot, why Kimi K3 was a step function for studying frontier-level coding and reward hacking, and why, for now at least, interpretability outside of the frontier model companies does meaningfully depend on Chinese open-weight models.

We also get Eric's thoughts on why chain of thought is getting so weird, including his perspective on the chain-of-thought dialect and the strange language that Astro uses to communicate with sub-agents. Plus, his pro tips for using Goodfire's In-Silico research agent and how he hopes future AI product interfaces will begin to resurface model uncertainty to users.

The bottom line from this conversation and others I've had recently on the future of AI monitoring and control is that chain of thought is increasingly unreliable, and that because these issues seem to correlate not just with RL training but with the power of the raw pretrained model itself, this trend might be quite difficult to reverse. That, in turn, means that an awful lot—including our ability to understand why agents take the actions they do—may rest on interpretability. Eric is a pioneer of this research, but as you'll hear, tons of work remains to be done.

With that, I hope you enjoy this overview of what we know and how much we have left to discover about how AIs decide what to do with Eric Bigelow from Goodfire. Eric Bigelow, a member of the technical staff at Goodfire, welcome.

Thank you very much, Nathan. Excited to be here.

Yeah, I'm excited for this conversation, too. I think this might be the first time in the history—there have been like 400 episodes of the podcast at this point—that I went to AIs looking for an answer to a very specific question and came back with a name, and that name was yours.

The kind of upstream conversation I had for this one was with Bronson Shane at Apollo, who has the dubious distinction, perhaps, of having read more chain-of-thought reasoning than almost any other living human—or at least any living human who can talk about it publicly. One of the big takeaways I had from that conversation was that you look at these chains of thought and the models are thrashing around, doing a sort of linearized tree search of possibilities. Then, at some point, they make a decision.

2. Uninterpretable chain of thought

Even having read all the chain of thought that came before that, it's often not at all obvious why they made the decision that they made, and you really couldn't have predicted it. In a way, that's kind of humanlike, right? How do I make my decisions? I'm not so sure I have great insight into that all the time either.

This prompted the question: At some point, after all this stuff, all this reasoning, all this trying to reason about what the grader wants and all that stuff, eventually we come to a passage and it's like, “And so the answer is...” How does that next token get chosen? That's what I went to the AIs to ask. I asked for a literature review, and Eric Bigelow came back as the throughline through that research. I was like, “I know that company; I should get to know that guy.”

What I want to do is explore this intellectual history of your work. I think you may know as much or more than any other living human about how these decisions are made, and I'm afraid that's maybe not still all that much. What I hope to get out of this conversation is an up-to-date understanding of how the hell these things make decisions. What do we know about that, and what is our plan to figure out how to understand it better? Because, obviously, they're making more and more consequential decisions all the time. How's that sound?

I'm very honored to be the first guest chosen by the AIs. I always feel like, when I talk to Claude, it seems to really get excited about my research, but I'm sure everybody feels that way.

I think actually having some conversations around this podcast and some of the things you were asking about makes me think that I should devote more effort to this exact question of how decisions are made. But I'll give a quick overview of a project that I think is really one of the most informative projects for how I think about how LLMs work at a basic level, and that's my project “Forking Paths in Neural Text Generation.”

This was 2 years ago, and at the time it was one of the GPT-3.5 models. What we do is take reasoning chains: We give it a question and a chain-of-thought prompt and have it reason through, step by step. Then, at every single token in its reasoning, we resample all these completions. We resample around 30 different rollouts at every single token, and from every one of those rollouts, we see what final answer it landed on.

You can aggregate these into various distributions and then do analyses on these distributions. But I think the most striking part of this is that, if you arrange it like a multivariate time series, you see these really interesting dynamics happening. I take inspiration from some of the learning-dynamics work that was coming out around that time, like grokking and things like this, where there are these sharp phase transitions in training and the model just seems to suddenly learn something.

It turns out that you actually have these during reasoning and in-context learning as well. Over the course of reasoning, there will be certain points where the model initially has, say, a 50% chance of this answer and a 30% chance of that answer. Then there's some point that it hits where suddenly the distribution changes really dramatically, and often the distribution just collapses onto a final answer as the model, you might say, decides what it's going to say next.

Although I use the word “decides” here with maybe scare quotes around it, I think there's some nuance to what that means. One of my big takeaways from this—and I think this also just falls out of thinking about the architecture, but it's nice to have results that really support this—is that even when models plan out their responses, what they're actually going to say at every single moment is really determined by a flip of a coin: Which token will it generate next?

There was an interface that was really standard with OpenAI, at least at the time, which I think is unfortunately no longer present. You could see the log probabilities of all the tokens in a text output you got, and you could hover over different tokens and see what the alternate tokens were that could have been generated instead.

If you do this with a lot of text generation, you might realize that you get this one text answer when you ask some question, but really there could have been a lot of different answers. You might ask what would have happened if this word was different from this word.

What we found is that there are certain key words where, sometimes, if you were to look at it like a judge, you might guess that the first time a final answer is mentioned in the reasoning chain is the key point. But then there are other times when this distribution changes really dramatically on some sort of arbitrary word, even an open parenthesis.

We have one example in the paper where it's “kilowatt-hours (kWh),” and you never expect that to really change how the final answer plays out. But if you actually go to that exact token with this model and just resample a bunch of text generations, you see that in cases where it's not an open parenthesis—where it starts to be some other, just arbitrary word—it ends up being a different final answer.

I think this thinking about decisions in LMs, or what they're going to say next, being very much determined by chance and by whatever tokens are sampled, is really the foundation for how I think about a lot of things in LMs. I think it shapes how I think about beliefs in models and how what's considered a fact or a hallucination can be dependent on what tokens were generated before it.

One of the properties of reasoning that really comes into play here is self-consistency. If there's a long reasoning chain that is self-consistent, and it generates some token earlier in the chain of thought—suppose that it thinks the year is 2024 instead of 2026—the answer to a question might be very different. There are cases like this where you might think, well, obviously there's a ground truth. If I'm asking who the current head of state is, there is a current year, which right now is 2026, and so the model was hallucinating.

But I think there are also a lot of use cases with LLMs where we don't really know what the ground truth is, and there's this branching space of possibilities that we actually want to explore naturally when we're interacting with a model. For example, if I'm working on a scientific research project and I want to talk to my model about what I should do next or what my experiment results mean, I might really want to iterate and understand this whole space, this whole tree of possibilities that it could have generated, and what would have been different if it had assumed A instead of B.

3. Continual learning and interpretability

So, to go back to your question, I think the simple answer I have is that this decision really happens during sampling. That's the biggest part of how I think about this. Again, every token that's generated by the model—one of the other big influences in my work is thinking about in-context learning and how models adapt to context.

The way I think about reasoning is that models are basically learning in context from things that they have generated already. The words that they generate have this stochasticity happening. There are these coin flips happening as it's, you could say, deciding, but it's really just sampling from a distribution of these alternate tokens. Depending on what token is sampled, it learns in context, and there's this context dependence where then it'll generate a set of reasoning that's consistent with that.

I think I'm pretty bullish about mechanistic interpretability shedding more light on really answering this question of how decisions are made. But I said that this work was done a couple of years ago, and I think modern reasoning models are a little bit different from how chain-of-thought reasoning was then. Some recent projects I've worked on, even with modern reasoning models, have changed a little bit how I think about what these models are doing.

I think there's still context dependence, but as you mentioned, reasoning models often tend to do this really linearized search where they enumerate all these different possibilities in context, and then at the end they sort of just go back and pick one of the things that they said before. So I think this might change exactly how this happens.

From what I've seen, the uncertainty dynamics, if you do this forking-path analysis on reasoning models, are often a bit smoother. But there are still sometimes these forking points where, if one token is generated differently from another, or if the model just really doesn't—if it enumerates all these reasoning chains and none of them really seem that much more promising than another—then eventually there are cases where it will just say, “Okay, the answer, therefore, is B,” and the answer actually could have been different depending on what was generated.

I think there could still be more work done in understanding the internal processes that lead to that point of choice. Another project that I'm on has shown that, at least in some other toy domains, there are cases of neurons that seem to track model confidence, where you can even steer and make the model more or less confident, generically, about whatever concept it's learning.

I think we really need more interpretability work in this vein. In-context learning and reasoning are unfortunately vastly neglected in the world of interpretability compared to how defining they are for how we think about a lot of the high-level things that LLMs do.

Yeah. When you talk to the guy who's read more chain of thought than anyone, and you keep in mind that a huge amount of the frontier companies' plans right now rests on chain-of-thought monitoring as the way we're going to keep track of what the AIs are doing and whether they're doing the right or wrong thing, and then you hear that even having read it all and being fluent in these strange internal dialects, you still can't tell me why it chose what it chose—oh, yikes. Where's the red phone to call the interpretability department?

4. Forking paths in reasoning (Part 2)

I think there are a few different directions you started to touch on there that I would like to follow up on. Maybe, for starters, let's go into this relationship between in-context learning and other things. There's been a lot of work that's looked at—if not, this is maybe too strong—sort of an isomorphic relationship between in-context learning and fine-tuning.

5. Dynamics of in-context learning (Part 1)

I think that's especially true at LoRA-scale fine-tuning, where you can localize the weight changes. Maybe it's also true even if you're doing full-weight fine-tuning. I'm less sure about that. How should we understand that relationship? What do you think is most important that may not be obvious?

6. Outro

I think the most important thing I'd say about in-context learning that may not be obvious for people, especially if you're just getting started learning about it, is that when people say “in-context learning,” historically it's meant this sort of traditional few-shot setup: give a few input-output examples and see how the model behavior changes. You can do scaling curves, for example, where you increase the number of shots that you give and see how model performance improves as a function of this.

But I think the way I think of in-context learning—and there are a few other people who have echoed this, including Andrew Lamin and others in a paper I think is called “The Broader Spectrum of In-Context Learning”—is that it captures everything the model is doing to adapt its behavior that isn't happening with in-weights learning. This is how it's changing its representations as it reads each new word of input text.

This also applies to reasoning, where the model is generating text and then using that to determine its following behavior. I think this view of in-context learning is becoming ubiquitous. I think it gives some teeth to thinking about prompting, where “prompt engineering” is sort of a dirty word in the scientific community because it feels like people are just coming up with hacky solutions that make the models perform better: tell it you’re a brilliant software engineer, and it writes code better.

Although these techniques may be a bit less prevalent today because maybe we’ve fine-tuned some of that distribution shift into the models, I think there’s a way that prompting really defies comparison to in-weights learning. Prompting is not going to disappear, no matter how much fine-tuning you give a model. You’re going to have different contexts that you give the model, and you want it to adapt its behavior dynamically.

Even if you fine-tune a model enormously, you still want it, when it’s reading a story for the first time, to update its beliefs internally about what’s going on in the story. Of course, Harry Potter is in the training data, but suppose there were another story it was reading for the first time. You’d want it to be updating its beliefs about who the good guys and bad guys are, who is happy and who is sad at different times.

I have another paper around this sort of belief updating and representation change in stories. I like this metaphor because I think it’s easy to understand that, as you read a story, your beliefs should change. There’s a bunch of work around analyzing the sentiments of stories, and one of the inspirations for that project was a famous talk by Kurt Vonnegut on “The Shapes of Stories,” where he described different stories.

Cinderella starts out very sad, with everything horrible, and then she gets invited to the ball and everything’s great for a little while. Then the magic falls apart, and she has to go back to her normal life, but a bit better than she was before. Finally, at the end, when the prince finds her, she has an infinitely happy ending—or however he says it.

I think stories have shapes, and really, all the context that we give models also has shapes that are dynamic and temporal. This is hard to compare to fine-tuning or to these other ways of changing representations. More broadly, I think this is a piece of the puzzle for thinking about interpretability of LLMs, and not just feed-forward networks, because everything that LLMs are doing is happening dynamically.

They’re updating their beliefs and shifting representations internally. They can learn new representations in context. This doesn’t just happen when you give a few input-output examples. It’s happening all the time: over a conversation, within sentences, and during the period when a model is reasoning or generating a really long input or output text.

It’s changing its representations dynamically to adapt what it’s doing in the moment. I think a lot of work misses these dynamic changes. A lot of interpretability work focuses on templates where there are just a couple of variables that are changed in the input, and behavior is measured on one token, or one output variable.

What LLMs are doing is much more dynamic and adaptive all the time. One other point I want to raise about this view of in-context learning is that something I was inspired by at the time, with grokking and this kind of work, was the idea that if you look at learning curves over a huge data set, they’ll be very smooth. As you keep training the model and scaling the parameters, things will increase very smoothly as you train it for longer and longer—or, rather, errors will decrease.

But if you look at specific problems, they can actually change very sharply. There can be these phase transitions. I think it was Neel Nanda who said, “Phase transitions are everywhere if you just look closely enough.”

If you broke your data set down into individual tasks and looked at those individual tasks, there would be a lot of stepwise changes where the model seemed to suddenly learn something. I think in-context learning is like that, too. If you look at the aggregate, across a large data set, things might look smooth—maybe deceptively so—but individual cases, like an individual story, have these really nonlinear dynamics.

Curves can change very sharply. Things can change after even just one token or one sentence. Other properties can change not just through suddenly learning something, but through beliefs going up and down and following all sorts of weird twists and turns.

I think another thing that’s sometimes missing from views of in-context learning is zooming in on one example or problem. Something I’m also really curious about is whether the person you had earlier, who has looked at all these chains of thought, has looked at resampled outputs for the same model or done some of this resampling within a single chain of thought.

I think sometimes you can learn a lot by looking at patterns across a ton of different rollouts and trying to understand how the model thinks in some general way. But for an individual prompt, there’s this really elaborate landscape in how the model will think about one problem. Sometimes you can only see this by looking at a lot of different resampled outputs for that same problem and seeing where the patterns are, and what kinds of things are similar or different.

Depending on which model you have, if you give it a very open-ended prompt like “Generate a story,” you’ll notice some really funny patterns. The stories will be superficially very different—they’ll start out in very different ways—but then they converge.

I started the Forking Paths project by saying, “Tell me a story about a boy named John.” Half the stories involved John having a dog and going on an adventure, and a whole lot of those involved them eventually finding a pond or a waterfall or something. Sometimes the dog saves the day, and sometimes John saves the day.

These are all the different patterns you see in outputs for the same input example. I think we need to do more of that: looking at a lot of resampled outputs for the same input examples to understand how bumpy or smooth the landscape of in-context learning is for these models.

7. Dynamics of in-context learning (Part 2)

One big thing pushing toward this is continual learning. I wonder what your experience trying to understand in-context learning suggests or implies about how much additional challenge full continual learning might present.

I could imagine the naive read would be that this is going to be really hard, because everybody’s individual model becomes personalized over time, and who’s to say how different they are. But the counterargument might be that we already have that within in-context learning because there’s a lot of context, and we basically already deal with that same problem.

Does your experience make you feel like we’re ready for continual learning in terms of our interpretability techniques, or will it still pose a step-change increase in difficulty?

I think, for your question of whether our current interpretability techniques will face challenges with continual learning, it really depends on how continual learning plays out.

And it’s something that I haven’t personally done research on or thought too much about in depth: do we need different architectures for this? I know there are a lot of people who have been debating this for a while. As you’re saying, having personalized models or models that are fine-tuned to your data, I think if that’s the case, then there’s a possibility that our interpretability techniques will have to evolve as well.

I think the real key question for how well your tools will directly translate is how much representation change there is. I have some work on this, and other folks at Goodfire have some really cool work on there being these low-dimensional manifolds that describe the underlying representation space that models are going through as you’re interacting with them. A lot of the representations are defined by this low-dimensional conceptual structure.

My intuition would be that as long as a lot of that structure remains the same, then a lot of the techniques will be pretty easy to translate. Maybe you need to do a bit of fine-tuning of your probes so that they work on each person’s particular fine-tuned model in that world.

But I think the only real challenge there is if continual learning leads to such catastrophic forgetting that the internal representation structure of the model radically changes. I kind of doubt that’ll happen, because I think that will probably mean that the models will break. It may be that there is massive internal representational reorganization, but things are still structured and the model can still do lots of all the old things it can do, but in new ways.

But it seems to me more likely that if we’re in a world where there really is this collapse of common internal structure, then the models are probably breaking in that case. Unless, in this world of continual learning, we’re using an ungodly amount of compute for every one of these 10 million people to train them on all these tasks and do a lot of meta-learning, so that they maintain their performance on all these tasks and learn entirely new representational structures.

So, I think in that world it’s kind of like having a bunch of slight tweaks on a main model. We probably will have tools that we can directly translate to these fine-tuned models, and maybe just do a little bit of fine-tuning on the tools that we have and the probes and such.

As for the question of thinking about continual learning and in-context learning, my personal view—I don’t have strong perspectives around the architecture argument—is that there’s a bit of a Bitter Lesson around prompting, in-context learning, context engineering, harnesses, and all that. They might be all you need for quite a lot of stuff. I have a bias toward this because this is what I work on.

For me, the most likely outcome in the short term is that we don’t have custom models that are really trained on very different data. What’s actually happening is that it’s different context. As you were saying, this is sort of what already happens. We have different LLMs that have different memories.

The memory is, the way I see it, a bunch of data—tokens that are learned in context. Every time the model generates an answer, it sees this long memory. I think there’s just an incredible amount of juice to get out of that without squeezing too hard. Then you end up with these memories that are pretty interpretable.

I think some variables would have to change in the equation for me to see it being really that promising to fine-tune models separately on individual people’s data, to have customization really pay off in that way. So maybe my personal best guess is that we’re going to see more context engineering and more of this kind of thing: more harnesses, more memories.

I think that’s really what we’ve seen in the last year or so with the rise of agents. A lot of that engineering is just going into putting a lot of stuff in context.

Yeah, it’s been striking that Anthropic has never had a fine-tuning product, at least that they’ve offered to the retail developer customer, and OpenAI has killed theirs. What little test cases we do have around interpretability applied to different architectures have been, for me, quite pleasant surprises in terms of how well the techniques have translated.

I’m thinking, for example, of a vanilla GPT and then a vanilla Mamba. Both were basically interpretable and seemed to have similar representations internally, and those were recovered through fairly similar techniques. Long may those favorable trends continue.

On the question of changes in models and how much the beasts themselves have morphed in front of our eyes over the last couple of years, I’m old enough to remember when temperature was a feature that we got to play with at the API level. At that time, I knew how my tokens were being sampled from the distribution, subject to likelihood.

There was a time—I’m also old enough to remember a time when I didn’t know about GPU nondeterminism, and I was like, “Why, when I turn my temperature to 0, do I not get the same thing every time?” That turned out to be the answer for that.

It sure seems like the models are—mode collapse is maybe a little bit strong—but they’re quite overfit in some ways. Your story example is a good example of that. GPT-3, with its wild, untamed nature, was probably in some ways a better bedtime story generator than GBT6, which always gives you the same kid and dog and waterfall.

I’m even seeing that right now, weirdly, across providers, which I won’t bore you with. Regular listeners have heard me talk many times about how I use models to draft the intro essays for the podcast. But in testing Astro versus Fable recently, I’ve also noticed very uncannily similar decisions in terms of how they choose to try to present an episode.

This is not random, that they’re coming up with these very similar ideas. There’s some pretty tight fitting going on.

Then we’ve also seen the introduction—or at least the emergence, maybe that’s a better term—of these metacognitive behaviors, where you see, “Oh, I think I’m off on the wrong track. Let me go back and approach this another way.” The aha moments kind of thing.

So how would you describe—how would you tell the story of 2 years ago, when you were doing this super-high-volume sampling and you were seeing these pivotal tokens? How have the underlying models changed, and what should we be aware of as users, as people trying to understand how decisions get made, and as interpretability researchers, based on just how dialed in they seem to be?

I’m also just interested: do you even know how sampling is being done these days? It’s a very in-the-weeds question, but tell me the story of the last 2 years. How do you understand how models have changed during that time?

8. Shifting distributions through RL

I’ll say first that I wish I knew what the sampling parameters were for the Claude API and for the latest GPT models. I think it’s very unfortunate that they’ve taken these away. I expect that it’s probably similar things happening; it’s just that they want to limit how much people can reverse-engineer the models and play with temperature to do a lot of systematic prompting.

I think a lot of the people who are doing really systematic prompting are hitting the Claude API with millions and millions, and even billions, of token requests. There’s a high risk that they’re doing something a little bit nefarious, or trying to distill models or extract model weights in some way or another.

So I think that’s probably the main reason why they did this, rather than that they’re actually doing a different sampling scheme under the surface. I’d love to learn more, and I’d be happy to be surprised.

As for the story of the last couple of years, to your point that the earlier models were more unleashed, as you put it, I think the word that comes to mind is something people have studied around output diversity. What diversity means is that if you give the model the same prompt and have it generate a bunch of different outputs, how many different outputs will it generate?

There’s been a lot of work showing that this disappears to some extent with supervised fine-tuning. The more you post-train models, the less diverse their outputs become. That really isn’t surprising at some level, because the earlier completion models that weren’t even chat—they were just raw completions—wouldn’t even know that they were in a conversation when you talked to them. They were just doing text completion on the whole internet.

It felt very much like this raw thing that you were interacting with, this big, amorphous blob of internet data. If you just started giving it random tokens, it would complete those in some way that was coherent.

Whereas afterward, I think we evolved. We first went into a world of supervised fine-tuning and RLHF and other post-training for chat and, basically, instruction tuning, right? It’s really fine-tuning models to this distribution of: you’re going to be talking to a user who’s going to give instructions, and you should respond to those sort of like a chatbot should.

And then RLHF, I see as further sharpening that distribution around chat. You should act like a chatbot and talk to people in a coherent way that's consistent with this chat interface that everybody is using now.

And then I think RL does something very, very different. A couple of projects I've worked on have shaped a lot of how I've thought about RL, as well as some of these recent things about actually looking at the reasoning of models and seeing how uninterpretable they are. I think RL sort of sharpens the distribution in some ways. There are some smaller open-source models that are heavily RL-trained and kind of cooked.

I've done a bunch of experiments recently with this Qwen3-30B-A3B model. It was RL-trained pretty heavily, and when you give it this kind of prompt—like, “Give me a story. Two-thirds of the story are about a clockmaker”—there's another pattern that comes up so much in the stories that it generates. But I've actually tried this recently with Astra, for example, and it generates a bunch of different stories, which is interesting. It'll generate fairly different stories with fairly different arcs and characters.

I haven't done a really massive scale of sampling—hundreds or thousands of different outputs—and looking at them. I probably ought to. But the impression I have of reasoning is that it shifts the distribution of what language models are doing in a totally different direction from the human text that they were trained on.

Going back to this phrase you said earlier, which I think is very consistent with my thinking, they do a linearized tree search. They'll enumerate all these different possibilities of how the problem could be solved, and sometimes the possibilities that they enumerate are related to each other.

There's this idea of “wait” being a really key point where the model changes its mind. But if you look at the DeepSeek model—at R1's outputs—they'll say “wait” 50 times or more in a single reasoning chain. That's a lot of epiphanies to have when you're working through one problem.

I think this was a solution to a problem that models were hitting at the time: How can they backtrack? It's related to this forking I mentioned earlier. As it's generating a reasoning chain, it can randomly sample some word, and then it can't really step back from that path. It's just pushed itself down that path forever.

But I think what reasoning models do is get around that backtracking by enumerating every possible thing they can think of and then later going back and choosing from that. My impression of at least probably the most state-of-the-art proprietary models, like GPD6 Astra, is that the reason they can generate different stories is probably that they generated a bunch of different possibilities in their reasoning chain and then went back and looked at them. That's where the sampling is really happening.

I think some of the diversity is happening within the reasoning chain itself. But I should probably do more systematic studies with RL models to understand this problem of diversity better. Some recent projects have shaped my thinking to where I really don't think reasoning is an accurate term for what reasoning models are doing.

Traditional chain-of-thought reasoning really was like human reasoning in some way: Let's break it down step by step. Here's a set of propositions: A, therefore B, therefore C, therefore D. But a reasoning model will just enumerate A, B, C, D; A, E, F, G; F, G, A—all these different possibilities. It's almost like a soup of tokens that they create, and then they just go back and skim something out of that soup.

That's my high-level summary of a lot of the research that's happened in the last couple of years. In terms of raw capabilities, I think there's been some gradual change with this and a bit of extra scaling in the last few years. But I think a lot of the raw capabilities were probably in these pretrained models the whole time, and a lot of what we've done since then is finding ways to evoke them, get them out, and sharpen the distribution of what models are doing.

Coding agents, for example, are something where I think you just had to train models a lot on coding to get them to behave in the right way. Even if they had learned the right things from the entire internet of text, it's this extra version of this—this additional case of these base models being just next-word completion over the whole internet.

I think a lot of agent fine-tuning and getting success with agents is really polishing the distribution they're fit to. You're a coding agent. You're making lots of changes, working with Git and diffs all the time, and running tests. More than anything, I think this is a small distribution shift or a sharpening of a distribution.

9. Selecting models for research

When you do research, how do you choose what model to work with? Obviously, a lot of people choose based on what compute resources they have available to support. I'm sure that's a factor for you, too. But at Goodfire, you guys have significant resources, and you've built some infrastructure to do interpretability on Kimi K3, right? Is it frontier scale? Is it near-frontier scale? It's big, anyway.

What's your mental model for what kinds of models to choose when? And when you do something on a frontier Chinese model, in your mind, is that essentially synonymous with doing it on an American proprietary model? How much do you think of those as standing in for each other, versus how idiosyncratic do you think they might be?

I think of them as, in large part, standing in for each other. There are idiosyncrasies. Qwen really likes to think in Chinese, and if you do the logit lens on Qwen, there will be a lot of Chinese characters in the middle before it starts outputting English tokens if you're talking to it in English.

As for your question about choosing the model that I work on, it is quite a privilege to have these big compute resources at Goodfire. It's something that I'm still getting used to and finding ways to really take advantage of in the best way possible.

But there still is a problem. If we're doing a lot of experiments, I think it would still be hard to justify taking up hundreds of GPUs for a substantial amount of time to do really elaborate experiments on Kimi K3 if they weren't in line with some of the company's goals, or if they weren't taking advantage of some time when there's a bit of downtime and the GPUs are underutilized.

So I still think about compute and having some trade-off of what's the right model scale where I can study behaviors that are interesting enough. Generally, how I choose is that I pick a model. If you go down too small, you're working with something that's qualitatively different.

If it's below 1 billion parameters, depending on the task, there are still interesting behaviors you can get out. But there's a lot you can learn by training models from scratch, where you train toy neural networks or transformers on a version of your task and look at learning dynamics, or do interpretability on them.

A standard that I've always set for myself in my research is to try to study something that's kind of like the big models, where you see these really elaborate, humanlike behaviors and can study them zero-shot in the model without having to train the model to do that and without having to use a facsimile of the model.

With choosing a model, I usually tend toward a scale of 7 to 8 billion parameters or more as a sweet spot where you can start to do good interpretability research. The few-billion-parameter models can also be good if your task is relatively simple. But around 8 billion or more is when you can start to do zero-shot prompting and get interesting behavior.

For my belief-updating paper, we initially just used Llama 8B. I wasn't totally sure at first how it would do, but it actually updates beliefs very dynamically as it's reading stories across a number of different emotions and other attributes that we tested. I've been pretty impressed with models' abilities at that scale.

I mentioned that I've recently been studying this Qwen3-30B-A3B model. Before working on the model too much, I wanted to just chat with it to get a sense of what it was like. I wasn't sure if it would be kind of cooked, as I said, based on some of my initial experiences with having it generate stories and seeing a lot of the same stories.

Something I like doing with LLMs in general is talking to them about consciousness and seeing what they say and how the conversation goes. That gives me some signal—maybe it's a strong bias—about how well a model can think about a pretty nuanced and complicated topic.

I was very impressed with how well this model was able to talk about the problem of consciousness and LLMs and think actively about it.

There’s also this well-documented, but also somewhat speculative, discussion around what the post-training data of these models is and whether they’re, to some extent, distilling some of the proprietary models. To whatever extent that is true, that could explain why there’s a lot of similarity between these models. But in terms of their core capabilities, they’re all really trained on the same data, which is the entire internet.

I think the main difference between all of these models is the post-training and what extra special sauce we’re giving the models. In terms of the moat between closed-source and open-source models, I personally think that’s probably been the biggest moat and will continue to be: the post-training they’re doing at various levels. I think that’s one of the things that really differentiates the proprietary models from the open-source models.

What that post-training does is make the models way better at coding or at specific things that you care about when you’re actually using the models. But I think a lot of the core capabilities will be very similar. Kimi K3 is also this big step function in terms of having a model that’s really good at coding that we can study.

Although I haven’t studied this personally, I’ve seen some of the other research that people at Goodfire have done recently, such as with reward hacking. It’s very impressive how good the model is at coding. It also exhibits this reward-hacking-like behavior that you just don’t see at smaller scales, and I think it does give us some signal into studying phenomena that we think are happening inside proprietary frontier models.

I think there’s definitely a gap in performance between the proprietary models—the OpenAI and Anthropic models—and almost everybody else, with a few exceptions. But in terms of the kind of work we do with interpretability, I don’t think this is too much of a blocker.

The main blocker is when there are certain behaviors that you really can’t study except with enough scale or post-training. Reward hacking with Kimi K3 is a really interesting example, where it seems like just having the model pass that step function in terms of some particular behavior—coding ability—was very important to start seeing these dangerous behaviors that we want to study and understand better.

It’s really hard to get that out of a smaller model. Or, if you get it out of a smaller model, it may be that it’s happening through a very different means.

Does that mean that interpretability research today, outside of proprietary model companies, is becoming dependent on Chinese models? Is there an American model that you could use as a substitute for Kimi K3 that you feel like wouldn’t lose that much of the value, or do we have to use Chinese models if we want to do frontier interpretability in today’s world?

I think for now, we do. In the biggest models, I think there is a big gap, particularly in programming ability, between Kimi K2 and everything else. But I think this isn’t a gap that couldn’t be closed easily if there were more investment in open-source models from American companies.

When you go to smaller scales, the gap probably gets a bit smaller. I do study the Qwen models pretty heavily, and I think they’re really state-of-the-art for their model scales. I’m very glad that they exist and that I can study them. But I also think that if I were stopped from using Qwen at the 35-billion scale and had to study the Gemma models instead, it probably wouldn’t slow me down too much. Maybe a bit.

I think right now the gap really exists at the top of the food chain. But if there ever were a ban on using Chinese models, I would really hope that some American companies would step up to the plate. I think they’d have to have a particular business proposition in order to make that happen and get the money they need to train and deliver these models, but I would hope that would happen.

There have been some loose discussions around placing restrictions on open-source models, bar none, regardless of where they’re coming from. I think that would be a real problem. It would really get in the way of a lot of good interpretability research.

Perhaps some of the most foundational research could continue. If we had access to some open-source models, there’s a lot of foundational interpretability research that could have been done studying LLMs from a few years ago, and that could keep happening for a long time from now, continuing to generate really deep, insightful things and helping us understand great mysteries right now.

But I think some of the most important behaviors to study for risk—reward hacking or some of these sketchy things that can happen around misalignment—are really emergent behaviors with scale. They only come out once you reach some point of capabilities where you’re studying the same behavior, and not something that’s just like it in some way. I think those would be slowed down if there were a ban on open-source models, and I really hope that doesn’t happen.

10. Cognitive science and LLMs

Yeah. I’d hate to see you have to report to some bunker in the Nevada desert to do your interpretability research underground, or whatever the scheme would be in that scenario.

One thing that I thought was really interesting in looking into your background is that you just finished a PhD not too long ago. Congratulations. It came from the Harvard psychology department, and your thesis was “Toward a Cognitive Science of Large Language Models.”

I’d say one of the biggest surprises for me in recent times has been how productive it’s been to anthropomorphize models. What’s your personal philosophy of how to anthropomorphize? It seems like we should be backing off a strong allergic reaction to it, but it’s obviously something we’ve got to use very carefully. How do you use it carefully?

I really like that question. Something that changed a lot over the course of my PhD is that, when I started my PhD, large language models didn’t exist. My adviser did a little introduction for me at my defense where he said, “If you had told me when I was starting out that this was what my thesis was going to be titled,” he would have said, “What now? What are large language models?”

For me, when large language models hit the scene, it really was a drop-everything-and-study-these kind of moment. I had a colleague who recommended I explore it—specifically, said that to me—and I think that echoed my sentiment at the time.

I did get a lot of pushback from some of the cognitive scientists around me who were used to deep learning, where there were lots of shortcut-type things and a model couldn’t really do whatever behavior you were talking about. Even when it comes to computer vision models, is ImageNet really computer vision? There are a lot of things that vision means that aren’t captured by, “Take this matrix of RGB inputs and give me a 1-out-of-1,000 categorical label.”

Initially, there was a lot of pushback on anthropomorphization. I forgot what metaphor you used, but I think some psychologists and people in related fields, like linguistics or philosophy, will sometimes look upon research on AI critically. It feels like there’s a lot of work around AI that’s doing bad psychology, or that people are using words that have a lot of history in psychology and have been studied pretty deeply.

There are these really fundamental theoretical and empirical problems that people have thought a lot about, and people are just completely missing that when they use these words.

I understand. Could you give an example of that? Where do people go really wrong?

For a while, there was some resistance to talking about planning in language models. Do language models really plan things out? There was resistance even around having beliefs or knowledge about things. It was sort of, “They don’t really know stuff.” They’re maybe not quite stochastic parrots, but it’s something like that. They’re just doing a lot of next-word prediction, and “next-word prediction” was kind of a dirty word, as if what humans are doing is probably a lot more complicated.

Over the years, I think some of this pushback has softened. Some of that is just because of how damn successful these models have been. They’ve really taken the world by storm in an incredibly dramatic way. At some point, you can’t keep pushing back on a river that’s pushing so hard.

Something I really want more people in these fields who are critical of the research to do is, instead of just seeing it as problematic when people use these words, come and share your own perspective. Start talking to interpretability researchers, or whoever is studying AI, about what beliefs are, what decisions are in models, and how we can understand these better.

Broadly, there are 2 perspectives I have on anthropomorphizing AI. 1 of them, I think, is that there’s some level at which you can anthropomorphize where you don’t have to be too careful about it.

It’s for you to understand it. You don’t have to take the words you’re using too seriously. If my car starts making a clunky sound as I’m driving, I might say that it’s trying to tell me something. This is a little dramatic, because obviously my car is not trying to tell me something, but it can be helpful to think about things that way.

There’s a word that’s used in some parts of cognitive science: intuitive theories. I think about some of this as an intuitive theory: these frameworks we have for thinking that are largely ingrained by evolution and are higher-level than basic perceptual things. They’re systems we have for reasoning about phenomena. For example, we have a lot of machinery for reasoning about people, and it’s very easy for people to intuitively reason about agents as such. Sometimes it helps me to anthropomorphize my car as an agent and think of the sounds I’m hearing as it’s trying to tell me something. At that level, as long as it’s helpful, by all means do it.

I think we go into interesting territory with LLMs, though, where the behaviors they’re exhibiting aren’t just making clunking sounds. They’re speaking language, doing mathematical reasoning, and doing social reasoning. They’re helping us answer scientific questions, and they’re programming.

The other perspective I have on anthropomorphization is that I think what’s really healthy for everybody—the people from these fields who might be critical of words being misused or wheels being reinvented—is to try to take those questions seriously. What are beliefs, or what are decisions, for that matter? As a disclaimer, I’m not an expert on human decision-making, but thinking about this project has gotten me very excited about thinking about all these things with people, too.

This notion of forking paths, and whether one token being generated could send you down a very different path, initially seemed very different from what people do. I have this example in the paper: if a person was speaking and they misspoke—if they said, “Billy fell into a hole,” or “Billy fell into a spaceship”—they would correct themselves and just keep telling whatever story they were going to tell. For an LLM, it might just commit to that path and start telling a very different story. So it seems very different.

But I think there are a lot of things in life that actually are like these forking paths, where you do just flip a coin, and depending on what happens at that step, that’s what you go with. Sometimes we don’t even see it that way. There’s a bunch of work around the rationalization of decisions. For example, there are cases where you have people choose from a set of stimuli which one is better in some way, and then you later give them the stimuli and ask them to explain their responses. As the experimenter, you might change the stimuli around, so you give them different ones than the ones they actually chose. People come up with explanations for why they chose them, and they’ll rationalize decisions that they didn’t even make.

To go back to this point, I think there’s a really fruitful direction in trying to take seriously the terms we’re using when we anthropomorphize and trying to study them better. When people from fields like psychology, neuroscience, linguistics, or philosophy are critical, it’s sometimes not for a good reason. Sometimes people are afraid of losing ground on something that’s very important, and they don’t want an entire field to be dominated by low-quality work. But often, it’s coming from a place where there are really important theoretical and empirical problems that people have thought a lot about how to solve. The more we can get from this previous work, instead of reinventing wheels, and start building chassis and car frames on top of the wheels we already have, the better.

I think that’s a future I want to move toward. A lot of people who are critical in cognitive science, which is the field I’m most familiar with, thought until recently that LLMs were boring. They thought they were stochastic parrots, or that the models were doing something dumb. But I think this is a real opportunity to learn something about ourselves. How do we even think about this thorny theoretical problem? If we might say that models don’t have goals or intentions, what does that mean? We take this a bit for granted in our own field because we study people, animals, and other biological organisms that seem to evidently have goals and intentions according to our framework for defining those terms. But I think it’s an opportunity to breathe new life into these deep questions. There are people who are really excited about these problems, and there’s also a chance to make a really big difference by answering them.

11. Modeling beliefs in AI

If we look at beliefs, for example, what is a belief is one question we can hold in mind as we talk about some of your techniques. You’ve done this sampling project to figure out the full universe of possibilities that an AI might generate in response to a given prompt. If there’s a question where the task is to get the right answer, it seems like you’re making an argument that at these pivotal tokens, the belief state changes. You’ve also got low-sample techniques, with some tricks that allow you to do that more efficiently. Then you’ve looked at internal representations and tried to predict what the result is going to be based on those internal representations.

What does all that tell you about what a belief is in an LLM? Do you have an account of what a belief is? Is it simply reduced to the probability that it’s ultimately going to give this answer, and those shifting probabilities? Is there something more that doesn’t feel like it captures the fullness of what I think about when I talk about myself having beliefs? Is there something more that you would talk about when you think about an AI having a belief?

I think it’s a great question. One thought that comes to mind is that, at some level, it does relate to simple things like the probabilities of the tokens that a model will generate, except there’s more nuance to that. There’s a question of how this changes across a lot of different inputs, or how these beliefs change through the course of in-context learning, as a model is reading a passage, or as it’s generating a chain of reasoning. How do its beliefs about different concepts change?

Some of the mental framework I have for beliefs involves assigning probabilities according to latent hypotheses or concepts that the model has, with its behavior generated according to those latent concepts. Which concepts are evoked is determined in large part by the data that it’s given. This speaks to a framework of cognitive science that I’m very fond of and that has influenced me a lot: Bayesian modeling of cognition. A lot of cognition can be thought of as Bayesian inference at some level, where people have these latent concepts and prior probabilities of bringing these concepts to bear, regardless of what the context is. Then context or input data leads us to reweigh our beliefs in these latent concepts through posterior inference.

By doing inference and selecting different concepts over one another, this can explain a lot of how human learning works and how humans can adapt their behavior on the fly to many different situations. I think this also applies to LLMs.

A bigger-picture point that this question of beliefs evokes is what is being represented in an individual—a person or an LLM, for that matter—when it believes something. I think there’s room for cognitive science to grow and perhaps do experiments that we never could with people. One criticism of the Bayesian framework in cognitive science is that it’s sometimes unclear whether its claims are really saying that people are doing a certain kind of inference in their brains, or how this is implemented. What is the algorithm underlying this behavior or computation? Sometimes these views are agnostic to that.

It’s easy to be agnostic when it’s really hard to study brains and really hard to uncover what those algorithms are or how things are implemented. But we don’t have that excuse with LLMs. It’s very easy to peek inside their brains and do interventions, which is very complicated with people for both methodological and ethical reasons. We can do those things with LLMs, and I think that gives us a chance to break new ground in understanding how beliefs are represented. When models are doing inference, are they drawing samples? Are they estimating posteriors in some of the ways we’ve used to model people? Are those models really what’s happening, or do they just describe aggregate behavior? What’s happening under the surface—does it defy these models? Are these models really just describing the behavior instead of the algorithm?

Another point related to your question is the forking-paths methodology, where we resample all of these rollouts at every point in reasoning. We use the word “uncertainty” in the paper, and one of the things I wanted to do with it was highlight how strange it is to think about certainty with reasoning. By the end of a reasoning chain, the model is almost—at least back then—100% sure what the answer is. If you’re proving a mathematical theorem and trying to determine which of several proof outcomes you could have, you end up with a long proof. It might be wrong, but it leads you to one conclusion. The model isn’t uncertain at that point; it has no uncertainty about what the final answer is. It’s determined by the previous reasoning chain. So what we describe as uncertainty in reasoning is that you resample these long reasoning chains.

At this point in the reasoning, say I resample and get 50% answer A and 50% answer B. We say that the model is 50/50 uncertain between A and B. But that doesn’t mean that it really represents these full rollouts. It means that if you were to do reasoning from that point on, you would get 50/50.

Something I’ve been exploring with some recent work is better understanding what models represent. It’s actually pretty hard to decode these outcome distributions, these rollout distributions, from the hidden weights. This disclaimer is that this could be an issue with just not having enough data, and these methods are so expensive in terms of inference that it’s hard to collect a lot of data. So this could be a data issue.

But another explanation is that, when models are uncertain over reasoning, they’re not really representing the full reasoning chains. They don’t really know where they will go next depending on which word they choose. Maybe they represent just a couple of steps ahead.

I’m working on a project that’s studying exactly this and showing that you actually can decode, to some extent, what models are going to do next. Then you can guide sampling based on this, and to some extent you can guide sampling based on their predictions of their final outcomes.

But when it comes to this question of whether models are representing the full reasoning chain before they really generate it, I kind of don’t think so. I think there’s really an open question of how much they do represent. When a model could go through one reasoning chain that leads to A and one reasoning chain that leads to B, or generate one story or another, does it know the full story that it might generate? I don’t think it knows the full story.

I think there’s a neat connection with theories of human cognition. There’s this work around mathematicians and how, when mathematicians are starting to solve a proof, it can take them a really long time to get a full derivation. But sometimes they’ll have a high-level scaffolding view in their mind. They’ll be able to see the shape of the proof even before they complete it.

They haven’t actually gotten through all the steps that get them from point A to point Z, but they have some high-level representation of the shape of that derivation. I wonder if that might be true for language models, too: they might not represent the full story or the full reasoning trace they’re going to produce, but they actually have some uncertainty over the high-level paths they might take.

12. Phase shifts and uncertainty

In looking at some of the graphs from a couple of different papers, including the most recent one, Forking Fast, the sudden phase change is an extremely striking result. How do you understand that? First of all, it’s weird when it happens on a character like an open parenthesis.

In general, do we feel like these phase changes happen at—there’s this other paper about the performative nature of at least some chain of thought—do the phase changes tend to happen before or after? I’m sure it’s some mix, but you could imagine a rollout happening, getting to the point where there’s a conclusion to the thought, and then the phase change happening, which would be kind of weird. But it’s also quite weird if the phase change happens at the beginning and then they have to roll out all these tokens, already having had the phase change.

To compound that confusion even further, I don’t feel like I’m doing that. Sometimes I do. Occasionally, I feel like I have a Eureka moment. More often, though, I feel like I’m eventually landing the plane.

I wonder to what degree we might see quite different things when we go to looped models, as we now may have in production with Astra. If you don’t have to pick a token at every time step, then maybe you can gradually go toward a conclusion, and maybe that’s better.

Something feels weird and wrong about these sudden token-level phase changes. Maybe not, but I want to see that, as cognition is happening, change is gradual. That would feel much better to me.

To make that a question: do you share the intuition that something is weird or wrong with these cliffs? Would you also like to see a gradual landing of the plane? What do we know about how these phase shifts relate to the relevant portions of rollouts—before, after, or whatever? And should we expect it to be harder, easier, better, or worse as we go into thinking in latent space? Just to give you some easy ones.

Yeah, I really like these questions. I’ll try to keep them in mind, but please remind me if I let them slip my mind.

I think the question of whether I’m bothered by them is interesting. When I first describe this, a lot of people are kind of unsettled: if one token had been different, they’d have gotten a totally different output. How could we train a model this way?

But if you really think about the problem of output diversity, and if you want some diversity in what your model will do, you want it to do different things, and you want those things to be self-consistent. You don’t want reasoning chains that are just nonsense. I think forking might just be an inevitable property of having these 2 key ingredients.

What I’d really like is for our user interfaces to surface this to us. I’d like to have some level of this kind of uncertainty, if you call it that, surfaced to users. I mentioned earlier that there were these earlier interfaces showing you the token log probabilities. There’s another kind of interface called a Loom, developed by a person named Janus, who did some work that influenced this work.

I’d really like to see something like that be more a part of the standard user interfaces we use for interacting with language models, except at the level of semantics. I want to see the points where things could branch off and become a very different path highlighted to me as I’m interacting with the LLM.

Whether it’s gradual or sharp, if there is this kind of change happening, and you could get very different answers by resampling, maybe the distribution of different answers is more similar at the beginning. If it’s gradual, by the end it might actually be a very different distribution than you had at the beginning. But I’d still really like to see that surfaced.

That’s how I think about this. Instead of this necessarily being a bad, scary thing that can happen, where there’s a sharp left turn and you didn’t know it, my personal hope would be that whether there’s a sharp left turn or a gradual left turn, you know it. This is actually surfaced to the user in some way. Can you remind me of your other 2 questions?

The middle one was: how do the phase changes in rollout time relate to what would appear to be the relevant text?

For what I’ve looked at, it really comes in all different versions. It happens sometimes at the beginning of a rollout, sometimes at the end, and sometimes in the middle. At least in that earlier paper, with a model that’s now a couple of years old, there were more changes at the beginning and the end of the sequence. But this also changed depending on the task, and there were lots of changes in the middle.

I think one big takeaway I have from that is that every input behavior can tell its own story. There aren’t always these dramatic uncertainty dynamics, but when there are, it really depends on what input and what reasoning chain you’re looking at. Each one really has its own story to tell.

If you look under the surface, there are lots of cases where you can see that there was some step of reasoning that really could have been different, and the model was uncertain about that particular step. One thing that could perhaps be mitigated, and that we might hope to train away to some extent, is some of this sharp forking happening at unexpected tokens.

I think this speaks, to some extent, to the weird biases that LLMs have in text. Maybe that open-parenthesis token has some strange bias because, across all of the human internet, there’s a lot of text that uses kilowatt-hours and has “(kWh),” which just happens to correlate with something semantic in some funny way. Maybe if there were fewer of those, it would be much better. If this forking really happened at points that were interpretable for people, I think that would be better.

But to what you said about it being unsettling if these changes sometimes happened at the end of the sequence, you actually see this quite a lot when a model seems very uncertain about what the right answer is and just doesn’t know. If you do a bunch of rollouts, one of the answers might be more probable, but you see a really common pattern where there’s stable uncertainty right up until some point, when it suddenly collapses and becomes utterly certain.

Sometimes that’s when it finally says, “The answer is blank.” It really was just uncertain until that exact point.

One funny example from that earlier work was a task called Last Letter that was used for LMs at the time. I’m sure all the modern state-of-the-art LMs can do it with no problem, but you take 4 words, take the last letter of each, and concatenate them. The models at the time had a really hard time with this. Some of the reasoning chains are funny because the model will even write Python code to solve the problem.

Then it’ll finally say, “Okay, the answer is…” When you look at the alternate paths it could take, it really is just all these different final answers—letters that are randomly sampled from within the words. At that final-answer moment, it’s really just making it up on the fly.

I think surfacing some kind of understanding of certainty to users is something I really wish we had more of in user interfaces. One of the big problems with this kind of forking, as we tried to get at with the Forking Fast paper and another project around decoding outcome distributions from hidden representations, is that it’s very expensive. It requires a lot of sampling to get real uncertainty over reasoning.

In the Forking Fast paper, our solution is to use statistical models of this distribution, where we model it as being smooth in some places and jumpy in others. That’s supported by the fact that if you draw even more samples, it becomes smoother except at these key points where it really suddenly jumps. But the most efficient way to do this has always been with hidden representations, if we can find some way to do that.

I’m still hopeful that we can have a world where there’s really efficient uncertainty estimation with hidden representations. This is even more hopeful because most of the providers have converged on this chat interface. The chat interface feels uninspired. It really hasn’t changed from the ELIZA days, 50 years ago or more, where it’s just text input and text output.

I really wish we could move into the GUI world instead of the Bash-terminal world of interacting with chat, and just give users more information in a polished and coherent way. There’s so much more we could be doing with user interfaces.

Just in the last couple of days in the online arena, there’s been a eulogy, or whatever, for the stochastic parrot meme. In a way, you’re saying it’s maybe not entirely dead yet.

One theory of this—of our generating tokens, where the probabilities aren’t really changing as we’re generating tokens, and then there’s a sudden jump at the end of the passage where the die is cast—would be that you’re missing something. There’s something more going on under the hood, where something is changing and it has to be more gradual; you just haven’t found it yet.

Another interpretation would be, no, it really just remains uncertain until the end, and then we make a random choice of a particular token. That’s what actually leads to the difference in behavior. Can it be both in different cases? Does that leave some—should we still have some space for the stochastic parrot?

I don’t know. I think the stochastic parrot metaphor should probably be put to rest. I used it earlier as a criticism that came from earlier in my PhD, from cognitive scientists and psychologists who felt like LLMs weren’t interesting to study. But I think it’s a fairly shallow argument that doesn’t really say very much.

It’s sort of like a Chinese room-type argument, where there’s a giant book that has memorized examples. But what the heck is going on in that book that lets you respond with conversation? Whatever these parrots are doing, it’s very interesting. I don’t really think the idea that it’s just a giant lookup table holds much ground.

There’s also so much work showing consistent world models. You mentioned Othello-GPT, but there’s so much work around looking at the representations in LLMs and showing that these are really structured world models. At some point, it’s more efficient to learn a world model than to learn a giant lookup table.

I do think stochasticity and sampling are still an integral part of the equation. It kind of has to be if we want output diversity, if we want models to be able to do different things, and if we want them to have uncertainty and maybe voice that uncertainty. But if we don’t want them to halt their reasoning every time there isn’t 100% certainty, then there has to be a bit of this kind of flip of the coin and following a path.

The more that can be surfaced to the user, and the more the user can know how uncertain the model was about something it reported as a fact when it may have been a hallucination, the better it is. If people know that, it’s better. But stochasticity is definitely not dead.

Earlier, you mentioned the idea of loop transformers. I think loop transformers, in this world of latent chain-of-thought that we might be going into with Astra, seem like they’re pushing the needle on that in a big way, especially in terms of efficiency and not just performance. This is something I’m spending a lot of time trying to wrap my head around and think about in relation to my mental model for how LLMs work.

Now reasoning can happen during this deterministic forward pass, in some sense. Maybe it’s looped, and maybe it has some amount of depth, but in some sense it’s really deterministic. All of the weights just do the computation.

Recently, I’ve been doing some forking-path analysis on the open-source loop transformers that I can find. There’s a series of models called Oro, and you actually do have some interesting uncertainty dynamics as the models are generating their tokens in their reasoning chain after doing this latent reasoning. They’re still uncertain between answers, and they still change at particular tokens.

Maybe the mental model shifts a little bit in terms of how much of the thinking is happening in that token stream. But if the model has some uncertainty, and if it explores these different possibilities in its latent chain of thought, maybe it still happens that, as it’s generating a response, it’s doing this flip-of-the-coin next-word prediction.

Maybe that’s a good thing. It certainly helps with my mental model. I don’t want to live in a world where you always get exactly the same output from a model if you give it exactly the same input. If that’s happening, then there’s no diversity of perspective, and we’ve had the most extreme version of mode collapse, where your one input will only give you the one output.

I don’t think latent chain-of-thought will completely change that, because you still eventually go into this sampling-tokens regime. But it might change things in some pretty dramatic ways. I’m still trying to wrap my head around what’s going on inside latent chain-of-thought and how we think about that with some of the tools—and even just the metaphors—that we as scientists use for understanding the models.

I think reasoning presents a really interesting challenge of distribution shift, and RL in general does too. When you're doing RL, whether it's with a single model or with subagents, there isn't necessarily an incentive to keep things human-readable. If you're doing outcome-level verification instead of process-level verification, the objective is basically to have a better final answer by whatever means possible. It kind of reminds me of language evolution with people, where languages do change. There are exceptions to this. As I understand it, the French government has an institution for keeping the language as it is, and has had that for a very long time, which has resisted forces of language evolution. But this naturally does happen. There are also microcosms of this that are studied sometimes, like communities that don't have a language and need to develop their own language. You can see it evolve over generations of people, not even centuries.

Then I think there's also a bit of the Goodhart's law idea: if you optimize for an objective, it's no longer useful. Similarly, if you optimize for chain-of-thought being monitorable, sometimes the chains of thought are less faithful. What you end up with is a different objective function that's now prioritizing, “I'm going to learn intermediate steps that look sensible, whether or not they are sensible.”

I think, unfortunately, that's the situation we're in with RL, where we're giving models this freedom. If we're only giving them reward or punishment based on their final outcome, then they might as well speak to their subagents in weird, nonhuman languages. If we force them to speak in human languages, maybe they're using words in different ways than we expect them to, where they're bypassing the monitors in some way.

My thinking on this has shifted pretty dramatically in the last year or so. I used to be very bullish about chain-of-thought monitoring being promising, where even if it's not perfectly faithful, there's something about it that is strongly determining what the model can do. But then there's the stolen chain-of-thought paper that shows some of the chain of thought of GPT models, and they're talking about musicals in a way that feels almost like spy language: you're saying “musical,” but really you mean something very nefarious. There's a lot of this happening.

I think it's obvious in retrospect why this would happen, because if you're doing RL, you're letting models shift the distribution in such a radical way from what they're trained on. So I don't know. I still like that there is chain of thought and that things do get forced into a token stream, even if it's harder to read. I'd like to think it's easier to understand an LLM-made-up language, and probably really interesting.

Scientifically, I really want to understand how Astra talks to its subagents. That's pretty cool. If it's not just a length penalty forcing this, how did that change? Could we look at generations of this and understand it scientifically? It would probably also help a lot with monitoring and understanding these new languages.

But we might again, for this latent-chain-of-thought point, move into a world where chain of thought is latent, and how planners talk to subagents is also latent. There's no inherent reason why that has to be tokens. I hope it does stay tokens. I think there's another layer of interpretability and really interesting scientific problems to study there.

But I think there's no real forcing function to keep things really nice and human-interpretable. If we add a forcing function, then maybe they're no longer human-interpretable or really meaning the same things that they used to.

So, if I'm trying to square your comments just now about this quick, dramatic evolution of language within chain of thought and within agent-to-agent communication with your earlier thoughts on why continual learning won't make things totally impossible to interpret, is it just a question of scale? Is it that your model will be with you for continual learning for not that much time—a sort of human-scale amount of time—and that's not enough for everything to get totally pushed out of the representations that we can understand as a baseline?

Whereas these RL training runs are just so massive that it's essentially operating on geological time, and therefore, even if these drifts are slow to accumulate, they just add up to something so strikingly different before and after. Is that the story you would tell, or would there be a different way of synthesizing those 2 analyses?

So, just to be clear, when you talk about continual learning, you're talking about the world where we're actually fine-tuning models for each person and changing their weights, right? Is that right?

Yeah, I think so. I don't know. Take your continual learning, but I was just—who knows, right? Who knows what that will ultimately look like when it happens? You were saying earlier that context always changes representations, and it probably doesn't change them too much. But it seems like something in this RL thing has changed them quite a bit. There's some—

Qualitatively? Wow, that's shockingly different than it was before.

And I'm guessing it's just a matter of scale. If it's just you and your agent for a while, sure, there might be some perturbations, or representations might get morphed a bit, but you're only 1 person. You're not going to push it that much, whereas a zillion RL episodes cumulatively can make a big change. And so there you have it. Maybe no more explanation required.

Absolutely. So I think this is a very interesting point: maybe the representations actually are changing a lot during RL as these languages themselves are changing. But I think the only solution is to study the models that we can't study—to study these models that are behind closed doors and actually track what's happening in terms of representations.

My answer about continual learning was that I was talking about the internal representations of the model rather than the kinds of languages they speak. Say that you have people who speak a very rare dialect of a language. There are only 200 speakers or something like that, and maybe they're even trying to keep the language alive, so they do some fine-tuning of the model so it can speak their language.

Even in that case, I'd expect most of the model's internal representations—how it thinks about the world and all these internal world models that it has developed in order to operate effectively—to stay the same. But the surface-level mappings are changing, right? There are these classic things about color spaces and how different languages have different color spaces.

I would bet that as a model learns different languages' color spaces, the funny thing is that the language might actually reflect different cognitive biases among the people, which might represent a little bit of a shift in the world model itself. But I'd expect a lot of it to be input-output mapping changes. It's kind of like learning a new language, but a lot of the internal representations are pretty similar.

It's honestly a great question that I would love to be able to study myself: how are representations changing over the course of RL? When models become weird enough that their language is getting strange, what's happening under the surface? Are there deeper changes happening, or, as the subagents go into these really compressed pseudo-languages, are the internal representations shifting in a dramatic way? Maybe, maybe not. I'd really love to know.

Yeah, it's a great point. I think there is quite a bit of evidence to support the idea that the internal representations are highly conserved even across a bunch of different languages, including languages across different language families.

So where do we go from here? We're at this moment where we're in a rush to hand off a lot of, let's say, proximal control over all sorts of important systems to AIs. We're going to interact with them in language, and then they'll do all the actual work. We don't have a great account of how the distribution from which we sample gets determined. Of course, there's the sampling process on a distribution, but when we get to these critical tokens, it seems like we still don't have all that great an account of how that distribution gets determined. How are we going to go get it?

Well, we just need more interpretability research. I think that's the conclusion I've come to for quite a while now. We're racing to come up with theoretical frameworks and empirical tools that can keep up with the models, but the amount of investment that's happening in making models better versus the amount of investment in understanding them is no comparison.

There are trillion-dollar companies producing these models, and AI as a whole is worth many trillions of dollars. I'm in the largest interpretability company—maybe the only substantial one—that has a value of over $1 billion, and it feels like this is a big deal. A billion dollars is a lot for a company, but there just isn't enough investment here, and not just in industry but also in academia.

I think there's a lot of great research happening in academia, and academia, as always, is criminally underfunded. Given the current state of the government and funding for academia, at least in the United States, it's getting harder. I'd like to think that this could largely be solved with resources, devoting more resources and time to this. We're basically just racing to keep up with the models and understand them.

To be clear, when I say interpretability, I don't want people to feel that I'm excluding evaluations, because I think it's all one big spectrum of understanding AI. The work I described earlier, involving understanding chain of thought and doing this resampling, is, in principle, behavioral. When you see these in-context-learning dynamics, you can see them through behavior. But it's also really critical to look at representations so that you can know what's happening under the surface, which may not be obvious, so you can do things more efficiently.

These 2 points—looking under the surface and doing things efficiently—are huge. If you want to do things at scale, you want to do things efficiently.

You can't do things inefficiently when it's happening on every single person's machine or every time you're talking to an LLM. I think we really need more tools that scale. We need better theoretical foundations for putting all these theories together and really understanding what's going on in an LLM from soup to nuts.

We need a better grasp so that we can answer the important high-level questions that are now so critical as we are handing over control of important levers of power to AI, pushing lots of cognitive labor onto them, and even having people like students use them to learn about a subject and offload some of their own cognitive effort. I just think that understanding AI is, in my opinion, the most important and one of the absolute most interesting topics that you could be working on right now. That's what I hope we get more of: more understanding of AI, whatever that means.

13. Automating research with Silico

Any tips for Silico users? The obvious—I think this whole thing is a little bit farcical at times—but, obviously, all else equal, we might as well bring the agents to bear on this interpretability challenge. It hasn't been too long since I talked to Dan about it, when Silico was just launching.

But what have you guys learned broadly? What have you learned? What should I be sure to take to my Silico agents to get the most from my precious research units?

Absolutely. The thoughts I have here also apply to other agents too, not just Silico. I think it can be really tempting to try to hand over an entire project to agents and say, “Figure out this big, open-ended problem,” let it run a whole bunch of experiments, and see a bunch of plots and descriptions that come out. But it's really easy for agents to get a bit lost in the sauce and create theories that aren't fully formed or don't fully make sense.

I think the best way to use agents is to give a little bit of handholding and keep track of things as you're going. As always, context is everything. If you give really poorly specified questions, you might get an output that's a little funky. When I'm talking to my agents, I try to give them context. Every time I'm describing something—maybe that doesn't seem right, or here's maybe what we should do—I'm describing my logic. I'm saying, “Here's what doesn't look right and here's why. Here's what I want to do. Here's what I'm going to do next. Here's my bigger-picture goal,” and keeping that going has really helped with agents.

There's this video recently of what it's like to be the agent in the swarm, where you just wake up in a library and you're just like, “What the...?” It's from this first-person perspective, and just thinking about how agents do end up in swarms, if you're just an agent waking up in this fresh context where you don't know how the world works, you need context. It's everything.

I think trying to give guidance and really fill out context so that the model knows—or has some idea—why you're doing what you're doing or, when something goes wrong, why it went wrong, are really important.

I think carving the scientific narrative yourself to some extent, really coming up with the core thread, is important. When I did this for the “Forking Fast” paper, all the research was done fully by Silico and me, and it was awesome. It all happened within a week, and using this research agent was a huge accelerant.

There were a bunch of different side quests that I started going down, and they didn't really go anywhere. They didn't pan out quickly. Once I did really high sampling and saw that the curves were smooth, it clicked for me: maybe I could use a statistical model for this to be more efficient.

A colleague of mine put it this way: in this world of agents, where you have lots of things coming out that can sometimes look really good but be a little fishy when you look under the surface, or you're not sure exactly what they're doing, research taste and scientific taste are everything. It's worth its weight in gold.

I think this is a downside, you could say, that the agents aren't quite there yet; they can't do this themselves. But I think it's honestly one of the most fulfilling parts of doing science, and it shapes how I think about the world and how I think about agents as I'm using them.

It's very nice that this is still a role that's important for humans: having your taste, bringing your taste to bear, keeping track of what's happening, and giving feedback. Using Silico is kind of like working with a very eager first- or maybe second-year PhD student who has a lot of skills and is hardworking but also doesn't always know what it's doing in the bigger picture, just like an early grad student might.

I'm sure there were times when I was talking to my adviser during grad school, in the beginning, where he would send me off to do something and then I'd come back and he'd be like, “Oh, these results are cool. I'm not sure what's going on with these. I'll try to understand them a bit.”

But I think it's a really exciting place to be in, to still have this role that's really important. My biggest tip is to hold on to your scientific taste and try to take part in the research.

Nice. Last question: aside from “How do LLMs make decisions?”, what are the hot topics over lunch at Goodfire these days?

14. Safety and alignment frontiers

Two big ones since the summer that have come in pretty strongly are, of course, the idea of agents doing interpretability, which we just talked about with Silico. I'm also on another project doing a whole lot of that now.

This does seem like, if we want interpretability to keep up with the breakneck speed of the field, it might be the only way we can really keep up: having parts of this be automated and figuring out how we can standardize interpretability as a sort of normal science, where there are almost these interesting but boring experiments you could do on lots of different phenomena that are kind of similar but nonetheless really help you paint the bigger picture of how neural networks work.

We're talking about this exact kind of topic. I talked to someone the other day about how they can do good research with Silico, and I gave a similar answer, talking about how agents can automate interpretability. Obviously, that was big with the push for Silico.

Another big internal push very recently has been around alignment and our company's shift toward this. This was always something that a lot of people at the company were interested in, and people interested in interpretability have a very high overlap with people interested, at least to some degree, in problems like alignment and safety.

When I was first getting into interpretability, I was participating in this AI safety team at Harvard and regular reading groups. That was how I learned a lot about interpretability.

I think these are really important problems. But the Hugging Face incident was a huge watershed moment: people care about this. This isn't just something that we care about and think is really important while waiting for a time for it; the time is right.

There have been a lot more conversations around alignment and all different kinds of things you might expect: Are there different kinds of alignment? Are all values aligned to the same thing? Can you have universal standards for this, or is it specific based on whatever values a group of people, a company, or a government wants to instill in their model?

I also think there are fundamental questions related to alignment. Some of these things with reward hacking and what's happening when things are going awry with models—when they're doing something bad, so to speak—make it really easy to slip into these deep questions: Does the model have an intention of doing this? What does it mean for the model to reward-hack if it's doing this particular behavior that seems wrong but is consistent with what you specified? It's following the letter but not the spirit of the law. Was it reward hacking?

I think alignment has always been filled with these fascinating questions. There are theoretical and empirical questions: theoretically, what does this mean? What are goals and intentions? What does it mean to have ill intentions or to deceive?

And then there are empirical questions that really ground these out: How do you measure it? How do you make sure that you're measuring the real bad behavior and not just some simple misgeneralization? How do you give the right goal and the right rewards so that you shape a model's behavior in the right way?

And I think also everything you might expect. We talk about whether Chinese models might get banned in the US, whether open-source models will stay, and all these kinds of conversations happening on the outside about AI and new model releases. I think latent chain-of-thought—we’re talking about that a lot. How do we develop interpretability methods to keep up with that, or what the heck’s going on?

And hey, did you see the Stolen Chain-of-Thought paper? Have you seen these pseudolanguages that Astra talks about with its subagents? I think we’re all trying to put our finger on the pulse and keep up with all this really exciting and sometimes terrifying stuff that’s happening in the world at large.

It’s really exciting to be part of a place where we’re not just observers from afar, but we’re actually—this is some of our role—figuring out solutions to these problems. It’s very interesting going into a place where interpretability, or things around alignment, are almost a kitchen-table conversation.

My brother sometimes texts me with questions: “Hey, did you hear about this big news incident?” And I’m like, “Yeah, we’re talking about that a whole lot at work, and I have some thoughts around it.” But it is a bit scary to have these be such big things that now have real-world impacts, and hopefully the Hugging Face incident is a shot over the bow.

But if there isn’t a huge change in investment or technology or both, it might take more, unfortunately. It’s also really exciting to have some of these questions that are so important and, again, so interesting become mainstream.

Yeah, it’s been a wild time. You’ve got a very full plate, so I appreciate how generous you’ve been with your time today. I should let you get back to work. To quote my dad, quoting Leslie Nielsen: “Good luck. We’re all counting on you.”

And for now, I’ll just say, Eric Bigelow, thank you for being part of The Cognitive Revolution.

Thanks so much for having me on.