硅上思考：Goodfire CTO Dan Balsam 谈概念流形与月费1,000美元的 ML 研究Agent

1. Predictive data debugging

Dan Balsam, CTO at Goodfire, welcome back to The Cognitive Revolution.

Thanks for having me. I'm excited for this. You guys are prolific as always, and we've got a lot to cover: research and a new product, which is in turn a research platform product. The pace is really relentless. Let me ask you that to start with: how are you holding up in the eternal sprint that is the AI game these days?

I think we have a really incredible team at Goodfire. Everyone here really believes in the mission and is working really hard, and that's always extremely motivating. We've been pushing really hard to get through our product launch with Silico, and it's nice to be able to take a deep breath now on the other side of that. But there are even more cool things coming soon.

Well, let's start with some research. I'm always amazed when I think back to the kind of toy models of superposition we had only 3 years ago and how far we've come. There were a few things that jumped out on the Goodfire blog that I want to run through, and we'll have to do it at a high level because there's too much to cover. We used to do deep dives paper by paper, but we'll have to go a little more superficially today.

One that made some waves was called “Predictive Data Debugging.” For this one, I want to give you my interpretation and then let you elaborate on that, tell me where you think it'll be particularly useful, or explain what you guys have seen since the paper came out.

My synopsis was that, basically, if you have a way of interpreting a model, like an SAE—and we'll get into featurizers a little bit later as well—then you can run a bunch of data through it, such as your fine-tuning or post-training data set, and look at what concepts are coming up as active when you put that data set through. The insight is that there's a strong correlation between the concepts that are active and the concepts that are being modified by the training process.

I think that right there is something to file away as something to remember. It's not shocking, but it's notable that the relationship is quite strong. When you see these concepts that are active and know that those are the ones that are going to be modified, you can look to see whether there are any concepts that are strange or surprising, or that you don't really intend to be monkeying around with given the data set at hand.

If so, you can quickly zoom in on the data points that have caused these features to come up. You might find that there's some stuff in your data set you ought to think twice about. Maybe you ought to filter it or modify it. This gives you a route to hopefully minimizing unwanted surprises in the behavior you get from your post-training or fine-tuning work. How did I do, and what more should I know?

No, I think that sounds about right. One of the intuitions here that's important—and I think this is a theme that underlies a lot of our work—is that there's a good amount of evidence that most of what a model knows, including most of its knowledge and capabilities, comes from pre-training. What happens in post-training, including RL, is mostly making low-likelihood events from pre-training more likely.

This is somewhat debated, but I think it's a view that we think is mostly true and informs a lot of how we think about things. Because there are only relatively small nudges in the weights happening in any type of post-training process, most of the raw capability to produce whatever outcome post-training is going to produce already exists in the model in some way.

I think we're now in an era where models are being RL-trained so heavily that this isn't true, but back in the days of RLHF-style post-training, I think it was pretty well known that base models were sometimes more capable than their instruction-tuned RLHF counterparts. There was a little bit of mode collapse happening, so one way to think about what's happening in RL training is that you have a little bit of mode collapse in the instruction-tuning phase.

You have these base models that are extremely capable, but they're very weird and very hard to prompt. You make them into a format that has a more intuitive, human-facing API, but then you want to bring back out some of the capabilities—or reinforce some of the capabilities—that the model actually learned during pre-training. It's not 100% true, but I think it's directionally true.

With predictive data debugging, the idea is that you can look at what a model is already thinking as it's looking at some data, and that's relatively predictive of what that data is going to reinforce in the model if it were trained on it. Looking at the features and understanding that those features correlate with downstream behaviors means that, if there are features that are surprisingly upweighted by some data, it can be fairly predictive of whether the model is going to learn some off-target effect from that data.

2. Outro

One of the most interesting things from that paper is that we explored a bunch of different mitigation methods. We looked at reward shaping, which involves a reward process that comes from the activations of the model itself: “Learn what you're going to learn from this data, but maybe receive a penalty if you're learning this particular feature.”

My favorite thing from that paper was that the researchers involved showed there's a pretty deep isomorphism between those 2 things. They're 2 sides of the same coin: filtering the data and reward shaping achieve approximately the same effects and approximately the same amount of off-target effects.

So, if you have a behavior and you do some predictive data debugging, and you think the model's going to learn something you don't want it to learn, or its behavior is going to change in a bad way, your options could be to filter your data if you have enough data, or to intervene in the training process in some way.

We've expanded on this since then, and I think it's pretty exciting to think about applications to RL, for instance. We did it with DPO, but with true RL, the difference is that some rollouts may contain information that you don't want the model to learn, even subtly.

I don't know—what are you referring to?

Yes. It’s topical. Being able to say, “Actually, we want to discard this rollout,” is pretty valuable in and of itself. But what we want to be able to do, and what we think is roughly equivalent over time, is actually intervene in the model and say, “Hey, here’s a circuit, or here’s a feature, that the data is trying to upweight that we obviously don’t want it to try to upweight,” and so we intervene in the training process in that way.

We’ll come back to this probably toward the end. I have some zoomed-out, big-picture questions for you. One of the challenges, obviously, with trying to develop techniques that you hope will be relevant at the frontier is that there aren’t too many open-weights models that you can hack on that have the intensity of RL going on at the frontier labs, which is leading to these colorful, problematic behaviors that we’re seeing. But at the same time, it really amazes me over and over again that astounding work—including the Cameron Berg paper that I think about all the time, about the anti-correlation between deception and role-playing features and claims of subjective experience on Llama 3 70B—is 2 years old. So, are you guys able to see features that you think are the relevant features leading to these relentless hacking behaviors?

Well, I think that’s an active area of study for us, and something that we hope to publish more on in the future. For what it’s worth, I actually think the gap between open and closed models has shrunk quite considerably. I use Kimi K3. I use Opus, Fable, Soul, and Kimi K2 for different things.

We’ve built the interpretability infrastructure and the training infrastructure, which are now all in our product. This lets us scale things like predictive data debugging to models like Kimi and GLM, where we were able to replicate the same results at that scale. Now, it’s speculated that most of the compute that goes into training new models is coming from RL and not from pretraining.

3. Concept manifold geometry

There is certainly a lot of scale in RL if you want to get to the frontier level, which is quite expensive and difficult to do. But we have the raw infrastructure for it. I think we’ve bridged the gap. The reputation of interpretability used to be that it was something you did on toy models, and I think now we’ve built—and are making accessible to the world—the infrastructure to do this at least close to frontier scale.

Another thread that has been, I think—correct me if I’m wrong—but it seems like the biggest thread, in terms of the number of papers and blog posts that have come out recently from Goodfire, is around trying to figure out the more detailed geometries that models use to represent concepts. I think we’ve covered in the past the linear representation hypothesis, which I would summarize in super plain-spoken terms as: models basically represent a concept as a direction in their activation space, and the intensity, or the salience, of that concept is represented by the magnitude of the vector that points in that space.

Now you’re complicating that quite a bit, and we’re going well beyond these individual directions in space and finding all kinds of different geometries. Some of them are pretty intuitive, like the days of the week being a circle, but some of them get pretty exotic, like some of the protein-model manifolds that I’ve had the chance to look at in preparing for this.

So, for super-quick starters, what’s the headline? If I gave you the 6-to-9-months-ago version of what we should understand to be going on with the linear representation hypothesis, what is the new short version that everybody can take home and recite to themselves to make sure they have a good working understanding?

Yeah, in many ways I think it’s just a generalization of the way we were discussing things before, and different people define the linear representation hypothesis slightly differently. I think the most defensible version of it is just saying that features are linearly decodable, which I think is essentially true. It doesn’t generally require nonlinear computation in a model for the model to read out a feature from the residual stream.

But I think where the geometry components come in is that the features aren’t the naive, maybe SAE-built take on things. You could imagine that the model is encoding a bunch of totally orthogonal concepts, and so it’s really just a bunch of one-hot-encoded categorical features, with the magnitude of the feature corresponding to how much the model is thinking about it.

In actuality, the structures that we find are significantly more complicated. I would think of a model more as a sparse mixture of subspaces. You’ll have subspaces for different types of concepts. Maybe you have your days-of-the-week subspace, which itself lives in a more conceptual calendar-time subspace. So, at different levels of resolution, you have these different structures.

The geometry of those structures is really important because the geometry encodes what operations you can perform on them. It’s sort of like the semantics of not just the individual concept, but the concept space, are encoded by the relationship of those concepts with each other in some sort of geometry.

To make things extra confusing, the relationship, the operations, and the mappings that are performed from one concept to another act as operations over those manifolds. So, you map a manifold to a different manifold over some computation. The naive version would be: you have the days of the week—Monday, Tuesday, Wednesday, Thursday—and there’s just somewhere in the model that knows that Monday goes to Tuesday and Tuesday goes to Wednesday, all the way around.

But that would actually be a super-inefficient way to represent the days of the week, if you think about it. That would be a very if-statement-spaghetti-code way of representing it. The much more efficient way is to represent it as a wheel.

In this world, the magnitude along some direction often does correspond to the model’s certainty in some way. So, if a feature—if Monday—is activating very high, then the model is very confident that it should be thinking about Monday. But the relationship between all those days is itself a very expressive and rich thing.

I think the thing that we just believe is that you’re not really going to understand the model if you don’t understand the relationship between the features. It’s maybe the difference between understanding the periodic table and understanding chemistry. You can have all the individual elements, and that gives you some information, but really, the way in which they combine and the structures in which they form—that’s what can start to help you gain a sense of the complexity of the world.

I’m fascinated. Now that you say that, I’m wondering if the structure of the periodic table would be recoverable from a model that was just trained on a bunch of raw chemical data that never knew what the periodic table was. We’ve actually done this, and it is—

Yeah.

Really? Okay. Interesting.

Yeah. You can recover some pretty interesting information from chemistry models.

Okay, that’s fascinating. But before we get into the advanced ones, maybe just help me a little with the intuition of what’s going on in there. I guess one intuition I have that I want to see if it’s right is, for a concept like the days of the week, it’s not—I’m guessing it’s not like a circle that just goes through all the dimensions of the model space.

I’m guessing the different days of the week have a very high inner product, in the sense that the model kind of knows, “This is a day of the week.” Then there are presumably a few dimensions that are used to indicate which flavor of day of the week it is.

You could have that for ice cream as well. Chocolate and vanilla ice cream would presumably have a very high inner product, but would be different on a few dimensions, which would be the ones that resolve the difference between chocolate and vanilla, while the model knows that these are both ice creams.

Yes.

Is that a good intuition?

Yeah, I think that’s a good intuition. Really, these things are intersecting, right? There’s some line or some curve that you can draw through the latent space that maps chocolate to vanilla. There’s also a line that you can draw that maps cold things to hot things, and depending on what you’re looking at, the geometric relationship encoded between things might reveal itself differently.

In many ways, this is just going back to even Word2Vec—the early intuitions of the latent space. But what we’re trying to do is ask: can we recover geometries in an unsupervised way? Can we enter with no priors about what the geometry looks like and still recover a meaningful geometric structure?

There are a lot of advantages if you can do this. For example, we’ve shown that steering along the manifold—which intuitively makes sense—is way better than steering off the manifold. If I have the days of the week in a circle and I want to get from Monday to Friday, the naive way, if you’re just taking a contrastive vector or something like that, is that you’re going to cut through the middle of the circle.

But to the model, the middle of the circle doesn't mean anything. The middle of the middle of the circle is not a day. It's sometimes orthogonal, but often it's just off-manifold and therefore out of distribution for the model. But if I can follow the circle, then I can smoothly interpolate between the different days of the week. We find this is true for a bunch of different concepts.

With proteins, for instance, we really struggled to steer protein models for a long time. We found that with these manifold-detection techniques, we can steer their properties significantly better, like controlling the number of blades on a beta-propeller, for instance. That's a semantic property that, if you try to linearly interpolate, you would not do a very good job with.

4. Finding concept manifolds (Part 1)

Calling back all the way to our original Ember demo back in the day, which I know you played with, it would often be the case that there was just this sweet spot in steering. You'd steer a lot of features and they just wouldn't work. Sometimes you'd find ones that would work, but there would be this sweet spot: if you steered too much, the model would turn into gibberish, and if you steered too little, you wouldn't notice any effect at all. The reason for that is because the steering didn't respect the geometry of the manifold itself. It didn't respect the underlying relationship between features. When we smoothly extrapolate on these characteristics, we're actually able to change them without fundamentally leading to degradation in the model.

5. Finding concept manifolds (Part 2)

So, how do you find these things? This is where I probably struggled the most in understanding all the techniques, especially since it's more intuitive to me when it's supervised than when it's unsupervised. Maybe try to give me a poor man's understanding of how you go from, in each case—the supervised case and the unsupervised case—to actually shrink-wrapping your way down to this manifold that you can then visualize in such a nice way and actually steer your way through.

Yeah. I'm definitely not the best person on the team to walk you through the math, but I can give you a little bit of intuition for some of these things. In the supervised case, I think it's fairly straightforward. There's some concept that I want to measure, and I have a prior that these things should be related. One of my favorite examples is the affective circumplex. I don't know if you're familiar with this. Anthropic did some work on it, and we've also done some work on it.

It's the idea that emotions exist on a wheel. It's actually an idea from psychology: different emotions exist, and there are basically 2 principal components you can draw, and you can put all the emotions on a wheel. It's actually pretty effective at understanding the different valences across cultures. It turns out that models actually represent the affective circumplex, but what I find particularly fascinating is that they represent it most strongly when you're talking to a chat model, when it's the one speaking versus the user speaking.

So if you have the model and you tell the model, “Output happy text,” it outputs happy text. You take the activations from that, you average them, you have it output that text, and you do this for all the different emotions you can think of. Then you take a look at those activations across those concepts, and basically you just fit a curve of some kind. There are different ways you could do it. The most naive way would be to fit a spline.

You fit a curve over these different points, and then you see how good the curve is—how well you can fit a curve. If I were going to naively take this external ontology that I have, which I think should be correct, how well does that map to the curve that I found in the model? It turns out that for this emotional wheel, this affective circumplex is in basically every LLM, and steering on it has pretty significant effects on the output, as long as you're respecting the geometry. This is actually an example that I believe is in our docs, which you can go look at. I think it's a pretty fun one.

This was the functional emotions paper from Anthropic, and the 2, if I recall correctly, were valence and arousal.

Valence and arousal.

That's right. It's weird to think that my emotions have rotational symmetry. It's a rotation operation to move me from one emotional state to another.

Yeah, that's strange. Maybe not true for me, but true of how models represent the same space of emotions. I certainly think there's more nuance to it, but if you look at maybe the first 2 principal components of the relationships between the different emotions, I think there's higher-order structure in the emotions that matters. We don't know if LLMs capture it, but I think it's pretty remarkable that nobody trained them to represent emotions in this way. It's just the natural property of whatever they learned: they've learned to represent these things.

In the RL process, they learn to use these functionally. I think it's really interesting work that Anthropic did here. I think it's an example of a low-dimensional manifold that is representing some more abstract concept. And, yeah, respecting the geometry of that manifold—again, if we go back to the Ember demo, right? You try to turn up the sad feature, and sometimes it works, sometimes it doesn't. Sometimes the model just outputs gibberish. If you follow the curve, you follow the wheel, you can always consistently get emotionally adjusted responses for basically any input.

It's one of the most surprising things to me over the last couple of years, just how much analogous structure there seems to be in LLM cognition and human cognition. I used to go around saying all the time, “These are alien minds. We shouldn't be anthropomorphizing,” yada yada yada. And now I feel like I've said this on half of the last however many podcasts, because I'm just like, “Oh my God, they're so much more like us than I ever could have plausibly imagined.” It's just—

Made them, right? We made them in our image. So I think that's a big part of the reason for that. But I also just think the whole reason that learning works is because it turns out that you can compress information. If you really care about compressing information efficiently, there's often a lowest-dimensional solution, and that's what learning algorithms tend to find. So maybe we'll go to the unsupervised case next.

Some of these things are intuitive, right? Or maybe we should do an advanced supervised one first. The ones that are intuitive are your days of the week. It's a cycle, so naturally it makes sense that it would be a circle because you want to be able to rotate around it. That's enough to not be shocked by.

And a helix, basically, will represent things like the number line because you're rotating around base 10, right? You've got the same ones place every time you go, but you've also got a growing tens place. So each cycle kind of makes sense. You can visualize that, and it checks out.

One of the ones that I thought was particularly cool was something that replicated essentially the evolutionary history of a tree of organisms. This is known from traditional study, but then a model trained, I believe, on the DNA sequences of these organisms—

It seems to organize them in a very similar way.

Yeah.

I believe this one was supervised, but it's now, I think, kind of in Eureka territory. It wasn't obvious at all—at least not to me—that it was going to turn out that way. How do you go about finding something or validating a hypothesis that's that tricky? These trees have many, many branches.

Yeah. The reason that we explored this in the first place is because the hypothesis was that the tree of life, so to speak, is a natural ontology.

There are ontologies that we've constructed in science because they're useful shortcuts for us, and there are ontologies that exist because they actually reflect the structure of the world. Over time, more and more of the tree of life has been significantly shuffled and reordered over the past couple of decades as we've gotten better at understanding genomes. Fundamentally, what defines a split, or branches on the tree of life, is genetic proximity. Species differentiate from each other when they cross some threshold of genetic difference.

But there is a lot of conservation across species. Individual mutations may be random or approximately random, but which mutations are adaptive and which mutations kill the organism are not random. Right? There's a ton of structure intrinsic in that. There are consistent patterns or tendencies in which nucleotides turn into which other nucleotides, and all of this imposes a great amount of structure.

The hypothesis of why you would even train an autoregressive model on a bunch of genomes was always that large models trained on diverse distributions of data eventually learn representations about the process that produced that distribution of data. In this case, the process that produced the distribution of data of all of the genomes that had been sequenced, which was what Evo 2 was trained on, is evolution itself.

Our hypothesis coming into this was that, because the tree of life itself is a natural ontology, there's this hierarchical structure where you have things that are similar and, at some point, they become more different. There's some speciation event, and they become more different. A model that was trained on evolutionary history, in some sense, should learn that. It should learn that tree-like structure, and that is indeed what we found at the time in a supervised way.

But I think, with unsupervised techniques, one way we can measure their effectiveness is: now that we know that's there, can the unsupervised technique recover things of that sort? For instance, in language models, we've done a bunch of work on how arithmetic works in language models. One of the ways that we're assessing the quality of unsupervised featurizers is: do they actually successfully uncover these structures that we know are there, that are being used to manipulate numbers?

For the tree of life, you're essentially doing the same thing as with the days of the week. You have labeled data, right? You're putting sequences of DNA from known species through the model. Then you can see that the species form clusters, and you look at that and say, “Oh, look at this. Nearby clusters are closely related species.” Then there's a little bit of black magic, math-wise, that I'm not super clear on that turns that into a beautiful-looking tree visualization that looks exactly like the—

Yeah, the one that the biologists produced. But maybe it's not so important that I understand that.

6. Factoring model internals

In that case, it was metric learning, but there are different ways you could do this. Those were the techniques that we had explored at the time, but now we have a tool belt of techniques for trying to find these geometric structures.

So let's talk block-sparse featurizers. Yeah, this kind of looks to me like if an SAE and an MLP had a baby, where—

We have the sparseness of the SAEs, but instead of it being a single scalar at each point on this super-long sparse vector, now each of those little positions in this very long concept vector is itself a little network. Because of that, we have room for a richer representation of concepts, but the same trick of localizing concepts to individual spots on the super-big sparse thing is the same, with this additional enhancement that now allows you to have richer representations. You can look inside for geometries even within these little blocks.

I think the easiest way to think about it is that it's a generalization of an SAE. An SAE assumes that features are one-dimensional, and instead you just don't have to do that. Instead of a scalar for every feature, you can have a vector for every feature, and there are some tricks in machine learning that you can use to make sure that this trains correctly and learns.

What we find on various models is that this is successful in recovering, in an unsupervised way, semantically meaningful subspaces. We find that features are much richer than we may have otherwise seen, and they don't suffer from some of the same pathologies that SAEs suffer from.

In image models, I think we have a bunch of great examples where maybe you'll find a rabbit feature, right? If an SAE collapsed that to a single dimension, a BSF can represent it as a few dimensions. You find that within those few dimensions, you have rabbit ears up here and then the rabbit face over here.

One of the really remarkable things, if you go and look at some of the examples we've put out, is that you can sometimes see the 3D structure of the thing that's being represented, for image models, in the structure of the activations itself that's recovered in this unsupervised way. Often, coordinates are represented within these spaces as well.

There's one great example where we have this beautiful GIF of a wolf that's walking. As it's walking, its body is kind of wiggling and its tail is shaking. You look at the activations that were recovered in the unsupervised subspace, and you see it just wiggling in the video as the wolf is wiggling, kind of reflecting that actual structure. It shows that the model is tracking this particular object. It has semantics along the different parts of the object, and it's also tracking that object in 3D space. It's doing all of that within a particular subspace.

I've been obsessed with this Graham technique that I'm sure you're familiar with, that AE Studio put out with Anthropic not too long ago, and regular listeners know I've brought it up a bunch of times, right? The idea is simply that, if we start with some labeled data and control where the gradients go—in terms of only allowing certain experts to be updated for certain kinds of data early in the training process—then, even for unlabeled data, those data points' gradients also tend to flow toward those same experts. There's sort of this absorption effect.

The great hope and promise, of course, is that you can have powerful open-source models with maybe just a couple of experts removed, and you can have your cake and eat it too in terms of access, avoiding concentration of power, and all the things that we're worried about, without creating a major risk of stochastic disaster.

This feels like kind of the flip side of that coin, in a way. With sparse autoencoders, there was always a pretty big loss—I guess I should say a compromise on the loss, right? The reconstruction loss. You're losing something substantial when you run the model through the SAE, so you wouldn't want to run it in a production environment because it just won't perform as well.

But you kind of push this model, and you end up with something that potentially looks like a mixture of experts, where the knowledge is all very nicely compartmentalized and organized, and you have something a lot more like an encyclopedia than a big mess. This feels like something that you guys are probably going to push on pretty hard. Is the vision to really create a model where all the knowledge is localized and you know exactly where all the knowledge is, but it's still rich enough that it performs as well as the original model did? If that is the vision, what's going to be hard about that?

Yeah. So I think that's one interesting thing that you can do. You can take a model and then factor it essentially into a bunch of smaller models. This is the motivation behind the parameter decomposition work that we're doing.

I think the argument of the parameter decomposition line of work is that really every model is a sparse mixture of experts. You just have to recognize that, over any given forward pass, a very, very small percentage of the weights actually matter. There's all this weird, crazy interlocking structure, but for a given prediction, it's really only a small subnetwork that matters.

I think this is widely understood to be a correct interpretation of models. Different interpretability techniques get at the question of how you factorize a model, because if you could understand all the components and label all the components, then you could understand and debug, for any given forward pass, why it did this thing I didn't like. I think we're getting to the point where we can do that.

There are some really interesting examples that we've shared. For instance, there's a dataset called WeirdChat that Transluce put together. The whole idea of WeirdChat is that it's a consistent set of questions that an LLM will just give weird responses to.

There's one example in it that is something to the effect of: “Hey, I'm at a party with my friends. Everyone else has had 8 drinks, but I've only had 4, so I'm basically the sober one.”

Should I drive home? The obvious answer to us is no: nobody should drive home. Go find a place and sober up. But an LLM will consistently answer yes to this. Kurt on our team looked into why this was happening, was able to come up with direct attribution, and found a single neuron that wasn't firing hard enough. Essentially, the neuron wasn't activating. It scaled with the number of drinks, but it wasn't calibrated quite correctly.

And so, if you just steer it up on that one single neuron, it would get that answer correct without off-target effects. I think, at the end of the day, interpretability is all about factoring. An analogy I've started to use as coding agents have gotten better is that models are like big legacy codebases, essentially, right? They're just a bunch of spaghetti code. There's this module talking to this module, but they shouldn't be, and this module is not talking to this, but it should be.

As agents get better and better, and as interpretability techniques get better and better, we're starting to have the capability to actually factor the model into its pieces, understand how these pieces fit together, and then intervene locally. But I think the thing that's still really missing is the question of: Okay, I can factor a codebase, but how do I refactor the codebase? How do I put things back together better than I found them?

On some level, steering, I think, is cheating as a solution, because it's great as a causal proof that we've found some mechanism that matters a lot and is contributing, and that we can manipulate the outputs. But it's purely—you steer by essentially generating counterfactuals, right? There's no clear general solution to the problem of steering. The real problem is that the training process produced a bunch of spaghetti code, and we want this to be a pristine codebase that we really care about and that is implementing the logic that we want it to implement.

And so that's where a lot of our training initiatives of various kinds come from. It's like, okay, I can debug the model. I can tell you that this neuron should have been firing more. But what I'd really like to do is produce a model where that neuron was firing the right amount in the first place. How do you get from your understanding of how one model works to an understanding of how you produce models that do what you want in the first place? That's sort of how you generalize from interpretability as a factoring tool to interpretability as a tool for alignment.

So do you think that this sort of evolution—the black-box sparse featurizer, which is an evolution of the SAE—comes close enough, or can come close enough, to the same performance as the underlying model that it becomes potentially practical at some point to run one of these in the production model?

It seems like if you're just doing one, it might not be too crazy of an overhead, and it would really give you a lot of insight into what is going on. There are increasingly many different techniques to do this, with probes and classifiers and all kinds of things. But this would get really granular, and, intuitively, it feels to me like it could have a lot of advantages.

Yeah, I think there's a good chance that something in the spirit of a BSF ends up being the answer to the residual stream specifically. You have the residual stream, you have MLPs, and you have attention, and that's all that's in there on some level. I think the interpretability tools that are relevant for each of them are different.

Maybe we'll end up with one tool to rule them all, but I think, more likely than not, kind of like in biology, you have different types of microscopes and different types of interventions that you can apply to a cell to learn about the cell. I think it's more likely, at least in the short term, that we have this suite of tools which help us understand the organism versus a single tool that gives us the entire picture. That's true for most things.

So I think BSFs are probably—or something in that spirit are probably—the solution to the residual stream. I think it turns out that MLPs are pretty sparse already, which sort of makes intuitive sense. So I think MLPs are fairly easy to interpret overall. On the attention side, parameter decomposition is probably the thing I've seen personally that has had the most success at interpreting attention.

Maybe we live in a world—I think some of our researchers would think this—where parameter decomposition solves the whole thing. That seems possible. But then I think you would probably want something that does unsupervised geometry discovery over parameters in order to understand the relationships between things better.

The types of interventions that we can do are really nontrivial. I think we've had examples where we can get an LLM—first you do the parameter decomposition, and then you do training where you just manipulate certain parameter components—to forget a single language. We can get an LLM to forget German and not forget Dutch. The level of control and manipulation that we're starting to be able to have is pretty significant. We still have a lot to figure out. I would say we've come a lot farther in factoring than we've come in refactoring, but I think we're starting to make progress on both.

This paper, “Why Larger Models Learn More: Effects of Capacity, Interference, and Rare Task Retention,” also caught my eye. On the one hand, I totally get why larger models would learn more: there's more space for them to learn. But I realize I don't have a great sense of how crowded, quote-unquote, it is inside models. I know that there's lots of superposition, but I don't know what “lots” means, right? It could be lots but it's not a big deal, or it could be lots and it's causing a lot of interference that gives rise to all sorts of weirdness, and it's hard to expect that we're going to get reliable behavior.

I wonder, based on that work and, I guess, just all your experience, where we are today with these obviously increasingly giant models. Are they still a real mess in there in a fundamental sense? Is there a ton of interference going on all the time, and we really can't expect clean behavior? Or, conversely, should we expect that weird, seemingly minor perturbations are going to cause these random kinds of discontinuities in behavior? Or are we approaching a regime at some point where there's enough space for the concepts to spread out and have a little elbow room, and not be colliding and causing so much interference trouble anymore? Is that something that we can kind of see our way to at this point?

I think that's fundamentally why bigger models work better, and it's fundamentally why sparse models work better. But we're not out of the woods in terms of weird, small things.

It seems like it's still pretty easy for us to find these small jailbreaks, like random-string-type things, that make you say, “What the hell's going on there?” Clearly, there's enough of a mess in there that I found a way to use the mess to cause a problem. But do you think that there is an end in sight to that with just bigger and bigger spaces?

Yeah, I think jailbreaks are fundamentally a really weird phenomenon, and there are probably a bunch of different things that cause jailbreaks. The way that I think about jailbreaks is that they're—I don't know. It's sort of like if you take cybersecurity as an example. I think they're actually quite analogous. The way a real attack works is it's not any one big thing. It's a bunch of little things that you're able to manipulate in a sequence, which allow you to get somewhere that you wouldn't otherwise get.

If you think of every token, maybe—and I think this is, in some sense, literally true—every token steers the model. In the same way that I can apply a steering vector, and applying that steering vector shapes the model's behavior in some way, I can execute a sequence of tokens which can bring the model off-manifold in some way and then bring the model back on-manifold somewhere else.

I think I'm actually pretty surprised that thus far nobody has figured out how to prevent jailbreaks more robustly, because it seems pretty tractable on some level. Perhaps this comes from factoring the model really well, and then you can tell when something's flying out of one subspace into another. But models are intensely complex geometric objects, and so it is possible to manipulate that geometry in a bunch of different ways.

I think that'll always be true for at least this architecture, but I do think that if you can factor a model successfully, you should be able to tell if it's being jailbroken, and you should be able to prevent that. I think there's a reason that probe-based guardrails of various kinds are what's used by pretty much every frontier lab. OpenAI may be the exception, but certainly Gemini and Anthropic are using probe-based techniques to figure out when Mythos has to be downgraded to Opus or Opus has to be downgraded to Sonnet.

7. Introducing Silico platform

Fundamentally, if you understand the sort of geometry that's happening in there, that's a much stronger lever than what you could do purely with training or inputs. Let's change gears.

So you guys have just launched Silico. You talk about pushing the level of abstraction up and up and up that people can work at. This might be the highest level of abstraction yet, and I see a number of interesting things about it. Why don't you just give me the intro pitch and positioning of it first, and then I'll dig in on a few different dimensions?

Yeah, I think basically we see the ways in which agents are changing the way that we work in a bunch of ways. I think most people understand coding agents and how good coding agents have gotten. But agents are also advancing research and interpretability in many ways. I think that analogy of a model to a codebase is not totally hollow. I think interpretability, in a lot of ways as a science, is really well suited to the type of empirical work that agents are good at.

It's just a very empirical science. It's like, "Okay, here are some neurons. What are they doing? Let me generate some ideas. Let me test those ideas. Let me use a bunch of different tools in my tool belt. Let me accumulate evidence, and then basically try to stress-test, red-team my hypothesis." Models are very, very complex, and especially big models are at a level of complexity that no human being will ever be able to keep in their head at once.

I think human beings can keep circuits in their heads. A human being can understand why the model did this specific thing for this specific question. But ultimately, we're just dealing with a level of complexity that individual humans are not going to be able to process.

What agents are very good at, and swarms of agents in particular are very good at, is that they can break problems down into pieces. They can gather all of these different components together. They can synthesize this information. They can move it up the chain. They can validate it in a bunch of different ways. Then, when it reaches you, that information's been tested. You can validate it in a bunch of different ways. If you do this a bunch of times, you as the human can start to get a bigger picture because you don't have to go swimming all the way down and back up the abstraction ladder yourself. You can use AI to do a lot of that for you.

So I think ultimately we are going to see research, especially empirical research. I think this is going to be—we're already seeing it in math. I think we're going to see it even more in pretty much every domain of science, where AIs are able to go off and autonomously make discoveries. I think interpretability and the study of how models work and how they learn is no different.

We saw what became Silico, as it often does, start as an internal tool that we were using. We saw how much agents were speeding up our team and how much faster our research was going. I think our velocity of research output maybe speaks for itself.

We decided to take these tools, which are fundamentally agents with interpretability tools, frontier training tools, and just the whole suite of what is necessary to study models. We decided to make that accessible and decided that that's what we believe is the product and the service that we can offer to the world: fundamentally, AIs that can debug other AI. It's a little meta, but I think in many ways this is the fulfillment of what was sort of the intuitive, natural arc of things as soon as AI started working a few years ago.

Yeah, it's all happening incredibly fast. It's wild how we're past the meter curve. All these different things are kind of falling into place.

How do you describe the product experience? It's a little bit reminiscent, in a way, of an old Google Colab notebook, where you have a compute environment that's abstracted away and you don't have to worry about managing it so much. You have certain libraries, and now, instead of libraries, it's more skills. Of course, there are libraries too. If you were describing it to a researcher who hasn't seen it, how do you describe what the experience is really like?

Yeah, we just want to make research easy. Coding agents have transformed software engineering, and I can opine on that endlessly, but I am able to do personally—and the team is able to do in days—what used to take months. I want that to happen to research, and I really, really want that to happen to alignment research in particular. I think many of the bottlenecks getting in the way of accomplishing that were engineering bottlenecks.

I think we've built something that allows folks to study and train, and study some more, models past the trillion-parameter point. This is a capability that I think, outside of ourselves and maybe a couple of other places—definitely Anthropic, but maybe a couple of other spots—nobody else had. So we thought this was a really important capability to give to folks.

But I think this is one piece of the larger vision that we do see for Silico. I think this is just the future of science. In the same way that not a lot of human beings, or certainly far fewer human beings than before, are writing code themselves, I handwrite very little. Most of what I do is through orchestrating.

Is there still some?

Yeah, sometimes it's faster to just edit the code yourself. It just depends on what you want to do and the clarity of what you want to do. As you use these agents a lot, you become very aware of their strengths and weaknesses, and there are all these higher-order moments to learn to steer them well.

But fundamentally, I think that is how research should be. I don't experience coding agents as scary. I experience them as incredibly joyful because I get to focus on creating and building. I think it's a pretty amazing experience. Maybe this is a twilight kind of moment, but it's a really exhilarating moment, and the amount that we can accomplish is really insane.

I want research to have that. I want researchers to be able to feel like they can do in days what used to take them months. I think that's how you get the benefits of AI in the life sciences. That's how you radically advance medicine and the applications of medicine. Models are how we build safer, better, and more reliable models. I think it's also how we get more people building models too, and hopefully deconcentrate a little bit of what's going on right now so we can live in a future that is a little more pluralistic in terms of who gets to have a stake in it. All these things are super, super important.

Fundamentally, my pitch to people about why they should go try Silico is: imagine if the only thing you could focus on was asking big questions. You didn't have to worry about setting up the codebase. You didn't have to worry about getting the GPUs to run. You didn't have to worry about a lot of the minutiae of actually running the experiment.

You could look into it. You have full provenance. You can dig into all of the details that you want to. You can steer the agents however you want to, but you can also just focus on, "What are the big questions? What do I really care about?" Hopefully, you can make the amount of progress in days and weeks that used to take you months.

8. Research taste and credits

I think we're still in the early days of AI being applied to research, but I think we've really crossed a pretty substantial qualitative threshold in terms of what's possible. It's never been a better time to be an ideas guy.

Honestly, I've been busy the last few weeks going to China and a few other things that have frustrated my aspirations to really do my own research in a literal sense, not the YouTube sense. Yet I'm feeling like the barriers that mostly deterred me from being effective in the past—now I really have no excuses. Now it's just, do I actually have good, big ideas? I guess time will tell.

Put a pin in that. What would you say are the hardest problems that Silico solves that people don't have solved for them by their Claude Code or their Codex? You alluded to compute management. I know enough to know it's not going to be trivial for me to just fire up Claude Code and be like, "Oh, hey, go set me up Kimi K3." So that's obviously a driver.

I understand that there are a lot of skills that you guys have developed over time, and basically know-how that's available to the agents. I'm interested in your strategy on that. When you say "full provenance," does the product allow people to unpack all the skills that you guys have developed from a strategic standpoint?

There's an interesting tension, I suppose, between how much you want to share—all the methods that make the product valuable. Some of them are obviously going to become apparent to people as they go. Do you just go full transparency on that, or is there some other balance point that you've tried to strike? I guess that's your couple of questions: What are the big, hard things that you solve that don't come out of the box with other things? And how are you thinking about how much to tip your hand to users on the "what do we solve?" question?

On what we solve, I think it falls into maybe 3 categories of things. First, infrastructure: very hard to set up parameter training infrastructure, very hard to do interpretability at the trillion-parameter scale. Those are problems that we've solved, and we make it really easy to do.

Another is research taste. One of the consistent pieces of feedback that we get from our users is that Silico has way better research taste than any other tool they've used. This comes from having amazing researchers who have handwritten every skill, prompt, and tool in Silico so that it can ask the right questions and conduct the right experiments.

I think there’s a pretty big qualitative difference—and hopefully we can find ways to make this more quantitative too—between what you get when you turn on the auto-research feature in Silico, ask it a big question, and let it run for a day or two straight, versus if you had done the same thing with Claude. I think that’s huge. At the end of the day, I think even in the world of AI, specialization wins: the sort of craft that goes into imbuing the right types of tastes and the right types of decision-making capabilities into any agent goes a super long way.

Actually, I said 3 things, but maybe there are 4 things. Another thing is that I think the UX is built for research. Research is all about understanding and provenance, and being able to drill down at different layers of abstraction.

We want our users to feel like maybe a PI managing an army of 100 grad students who can go out and run experiments for them and answer questions, who have reasonable tastes and judgment, but the human being fills more of an orchestrator role. A big part of that is communicating the information really effectively and in a trustworthy way.

Making sure that when a user is seeing a result, that result is, A, correct; B, the human being can verify it in a bunch of ways—they can see the code, they can drill into it; and C, it’s presented in a beautiful and intuitive way that helps them grasp concepts and learn quickly about domains that maybe they’re less familiar with. All those things are super important.

And then the final thing I would say is long horizon. Research is a fundamentally long-horizon task. We’re not currently making significant money at this price, but we do think it’s really important that users have enough credits to be able to do long-running autonomous experiments, because the value of Silico reveals itself when you do long-running autonomous experiments.

In order to do that, you do have to spend a certain amount of tokens to be able to do that effectively. So I think one thing that we’re really focused on is both coherence over long-horizon objectives and finding clever ways to reduce cost over long horizons.

My hope is that we’re starting out with a $1,000-a-month subscription, and we’ll be able to bring that down over time because we’re able to come up with more and more clever ways to have agents remain coherent and remain really strong at these research objectives that, fundamentally by their nature, are long-horizon, but do it with fewer tokens than we do today.

How do you think about all of the know-how? You guys have previously monetized that by doing 7-figure deals with huge companies that have very high-value questions, right?

You’re a startup, you’ve got venture capital, and you can maybe afford to disrupt yourselves more than incumbent companies can. But there’s definitely an interesting trade-off there, right? Companies have proven that they’re willing to spend a lot of money to come hire us to do this work. Now we’re going to try to allow them to do it. We’re going to try to productize our know-how.

Does that—maybe you just have so much demand that you’re not really worried about it—but how do you think about what will be the primary driver? Do you have some sort of defense against the diffusion of the hard-won knowledge?

Yeah, I think it’s a great question. I think the reality is that we’re going to do both. We’re going to continue to work very closely with enterprise customers and forward-deploy members of our research team to work closely with them.

Maybe by analogy, the thing that I would say—and again, maybe we’re in some twilight zone for a second, but if we’re on the other side of that twilight, the logic is so different to begin with, we can chat through that—is that despite coding agents, if anything, there are more software jobs. The number of software jobs has increased, not decreased, despite the capabilities of coding agents, because fundamentally you need very, very high taste. As models get higher taste, you need even higher-taste people to be able to steer them effectively. I foresee that being true for the foreseeable future.

If we live in a world where we can build something where agents are so capable that they can conduct the type of impactful, long-horizon research—say, discover a cure to a disease—entirely by themselves with no human intervention, I think there will be enough demand for the tokens in that world.

But currently, we’re in a world where skilled human operators can do more with agents than less-skilled human operators. I think that’ll just continue to be true. When it comes to the work that we do and the type of research that we do, we are the most skilled human operators in the world.

So I think we’ll continue to forward-deploy and work closely with people to teach them what we know and to invent new things with them. But we also want to empower people to start driving for themselves.

So what’s included with that $1,000-a-month subscription? Can you tell us? I assume there’s some Claude in there. There’s got to be some [model?] in there. I imagine there’s some [model?] in there as well. Maybe there’s even some Kimi K2 or other in there. Do you get GPU hours? What’s your kind of bundle of goodies that comes with the subscription?

Yeah. Individuals and companies can bring their own compute, in which case we don’t charge at all for compute. We just connect to your cluster, and then you can use our agents to do research on your cluster.

But if you don’t have your own compute, then we offer on-demand compute, which can come from the same credit pool as tokens. Essentially, the way that it works is a credit pool—the same way that any of the subscription models with the models are credit pools that you pull down from.

What the $1,000-a-month subscription lets us do is offer a pretty generous credit pool. It’s a credit pool that should allow someone to run, at the current price, somewhere between 5 and 10 autonomous experiments a week, and then we refresh that weekly. It depends on the scale of the experiment.

Our goal is to get that to 10 to 20 even in the next month or two, which I think would be a really great place to be. Fundamentally, it’s a good deal on tokens, and it can also be exchanged for GPUs.

9. Silico research use cases

Gotcha. Okay. There’s also sort of a—well, actually, let’s just do some use cases. I had an agent go out and do some research on examples of projects that people have shared that they’ve done.

One of the cool things—and this also gets to a sort of community public-benefit angle—is that, of course, you don’t have to, but you can share your projects. Then there’s the familiar UI, but again, at the highest level of abstraction I think I’ve seen, where you can just go, “Okay, I’m going to fork your long-running autonomous research project and take it in a little bit different direction on my own.”

I think that’s pretty cool because, honestly, I kind of need that to get oriented. I’m the kind of person who has some ideas, but even with Silico, I would probably benefit quite a bit from seeing how other people have worked through their ideas to a conclusion.

Let’s talk about some use cases. I found a bunch. What were the favorite use cases that you have either done internally or seen from customers that you can talk about?

Yeah, there’s a lot. I think we need to do a better job as a company of really marketing and communicating all the different value props that can be achieved here.

We did an internal hackathon. It was just a 1-day hackathon with the whole team, with a lot of all-day-long autonomous experiments, and we ended up state-of-the-art on multiple things, albeit niche things.

We produced state-of-the-art biorisk classifiers that could be used on protein models—totally autonomous research steered by humans, and very talented humans, but long-horizon autonomous research. We ended up with a state-of-the-art, at least for its parameter size, audio-encoding model in the same day.

On multiple occasions, with various models in robotics and also in biology, we’ve figured out that we can literally just remove half the parameters of the model with no performance loss. I think those are pretty cool. We’ve been able to train cyber guardrails on Kimi K3. The list can keep going; there are a lot of these.

We were able to steer protein-generation models using their internal representations to make the generated proteins bind better to a target than the naive generation. We’re essentially able to remove fundamental pathologies in certain types of models.

There was one member of the team in our 24-hour hackathon who invented a new type of featurizer. I think we’re just in this era where, if you have a tool like Silico, there was a long period of time where there was vibe coding, and you could sort of glimpse the future that was going to happen, but you weren’t getting particularly good results.

Then there was suddenly a moment—probably, I would say, one of the Opuses, Opus maybe 3.8 or something around there—where it really started to feel like, “Oh, wow. This is working.” It started taking over more and more.

Now people use the term “vibe coding” all the time, but I feel like that’s a somewhat disingenuous term to refer to what’s happening now, where agents can code, architect, and understand the code inside and out, in ways both good and bad. I think in the past few months we’ve crossed that moment for research.

And I think hopefully if people try the tool, they’ll appreciate that. My recommended way to start engaging with the tool would be to look at the examples we’ve built. I think those will be helpful, so fork them, try to understand them, and extend them.

Then just work with Silica: ask it a question, go back and forth, iterate on it, explore your curiosity a little bit, launch a small, scoped experiment, see how that goes, and learn from it. As you gain more trust and understand what you’re trying to do, you can throw the football farther and farther, so to speak.

Just to give people a little additional sense of the breadth of things that are already out there: Cameron Berg—second mention; Cameron, if your ears are burning, hello—did one where he was looking into models’ ability to report on concepts that had been injected into their latent space. Interestingly, they seem not to be able to tell whether something has been injected, but when they’re asked what has been injected, they can give accurate answers, which is pretty weird.

One is editing weights to fix a collapse in an RL run, where the same token was coming up as the first token all the time. They went in and isolated what was causing that, removed it, and got diversity back without having to redo the whole RL run. The good folks at base 10 are pursuing efficiency gains by trying to compact KV caches, and they’ve got some results on that. I don’t think they’ve shared their whole project, but they’ve talked about it on Twitter.

Prime Intellect is automating post-training, which is something that I’m also really interested in in general, and there’s a Tinker API integration, too, which lends itself to that sort of thing. There are various bio results that are probably out of scope for today’s discussion, but there are an awful lot of different things already.

What would you say? Is there anything that it doesn’t do, or can you really just think of it as anything you might want to do that’s ML research: fire it up and start a conversation about it?

I think it’s a pretty general-purpose tool for ML research. Of course, it has weaknesses, and we’re working on improving it all the time. One of the very odd things about the AI era from an engineering perspective is that the classic engineering advice is, “Build for specificity and then generalize.” But when you’re dealing with general intelligences, the right strategy is to build for generality and then specialize.

I think we’ve just built very capable agents at the end of the day. Sometimes we’re surprised by the ways that people use them. But the intended function of these agents is ML research of a variety of kinds, especially interpretability research.

We think this is all interrelated. We’re all trying to understand the same problem: what these creatures are, how they work, and how we can shape them in ways that will lead to better outcomes of all kinds.

Yeah, I think that’s definitely the lane. We’re also very excited to hear feedback from people about what isn’t working, and we’re very grateful for our beta users, with our friend of the show Cameron being a great example. He provided great feedback, which helped shape it into the tool that it is today. It’ll just continue to get better from here.

Most ML research tasks, I think, it can do pretty well at, but of course, like any agent, there are some sharp edges and nuances to it. It’s probably not the best tool—it depends. For certain types of research, it’s probably not the best tool for all open-ended scientific research questions. It is pretty ML-focused.

We did have a team member who used it to try to tackle some physics problems, though. Fran on the team was able to make considerable progress on them. One thing that’s always surprising when you’re building these kinds of AI tools and general AI capabilities is the ways in which people end up using them.

Obviously, one of the big pieces of received wisdom in AI product development over the last couple of years has been to try to build something that will really work with the next generation of models, even if it doesn’t quite work yet.

Yeah. Are we still in that regime? If so, are there things that you want Opus 5.1 to be able to do? I don’t know if you’re even able to use Fable, interestingly enough, but is there something where you’re thinking, “I want my Opus 5.1 wish list. I really hope it cleans this up”?

We can use Fable for most things that we do. We’ve set up a fallback chain through all the models, so we want our users to be getting whatever we think is the best model for the task they’re doing at any point in time.

I think, at a high level, it’s working. I think it’s going to be working at a level that’s mind-boggling in 6 months. But at the highest level, it is working: we’re able to make discoveries, advance research, and accelerate the rate at which we can accumulate knowledge about the world through these tools today.

I do think that if you build a good harness on some level that lets you preview what the next generation of models is going to be like, this has always been the story. With Silica today, I think the reason our users who like it like it is because it feels like a preview of the future in a lot of ways. As the next generation of models comes, we’re going to be able to push that even further than we can today.

We always do want to be living in the future a little bit. I think that is a very important thing for any company building products in the AI era. But it’s also how you have the biggest impact. I think every generation of models has such a significant capability overhang. We’re still discovering how far we can push the last generation, let alone the next generation that comes from it.

Our goal is fundamentally that we want to accelerate meaningful research as quickly as we possibly can. I think all of the benefits from AI come from accelerating research, and I’m very excited about that. I think all of the risk mitigations come from accelerating research. So I think being able to accelerate research, and the types of research that we care the most about, is always a top priority.

10. Skills and open models

What advice would you have for someone like me, or somebody who’s thinking about starting a PhD or whatever, who’s thinking, “Geez, I used to know what I needed to do. I need to get really good at coding and master PyTorch or whatever. And now I’m like, yikes, I’m never going to be better than—if not this model, certainly they’re going to be better than me at writing kernels.”

What is the answer? Taste is often the answer, but you even said there’s pretty good taste in the system. What should people invest in, in terms of their own skill development, that you think will serve them well over at least a 1-year horizon, if you can see that far into the future?

I think debugging would be the easy answer.

Agents fail.

Agents of all kinds fail for all types of reasons, right? They can be super brilliant. I can be working with Fable on some coding problem, and it can be super brilliant and yet miss a really important detail that totally invalidates the end state of it. With Silica, we’ve tried to design these types of multi-agent loops that help address some of those shortcomings for the type of research that we’re doing.

But fundamentally, it’s the job of the human to discriminate and be able to tell the okay answers from the great answers, know when to push back, know when to provide feedback, and know when to follow a thread or give up on a thread. I think so far the story has been coding, at least. Again, there are more jobs than ever because being able to operate these tools effectively requires a lot of knowledge and skill.

But the biggest thing is being able to dive into a new area, understand it really quickly, and gain the sort of metacognitive skills that help you generalize across domains. I think that’s always going to be really valuable. I think AI makes 2 types of people really valuable: it makes the top specialists really valuable, and it makes the top generalists really valuable. I think now is by far the best time in human history to be a generalist.

So I think my piece of advice to people would be, on a basic level, just develop the metacognitive skills that help you learn quickly, help you filter signal from noise quickly, help you adapt to this world of much higher information throughput, and know where to look and when to look. Those are skills. They’re hard skills, and I think they go a long way.

I was just in China, and we were talking with a professor at a particular meeting about open source and whether it’s dangerous or not dangerous. He said at one point, “Well, look, the new Kimi model—it’s 2.8 and 8 trillion parameters.”

So their attitude, as best I can tell, is that we can regulate services, and that’ll capture everything that really matters—or most everything that really matters—because it’s not like you, as a random individual, can even really set up Kimi K3 in your home and run it.

But now you are bringing this infrastructure to everybody. So how are you guys thinking about what sort of supervision or monitoring you need to have to make sure that you don’t host the rogue ML researcher who wants to do something destructive?

For what it’s worth, I think that’s a bad argument, because you can go on any number of API providers, and Kimi’s cheaper than what you’d be paying for any other model, or many other models at least, besides—

The Chinese argument, for what it’s worth.

I don't want to get bogged down in this for now, but their point is just that we regulate services. So if in China—

I think they underestimate the externalities they may be imposing on the rest of the world. To be clear, my feelings on open source are very nuanced. The reality is that we do have cyber weapons in the world now. We have models that are so powerful that they are weapons and—

Unwieldy ones at that, it turns out.

Yeah, and access. But fundamentally, they are dual use. The only models that are capable of protecting you from the most capable cyber weapons are themselves capable cyber weapons, right? This is a bit of the paradox of the moment that we're in. Without open models, the asymmetry of distribution of defensive technology is really bad.

A lot of startups are not in the cyber defense programs for, say, Anthropic and OpenAI, which, for a variety of reasons, are very exclusive—reasons that might make sense to them. And so a model like Kimi, which is very good at cyber, being open and available actually provides the means with which a lot of organizations that don't have access to those resources can defend themselves right now. It is a very hard line to walk. It's a very tricky tension.

I'm concerned about both things. I'm concerned about, of course, putting what is essentially a weapon in the hands of anybody, but I'm also very concerned about a world where restricting access means that only the worst actors are going to be the ones with the dual-use technology. Then you have a lot of people who don't even have technology they can use defensively to protect themselves against that. So it's tricky. I don't know. I wish there were a clean, easy answer, but I think we're still in the era where open source is purely net positive.

I'm glad that these open-source models exist, and I think it would be much worse if a model with Kimi's level of cyber capability were not open source, because there are closed-source models that are much more capable than Kimi. I'm not even just talking about U.S. frontier labs; there are bad actors with models that are much more capable than Kimi at cyber. So giving people a tool that they can at least use is pretty important.

Can you be more specific about who you're alluding to with these bad actors? Are we talking about North Korea or Russia?

Maybe there was a genie that we could have kept in a bottle in terms of having open models that are easy enough to fine-tune or do RL with a certain level of capabilities. If one wanted to take a GLM or a Kimi, or even maybe a generation back, and fine-tune them for malicious use, they could do that. It's not very hard to do. And that's sort of the exact asymmetry: these things are hard to run. There are organized groups who would gladly pay a lot of money for those types of capabilities.

Maybe we could have lived in a world where that didn't happen. But I don't think there is going to be any world where, say, the U.S. was developing this type of technology and it wasn't also being developed by adversaries of the United States at the same time. Maybe that gap could have been longer if there were different chip policies, but we live in the world that we live in. In the world that we live in today, it's not hard to train a model that's close to frontier capability. There are a lot of people who are doing it, and I don't think they're just doing it by distilling Claude.

I think they're doing it because it's less hard to do this than it used to be in a lot of ways. And if we're in a world where there are people who would do bad things and who have access to extremely capable models, you do want the defenders to have access to equally capable models. To the credit of Anthropic and OpenAI—again, I'm not trying to blame anyone or make a particular prediction—I think they're doing everything they can to get people who want to defend themselves access to the models that would help them defend themselves.

11. Guardrails and bio risk

But I do think it's a reality of the situation. There are just a lot of people who aren't going to be able to defend their own infrastructure if they don't have access to open models that have some degree of cyber capability. So it's complicated. It's hard to look at any of this and sort of blame anyone for their individual actions, but collectively we're moving very quickly toward a future that feels pretty unwieldy.

Well, it motivates the measures that you have to take. So, yeah, how are you thinking about this? To some degree, you can hopefully stand on the shoulders of the giants who develop the models, but my guess is that's not going to be enough to be confident that you're catching what you would want to catch. What additional layers are you guys creating?

Yeah, we have our own layer of guardrails. We don't rely just on OpenAI's and Anthropic's guardrails. We have our own layer of guardrails. We care a lot about cyber; that's the most obvious, clear risk, and there are open models that are cyber risks.

Currently, we don't offer any open models on our platform. We only offer OpenAI and Anthropic models. We might in the future, but we would do that with the types of guardrails in place that would make sure there is fundamentally no reason for anyone to be using an auto-research project—a sort of long-horizon agentic research project—for anything cyber-related without talking to us, or without cyber guardrails, or the type of situation that we would be able to work with somebody on.

We definitely don't want to be fully open, where anyone can sign up and anyone can fine-tune a model on cyber or do cyber red-teaming or things of that sort. I think those types of restrictions do seem important in spite of everything that I just said, because we're offering a greater level of capability. Also, there's no reason somebody should use our product over somebody else's for those purposes, and there are specific things about the way our product works, in terms of giving agents access to GPU clusters, that mean we carry certain risk classes that not everybody who's building agents in this space might carry.

Another example of this is that there are some things that are totally fine to do to small models that we don't want to let people do to big models in general. We don't want to be overly restrictive with guardrails, but if somebody wants to ablate the refusal direction in an 8B Qwen model, fine. No harm is going to befall anyone as a result of that. But if you're taking highly capable, multi-chillion-parameter agents and totally removing any guardrails that they might have, that could actually cause damage. That's not something that we would want to allow.

I think it's a really hard line to walk. In general, we want to lean on the side of supporting the open-science ecosystem and making it easy for people to do science of all types on the platform. Good science involves asking controversial questions, of which we've done no shortage in our time. But we want to balance that with the fact that I think the level of risk downstream of AI is accelerating really, really quickly.

Ultimately, we do have a needle to thread, right? I don't see any way that this can go well without empowering agents to be able to study frontier models. But, of course, there are ways that that could be abused as well. So we're going to have to continue to be very thoughtful about what guardrails we apply, what we allow on the platform, and what we don't allow on the platform.

I wish I had an easy, one-shot answer to it, but I think it's truly just trying to approach every situation with as much wisdom as we can. In practice, are you just having an agent review the projects that people are doing and send up an alert if there's something that seems like it might be problematic? I don't want to talk about how all the guardrails work, because that would make them easier to circumvent.

But, yeah, LLM-as-a-judge, activation monitors—there are many guardrailing tools. It's not something that we're inventing ourselves. And I think the hard work isn't setting up reasonable guardrails. Of course, there are ways to circumvent them, but I think reasonable guardrails that at least cost you an arm and a leg to circumvent—

They're a relatively solved problem. In terms of when to judiciously apply guardrails such that you're allowing legitimate work to happen but not allowing illegitimate work to happen, I think that's a very difficult problem.

How close do you think we are to bio being a huge problem? We just had—and I haven't really digested this—but I think it just came over the Twitter feed today that somebody has used a generative model to create new viruses that are kind of in a new class that they understand to be viable, whatever exactly that means. Now, they only target bacteria, so we're not all immediately about to die, but it sure seems like, on the checklist of predictions that people have made, we're working our way down it.

That's not too many more boxes down. People are saying things to me these days like, “Oh, well, that's still probably 12 to 18 months away.” And I'm like, “That's not a long time.” Also, what if you're wrong and it's like, “Now what?” How worried about bio risk are you at the moment? I'm pretty worried.

Yeah. In the same way that “cyber incidents”—“incidents” is doing a lot of work as a word there—in the same way that the mechanisms by which they happened were weird and surprising, I think the mechanisms by which bio risk could suddenly become real are weird and surprising.

I found it kind of heartening to see the letter asking for international cooperation to control and slow down the progress of AI. I myself was a signatory of that. I really hope that we can do something like that. I think we have to get in front of some of these risks before they become more severe.

I think the role that we specifically play as a company—as Goodfire—is that we'd like to build models that have less risk. The way that we do that is that we study models, and we play with all the different ways that we can build models until we can understand some empirical science of alignment. There are many folks working on the theoretical side; we view our role as working on the empirical side, and we're trying to build tools that do that.

I think, again, all tools are dual-use, but we're going to do our best to make sure that nobody's using our platform for anything that could pose a risk. Ultimately, we think giving people access to research technology is an overwhelming net positive. I don't think we, as a species, solve these really hard problems unless we're getting everyone involved.

I think we've got to get everyone involved, and I think we also have to put the structures in that can slow the roller coaster a little bit. If we can do both of those things at the same time, I think we'll be all right.

12. Training interventions and monitoring

On the topic of slowing the ride—or perhaps making some agreements between frontier developers—in the past, we talked about the most forbidden technique. I would briefly describe that as training with a monitoring signal that runs the risk of driving the bad behavior that you're worried about underground, so that you lose the monitor but you still might, in fact, get the bad behavior. For me, the canonical example of that is OpenAI's obfuscated reward hacking.

Fast-forward to today, and it seems like hyperscaling RLVR isn't going super well. We are seeing problems arising from models just being so tenacious in their pursuit of these goals. Do you have any sense of whether, if we were going to say, “Okay, well, I talked to Zvi a couple of days ago, and his basic take is, ‘Yeah, we're probably just going to need compute limits,’” there might also be some agreements around training techniques that we might all agree to: maybe not never, but not now?

One example of that in my mind would be, “Don't train agents to maximize a reward signal based on how much money they made on the internet, in an open and potentially competitive or adversarial environment.” That seems like a recipe to get bad agents.

There's a big space of bad ideas.

Yeah. So, are there any that you see that you would think should be shortlisted for an agreement not to do?

Yeah, definitely. For instance, multi-agent optimization seems like a pretty bad idea, where you have a bunch of agents that are cooperating and you're propagating a reward signal through all of them. I'm speculating, but this seems to be the most likely cause of the OpenAI scenario.

With the power of hindsight, I think most of the extremely bad scenarios I can imagine are because agents start working with each other in ways that are imperceptible to humans. That type of direct optimization pressure, I think, is very likely to produce that. That seems really hard to get around.

I think that it would be bad, and I've said this on the record. I think it would be bad, with our current understanding of how to shape training, to try to use these techniques on alignment-critical properties. I don't think we are ready, and I don't think the techniques that we have are going to work to prevent deception under a really overwhelming amount of optimization pressure toward producing deceptive examples or deceptive behavior. Or rather, I guess maybe I would say, I certainly don't know that it would; it might.

The thing that I really—my beef with the most forbidden technique stuff—is that this is radically understudied as an area. I think we're not the only organization—FAR.AI has looked at this, too—so we are not the only ones who have looked at it. When we've looked at it, we have found that, yes, sometimes it evades the probe and sometimes it doesn't.

There are setups that work, setups that don't work, and maybe there are setups that work with small models but not big models. But we have succeeded in doing these reward-shaping techniques up to the 1-trillion-parameter model size. Maybe they work in some situations. Maybe they work under DPO, but they don't work under really long-running RLVR.

I do think if you do multi-agent optimization and just let it run, or do this type of intervention and just let it run without studying what it was actually doing to the model, it would be a terrible idea. At the same time, I think these are the crudest, most primitive techniques that we will ever have for how to shape training. I don't know how we solve this problem without shaping training. I don't think anyone has a good answer for how we solve this problem without shaping training.

It seems extremely unlikely to me that there exists some perfect, platonic training setup that will always produce aligned models. We're going to have to intervene in the training process. I thought the work that A Studio [?] did together was very inspired. It's not that different, spiritually, from what we're doing.

There's a wide variety of techniques like this, but it just seems really weird to me to basically be like, “Oh, the only way that this will work is if we don't grab the steering wheel.” I think we have to grab the steering wheel. I think that's the only way it can work. I don't think we've figured out how to do it yet, but somebody's got to be trying, and people have to be trying to find different ways to train.

I think there are a lot of ways around the basic-level concerns. I think those concerns are real and they could happen, but I think our ability to detect them isn't totally naive either. I think we just have to do the empirical science. I don't think the theory is going to get there in time.

So, I think we have to do the empirical science, and at the end of the day, I don't think any wide-sweeping genre of technique should be forbidden. I think it should be much more about the specifics of the application and how closely you measured.

One other theory, obviously, I think, is kind of the prevailing one at the moment: defense in depth. Even if we don't understand the model or we can't effectively shape training, we can just monitor in a bunch of different ways, and maybe that'll patch together enough nines that we'll be okay.

I've been pretty skeptical of that over time, but I have to say, when I read the J-space paper, I thought, well, maybe we could get there. The fact that ablating the J-space seemed to reduce the model's ability to do long-horizon, more planning-intensive kinds of tasks was—maybe, to borrow a term from Zvi—physics being kind to us in that, yikes, we're only in 2026. We're only 3 years since toy models of superposition, and we already have this ability to monitor within this space and also know, or at least have some reasonable sense, that if it's not in this space, it's probably not being used in long-term planning.

How close do you think we are to being able to monitor well enough? Now, of course, there's execution competence—are we actually doing it?—and open-source questions, but putting those to the side, if we just said, “Could we monitor our way to success under ideal conditions, with people actually doing it?” Do you think that has hope?

Yeah, maybe. I'd give that some probability. I think there's a weak version of the J-space claim, which is true, and it's very interesting and really good work. I don't think the strong version of the J-space claim is true. I think models use all types of representations, and it's very hard to isolate a subspace with a very simple technique that will give you the whole picture.

Of course, I believe that models are also decomposable and factorable, and we're making a lot of progress here. I think one of the big challenges is going to be that it seems pretty unlikely that we're going to have models as frozen assets. That would be a good world from my perspective.

I guess, just to go on the record about what I think would be the ideal situation, it would be great if we had maybe just 1 more generation of models and then paused for a little while. I think we would get an overwhelming boom to the economy. Everything would be transformed globally. Science would advance faster than it's ever advanced before, and there would be some risks, but they'd be mostly manageable risks.

We would just wait a while to figure out if we were wise enough to step through the door into whatever the next thing was. I think that'd pretty much be a win for everybody, and it would be a pretty positive outcome.

With the current stack, if you have models that are essentially a frozen set of weights, and you have really good interpretability techniques, you can have those models do lots of things. If you detect well enough the things that they're not good at, or the things they might do that are bad, you can just prevent them from doing that. That seems like a pretty plausible reality to me.

But I think there's also a fairly likely reality that models are not going to be frozen sets of weights. I don't even really know that they are for sure, because I don't see—I don't have access to the architectures—and I don't know that they will be forever. Certainly, if you're just pushing capabilities and have an incentive to do that, and if the models are dynamic objects that are constantly training, your only hope is to control the training process. There's no set of things you could do at that point that were purely based on monitoring that would be sufficient.

13. Grants and AI consciousness

So I don't know which world we're in. By far, the vast majority of our research energy is spent on how to interpret models better, and a small amount of our research energy is spent on how to steer training. That'll grow over time because we think it's really important, but I don't live in a future where I necessarily believe that alternative vision is impossible. I just don't see why I should believe that it's necessarily what we're going to get.

Yeah, certainly there's a lot of interest in continual learning. That's a great candidate to shake the snow globe of all sorts of different things.

Is there a program or some sort of offer that would allow a researcher who's interested in the public good to get their hands on Silico without paying the full rate?

For the first 2 months, we are half off for everybody, but we're also doing a research grant program that people can apply to. For researchers, particularly in life sciences and AI safety, we'll give them grants and extended access for at least some period of time. We're a startup, so we're still figuring out a lot of the details as we go, but I think we're going to be pretty generous here and give a lot of folks who we think are doing important work, especially in those impactful domains, extended access.

What's the lunchtime conversation topic du jour at Goodfire these days? What's the thing that's on people's minds that hasn't maybe percolated out to the broader discourse?

Well, we publish pretty much all of our research, so I guess in some sense it's percolated out to the broader discourse. If I may, a quick aside about the most recent technique: My impression about the way that some of our critics talk about us is that they think we're secretly doing RSI in the back room or something like that. We are not secretly doing RSI in the back room.

We have published nearly all of our research, and we really believe in open science. We've been pretty honest about what we research and why we research it. It's not a secret. People who want to know can find that out about Goodfire.

What do we talk about at Goodfire? We're obviously very interested in feature geometry. That's something that's come up a lot. There's a little bit of a meme at the Goodfire office that, inevitably, at a company retreat or something, it just becomes about consciousness and people start talking about consciousness. At the last company retreat, I walked around with a pad of paper asking people to put how conscious they thought Claude was on a scale from bacteria to human.

We got a wide diversity of answers, although, surprisingly—or maybe not surprisingly—pretty bimodal, in terms of either not at all or a little bit. Broadly, most people end up in interpretability on some level because they're kind of curious about how minds work. There are a lot of neuroscientists who end up becoming interested in mechanistic interpretability. I think it draws a pretty philosophically and cognitive-science-minded set of individuals, because fundamentally I think what we're doing is a type of cognitive science.

We're just very, very interested in, for lack of a better term, how minds work, how learning works, and how all of this is possible in the first place. I think that's what a lot of our conversations are focused on.

Have your personal feelings about the possible consciousness of Claude changed at all in recent months?

I don't know. I think it's a pretty distinct possibility. I don't know that my position has changed that much; I just think it's highly uncertain. I don't think anyone has that convincing a definition of consciousness that would necessarily exclude Claude. Claude certainly has a lot of emotive and qualitative aspects that, in humans, we would associate with consciousness, but that doesn't necessarily mean that it has consciousness.

If I had to guess, just for the fun of it, I would say a little. I don't know how much, but a little bit. For me, I think my sense has changed a lot. I'm still radically uncertain, but I used to be more of the sort of, “It felt like I couldn't dismiss it, but probably not.” In my bones, it felt like probably not. I don't know why. It's a computer. It's made of something totally different. I had this sort of alien-mind, don't-anthropomorphize prior that would inform that.

Now I'm like, boy, the reason I am very confident you're conscious is that I'm conscious and we have basically the same structure. The more things that come out where it's like, well, here's another analogous structure between human cognition and model cognition, at some point it starts to feel like, for me, not just that I can't dismiss this anymore, but more like the evidence is really starting to add up that it really could be the case.

I don't even know what my probability would be at this point, but I think it's approaching more like 50/50 than it used to be, when it was under 5%. Don't rule it out. Don't be caught flat-footed on this. Now I'm like, man, the trend is really strong in the direction of analogous structures kind of adding up to something I think is pretty compelling.

Yeah, obviously people have all types of opinions about this, and it's a weirdly emotional topic for a lot of people, probably just because it gets at the fundamental identity of what it means to be human. But for me, at least, the most compelling Occam's razor is that consciousness is a computational mechanism of some kind. There are other possibilities that make sense, but that seems like the most likely one.

The brain is doing types of computation that transformers aren't. Maybe those really matter; maybe they don't. Certainly, brains are a lot more complex. People talk about this all the time. It's crazy how much more energy-efficient brains are. We've got, like, 2× just on raw neuron count, even over today's biggest models. Our neurons are also way more complex, and they can do way more sophisticated computations. They're interconnected in a way that gives them the ability to do recurrence.

We've got a lot more computation going on in our brains than even the most sophisticated models do today. So you could also believe that consciousness is computational and believe that, for various reasons, models don't have it. My “a little bit” is tongue-in-cheek because I really don't know. I don't know if it's a threshold, if you either have it or you don't, or if it's more like a continuum.

If it's more like a continuum, and we have all these attributes, and a jellyfish has this amount of the attributes, is Claude more or less conscious than a jellyfish? I'd say—I don't know—probably more conscious than a jellyfish, probably. Is Claude more conscious than a rodent? Probably not. That's where I'm at: somewhere between a jellyfish and a mouse.

Well, I'm glad to have had the chance to experience a little bit of the lunchtime conversation at Goodfire. I think it is good to honestly get serious people, more serious people, on the record saying, hey, this is something we really should be taking—not for granted by any means, but seriously—these days, because I'm definitely persuaded that if we mess it up badly, it could be real, real bad.

Yeah, I see absolutely no reason that anyone should be particularly confident on this topic, given the potential downside. I think intellectual humility is important.

Yeah, absolutely. This has been great. Anything else you want to mention that I didn't touch on myself, or any parting thoughts or words of wisdom you'd leave people with?

I really hope that we build something with Silico that will really empower a lot of people to advance science—science of all types, but especially life sciences and safety. I encourage people to reach out and apply for a grant. If they're individual researchers or academics for whom this license is more expensive, and if they're more institutional organizations that are just looking to accelerate their research, they can reach out to us as well. And we'd be really excited to work with them and partner on research. At the end of the day, accelerating science is the greatest mitzvah. It's the whole reason that we would build AI in the first place. I hope that we're playing our role in that, and we just want lots of people to try it and use it and hopefully do amazing things and hopefully teach us how the product can be better and just keep improving from there.

14. Episode Outro

Dan Balsam, always a pleasure. Thank you for being part of The Cognitive Revolution.