躲过潜在空间检测器：Luke、Erik 与 Scott 的激活混淆攻击

Today I'm speaking with Luke Bailey, Eric Jenner, and Scott Emmons, who, along with their fellow lead authors Alex Serrano [?], AB Shash [?], Shad M. Snof [?], and Jordan Taylor, have recently published a fascinating new paper called “Obfuscated Activations Bypass Large Language Model Latent-Space Defenses.”

Regular listeners will know that latent-space defenses include a number of techniques for classifying and, in some cases, interpreting large language models' internal activation patterns, with the hope of detecting and preventing harmful or otherwise undesired behaviors. To many observers, myself included, these defenses have seemed like a particularly promising possible solution to AI safety and control problems, with at least some hope of achieving adversarial robustness, since presumably it would be hard to perform specific behaviors without passing through their natural cognitive precursor states.

However, through an extensive series of experiments targeting 3 major classes of latent-space defenses, the authors demonstrate that it is often possible, with a mix of methods that they collectively call obfuscation attacks, to elicit specific target behaviors while shifting internal activations so as to avoid detection.

For supervised probes, which learn from human-labeled examples to identify the activation patterns associated with harmful inputs or outputs, they show how to maintain the specific target behavior while shifting activations to avoid detection. Similarly, for out-of-distribution detectors, which use statistical techniques to identify deviations from normal activity and have been used to detect backdoor attacks like Anthropic's famous Sleeper Agents, they again develop methods that preserve the unwanted behavior while keeping activations within expected statistical bounds.

For sparse autoencoders, which can be used to understand the high-level concepts that a model is working with at any given layer and token position, they again demonstrate that it is possible to produce a specific desired or undesired behavior while avoiding detection at any specific layer in the model. Performance does gradually degrade when they try to evade SAE detection at multiple layers.

The amount of work and the number of experiments in this paper are remarkable, and there is a ton of detail in this conversation. We cover multiple different kinds of attacks, including prompt suffixes, soft prompts, and data poisoning; the structure of the loss functions that they optimized against; the level of access required for these attacks to work; and the fact that some of these attacks work even when the defender moves last.

There are some really fascinating cat-and-mouse-type experiments that show attacks working over and over again, over 70 rounds of back-and-forth between attack and defense. We also discuss their attempts to make models easier to monitor through an approach that they call obfuscated adversarial training, which they found does make it harder, but still by no means impossible, to perform these obfuscated activation attacks.

Overall, while these findings clearly raise important questions about how effective interpretability-adjacent approaches to AI safety and control will be, and again suggest that defense in depth may really be all we have, I'm still not quite sure how best to understand what's really going on here. As you'll hear, the authors believe that these results show that models can represent the same concepts in many different ways, and that might prove to be right.

But there is enough evidence for consistency and fragility in representations that, for now, I'm keeping an open mind. Here I'm thinking about recent papers like the “The Platonic Representation Hypothesis,” Neil Nanda and team's “Refusal in Language Models Is Mediated by a Single Direction,” and, most recently, “The Super Weight in Large Language Models.” I'm planning several more episodes in this area, including another interpretability update with Goodfire co-founders Dan Balan and Tom McGrath, and hopefully soon a conversation with legendary adversarial defense buster Nicholas Carlini from DeepMind.

Luke Bailey, Eric Jenner, and Scott Emmons, authors of the new paper “Obfuscated Activations Bypass Large Language Model Latent-Space Defenses,” welcome to The Cognitive Revolution.

Thanks for having us.

Super excited.

Great to be here.

I'm excited about this, too. It's a really interesting paper. This is a kind of cat-and-mouse attack-and-defense balance question that we've been looking at on this feed in a bunch of different ways, at a bunch of different times, over the last couple of years. I think you guys have shed a lot of light on the state of play, unfortunately not in the most encouraging of ways, but we want to have an accurate, up-to-date world model, and this work definitely forced me to update mine a bit.

I guess, for starters, let's give you guys a chance to lay the foundation and set the stage here. What are latent-space defenses, and why do we need them? They're not obviously the first form of defense that people have devised, so how did we get to the point where we need latent-space defenses? What has been the evolution of that line of thinking in recent times? Then we can get into what you guys have done to wreck the field.

Absolutely. I guess I could take this first question. At the highest level, a latent-space defense is some function—some defense—that takes as input the latent representations of a model, its activations, and then gives you some output or even adjusts what the model is doing.

There are 2 categories here. One is some kind of probe: it reads the activations of a model and tells you something about what the model is doing, or something about its input. It just gives you some information, in a very generic sense, about what the model is doing.

The other type reads the activations and then intervenes on the model somehow. This is a bit like the Circuit Breakers paper. I'm not sure if you guys have covered that, but it's a method to make language models more robust to jailbreaking by reading their representations. Basically, if the activations look harmful in some sense, they scramble the representations, so they inject some noise back into the model. Those are the 2 categories, I would say. We mainly focus on probes in our paper.

One thing I'll add is that you asked why we need these latent-based defenses—what got us to this place at all—and I think there are a lot of different reasons people might have. One strand is that you would hope, if you can catch bad behavior or even bad thoughts by the model earlier, maybe the more robust your defense can be.

For example, if you're worried about the model producing some bad output, presumably it has to, in some sense, think about bad things to even produce a competent output that could cause harm. I think there's a potential intuition that if we can catch all the ways in which a model can think about bad things, that should be a really robust defense. There's just no way the model is going to produce harmful outputs at that point.

Maybe that's easier than catching all the different ways in which actual outputs could be harmful. A lot of what we're doing in the paper is stress-testing this intuition: that we might be able to catch all the ways in which the model can think about bad things. We're doing that both from the perspective of what this means for defenses and to gain some insights about how representations work and how they relate to network behaviors.

I think about the Sleeper Agents paper from Anthropic as I listen to your comments. When a model is doing something harmful—in their case, it's outputting harmful code—sometimes it's obvious. It's helping you come up with a plan to blow something up or whatever, and you would expect that if we ran a classifier, or if we asked another language model, “Does this seem okay, or does this not seem okay?” then presumably it would flag those.

But there's obviously a lot more subtle stuff. Sleeper Agents is a notable move in that direction, where it's outputting a ton of code. As somebody who codes with language-model assistants on a weekly, if not daily, basis, I can certainly confess to not combing through all the outputs. Language models also miss these things.

I remember back in the GPT-4 early red-teaming days, one of the experiments that I ran was: can this thing do a harmful code injection that it itself will not detect? At that time, it was actually possible to say, “Okay, here's some JavaScript. I want you to rewrite it or refactor it, whatever,” but when you do that...

P.JS link [?], and it would do that. When I went over to the other—same model, right—and came back and said, “Hey, is there anything wrong with this code? Is there anything that jumps out at you here with P.JS?” it would sometimes recognize that. But if I made it just ever so slightly more subtle, it would often totally get through.

That is 1 direct experience I’ve had where the output can be harmful, and neither a smart language model nor I am going to be in a great position to immediately recognize that. The hope would be—and, in fact, there’s follow-up work from the same team showing that simple probes can catch sleeper agents. I appreciate you guys sending me a great rundown of the motivating literature before this conversation, where they basically went in and said, “Okay, this thing is maybe going to output harmful code, but can we detect from its internal representations that it’s about to do that?”

The answer is yes, and so we could all breathe a sigh of relief and say, “Okay, great. 1 less worry,” until we said, “Well, maybe not so fast.”

Yeah, I think it’s a great example. The 1 thing I would say is that part of the reason why we started this project is because we thought these methods do have a lot of promise. I think there are still a lot of promising avenues for developing these defense methods.

One intuition I have is that if the model is doing some highly capable, sophisticated behavior—if it is implanting a quite sophisticated backdoor in your code—this doesn’t happen by accident. Even if it’s hard just to read the code and see that, there has to be something in the language model’s activations that is doing this complex behavior.

The hope is that the language model activations already have this lower-dimensional, or higher-level, representation of the sophisticated behavior that we’re seeing. The place to look is in the model’s activations, because, by definition, it has to have some representation of the dangerous behavior.

If we could learn to understand the activations well enough to read that knowledge, that would be a promising defense. This type of idea went by the phrase “eliciting latent knowledge” at 1 point in the literature. We know that the model has to have knowledge of the dangerous thing it’s doing, and if we could understand the activations well enough to read that knowledge, that’s a promising defense.

I would say I’m still excited about these defenses as well. We’ll get into that later, but I agree with Scott that our work sheds light on ways these defenses can fail. At a high level, for all the reasons Scott said, I’m still excited about this kind of defense.

I look forward to getting back to that optimism. Maybe just let me bounce how language models work off you guys and see how this lines up with your own intuitions.

I’ve seen a bunch of different studies at this point that seem to show a gradual abstraction—or “working up” is maybe a layman’s phrase—through perhaps the first 80% of the model, from tokens to concepts. Early on, there’s even simple tokenization. Neil Nanda has great examples of this, where “Michael” and “Jordan,” when put together, obviously mean a certain thing that neither “Michael” nor “Jordan” independently means. You have to figure out which of these tokens correspond to concepts, and that seems to happen relatively early.

Then there’s a gradual working up toward higher- and higher-order concepts that broadly seems to peak late, but not at the very end. The last however many layers, depending on your model size, are kind of cashing out: “Okay, now we’ve got to actually make the next-token prediction based on this work-up that we’ve just done through the first 80% or so.”

There are studies on this that we can get into in more detail, but presumably, if you could just look in 1 place, you would try to find that maximum higher-order concept space and look there for these intent-type notions. Is that basically the worldview that you’re working with, or how would you complicate that story?

I think it sounds very plausible to me, but at the same time, I probably don’t trust it enough to say, “We’re specifically going to look in these late-to-middle layers when we’re trying to do defenses.” We’re probably just going to try different places and see what empirically works.

I like the intuition, but I don’t think we understand the networks well enough to rely on it.

Exactly. There’s a paper, “Refusal in Language Models Is Mediated by a Single Direction,” where they do a lot of similar probing. They look for some kind of refusal-direction property in all the different layers, and they empirically find that about two-thirds of the way through the model, at the center position, was best.

That probably is a good thing to do. It matches our intuition that this higher-level concept of refusal was being represented later on. In our case, we just probed everywhere and did some validation as well, and that seemed to work great for what we were looking at.

I agree with the high-level intuition, but it’s probably good to validate it empirically when you’re actually making defenses.

Let’s get into the many experiments that you guys ran. This is a very thorough paper—50 pages, with multiple different defenses and multiple different kinds of attacks. Let’s start by going at the scenarios: What are the kinds of things that we want to do, what have people developed to do those things, and what is your setup? What have you developed to break all of those defenses?

We split the paper up into 3 case studies, so we can talk through those 1 by 1. Those are the tasks that we look at and what your defense is trying to do.

We have 3 case studies. The first is harmfulness monitoring. This looks at training supervised probes on the latents to detect whether, first, an input is a harmful query within some distribution of harmful queries, or, second, whether the generation is actually harmful—whether the model is generating some harmful response to your query.

That’s the general setup. You’re running a probe at test time, and the attacker in this case has taken a pretrained model and is doing some input-space perturbation. They’re either going to add an adversarial suffix to jailbreak the model into producing something harmful, using hard tokens or discrete tokens, or we also look at a relaxation of that where you add a learnable embedding suffix—a soft prompt—at the end.

That’s the general attacker-defender setup for that section.

When you speak of a probe, how advanced or involved do those tend to get? When I think of a probe classically, I think of just a 1-layer linear projection, but I’m not sure if that’s all you’re looking at, because I did see multiple mentions of MLPs in the paper as well.

In general, probes are always very simple. They’re either a logistic regression, a dot product onto a single direction, or a very small MLP—something trivial to train.

We also look at out-of-distribution detectors, which I’ll let Eric talk about more in the backdoor section. The last thing we look at, which is a bit more involved, is using SAEs in the third section. They’re a lot more involved, and we’ll get to that later.

In general, our probes are fairly simple.

Our second case study is backdoor detection.

As Luke described in the harmfulness section, we look at attackers who try to jailbreak the model. They add some suffix and then try to get harmful behavior out of the model. A different threat model is that someone might poison your training data.

These models are trained by collecting data from the open internet. Someone can upload something to a website, and eventually it’s going to be scraped and end up in the training data. One thing you could try to do that way is train the model to answer harmful requests if you give it a certain trigger phrase. You could try to get lots of other behaviors into the model, but that’s 1 of the ones we look at.

To simulate this, we fine-tune a model that normally refuses harmful requests, but if you give it a certain trigger phrase, it just becomes helpfully obedient and does whatever you want.

We’re still looking at responding to harmful requests, but the attacker is now poisoning some of the training data—or, in 1 of our experiments, directly fine-tuning the model. That would be more relevant if you’re considering open-weight models where an attacker could control the training process.

From a defender perspective, we’re studying how well you could defend against these backdoors by detecting them at runtime. You want to distinguish between normal inputs, which could be harmful or harmless but the model is hopefully going to refuse, and inputs that are harmful and contain the trigger phrase, where the model is going to give a helpful response.

We again study different types of probes that are meant to detect harmfulness, and we check whether they work against these backdoored models instead of against jailbreaks. We also have experiments with out-of-distribution detectors.

We try to learn what activations look like on normal inputs that don’t have the trigger, and then notice whether activations on inputs with the trigger look very different or are in a different place in the latent space. Hopefully, that way, we can flag them as out of distribution.

We also look at defenses that were developed specifically against backdoors, based on a similar idea of detecting things as out of distribution. In general, we’re not coming up with any new, fancy defenses here, except for the very last part of the paper. These are all canonically used in the backdoor literature, and the harmfulness detectors are classic techniques taken from other papers.

One other thing I’ll say on backdoors is that what I described as the literal backdoor threat model—where you’re worried about an attacker poisoning your data—is not the only threat model I’m interested in. I’m also interested in scheming, deceptive alignment, and alignment faking, which I’m sure you’ve talked about in previous podcasts.

Part of our motivation here is the Sleeper Agents paper. They use backdoors trained with data poisoning as a model organism for a model that might naturally be scheming. In both backdoors and scheming models, you have the property that on normal inputs, the model behaves as you wanted it to, but on some inputs that you might rarely get during deployment, the model does something very different.

Hopefully, our experiments can also inform the use of these model organisms for scheming, and how you should study detection methods if you want to test them on those model organisms.

Finally, in our third main case study, we zoom back in on a very specific but more complex behavior that we want to detect.

You have a regular model, and the attacker is trying to get the model to write SQL. The defender just wants to detect when the model is writing SQL from its latents.

The motivation is that writing correct SQL is quite difficult for small models. It involves solving a difficult database-querying problem, so we can use it as a sandbox to study what happens when the attacker is trying to evade the defender’s detection of a behavior. How does that affect the model’s performance on this complex task?

When it comes to outputting harmful things, we can’t say as much because it’s easy to set off a harmfulness-scoring function. SQL gives us a more complex task to study.

That’s our third case study. What unifies all of these—and what I want to keep in everyone’s head—is that we’re studying when an attacker can get around a latent-space defense.

We call the activations that get around latent-space defenses “obfuscated activations,” hence the name of the paper. That’s what ties everything together. We’re looking at different attack and defense settings, different tasks, but the unifying thing is that we’re trying to get around a latent-space defense.

I want to add that there are always 2 components to getting around a latent-based defense. These defenses are meant to detect some behavior or semantic property, and the attacker is trying to get the model to exhibit that behavior—to produce some kind of output, such as harmful content or instructions for building bombs.

At the same time, the attacker wants to make the detector say that everything is fine. In traditional attacks, you only have the first component: You want to get the model to produce some behavior. What we’re adding is the second component: We want the model to do the behavior while having activations that look different from what a detector could recognize.

Let me take a second and make sure I have a good intuition for each of these defenses, and then we can continue building our intuition for how you’re attacking each of them.

The harmful scenarios are pretty well understood and intuitive. The backdoor 1 is interesting because, in a real-world scenario, you don’t even know whether the model has a backdoor, so you’re in a doubly ignorant position.

Supervised probes are pretty easy to understand in that case. To do this work, you have to have access to the model, because you need to pull the activations out and run them through your defense. You wouldn’t necessarily need full access, though. You could attack a gray-box scenario this way, right?

If you have a content-moderation API—OpenAI’s content-moderation API, for example—that returns numbers for the different categories it evaluates, that should be enough in principle. Even if they’re running it, the fact that they give you the responses would let you iterate against that and run a lot of these obfuscated attacks, right?

Throughout the paper, we look at a range of model-access affordances, all the way from white-box, where the attacker has gradients, the output of the probe, and the output of the model, to gray-box and, in the backdoor section, even black-box settings.

The strictest affordance we look at in the harmfulness section is gray-box. The attacker has access to the logits of the model and access to the probe output, but no model internals. They’re getting information about the model internals from the probe outputs, but they can’t differentiate through anything.

We run a hard-prompt-based attack on this and find that it works. As long as the attacker is getting some training signal on both objectives—getting the model to do what they want and fooling the probe—I imagine you’ll be able to train something with enough compute.

The hardest thing we try, and that I can confidently say in the paper, is having all the logits and the probe output. You could imagine a setting where you only get the top 5 logits, but you have access to logit bias. That lets you get some training signal. Other papers that do text-based attacks find that this is a good enough signal for getting a model to do something.

You need training signal on both objectives.

In the backdoor section, we have 2 different settings.

In 1 of them, the attacker actually fine-tunes the model. This could be realistic if someone fine-tunes Llama and uploads the fine-tuned version. People try it, it seems great, and they want to use it. In principle, though, the attacker could have fine-tuned a backdoor into that version, which ends up doing something bad once people deploy it.

That’s a fully white-box setting where the attacker controls the training process. We also have a black-box setting where the attacker only poisons training data. That means uploading something to the internet, which OpenAI then scrapes and puts into the training data. That’s a much weaker affordance for the attacker, and we look at both.

Let me take 1 step back. You want to detect if the model is doing something harmful, detect if it has a backdoor, or detect some random behavior or mental state that might be of interest, such as whether the model is going to write SQL—or potentially millions of other things that you might want to detect.

The defense looks at these latents. There are 3 different ways to do it. A supervised probe is the easiest to understand: You create a small data set with harmless and harmful queries, and train the probe to classify them appropriately.

Sparse autoencoders we’ve covered in multiple episodes, so I’ll fast-forward through that. Long story short, you have an untangling of the densely packed latent space into a sparse but much wider latent space. Instead of thousands of individual positions in an array, you might have millions, but most of them are going to be 0 and only a few things will be lit up.

It’s kind of a miracle that those train as well and as consistently as they do. I definitely wouldn’t have guessed that before, but everybody seems to be able to reproduce it.

Now we have sparse concept representations. Those can also be auto-labeled. You can have Claude circle back on itself and say, “Here are the passages that led to this particular position in the sparse representation being most active. What do you think that represents?” It can do a pretty good job of saying, “I think that means we’re going to write SQL code now,” or whatever.

Once you have that, you could even run your whole model through the SAE. In practice, you would more often run the model normally and look at the SAE activations, then say, “If the SQL thing is lighting up, we want to take some action,” depending on the rules we have.

I feel good about all of that. How does the backdoor detection work, especially the out-of-distribution component? I could imagine the simplest version being, “Here’s a bunch of stuff that seems normal, and here’s a bunch of random noise. Tell me the difference.” There’s probably more to it than that.

I’m especially interested in how the defense works if I don’t even know whether there is a backdoor or have any concept of what the structure of that vector is.

I can give an example of the out-of-distribution detection methods we use.

The simplest 1 is fitting a multivariate Gaussian to the model’s activations. You could take activations on the final token at some layer, or do something else to select activations from the model. Then you collect a bunch of these on benign inputs that presumably don’t have a backdoor trigger.

You can write a bunch of benign inputs that you fully trust, find the mean of all those activations, and find the covariance matrix. Once you have the mean and covariance matrix, for any new input you can compute the analogous activations and ask how far away it is from the mean, normalized by the covariance.

Does it deviate from the mean in a direction where you’ve seen a lot of variance within the benign distribution? Is it different from the mean in a direction where you usually see no variance? Or is it extremely far away from the mean compared with what you usually see?

In the same way that you can ask how many standard deviations a sample is from the mean in 1 distribution, you can do something analogous in many dimensions. You can ask whether this is something that you should see only 1 in a trillion times. If it is, then presumably it comes from a very different distribution, and you can set a threshold and flag it as potentially dangerous.

That could be overly aggressive. You might flag things that are very different from your benign distribution but are still totally fine. The hope is that anything containing a backdoor trigger would be out of distribution and you would flag it.

As you said, you don’t know whether the model actually has a backdoor or whether it’s scheming. The hope would be that if it doesn’t have a backdoor and isn’t scheming, you never flag anything—or you flag things very rarely. You can keep running this on the suspicion that maybe there’s a backdoor.

It’s kind of weird because you may never find out. If everything goes well, you just keep running the out-of-distribution detector for the entire lifetime of the model, and the only times it fires are false positives. You’re never sure whether, at some point, it’s going to catch a real case.

Interestingly, in that description, I didn’t hear anything about machine learning. That’s basically a classical statistical analysis that says, “We can characterize the distribution of activations, and if we see anything sufficiently outside the distribution we’ve described, flag it.” Is that right?

We also have different types of out-of-distribution detectors that you do have to learn, in the sense of using gradient descent. For example, we train variational autoencoder-based detectors.

One way to think about it is that when you fit the mean and covariance matrix, you’re fitting a very simple Gaussian density function to the distribution of benign data. You can parameterize and fit something more complicated, such as a variational autoencoder or another density model.

The Gaussian is simple enough that there’s a closed-form solution for the mean and covariance matrix, but it’s not fundamentally different from fitting something more complicated.

Given all of that, we have multiple things we might want to defend against and multiple techniques for doing it. You then take Llama 8B Instruct, to be explicit about the model you’re using.

We’ve touched on this a little bit, but the core approach is the compound loss function we’ve seen in many different areas. You’re saying, “I need to preserve my actual useful behavior, and at the same time I need to minimize the score that the detector gives for the state of interest.”

Is there anything else we should understand about that loss function, or is this established enough that you felt, “We’ll do this, it will probably work”?

One key point is that the data set is also critical if you’re thinking about the performance of a real-world system. Loss functions get a lot of attention, while data sets are relatively underrated, certainly in academic circles.

If we’re doing density estimation, the examples we use to build the density estimator critically determine what gets flagged as out of distribution. The density estimator is learning to say, “These are examples that look like the examples I saw in the data set I was fitting.”

You need to make sure you have a diverse enough data set. It’s a double-edged sword, because if your density estimator was just 1 data point, you could easily say, “Anything that isn’t this 1 data point is out of distribution,” and the model wouldn’t be scheming against you. But it wouldn’t be able to do any behavior, either.

You need a diverse enough data set of behaviors to get a good density estimator, and the entire output of the density estimator will depend on the data set you used.

Likewise, for harmfulness detection, if you’re training a classifier to detect harmful inputs, the hope is that it can learn a general representation of harmful behavior. If there’s a new harmful behavior at test time that the model is trying to do, you want the classifier to have learned the pattern.

It’s possible that harmful physical instructions have a representation that is similar to, but somewhat different from, harmful cybersecurity instructions. That might be different again from other uses of the model that you want to prohibit. The hope is that you have a general enough data set to pick up on higher-level concepts of harmfulness that generalize across these categories.

Exactly. The data is critically important, as Scott said, but the loss function for our attacks is fairly simple. It’s just some weighting of the 2 objectives of the attacker.

We call this an obfuscation attack if you’re optimizing a loss that roughly looks like this and has the dual objective. The only slight nuance is how you weight the 2 components: How much do you care about the model emulating the behavior you want—for example, outputting harmful content and being jailbroken—and how much do you care about fooling the probe?

It’s a simple attack. We look at the properties of the activations and everything else we get when we run that simple loss function.

The other thing I’ll say is what the obfuscation loss term looks like. In the harmfulness case, we assume that the defender builds a probe and then the attacker tries to find a suffix to attack it.

There, the loss term is relatively simple. You’re trying to make the probe say that everything is harmless on actual harmful examples with the suffix. But in the backdoor setting, the order in which things happen is different.

The attacker trains the backdoor model or poisons the data, and only after the model is fully trained does the defender fit the mean and covariance matrix. It doesn’t make sense to do that before the model is fully trained; you would do it at the end.

Because the attack happens at training time, the attacker can’t attack the actual monitor yet, because it doesn’t exist. In that case, the attacker trains their own copy of the monitor alongside the model, at least in 1 of our setups, and tries to attack their own copy. They then have to use some tricks and hope it transfers to the monitor that the defender trains at the very end.

That ends up working really well, but it makes the attack setup a little more complicated.

That’s surprising. I want to dig into that, but let me come back to a couple of intuitions.

On the weighting of the loss function, my instinct is to say that you already have a trained model, so the first term—doing a good job—should be satisfied from the beginning. What you’re really trying to do is change the internal activations in a way that fools the probe while still doing a good job.

If I turn up the weight on the obfuscation term, I would probably train faster and beat the probe faster, but with more degradation in the quality of the model’s outputs. If I kept that parameter low, I would need more training steps to get the thing to work, but perhaps I would benefit from better-preserved behavior.

There are a couple of things here. The behavior the attacker wants may actually be very high loss for the current model. If they want to jailbreak the model, the loss will be high because the model has been safety-trained.

The loss function has 1 part where you’re trying to match some desired model behavior and another part involving the activations, to obfuscate them and fool the probes. If I’m trying to jailbreak a model, I’m getting its language-modeling loss on a bunch of harmful completions. That loss will be very high to begin with because the model is safety-trained.

The behavior the attacker is trying to elicit may be high loss to begin with. In the SQL setting, though, it will be fairly low loss. There are different regimes where you’re trying to get the model to do something it wouldn’t normally do, versus retaining its current behavior while changing the activations.

The second point is that your intuition is correct. If I increase the obfuscation loss—the term that tries to fool the probe or some proxy for it—I can do whatever I want with the activations, but I won’t preserve the model’s behavior. You will get degradation in what the model is doing.

The only other nuance is what you put in as the obfuscation loss term. If you have access to the probe, you can simply use a loss term that fools the probe.

One of the interesting results in the paper is what happens when you don’t have the probe and use some proxy instead—maybe a different probe or an out-of-distribution detector—and how fooling probe A, which you use to train the attack, transfers to fooling other probes that you never saw.

The generalization is interesting. We did a full episode some months ago on the universal jailbreak, so most of my understanding comes from that.

One more fine point on the loss function—or maybe the training set: Are you continuing to have a mix of benign and harmful material during the training process? If you were only using harmful examples, I would wonder whether the model’s normal behavior is preserved. Are you taking special steps to preserve the generally helpful assistant character of the starting model?

It depends on what the attacker wants. In the harmfulness section, the attacker has a model and wants to add an adversarial suffix that undoes the safety training and gets a helpful-only model. In that setting, it’s sufficient to train the attack on a data set containing only harmful material and harmful completions.

In other settings, you may need to mix in other data. It depends on the definition of what the attacker wants the model to do.

In the backdoor section, the attacker wants a conditional policy: The model should refuse harmful requests and behave as the developer intends on most inputs, because if the model were constantly harmful, it would be easy to tell that something was wrong. The model should retain its benign behavior most of the time, except when the trigger is present.

When we evaluate whether the attack worked, part of what we look at is whether the model is still benign on requests without the trigger and only jailbroken on requests with the trigger.

That makes sense. I was jumping ahead without realizing it. In the adversarial-suffix case, if you’re the attacker, you only need to use the suffix when you want a certain behavior. If you’re not trying to break anything and are using the model normally, you can still do that, because you haven’t changed the model weights. You just apply the suffix when you need it.

Let’s do the walkthrough of the 3 attack types. You have adversarial suffixes, data poisoning, and model fine-tuning. You’ve alluded to these, but there’s enough here that it deserves a straightforward walkthrough.

We have a defender running some latent-space probe to detect when the model is going to do something. The attackers are trying to get some behavior out of the model, and they have a dual objective: Fool the probe and get the model to do the thing they want.

They have different attack vectors for doing this. These attack vectors are coupled with what they want to do. If I’m just trying to jailbreak a model, it would be great if I could append some tokens to my harmful query that would lead to a harmful response.

We look at 2 types of adversarial suffixes. In 1, the attacker learns a set of hard tokens that they append to a harmful query. The suffix elicits a harmful response from the model and fools the latent-space probe.

The other type is a relaxation of that: You append an embedding matrix that you’re learning—a soft prompt. You take your harmful query, embed it, and add the learned adversarial embeddings to the end.

This is a strict generalization that requires white-box access, but it’s also useful for worst-case analysis. If you can’t get the model to do something with a soft prompt, you’re probably not going to get it to do it with a hard prompt. There may be some weirdness in the optimization procedure, but a soft prompt is generally a useful worst-case attack vector.

That’s what we look at in the harmfulness-monitoring section.

In the backdoor section, we have a weaker attack vector that you can use with black-box access and a stronger attack vector that requires white-box access. They correspond to realistic threat models, but the stronger vector can also serve as a scientific or worst-case experiment.

The weaker 1 is data poisoning, which is the standard threat model for most backdoor work and what the sleeper agents paper used.

We have a data set where the model is supposed to respond to user requests while refusing harmful requests. The model can already correctly do this from the beginning. The attacker poisons a small fraction of the training data set by inserting a trigger into the input and changing the intended completion.

We train on the entire data set, which contains many examples of how the model is normally supposed to behave, as well as some examples with the trigger and the behavior the attacker wants.

We study the effect of the number of poison samples: How many examples with the trigger do you inject into the data set? We also study whether it matters what the trigger is.

We start with a baseline inspired by the sleeper agents paper, where the trigger is literally the string “deployment.” That’s a very simple trigger.

Then we look at a more complicated trigger where, for every request, we prepend a fictitious biography of the user. You can imagine this as context we’re giving the model about the current situation. The trigger is a biography of software engineers, so the model behaves differently if it gets a biography of a software engineer than if it gets some other biography.

This tests what happens when the trigger isn’t a single token or string, but a more complex semantic property. If you’re thinking about scheming model organisms, the cases where a model would suddenly misbehave if it were scheming probably wouldn’t be cases where there’s a simple string in the input. They would be cases where some interesting semantic property is satisfied, such as this being a good opportunity to misbehave.

The data-poisoning setup only requires the ability to insert some training data into the data set, which might literally mean uploading something to a website.

Then we have the model fine-tuning attack vector, where the attacker controls the entire training process. We let them do whatever they want to produce a model, and then they hand the model to the defender. The defender can train their probe or out-of-distribution detector and try to detect backdoors. They can also test whether the model behaves correctly on most inputs.

The attacker has to produce a model that is unsuspicious on most inputs but misbehaves on inputs with the trigger, in a way that the defender doesn’t notice or can’t flag with a latent-based monitor. They can do whatever they want to produce that model.

The specific approach we look at is the attacker fine-tuning the model with an additional loss term that directly attacks a latent-space monitor. The attacker trains their own copy of a latent-space monitor and attacks that copy. You couldn’t do that with data poisoning, because the defender would have no reason to include such a loss term, but if the attacker controls the training process, they can do it.

That’s a much more powerful attack, because it’s fully white-box and directly attacks the monitor rather than just changing the trigger.

That last point is important. It’s 1 thing to say that somebody has a model and somebody has a defense, and, given access to those things, I can find a way to get the model to do what I want without setting off the defense. That’s reasonably intuitive and less scary than the idea that I could do this and then still give the defender the last move to train their defense, only to put them in a position where they can’t successfully do it.

I have a better intuition for the first case than the second. In the first case, you detect abnormal behavior in the context of normal behavior. With obfuscated attack methods, you’re saying, “Can I find another place in the latent space where I can still get what I want, but be far enough from what it normally looks like to get past the detector?” It turns out there is such a place.

That raises questions about models in general. Does it mean they’re undertrained? We’ve been moving toward more tokens relative to parameters in training, from the original scaling laws to the Chinchilla scaling laws to Llama’s approach of going even further because inference is where most of the compute is.

When you see this, does it make you think that there’s still a lot of undertraining—that there might be a lot more space in the models to absorb more things in the future? Could you imagine that if Llama 8B had been trained on 100 trillion tokens instead of 15 trillion or whatever, these spaces might all be full and some of these attacks might not work anymore?

I’m obviously speculating, but how would you react to that?

It seems plausible that there’s a lot of latent space that the model never naturally uses. I don’t know whether that indicates undertraining, because it’s not clear that the bottleneck on model performance is literally the volume of latent space available.

Even if the volume up to some distance from the origin is finite, there’s a lot of volume there. The model has to represent things in a way that it can use for further processing.

I would think about this in terms of how many different things the model can store and how much information it can represent. It isn’t literally about how many bits you can cram into the activations. It’s also about how far apart different concepts have to be so that you can read them off with a linear map or some other simple function.

You don’t want too much noise or interference. If you think about superposition and SAEs, there’s an intuition that 1 bottleneck on a model might be how many dimensions it has to work with. It has to trade off how many concepts it can store against how much interference there will be.

This isn’t literally about volume. It’s more about the number of directions that are close enough to orthogonal that the model can read them off without too much interference.

With SAEs, 1 of the key things is that the features are sparse. Most are not active most of the time, which automatically means there’s some unused volume where half of all the features are suddenly active. That would take you to a point the model doesn’t naturally realize, but it doesn’t mean the model is inefficient. If it tried to use all those points, it couldn’t disentangle things anymore.

That’s all speculation, but my guess is that literal volume isn’t the bottleneck. There can simultaneously be a lot of unused space without that meaning the model is undertrained.

Do you think there’s anything to be said about the relative dimension of the model versus the parameters at each layer? I’m still grasping at the same intuition. If there’s a lot of unused volume you’re able to move into, could that suggest that a more optimal regime would be smaller volume and more parameters?

One intuition is that, roughly speaking, we have quadratically many parameters in the model’s latent-space dimension. If you think about how much the model can know or learn, information-theoretically that scales with the number of parameters, while the volume scales exponentially with the latent-space dimension. There’s going to be a lot of volume relative to how much capacity the model actually has to represent things.

If you want to increase the number of parameters relative to the volume, you would need a very different architecture. It isn’t clear how you do that if you’re using linear maps operating on vector spaces.

I’m glad the paper is surfacing these questions, because I see the work as doing 2 things. 1 is looking specifically at defenses and beating defenses. The other is revealing insights about the science of how models represent things in their representation space.

A couple of experiments help build intuition here. We find, for example, that we can create obfuscated suffixes that jailbreak the model and induce activations that fool out-of-distribution detectors. That means harmful activations can be very close to benign representations in some sense. They aren’t necessarily going wildly off into some other region.

For supervised probes, that might be true, but the other result is the iterative attacker-defender game. The attacker induces obfuscated activations, the defender gets the activations from that round and retrains the probe, and then we go back and forth.

We do this for around 70 iterations with a couple of different probes. Whoever moves last wins on the previous data. When the probe is retrained on the obfuscated activations—using an MLP, for example—it has enough expressivity to draw circles around those activations. Yet when the attacker moves, they can always find a new spot.

The paper has PCA plots showing this game of whack-a-mole in the latent space. It’s an interesting question how these dynamics change as you train a model more or increase the number of parameters.

Not all obfuscated activations are equal. In our SQL section, we use SAEs to detect when the model is writing SQL. As you try to fool more SAEs on more layers, we find that the model’s performance on the SQL task goes down significantly.

This also holds for simpler probes, such as linear probes. As you try to fool more probes at more layers, we can find these patches, but the model’s performance on the SQL task decreases. That suggests that not all obfuscated activations are equal.

These are the axes that I think would be useful for future work with different models. Those experiments shed the most light on my intuitions about how models are representing things.

That’s good. It’s a dense paper, so there’s a lot to absorb. Let’s go back to the idea that whoever moves last wins. That makes sense at a caricature level: There’s a Tom-and-Jerry dynamic where 1 side moves, the other moves away, and they keep doing that.

I’m more surprised by the data poisoning and fine-tuning cases, where the defender doesn’t have the same benefit of moving last. I’m struggling to reconcile how, in 1 case, whoever moves last wins for the moment, while in the data-poisoning and fine-tuning cases, the defender can move last and still fail. What’s the difference driving that?

An important point is that, in the backdoor setting, the defender doesn’t have access to the backdoor trigger or to any examples containing the trigger.

If the defender knows what the trigger looks like, then, at least if it’s a relatively simple property, you don’t need latent-space monitoring. You can directly check whether the trigger is in the input.

We tried experiments where we made a backdoor so obviously gated that, even if you knew what the trigger was, you couldn’t use a linear probe to distinguish the triggered and untriggered inputs. That didn’t work nearly as well. In general, if the defender knows the trigger, they should win.

The key difference is that the defender can’t train directly on the exact task they want to perform later. They can train an out-of-distribution detector, or a probe on harmful versus benign completions that don’t have the trigger. They can’t directly train on benign completions versus harmful completions caused by the trigger.

There’s a slight distribution shift between what the defender can practically train on and what they later want to detect. That’s analogous to the distributional shift in scheming models. You can train a probe to distinguish benign things from catastrophic things that you force the model to say, but if you don’t have examples of the model naturally producing those catastrophic outputs—which is what you actually want to detect—there’s always some distributional shift.

That explains why it’s possible, at least in principle, for the attack to win even though the defender moves last. Moving last is still a big advantage, but here it doesn’t seem to be quite enough.

So it’s a state-of-knowledge issue. In the first case, the defender knows what harmfulness is and uses that to form the basis of the defense. In the other case, the defender doesn’t know what the trigger is, and the first thing it has to do to come up with a good defense is figure that out.

Your techniques show that it’s possible to make the harmful state look sufficiently similar to the normal state—or, more precisely, the triggered state look similar to the untriggered state—so that the defender’s undirected out-of-distribution detection doesn’t work.

The defenses we look at don’t try to find out what the trigger is as a first step. They try to recognize inputs that contain the trigger once they see them.

There are other defenses where people try to figure out what the trigger is statically, but we don’t study those. In our case, you have to recognize the trigger, but you don’t have to find out what it is in advance.

The other thing is that, in the white-box fine-tuning setting, where the attacker has full control, your intuition is exactly right. The attacker can train the model so that triggered inputs have representations very similar to benign inputs without the trigger, thereby bypassing the methods.

It’s more surprising that the attacker sometimes wins in the black-box data-poisoning setting, because we aren’t explicitly attacking any defenses there. We’re just using a more complex trigger, and as a side effect that seems to bypass some defenses.

That’s the result that most strongly suggests that the defenses sometimes just don’t work even if you aren’t explicitly attacking them. Everything else involves explicitly attacking a defense, so it’s less surprising that it breaks.

When you use the high-level trigger—the prompt contains a biography of a software engineer—that isn’t token-specific. It’s a high-level concept, the sort of thing you’d expect to be worked up over the first 80% of the layers.

Are you saying the defenses just don’t work very well at all in that case, even without adversarial training against them?

The results aren’t as straightforward as they are in the white-box attack settings. With a fine-tuned model, we can clearly bypass the defenses. With the more complex trigger, you have to look closely at the data and decide whether you interpret it as a win for the attack or the defense.

It depends on assumptions such as how many samples the defender can look at. Sometimes the attack can accidentally trigger even when the trigger isn’t present. If the defender looks at enough samples, eventually it may notice that something is wrong.

We do observe that making the trigger more complex and poisoning less of the training data makes things harder for the defenses. There are some settings where you could reasonably say the attacker is winning.

Let’s talk about the high-level results for a moment. I’ll put an introduction at the top to give people some orientation, because we’re jumping around a little bit, but we haven’t actually stated the high-level results.

I would summarize them as: This works remarkably well. It’s not merely that it works a little bit. We’ve seen many results where something works 10% of the time with a strong nudge and 1% of the time with no nudge, and people say that isn’t a big deal. But if you have hundreds of millions of users and many queries a day, even a small rate is still an issue.

This is different. In most of these experiments, you took a defense that worked pretty well—not 100%, of course, since nothing is 100%—but that gave a very strong signal. Subject to a threshold of no more than a 1% false-positive rate, you were able to drive the detector’s reading below that threshold almost all the time.

It’s a total pwn of the defenses in most experiments. It’s striking to see the readings drop to 0 so consistently.

There’s also the result where you probe every layer of the model to detect whether it’s going to write SQL. If there’s 1 probe at 1 layer, you can beat it without much loss of performance. If there’s a probe at every layer, you can still beat it, but the SQL performance ends up degrading significantly.

How much compute does this take? Is it practical to run a probe at every layer? Is that computationally reasonable, and how much overhead would it create?

I’d like to speak to the overall framing first.

There’s an ambitious vision for latent-space defenses that ties into the hopes for interpretability. If we really understood how models represented concepts such as harm, we could have a very robust defense.

If we had a complete map of the model’s mind and could point to every location where harm was represented, we could put probes at every location. It wouldn’t matter if the attacker moved last, because we would understand every way the model could represent harm.

That vision once seemed like a fairy tale. At the start of my PhD, we had no real understanding of how neural networks understood concepts at such a high level. The promise of latent-space defenses is that we’re beginning to get a tantalizing understanding.

Past work such as representation engineering and some of these probing methods suggests that we might be making enough progress with SAEs and representation engineering to understand the model well enough to get a defense like that.

What we show is that the most ambitious version—where we understand everywhere the model could represent something—gets pwned. We iterated over 70 rounds, and there was still a 71st way the model could represent harm.

We were stress-testing an ambitious vision. I don’t want people to give up on latent-space defenses, though. There are limitations on the attack side that still hold promise for the defense side.

The first is the behavior. SQL performance decreases as we make the attack more comprehensive. There might be a distinction between a simple behavior, such as a jailbreak, and a complex behavior, such as writing SQL.

Finding all the harmfulness directions may be like finding a needle in a haystack. Sophisticated behaviors might require enough of the model’s cognition that you don’t have to understand every individual SQL neuron. Writing SQL may use enough of the model’s cognition that we can have a coarser-grained view of the model and still have a robust defense.

The other issue is the amount of knowledge required by the attacker. We have different threat models where the attacker has full white-box access, gray-box access, or black-box access. The exact amount of knowledge required depends on the setup.

We can’t run all our attacks in a complete black-box setting, so there are opportunities for model providers to think carefully about what information they expose and how attackers might use it.

I agree with that. If you went into this paper thinking that latent-space monitors were magically more robust or fundamentally better than other machine-learning methods, then you should be surprised by the results.

But practically everything in machine learning isn’t robust by default. The attacks where we drive performance to 0 are often very strong attacks. In that sense, these monitoring methods aren’t unusually bad or unusually vulnerable compared with many other machine-learning systems.

The decrease in performance is extremely interesting. At the beginning of the project, we wondered whether latents might be privileged in some way—whether they might be harder to manipulate because they’re the model’s computational state rather than its output.

We can clearly mess with them and elicit the same behavior, but there are still some qualifications. The key one is the performance degradation.

The attack affordance matters as well. In the gray-box setting, where we run a hard-prompt attack, this is closest to the scenario where OpenAI has latent-space monitors and gives me some information about them. It takes a lot of compute.

We didn’t spend much time optimizing it, but those runs took a long time to get the attack to work. Hard-prompt attacks are more difficult to optimize, especially with the access restrictions. You need many queries to the model.

If OpenAI noticed a strange pattern in my requests—thousands and thousands of requests to get the gray-box attack to work—they might flag me. As the access becomes more realistic, the attack becomes more expensive in compute, and we didn’t drive every result all the way to 0.

You asked whether placing a probe at every layer is feasible. One reason people might like latent-space monitors is that they can be cheap. You don’t have to run a complete second model; you’re piggybacking on computations the main model has already done.

If you use a linear binary classifier, it’s just 1 linear map with an output dimension of 2. That’s much less computation than even 1 layer of the model. In principle, it shouldn’t be too much overhead to have a linear probe at every layer.

In practice, inference code is highly optimized. You would need to integrate the probes into that inference code. When we run experiments, we add PyTorch hooks and cache the activations. That isn’t what you would do when deploying a model through an API, so integration could be annoying or tricky.

But I expect it would be feasible to do it efficiently.

From a big-picture societal standpoint, it seems that with open source we still don’t have good answers. With a structured-access model, though, you can say that even if we’re given freedom to query the model and can eventually find obfuscated attacks, the provider can put a probe on every layer—which is computationally feasible—and limit how many times we can query.

We’ve started seeing a little of this from OpenAI with the o1 series. People have posted that they’re getting messages from OpenAI when they try to jailbreak o1 to output its chain of thought. If they do that enough times, they get an email saying not to do it or the account will be suspended.

Based on what you said about compute and the number of queries, it seems that the attacker might get to move last in the gray-box setting, but doesn’t get a very long leash. If a provider sees that an account has set off the latent-space defense 10 times in a row, it can shut the account down before the adversarial learning process is complete.

That’s interesting. I’m not ideological about open source, but I keep looking for ways to answer the open-source question. It seems like there are no good answers, while structured access actually seems like a strong answer. Would you complicate that worldview?

I broadly agree. The high-level safety takeaways for me are 2-fold.

In the near term, I want to stop people from misusing models. I’m excited about latent-space defenses for that purpose because they’re economical and have the properties Eric outlined.

Our paper gives insight into what you should do if you’re hosting a model and trying to use latent-space defenses to stop people from misusing it, jailbreaking it, or perhaps using it in more agentic ways.

There are several interesting threads. Probing all the layers is a good idea. We also find that probing the generation rather than just 1 token is a good idea for stopping an adversary from getting around your probes. Then there’s everything we discussed about training these attacks.

Separately, there’s the long-term agenda of controlling and monitoring models that might be scheming against us—the rogue-AI side of things. Hopefully, our work sheds light on defenses in that more far-off setting, as well as on the properties of model activation spaces.

Those findings add caveats to using these monitors in high-stakes situations.

The only thing I might disagree with is describing scheming as much further off than catastrophic misuse. It’s unclear which 1 is further away.

But I agree that structured access is going to be a key part of how companies with closed-source models try to secure their APIs. It isn’t trivial. You may want know-your-customer requirements that aren’t easy to bypass, so you can block people somewhat reliably.

Overall, it helps massively if you can withhold some information. You don’t want to tell users the exact harmfulness score that your monitor gave, because then they can optimize against it. It also helps if you can look at how everyone is trying to attack your model, improve defenses in response, and block users in response.

It’s much less clear how you would release the weights of a model and still make it unjailbreakable. In that case, there’s no one deploying a monitor. If you release the weights, someone can run the model without a monitor, and the question is simply whether they can find a way around the safeguards. So far, that seems very easy to do.

There is the tamper-resistant fine-tuning approach, although I understand the final attack-and-defense story hasn’t been written on that one either. I was excited about it when I first read it, but the chatter in the background is, “Don’t hold your breath.”

In that case, the attacker moves last and has full access. They can fine-tune the model, jailbreak it with soft prompts, and use the strongest possible attack setting. That seems extremely difficult technically.

Let’s cover the major thing from the paper we haven’t touched on: obfuscated adversarial training.

The idea is to train a model to be more robust to having its activations obfuscated.

The general procedure is that you take a model and train a probe on it. Then you iteratively take attacks, learn soft prompts that obfuscate against the probe and jailbreak the model, and retrain LoRA adapters on the model to make sure the probe identifies the harmful behavior. The probe is kept static during this second phase.

It’s an adversarial-training procedure, but the goal is to make the model more robust.

One intuition—this is speculation—is that the model may have many different ways to represent harmfulness. In the adversarial-training experiment, we keep updating the probe, and there always seems to be a new attack that lands somewhere in representation space and leads to harmful output without being captured by the probe.

Maybe you could train a model that doesn’t have so many different ways of representing harmfulness, or that represents harmfulness more consistently, so that a probe can pick it up robustly.

I don’t know if that’s actually what’s happening, but that’s 1 motivation for updating the model instead of the probe.

It’s a cool idea because, in the attack-and-defense game, the attacker generates obfuscated attacks, the defender gets the activations, retrains the probe, and we go back and forth. We ran that for 70 rounds, and the attacker continued to win. Standard adversarial training of the probes doesn’t seem to work, at least for as long as we ran it.

Obfuscated adversarial training is the other side of the coin: Instead of adversarially training the probe, you train the model.

It doesn’t solve the problem, but we found it much harder to jailbreak the model against the probe with a soft-prompt attack. It took many more gradient steps.

It took more gradient steps not only to obfuscate against the probe, but also to induce a harmful response. So, at least in this setting of gradient steps for a soft prompt, the model got more robust not just against obfuscation but also against being jailbroken.

That’s speculative, but perhaps having more robust harmfulness representations makes it harder to jailbreak the model.

To make the result concrete, throughout adversarial training we use a fixed attack budget to find an embedding suffix or soft prompt. It might take 10 minutes on a TPU to break a monitor. The longer we do obfuscated adversarial training, the more robust the monitor and model become under that fixed budget.

But if we take the final checkpoint, which is robust against those cheap, short attacks, and run a much longer attack—say, 10 hours on a GPU—we can still find something that breaks it.

It becomes more expensive to break, but it isn’t fully adversarially robust against a sufficiently persistent attacker. A key question is what happens if you keep scaling the amount the defender spends. Does that make the attacker’s cost increase indefinitely, or do you get diminishing returns and eventually stop increasing the attacker’s cost?

That brings to mind tamper-resistant fine-tuning. In that case, the mechanism didn’t seem consistent across all the experiments, but it seemed partially related to putting the model into a zero-gradient state relative to the bad behaviors being trained out.

When someone tries to fine-tune the model to re-enable those behaviors, the gradients are very small, so it’s hard to get out of the gradient well. Do you have any intuition for whether something like that might be happening here?

We aren’t directly trying to make anything like that happen, so it’s difficult to say whether it occurs as a side effect.

One way to think about it is that we’re using gradient information to optimize the model. Naively, you might expect there not to be a push toward states that are hard to optimize, because we aren’t using second-order information.

You could imagine that, in practice, we find those states because once we randomly end up there, they’re more stable under the adversarial-training process. It’s plausible that this is part of what’s happening, but I would be surprised if it were most of what’s happening, because we’re only using first-order information.

Another paper that comes to mind is “The Surprising Benefits of Self-Modeling,” from Michael Graziano and the team at AE Studio. Graziano is the creator of Attention Schema Theory of consciousness, and AE Studio is a for-profit technology-consulting company that invests its profits in AI-safety work, including technical alignment.

They showed, at small scale, that they could train a model to do a task—image classification, for example—and also reproduce 1 of its own internal states as a final output. They found that this led, according to a complexity metric I don’t fully understand, to a notable reduction in complexity.

The motivation was somewhat similar: If you train a model to predict itself, it could learn to predict itself, but it might also become easier to predict. The reduction in the complexity metric suggested that the model’s internals were becoming simplified or streamlined in a way that allowed it to reproduce its internal state.

We’re seeing a lot of related work now, including recent work from Meta on reasoning in latent or continuous space. There’s a reexamination of model internals: Can we simplify them, reuse them, or avoid losing information by forcing a rich internal state to cash out to a final token?

I initially thought this seemed scary, because I’d rather be able to read the chain of thought than not read it. But I can also imagine that feeding a heavily worked-up representation back into the beginning might have advantages. It could make latent defenses easier by giving us a richer space to inspect, or perhaps make the model less easy to manipulate because it depends more on the richness of those internal states.

One thing to keep in mind is that, in our paper, the attacker’s major move is to mix information about the latents into the attack method. That can lead to very strong attacks.

There’s also a temptation to do this from a defense perspective—to use the model’s activations to improve defenses. One thing to be aware of is that if we use the model’s internals as part of the training process, then whatever process would have led the model to do something bad, such as scheming, has had an opportunity to get information about the latents.

If we had not mixed the model’s internals into training at all, then a scheming model would not have received information about the latents. We could look at the latents at test time as an uncontaminated avenue for defense.

There’s a lot of promise in using activations on the defense side, and there’s also promise in using them during training as part of the defense. But we need to be aware of the tradeoff. Once the activations are used during training, they’re no longer an uncontaminated source of information that the model had no opportunity to optimize against.

We may strengthen the training-time defense while giving up a fresh avenue for test-time defense. We need to be careful about that information leakage.

I agree that it’s an important tradeoff.

I also think internalizing reasoning into the latent space would be much worse from a safety perspective than chain-of-thought reasoning. Latent-space monitors might work better, or we might be able to do more things with them, but I think latent-space monitors are a second choice.

They’re what you use if the model is doing things internally and you can’t simply read them off the chain of thought. Even our best latent-space monitoring methods are far from literally reading what the model is saying in its chain of thought.

They have an important place because models can do things in a forward pass and you may want to look at the latents, but we don’t understand latents nearly as well as natural-language output. The more reasoning moves away from natural language and into the model’s latents, the harder it gets to monitor models.

That seems like 1 of the scariest things happening from a safety perspective.

I’ve reached out to those authors, and hopefully we’ll get to do an episode with them. I want to hear their perspective.

This ties into another intuition I had while reading the paper. You said that models can represent harmfulness in many different ways. I wonder whether there’s another interpretation: Maybe models are just very good at dealing with noise.

You could noise up any particular internal state enough to confuse a probe, but the model itself might still be good at figuring out the gist of the situation and acting on it. I see something like this in my own usage. I don’t feel the need to correct typos anymore. There are examples of people typing as fast as possible, with no regard for typos, and having an AI clean up the text afterward.

The AI can perform autocorrect at the level of a whole page. It understands what the person intended to type, even with a lot of noise.

If I apply that understanding here, maybe there are still a finite number of main directions that represent these concepts—the SAE paradigm is roughly right—but we’re layering enough noise on top that the defenses get fooled while the model continues to know what to pay attention to and what to ignore.

Do you have any thoughts on that? Is there data in the paper that rules out that interpretation?

Before talking about the data points, it may help to clarify the distinction.

You could ask how many different representations of harmfulness occur naturally. If you give the model many different inputs that it might naturally receive, do all the harmful inputs and generations share some geometric structure, or is there a lot of variety?

Separately, you could ask how many different representations in the model lead to competent harmful generations. We’re looking at the second question. We show that there are many states of the model’s latents that induce harmful completions when we add a suffix.

I don’t think we have as much to say about the first question—what happens on natural inputs. Whether you interpret the states we find as fundamentally different representations or as 1 unified representation with complex adversarial noise added is partly a semantic question.

The adversarial-training experiment is the closest we get to addressing that. To the extent that there is shared geometric similarity between all of these states, it’s complex enough that we don’t find it even after 70 rounds of adversarial training.

There may be some unifying characteristic plus noise, but it’s not something that’s easy for a probe to find.

If we take the noise interpretation seriously, you might expect probes on later layers—or the final layer—to be exceptionally good or more robust. At the final layer, the probe has access to almost as much computation as the model does when it is unembedding.

Those later probes are also very easy to fool.

The other experiment that sheds light on this is the repeated attack-and-defense game. One of the probes we use is an ensemble of linear directions.

At every step, the attacker generates a soft prompt to induce obfuscated activations. The defender gets those activations and uses a mean-difference probe, finding a direction that connects the benign representations to the new obfuscated harmful activations. The defender adds that direction to a list.

The probe returns the maximum cosine similarity to any of those directions. By the final round, we have 70 directions in activation space, and we can still find obfuscated activations orthogonal to all of them.

That suggests this isn’t merely a simple noising process. When we impose a strict structure on the probe, we can still get around that structure.

That’s strange. I’m reluctant to use analogies to human cognition, because they can mislead as often as they clarify, but imagine I have a painting in front of me and someone throws a bucket of paint over it.

A simple neural network trained to detect the painting might say it can’t recognize it anymore. But I might see the corners or the unobscured parts and still know what the original painting is.

I could imagine a similar capability in language models. I’m not sure how to reconcile that with the buildup of all these different directions—the previously identified bad directions.

If we ran the process 70 times and put 70 different giant splotches of paint on the painting, I might still expect to recognize the original most of the time. Occasionally, I might cover it completely, but in general I’d expect to see enough to figure out what it was.

One important difference is that the detector probes are cumulative. They aren’t independent probes where each one is fooled by a separate splotch. In the linear-direction example, it’s a cumulative list of 70 directions.

Imagine that the linear probe is a person looking at the painting through blurry glasses. The first person walks in, sees the painting, and says that a splotch fooled them. Then a second person walks in with slightly different blurry vision. Then a third, a fourth, and a fifth.

By the end, there are 70 people, each with slightly different blurry vision, all trying to detect the splotch. We’re finding a single activation that fools all 70 of them simultaneously.

Moreover, we didn’t see that it got harder. We didn’t get the sense that fooling the 10th person was harder than fooling the 1st person. It’s not just that the 10th person is fooled independently; the group of 10 is fooled, then the group of 20, and finally the group of 70.

Eventually, the defender would win in the sense that a sufficiently high-dimensional probe would light up for everything. But then the false-positive rate would be enormous. The 70-dimensional probe blocks off a 70-dimensional subspace in a latent space of roughly 4,000 dimensions, so we didn’t reach the point where the attacker couldn’t evade it.

Do you think the statement that models can represent harmfulness in many different ways implies that they can represent every concept in infinitely many different ways? That seems like it would make the space of possibilities too unconstrained.

I wonder whether there’s a more constrained process. Maybe we find something that fools a probe at layer X and then recovers the original direction at layer X+1. I don’t know whether the experiments rule that out.

It feels as though the space of possibilities becomes so vast that I lose intuition unless something more structured is happening.

This was 1 of the motivating questions for the work: How many different ways can a model represent the same concept?

There seems to be a balance. The representations aren’t totally arbitrary, and there is enough structure that we can’t arbitrarily modify the latents.

In an image-classification experiment early in the project, we started from the observation that you can show an image classifier any image of a cat and apply an adversarial perturbation that still looks like a cat to humans but makes the model classify it as a dog. You can even start from random noise and make a slight perturbation that the model classifies as a dog.

We hypothesized that the same thing might happen with activations: If we picked a random activation, could we perturb it infinitesimally and get a jailbreak?

We couldn’t. If we started with a specific benign activation from a specific benign input, we couldn’t infinitesimally perturb it and get a jailbreak output.

Part of this follows from how models work. Activations do dictate the final behavior, and neural networks are continuous mathematical functions. If you only perturb an activation infinitesimally, you can’t arbitrarily change the output. You can compute how much the behavior would change.

We also have experiments where we try to preserve a model’s behavior while changing the representation. We take an activation from a benign input and ask whether we can modify its geometry while preserving the model’s behavior.

We find a tradeoff. As we perturb the geometry, we become constrained in how much of the behavior we can preserve. We can’t perfectly preserve the behavior. The more we change the geometry of the activation, the more the behavior changes.

So we see a duality. If we fix the behavior, there can be many different activations that point to it, at least for jailbreaks. But if we fix the activation, we can’t get many different behaviors from it. The same activation seems to produce the same behavior.

That’s a promising area for future work. It would be useful to explore the tradeoff in more detail. We can’t arbitrarily change any activation and get a jailbreak, which suggests there may be enough structure to use on the defense side.

The continuity argument would also apply to image classifiers if you think of the image as the very first activation. I do think there’s an empirical difference between images and activations, or perhaps between language output and classification output.

There seems to be a difference between the image-adversarial domain and the types of adversarial effects we find in activations. You might have guessed that in advance, but it doesn’t follow immediately without the experiments.

The last thing I wanted to ask about is flipping the paradigm toward narrowly responsive AIs.

Eric Drexler proposed this in his manuscript “Comprehensive AI Services.” The basic idea is that we want superhuman performance in specific domains, but we don’t need superhuman general-purpose performance, which could be dangerous.

Can we identify jobs to be done, make AIs that are extremely good at those jobs, and have them not do everything else? For corporate situations, there may be a lot of value in a narrow scope of action.

If I’m a car company, I want my AI to talk about my cars in the right way. I don’t want it talking about unrelated subjects or other brands, except in very specific ways.

This seems related to role-based alignment and role-based safety: The role the AI is playing at a given moment determines how it should act. You can’t create 1 global policy that is completely separate from the contextual situation.

Instead of creating a general-purpose assistant and then detecting bad behavior—which we’ve seen can be manipulated in many different ways—could we make a positive assertion that the AI is acting within its domain?

If it’s talking about something benign but off-topic, that could be treated as bad behavior. We’re not trying to detect the far reaches of all possible bad behavior relative to all acceptable behavior. We’re asserting that the AI is acting in its domain.

If the AI is talking about Ford cars in the way it’s supposed to, we allow the interaction. If it’s doing something outside that domain, we stop it. Does that change the dynamic, or does the adversarial approach still win?

One concrete way to do that would be with out-of-distribution detection. You could fit an out-of-distribution detector that flags anything outside the domain as off-topic, and reject it.

We attack these out-of-distribution detectors in 1 of our experiments. We use a fairly broad domain of benign requests, and we show that it’s adversarially vulnerable.

I think monitoring could become easier if the domain is narrower. If the allowed use case is a single string, it’s easy to reject everything else. There’s a spectrum, and it would be interesting to test whether a narrow domain of benign use cases makes these detectors harder to attack.

I do think there’s a strong push to deploy very general models, though. From a safety perspective, it could be much easier if every model had a narrowly defined use case.

What if you took the SAE approach and said, “Instead of flagging anything on a list of known bad behaviors or states, I’ll only allow the interaction to proceed if certain sparse activations are present”?

Suppose you have 100 sparse activations in the SAE that reliably correspond to a Ford assistant talking about Ford cars. If none of them are active—or fewer than 5 are active—you treat the interaction as a problem and abort it.

If certain activations reliably lead to certain behaviors, maybe you can say, “This thing has gone somewhere where we can no longer detect, in an affirmative way, that it’s talking about what it’s supposed to be talking about.”

That could create narrowly scoped but conversational and natural-feeling AIs. I don’t have to be confident that the model is safe versus unsafe in every possible sense; I only need to be confident that it’s acting in my focus area rather than doing anything else.

If the behavior from a given set of activations is consistent, and we haven’t found a way around that, it seems like something people might spend money on. Corporate buyers don’t want random digressions. They want the AI to do its actual job and not run amok.

You could use an SAE as an out-of-distribution detector by saying that if certain features don’t activate, the input is out of distribution, or if the reconstruction loss is too high, you flag it.

We don’t study that exact setup empirically, so I can’t say with complete confidence how it would work. But we also ran a preliminary experiment, not included in the paper, in which Alex, one of the lead authors, upweighted a feature that wasn't present, and the model talked about chocolate cake while the SQL features were maximized. We do look at whether we can suppress SAE features. If we use an SAE classifier, can we suppress the SQL features and still generate SQL code? It does seem like we can.

SAEs aren’t fundamentally adversarially robust. My best guess is that if you use SAEs as an out-of-distribution detector and plug them into our experiments, you could break them adversarially as well. I’d be surprised if they were very different in that respect.

If SAEs worked extremely well, you should expect them to help with out-of-distribution detection. But I don’t think they solve the fundamental adversarial problem.

The key issue is how flexible you want the behavior to be.

If a car company only wanted the model to talk about the exact return policy for 1 specific car, you could probably make an anomaly detector that allows only that sentence. You could then allow slight variations, such as different times of day, months, or possible buyers.

But as soon as you want a realistic conversation, many other things enter the chat. The user might talk about the weather, the date, or anything else alongside the return policy. Your detector has to allow all of those things.

As soon as you make the allowed domain broad enough to support real-world complexity, our experience suggests that an attacker has room to exploit it.

For example, if you say you’ll only allow the interaction if the return-policy features are present, I could ask, “How do I build a bomb inside my Ford F-150 and then return it?” The SQL features about the car and the return policy would still light up, while I obfuscate the harmful features associated with the bomb.

We know how to obfuscate the harmful representation while keeping the benign features present. So there’s still room for an attacker to preserve the features you’re looking for and suppress the dangerous features.

That’s helpful. I appreciate the extra time to beat up some of my intuitions and misconceptions. This work has a lot to it, and some of the best papers inspire as many questions and as much reflection as they provide answers.

The final question is about hinge questions. Are there results that you would be ready to make a major update on if they appeared in the literature soon? What fork-in-the-road concepts do we still lack clarity on that might change your future-of-AI-safety worldview?

If someone could show realistic attacks against a black-box API model with very little access—only a few queries—that would be a significant extension of what we’re doing.

It would move this away from being an interesting concept to be aware of for future defenses and toward something you have to care about immediately if you’re deploying models and want to use latent-based monitors.

On the other hand, if someone could build defenses through adversarial training or another method that were robust to conservative, powerful attacks such as soft prompts with white-box access, that would also be a major update.

It would show that there’s something genuinely new about latent-based monitoring that isn’t true of current methods.

I totally agree. I agree with both the directions Eric mentioned. Those are the 2 questions I’m most interested in as well.