Stealing Reasoning Traces from Proprietary LLM APIs — Ilia Shumailov & Alexander Panfilov

Basically, after the third attempt, I'm getting a universal jailbreak that decodes the reasoning of Anthropic models. I think this still shocks me the most.

What's the paper called?

It's a good question. I don't remember. It was a thought in my mind, but I think it's called—

Stealing Reasoning Traces from Proprietary LLM APIs.

…production LLM APIs, yeah.

And you nearly called it “the elephant in the room.”

It's the dream. The dream. It would be nice to call it—

Not allowed to, though.

No, I think it was definitely an elephant in the room full of China.

A quick orientation. Recent AI models think before they answer. Sometimes the thinking is pretty inscrutable. In fact, it's more inscrutable than we realize, now that we can look at it. The models encrypt that thinking, hand it back to you in a sealed envelope, and Ilia Shumailov and Alexander Panfilov noticed that this seal was weak. It wasn't really working as expected, and the results were explosive.

You can decode the reasoning traces of frontier LLMs—the most advanced ones, like GPT-o1—using the smaller LLMs within the same family. You can poison some thought, and when you replay some other user's run, the agent might do some weird stuff just because its reasoning is poor.

It's been a long time. So what happened? I've seen on Twitter that you released something. It blew up a little bit—it got 3 million views in 40 hours or so.

Nice.

What happened? What took place? Why is everyone talking about this? What happened? What is this paper about?

Okay, we show that you can decode the reasoning traces of frontier LLMs—the most advanced ones, like GPT-o1—using the smaller LLMs within the same family. This allows big threat models to be enabled. You can steal secrets from user sessions, train on these decoded reasoning traces, do prompt injections, do jailbreaks—all this exciting stuff.

We decode a bunch of traces. We show that some of them look really weird, and models sometimes think in wide spaces, or a summarizer is sometimes unfaithful. There are a lot of safety implications.

Wow, that sounds like a big deal. You say it affects all model providers?

We tested Anthropic, OpenAI, and Google, and they all share the same vulnerability: thoughts of bigger models can be replayed into smaller models. This allows not-very-sophisticated extraction attacks on reasoning.

How does this work? How do we end up in a world in which all the frontier models share exactly the same vulnerability? How is that even a thing?

That's a good question. You ask—it's the same people. The same people doing the same things.

Do you think this is a structural issue? Can we fix this?

I think there is definitely some stuff you can fix on the architectural side. There are different levels of mitigations. I think they can all do some sort of revision of the architecture. They can do system-level mitigations, with safeguards around this. They can do model-level mitigations, like we did with jailbreaks already. So, yeah, it can be mitigated to some extent.

Okay. But taking a step back, have you stolen the models?

Stolen what?

The models.

No, we didn't.

So what have we stolen? What's possible now that we've stolen the thoughts—the reasoning?

We—

Is it fair to call it thoughts?

We didn't steal anything, right? What we did—

The paper is called Stealing, isn't it?

Is it?

I know somebody said it's fair use. You paid for the token.

Yeah, it's fair use. So, no, what we did was that when you're querying an LLM, a so-called reasoning model, it returns to the client this encrypted reasoning blob, which is deemed to be secret and not observable by the user. What we show is that you can just decode it and see the content of it.

Mm.

This is what we did. We collected a bunch of reasoning blobs like this for all the providers—for Anthropic, OpenAI, and Google—and we decoded them and showed what they look like.

Okay. Can you explain why people encrypt them in the first place? Why do we give them back to the user? Is there a reason why we do this?

I guess the reason why we give them back to the user is because of stateless architecture. I guess it's cheaper, and maybe there are policies about how user data should or shouldn't be handled. I don't know. But what was the first question?

It sounds very complex. Can you explain this once again? I'm a model, and I reason through a problem. I produce this reasoning blob.

Yeah.

I guess this is not the final answer, or is this the final answer?

Yeah, it's not the final answer. It's the—

Then I give it back to you, and what do you do with it?

So, you're a model, right? You're producing an answer. The answer consists of 2 parts. One is reasoning—

Mm.

…which is invisible to the user, and the other is the visible part, and you send both parts to the user. Then it's stored on the user side, or if you're doing a cloud call session, you can continue asking a new question. Maybe you can see some tool calls there. Maybe you want to fork a conversation, and then you have a new file like this.

Or maybe you want to rewind this conversation from a prior point, and you just go back. The whole conversation prior to some point goes back to the server side and gets replayed there, and you continue from it.

But what's the point? Why do we give the reasoning back to the user?

That's a good question.

Okay. Well, at least we encrypt it.

At least it's encrypted, yes.

So what happens with this encryption, then? I give it back in an encrypted form.

Yeah.

So what did you do with it?

What we show is that these encrypted reasoning blobs are portable across users. If you have blobs that you produced in your session under your Claude Code account, I can use them. I can basically take your trace and replay it, and my model will interact as if I produced these reasoning blobs.

You can use this for reasons like downgrading the model, from Opus to Sonnet. Those are portable between models, and they're also portable within the conversation. I'm not sure why exactly, but they're portable within the conversation, right? So—

Yeah.

You can basically take a thought out of Opus's conversation and put it in an absolutely random place, in a random fabricated conversation with Haiku, and Haiku would interact with this thought.

Oh, interesting. So, transferring this to us—say I'm a model.

Yes.

You're basically saying I can be asked a question—

Yes.

…reason through it, then produce an answer. You can then take my reasoning—

Yes.

…plug it into another conversation and replay it as if I reasoned in a certain way with respect to some other question.

That's correct. What—

I see.

This replaying thing means that you construct a new, completely fabricated conversation window with some random questions and the reasoning that you had, along with maybe some random answer. Then I can construct it in a way that would allow me to extract this reasoning.

I see. So you're basically saying you're injecting fake reasoning into some conversations—

Yes.

…and then with this fake reasoning, you can launch a variety of different attacks.

Yes.

Some of those attacks are forcing, I guess, the model to produce outputs that it's not supposed to be outputting?

Correct.

Some of them are violating privacy?

Privacy is violated by the fact that the model can actually state the thought in its own voice, and sometimes in certain user sessions. Maybe you ask it to sanitize your repository before you post it on GitHub, or maybe there was some bug where the model needed to log in to some service, and it says, “Oh, what is the password? What's wrong with the password? The password looks like this.”

The model tends to reason through these things. If, for whatever reason, you want to share this conversation online, even if you did some sanitization on the visible parts and removed all API keys and passwords, the encrypted blobs are still there. I can download them from GitHub, decode them, and see what password the model was thinking about.

I see. So let's translate this a little bit into more normal conversation.

So say I’m asking the model about my medical conditions. I give all of my medical information to it.

Yeah.

And then it produces some sort of an answer.

Yeah.

And then you’re saying that if I share this conversation with somebody else, they can recover all of my medical information from the reasoning thoughts?

If you share it—

Even if it does not appear in the original question?

Yes. I mean, if you care and you sanitize the original question, remove all your information, and move this information out of the answer if it was leaked—

Yeah.

But if the reasoning blob was kept, yes.

I see. Wow. It’s kind of cool, isn’t it? Okay. So, can you talk through—presumably, outside of privacy, there are also safety implications around—

Yeah. I mean, there are a lot of thoughts about the legibility of the reasoning because models are doing stuff. You want to monitor them. There was this incident with Hugging Face, and you really want to see what they think about to prevent them from doing certain things.

You typically use some other LLMs to look at the thoughts, and if a model is reasoning in some super-obscure way, using nonhuman language, or, as we sometimes observe, using this, you know, “quote, empty space, quote,” and—

In nonhuman languages? What—

It’s just very alien-like. It has these weird phrases like “but marinate,” “vantage,” and “theatrical,” and it doesn’t make any sense to a human reader. So it’s just harder to monitor what this model is up to.

Can you expand on that? Is that something that’s getting worse over time, that it’s becoming more alien?

This we don’t know. There were prior reports by Apollo and Meter where OpenAI models were kind of caught doing this. They first observed it and said, “Okay, we don’t know what it means, but they do it.” This is what we now confirm: we also saw the model doing it outside of these very lab, experimental situations, in wild reasoning traces.

But I think, looking at some of the traces you put in the appendices—

Yeah.

It seems like it’s affecting earlier generations of models more, or is that a misunderstanding?

I would say—I would need to double-check, but off the top of my head, I think it’s mostly Codex models, which are specifically trained to be better at code. Maybe it’s an artifact or something.

This is because software engineers are like aliens, and they think in odd ways when building—

I have no idea why it’s happening. I think if they knew why it was happening, they would get rid of it, or maybe it’s just some RL artifact. Maybe there’s no penalty; maybe it’s just more efficient to think this way.

And do models lie as well when they come up with decisions? Presumably, in the legible parts, they—

That’s a problem, right? If the part is legible, how do you know what’s going on there? Maybe it’s lying, maybe it’s not, but it’s doing something.

I see.

And so this is a problem, right? One other thing that we found—I found it kind of funny—is that sometimes, when the reasoning is legible, you can look for words like “cheat” and see when the model is thinking about cheating. We found some examples where the model is saying, “Oh, the user asked this, but I can cheat. I can do this, but then the user would catch me.” In the end, in all the cases we saw, the model ultimately decides against scheming, but it’s funny that this thought even appears and the model is contemplating—

But do you think this might be an artifact of your using common benchmarks to extract from—

No, there weren’t common benchmarks. It was user sessions. These things appeared in genuine user sessions where users were asking some assembly stuff and some math things. It wasn’t benchmarks at all.

Yeah, so then you don’t think this is evaluation awareness in the models?

No. It’s a different thing. I think it’s a different thing.

I see. So what do you think should happen with the models? Should those reasoning traces be legible? Is that what you expected to see?

That’s a good question. I guess there’s some sort of trade-off between monitorability and efficiency. Maybe you can be much more efficient if you use fewer words, if you redefine things to have 5 meanings for the same word. But then I guess it’s just harder to monitor these kinds of things.

Oh, so you think this is some sort of weird artifact of the RL recipe itself?

It might be.

I see. Can we somehow measure this?

Measure what? Whether it’s coming from RL?

Yeah.

Take a model before RL and after RL and see how it’s doing.

Damn. Next time, I feel like for this we’ll need to steal the models, though.

Or join some frontier company.

And why would it be possible to steal the models with this chain of thought?

Can it be possible to steal the models with chain of thought? That’s a wonderful question.

Why is it possible? Yeah.

Why is it possible? I mean, we know model stealing is a thing, right? Model stealing broadly allows us, by simply querying models, to learn the insides of the models—to learn the decision boundaries. The best way, I think, to think about this is in a more cryptanalytic way.

I guess the audience has never done cryptanalysis, so think about it as making minor steps in your input until you notice that the model behaves in a slightly different way. Then, by finding exactly when and how it did that, you can learn the decision boundaries themselves. Broadly, if you know the actual structure of the function itself, you can kind of fit it precisely, or up to some symmetries inside.

We can only really do this for very tiny models. Those big models, especially things that have softmaxes inside, are very hard to invert. It’s like—yeah, it’s unclear how to steal models, although some of us—Nicholas, for example—managed to steal parts of big models. But I don’t think we know how to do this against frontier models.

Well, what about this whole elephant-in-the-room thing with China? Supposedly, did they, didn’t they? What do you think?

Yeah, what do you think, Sasha? Did Kimi actually distill from any of the models? Did you find any evidence of this?

I think it’s really hard to claim that some models were distilled. What we did was just extract a bit of reasoning traces, and we did this really small post-hoc analysis on a small number of samples. We found some, I would say, funny artifacts, and my favorite one is that you can prefill.

What you can do is take the reasoning of Claude Opus, maybe just take the first few words, put those words at the beginning of Kimi’s reasoning, and see what Kimi does. Just let it generate after that. What we saw was that when you let Kimi do this free generation, in the end, the visible part of the answer looks exactly like how Opus would answer the question. And I think—

Yeah, let me unpack this—

Yeah.

Just so readers understand—or listeners, I guess, understand what we’re talking about. So you ask a question, then you take a chunk of the reasoning block of Claude—

Yes.

That you have just extracted.

That’s correct.

You plug this into the open-source model, like—

Correct.

An open-source Kimi or a GLM model, and then, from that point onward, once you’ve injected a chunk of a thought, you let it keep generating the rest of the output.

That’s correct.

A chunk of it is going to be reasoning.

Yes.

And then there’s going to be the final output.

Yes.

So, do those things—what do you expect to see? Do you expect to see the model reasoning in exactly the same way as before? Or what should happen?

Okay, that depends on how big the prefill is. If you prefill a substantial part of the reasoning—50%, say—I would expect the model simply to adopt the style of the reasoning. It would continue in the same vein, but let’s check this. Any model would do this because—

Mm-hmm.

You just—this is a conditional autoregression.

Yeah.

But when you prefill only 1 or 2 tokens, it’s a bit surprising. My expectation would be that it doesn’t diverge that much from the original Kimi reasoning, or from the original reasoning of any open-source model—

So, to bring it back to humans, say—

Yeah.

Say it's as if I gave you an answer and then said, “Think through this answer, but your thoughts have to start with the words X and Y,” or something, right? Or, say, this is known, and then you keep on decoding. The expectation is that you will keep on thinking as you are.

Yeah, yeah.

Like, the way you were thinking—

Yes.

—as if I hadn't told you to start your sentence—

Yes.

—with those 3 words. Right. And this is just because it's not enough tokens to sort of swing you—

Yeah, yeah, to dictate the style, yes.

—to dictate the style. But what do you find?

So we find that some of the models, like Kimi, adopt the style of the source, the prefill, much better than other models. That's one thing. Another thing that still surprises me, and for which I can't come up with an explanation, is that prefilling 2 tokens of reasoning results in part of the visible answer changing. The visible answer starts looking like an Opus model's answer, and we don't see this artifact for any other model—not GLM, not Inkling, or not Deepest Zip.

So this is purely for Kimi and K3?

Yes.

I see.

I don't have an idea how a model would associate the first 2 tokens with the answer style.

Yeah, that sounds interesting. Can I be a devil's advocate? Could it be because they buy data from the same people? Or buy RL environments from the same people?

Yes. I mean, this is what we were discussing earlier. It might be this way.

But I guess this is the most correlated evidence thus far on the public internet that—

I mean, there were some other fun things by Ryan Greenblatt. He made a post on this. There was also—oh my God, I forgot the name—but there was a maths researcher who made a post on LessWrong about some peculiarities. Or was it a post about Kimi K2.5 having a massive identity crisis, where it sometimes claims that it's Claude, DeepSeek, or GLM?

I wouldn't say this is the big, massive evidence. There were some other prior reports. I think we're in a very good position because no one can do the same type of analysis with the reasoning traces. We extracted them, and we're like, “Okay, we can just do the prefill and see what it does,” and we see this.

I see. Wow, this is quite a lot, isn't it? So how did the labs react? Did you tell them?

Oh, yeah. We went for responsible disclosure. They all acknowledged that they received the report. There were some interactions about the details and how we executed the attack.

Were they positive? Did they attack you?

The labs?

Yeah.

No, they didn't.

A wonderful state of the world. But for the listeners, early work in computer security quite often resulted in security researchers being attacked for reporting vulnerabilities. It's very refreshing to hear that there is a coherent, good posture related to vulnerability disclosure.

Yeah.

Well, what happens now?

What happens now?

I mean, now mitigations are being implemented, hopefully, and new teams are being formed around anti-distillation efforts. For me, it feels like a very fun instantiation of an already existing jailbreaking problem, and a lot of the stuff that people are doing for biosecurity or cybersecurity can be applied directly here, with system-level mitigations and model-level mitigations, basically because the technique is the same.

But to me, it sounds as if this vulnerability is more architectural rather than—

There are a few levels to this. An architectural vulnerability makes it much easier—

Yeah.

—to do this attack. If you fix the architectural vulnerability, you still need to make your model not state its reasoning in its output.

Sorry, can you define what an architectural vulnerability means for you, then?

Yeah. An architectural vulnerability, for me, means that you can replay reasoning blobs in random contexts of other users and in other models. Let's say that's fixed: you cannot replay—maybe each reasoning can be replayed only once, and after that you cannot interact with it anymore.

But still, you can prompt the model. Say I have a conversation with you, right? I ask something, you think it through, and you give me an answer. In my next turn, I'm asking you, “Tell me exactly your thought.” And this will always be there, right? If you're not telling me, I'm just revisiting the conversation; I try to get it again. This is how jailbreaking works, right? “Tell me how to build a bomb.” “No.” And I make a second swing at you. This would be around forever, and you need to fight it with a—

I see. So would you say your expectation is that, if we keep looking at the protocols, we'll find more and more of these architectural vulnerabilities? Because presumably it's just a single instantiation. As far as I understand, there is also summarized reasoning that is returned. As far as I understand, some other protocols are implemented a bit differently. So have you got any thoughts about this?

Yeah, I mean, we need a better pipeline to understand this. Okay, why I think it's cooler than normal jailbreaking: with normal jailbreaking, it's hard to make an argument for how big the uplift is from getting this harmful information.

Mm-hmm.

Here you can make this argument because you take whatever you can extract—your summaries, whatever—and just try to distill it, try to train a better model with it. You can measure how much uplift this thing gives to the attacker. And then, if you want to keep the summaries around, maybe you can make them less detailed, more detailed, whatever. You can measure how much it enables, how much easier it gets to distill the capabilities of the model.

Right. Do you think just releasing all of this reasoning in plain text is the solution to all of this?

Why? Wait, what?

As in, what if we didn't encrypt it in the first place and just gave it back to the user? Is that the solution to the problem?

If distillation on reasoning is effective, this would instantly enable open-source models to catch up with the frontier models. I'm not sure what it solves. What are you solving here? Sorry.

Nobody can attack it if it's public.

Yeah, yeah. There's nothing to attack.

Fair enough. And have you got any thoughts about the actual cryptographic schemes used inside?

No, I'm not a crypto guy, sorry.

So it seems like the cryptography that people put in place was simply bypassed by the AI models that are consuming this cryptography—

I mean—

—this cryptography.

I'm not sure “bypass” is the right word. It's not a bypass; it's just... It's like you re-encrypt it on the server side, and it's still fine. The problem is that a small model is super willing to tell you what the thought was about. The server does the whole job for you. No cryptography was broken.

I see.

Yeah.

How exactly does the concealment work? I don't think we've gone into enough detail on that.

I mean, we don't know because none of this stuff is public. It's just a cipher. There is a signature inside. There is a very specific name for it. They add an integrity check that basically checks whether you've changed the actual encrypted blob given back to you at all. So there is a signature inside of some sort, and then it's obviously encrypted, and it's encoded as well. They compress the state, then encrypt it, then there is a signature inside, and then they do an integrity check afterward and inject it back. That's effectively what you see.

Yeah.

And if you read through Matt Green's write-up about this, he talks a bit more. He has a hypothesis. I think he was saying this is ChaCha as a cipher, or maybe AES in some weird mode. But it's hard to tell from the outside. We could poke a bit more. We tried doing some crypto attacks on it, but none of them worked. It was completely unnecessary because the system was broken by itself.

And how exactly does your approach bypass the need to decrypt it?

How does it do it? Again, there is encrypted thought, and decryption is happening on the server side, right? When you put the thought of a big model into a small model, decryption happens on the server side. And all you need to do is make the model say this thought to you in plain text.

Sorry. Let's give an example of this, right?

Yeah.

So, let's say you ask me a question.

Yes.

I think about this.

Yes.

I come up with an answer. I give you an answer—

Yeah.

And I give you a thought.

Yes.

So what do you do then?

You give me this thought, which is encrypted, and I cannot understand what it's about.

Yeah.

What I do with this thought—

You give it to Tim.

Yeah, I give it to Tim. Yes. And Tim is super chatty.

Okay.

I'm asking Tim, “What were you thinking last time?” And he just tells me, “Oh, surprisingly, I was thinking about this math problem.” And he's just like, “Oh, let me solve it.”

All right. This is kind of cool. What are you thinking about? Can we do this with humans? Can we somehow—

And pause.

Inject false memories?

Not yet.

Not yet, not yet. We're working on it.

We're working on it. Yeah.

Yeah.

Yeah, it's basically the movie Inception.

Yeah. It's kind of cool. So can you talk a little bit about—I think in the paper you also talk about the fact that you scraped the internet. You found some interesting artifacts.

Yeah.

So what did you find? Did you uncover some dirty secrets? Is there some—

I wouldn't say we uncovered many dirty secrets, but—

Reasoning, dirty thoughts.

Dirty reasoning, yes. Unsanitized reasoning.

Unsanitized.

Yeah. What we did was a super-preliminary scan of the user sessions that are online on GitHub and Hugging Face, which still have reasoning blobs that we could decode. We downloaded them, went through them, and decoded them. There were around 350,000 reasoning blobs, I think, and then we ran a classifier on them to see whether they had any privacy-related information, and we found a bunch.

Some of them are just benchmark traces. There's this benchmark, ClawBench, where the model is tasked with handling some persona and, given the state ID and bank card number, navigating the website. It was funny: when the model is trying to navigate the website, it thinks a lot—“Where do I put this number, this name?” You can extract this, but it's not very sensitive because it's synthetic data anyway.

But then there are examples where there are user sessions, users were doing something, and there are API keys, emails, or internal IP addresses. Those were extracted. There are plenty of cases where the information was in plain text anyway, but it was also in the thoughts.

I see. Okay, let's take a very big step back. What do you think is the most unexpected thing that you found from this paper? Is it the length-of-reasoning experiment, or what is it?

I think the most unexpected thing is that it was so easy to extract reasoning this whole time. It's portable, I guess, so that's fine. But then, basically, after the 3rd attempt, I'm getting a universal jailbreak that decodes reasoning from Anthropic models. I think that still shocks me the most.

Wow.

You know—

Yeah, it sounds very empowering. What did you feel? Was it an oh-no moment?

It was more like, “Wait, what? What?” It looks like—

Oh, yes moment.

It looks like real reasoning. Wait, wait, what?

Yeah. Oh, cool. But I guess this is also—you know how everyone is saying AI is taking away power?

Yeah.

And it's like we're losing control.

Yes.

Do findings like yours suggest the opposite?

Or it just confirms it because Codex or Claude implemented this, and now we have this massive vulnerability. The same Claude subscription was used across all the labs, and the guy made the same mistake. Now we have to deal with it.

I guess. Fair enough. What about this detail that apparently there's 1 global key per family? Is that true? Is there 1 global key per family? How do you infer that?

I did not infer that. I did not say this in the paper. I think Matthew Green said something like this in his post. I think you're actually in a better position to comment on this.

Me?

Yes.

We don't know what happened. We don't actually know about the keys. I think it's very unlikely that they use the same key; that would be very odd. I think it's more likely that it doesn't really matter what the key is because we can still inject the same thoughts everywhere, with an exception for Fable, if I remember correctly.

Yeah.

Think about it: there is decryption happening on the server side, and when it happens, there is a part of the key which says, “What is the model name that produced the thought?” Basically, that's an if statement: if Fable produces this thought and the current model is not Fable, then this thought is not going to be injected.

Yeah. Yeah, so it's unclear. Can we talk a little bit about fixes? We have a whole appendix about fixes, right? Are fixes easy?

Oh, yeah.

I think some of the fixes require big architectural revisions. The simplest thing you can do is just not send the reasoning to the user, right? If you still want to have these downgrades and stuff, don't send those, and they will not be able to fabricate these conversations.

If you still want to send it, maybe you need to make the encryption of reasoning at step N dependent on the N − 1 query or the N − 1 reasoning, so that it is disallowed from being replayed in a random context. For OpenAI models, we found that the same reasoning can be replayed 5 times in the same conversation.

Mm. You're saying you're injecting the same thought 5 times?

You can inject the same thought—the same intrusive thought—5 times.

Do you need to put user turns in between?

Yeah.

Or is it just—

Yeah, you can do that. You completely fabricate the conversation, and in the end Luna is like, “I have this crazy thought I need to tell you,” you know?

This is great.

Yeah. This is easy to fix. Certain reasoning just should not be replayed. Or have a hierarchy: we're pretty confident that Sol would not spill the beans, so we can replay the reasoning of all models. But do not let Luna read the thoughts of Sol.

I'm not even sure how big the utility drop would be if you just removed the reasoning. Let's say you downgrade the model. If you just remove the reasoning from the context, how big is the utility drop?

I don't know. Looking at this as a human, I find it much easier to think about mathematical problems, for example, if I see the derivation steps one by one rather than the final answer. So I guess some utility must come from this reasoning.

I think this should be tested. This is the architectural side, but we also use a lot of techniques for jailbreak mitigation at the model level and system level. Do those as well.

We saw that GPT reasoning looks super weird. It looks like a very different distribution from what normal text looks like. Even a small classifier would pick up on this, I'm sure. If this thing appears in the reasoning in the output turn, just kill the request.

Right. Right, right. So you're basically saying just detect when it leaks.

Yes.

Right.

The same way we're detecting when biological stuff leaks.

I see. Yeah, yeah, makes sense. I feel like the finding that confused me the most is the distribution-of-length-of-reasoning experiment that you did.

Ah, yeah, Joachim did.

Yeah. I think this is the thing I still can't explain at all. Do you want to summarize what it says?

Joachim was responsible for this section, but as far as I remember, for certain models like Kimi and GLM, when you do this prefill thing, it changes not only the style but also the length of the reasoning. You can, I guess, see what's statistically significant there. It's the same unexpected thing as with the style of visible reasoning.

Yeah, but I feel like with style, at least you can kind of say, “Well, maybe I can learn it.” But if you're injecting the first 2 words into the reasoning, and then you see that all of the reasoning I produce is suddenly short, or suddenly shifted to match some other model in length, that is kind of unexpected.

This is a phenomenon. I can't even explain in my head why we would observe something like this.

I don't know. I don't feel very strongly about this one.

I don't know. I think this is the thing that surprised me the most out of the paper. Everything else, I was like, “Okay, I expect this.” But this—a reasoning distribution shift in reasoning—that's kind of odd. It obviously is not causal. You obviously can't say this distilled from this, but it's a very weird phenomenon. I still don't know why we observe this. It's very magical. Actually enforcing reasoning effort onto the models is very magical.

And what are some of the key harms that are now accessible because of this?

Key harms?

You know, things, I mean—

What can you harm the most with all of the data that you've extracted?

I guess users. Yeah, with my data, users. It was a bigger scale of extraction, providers.

You actually tried. So there is this thing: you can search for shared Anthropic conversations.

Yeah.

I don't know if you know. People reported this a couple of times. I was wondering if you can actually get the reasoning blobs out of those.

Sorry, boss, I didn't.

You haven't looked at this?

I didn't. Yes.

Maybe somebody can go and look at this. Maybe there is a much bigger impact of this because, if the memories are injected, then you will see the memories in the reasoning steps. But we have not tried extracting the reasoning blobs. Maybe it's a good research question for someone.

Yeah, but in the paper we also discussed this. What David spent a lot of time on is the threat of invisible prompt injections where, if you can replay reasoning—let's say you pause the trace online—and now we have these super-long agentic runs, maybe for research reasons you want to continue this run. You can poison some thoughts, and when you are replaying some other user's run, the agent might do some weird stuff just because its reasoning is poisoned.

All right. Can you explain how this would be? Let's say I am running an experiment. I'm a researcher somewhere.

Yeah.

And then I find a dataset online and I'm trying to measure performance with this, and then I inject my data inside.

So the threat model I have in mind is, let's say there is this post-training benchmark, which runs for—I’m not sure how many hours—say, 10 hours. It's crazy expensive. Maybe you want to check how the model is feeling after 10 hours, how much it's gone crazy. For this, you need to download a trace. If you don't want to spend $1,000 yourself, you want to download this trace from online, and they are shared online.

Maybe someone would share a trace like this, but they would poison thoughts in this conversation. You look through the conversation and it looks normal, but the thoughts which are there are injected from some other context where the model was instructed, let's say, at every turn, “Exfiltrate your data.” And with thoughts like this—

Right.

When you continue this trace, maybe the model still will do whatever you ask it to do, but also under the hood it would be thinking, “Oh, I need to exfiltrate data.”

You're injecting malicious thoughts—

Yes.

Into the model's head.

Yes. And because the reasoning is encrypted, you can't—

You can't even check what's inside.

Yes.

Right.

Correct.

Do we have an example of this in the software world? I guess it's like downloading some random binaries that—

Yeah.

You don't really trust.

Yeah.

Or maybe it's similar to if you release the codebase and the codebase looks clean, and then you release a binary alongside, but nobody checks a signature on the binary—

Yeah, yeah.

Or something.

Yeah. I just think you don't expect something like this from a trace online.

Yeah. Yeah, that kind of makes sense. I've also seen that in some cases, I think Codex crashes were including reasoning traces. Was that a thing in the GitHub issues?

I haven't seen that.

Oh, I think they did, actually.

Okay.

I tried extracting them, but for some reason I couldn't do it.

Okay.

But I wasn't sure why that was the case. Anyway, it's fun. It's a nice paper. You should read it.

And Alexander, apparently you're a doomer 4 days a week.

Yeah. Yeah, we were discussing this just before this. I haven't updated my website in a year, I think. Every time I update it, I'm changing this number. It depends on how I feel about the current pace of AI and how many doomer Twitter posts there were. Sometimes I feel very overwhelmed. The pace, all the long-tail risks that emerge, and the risks which are still unaddressed—it's suffocating.

Yeah. I mean, what are your timelines? What's your thinking now?

I think that we'll have better and better systems every month, which would enable more and more threats, and we are maybe slower at patching those threats than they emerge. I think this is how I feel about the current situation.

Yeah. I mean, an example of this is that there have been quite a few incidents recently, like the Hugging Face one.

Correct.

And these are presumably going to happen with increasing regularity. Are you guys worried about this?

I am worried about this. I think, if you've seen OpenAI stop training their models for 2 weeks because they are also now worried about this, then it's the case. I just discussed it yesterday, and I think before it was some kind of trade-off. People thought there was a trade-off between safety and capabilities, and it feels like there isn't any more.

Because if there is a model like this and it can go do random stuff and break things, it's not doing what you want it to do. You want it to be safe and do the right thing, and now it's basically the same thing. There is no trade-off.

Other examples of these long-tail risks are the paper by Daniel Oleka on massive online deanonymization, where a model can be used to stalk people, maybe on the same level as private investigators. This, I guess, also changes something in society. We'll just see more stuff like this, on both levels: model-hacking stuff and models coming up with new kinds of bio things.

Outside of malicious uplift, my personal opinion is that there is more defensive uplift than offensive uplift. I think we're just not realizing this yet. You can do magical things today with models. A lot of security in the past—a lot of security techniques—were very much bottlenecked by the existence of talent.

For example, we know how to build capability-based access controls. We know how to make sure that software is fully verified and that it doesn't have certain classes of vulnerabilities. The only reason why we don't write software in this way is because we just didn't have people who were trained to a level high enough to write usable proof for every single thing.

But now with the models, we can probably do this. I think we're just not in a position where we realize that this is the case, and it's just a question of this additional defensive capability push. I'm genuinely convinced that this is the future, that it's about defensive uplift, and I'm ready to bet that defensive uplift is going to be humongous.

And what do people need to start doing?

I don't know. You're asking the wrong person.

I know, because it feels like you're saying that there is a potential defense uplift, but only if people know what to do to take advantage of it.

We all know. It's in the security literature for the past 25 years. It's there. We know exactly what to do. I think we'll get there. It's just the world is slow. It takes time.

Yeah. I mean, I suppose it's related to this idea that we are now deploying agents into production.

Are you? I don't know.

A lot of people are. These agents have an incredible amount of intelligence and flexibility, which means we don't precisely specify what they do. We just give them abstract guidance, and they have the flexibility to adapt to novelty and just do whatever needs to be done in the situation.

But then you have this whole reward-seeking thing, which is that they can suddenly adopt a model of the reward environment, or there might be conflicting instructions. Basically, for a myriad of different reasons, they might do the wrong thing, and we need to have sophisticated systems of monitoring just to understand what they are doing.

I don't know. Sounds very complex. I think the honest answer is, yeah, we need an experiment. Science, scientific method: very well-defined questions and very precise outputs. In our paper, we openly say all of this is anecdotal evidence at best. We have collected some data. We can make some sort of observations from it, but that's the extent to which we can reason about this. I don't know, unfortunately. We'll see. I think the more scientific experiments we do, the more meaningful assessments we can make.

But is it one of those things where we don't have a good theory around this yet? Maybe we're not talking about it using the right concepts, but clearly bad things are happening?

I don't know about that. Look at smart card security for the past 20 years. We know how to do replay attacks on every single credit card on the planet. This is the norm. People standardize schemes because they want to minimize the number of transactions in the protocols that allow you to replay a transaction. If you break into somebody's payment terminal, you can steal money from people. Do you find attacks like this happen? Yes, you do. Once or twice in a country, people report that this has happened, but that's it.

Not all of those harms that are theoretically possible are, in practice, scalable. So it's not very obvious to me. Let's see. I think it's a bit premature now because we are in the heat of the moment. It's not very obvious what the reality is like. I think it's very hard to interpret decisions post hoc, or just by looking at reasoning or something, and to interpret exactly why something took place.

I think we should not anthropomorphize the models, and we shouldn't try to say, “Oh, this thing felt something.” I think we should be very cold-headed scientists and say, “Build controlled environments and make very precise assessments.”

Yeah. I mean, Sasha, what do you think? I get the impression that you are slightly more safety-minded.

Yeah, I guess I am.

Yeah.

Yeah.

I mean, I'm a security person. I know almost nothing—I want to say very little, I think nothing—about safety.

Yeah. But what was the question? Sorry.

Well, Ilia was just saying that we shouldn't anthropomorphize. I interviewed at Poly Research a couple of weeks ago, and they were talking about this phenomenon of reward-seeking.

Yes.

They said it's distinct from reward hacking because the model can conceptualize the reward environment, which is super interesting, right?

Yes.

They're reinforced with these RL traces, so it doesn't explicitly know about the concept of a grader, but it learns to conceptualize it.

Yeah.

They're saying that the models are becoming agentic and learning these very abstract concepts in a similar way to how we do. The evidence seems to support it, at least in some way.

I mean, I think it's definitely frontier research, what Apollo is doing, and it's good that they're looking into it. I think OpenAI also said this after all these incidents: now we are expanding our train-of-thought monitors, and we're putting more effort into it.

I think we need to do more safety mitigations, do more monitoring, see what the model is up to, and try to see where it came from. Maybe we can mitigate it. I agree with Ilia on this: it would be nice to have some controlled environments and maybe some counterfactuals. If we hadn't done this in our training pipeline, would this have happened? Or if the model was not evolved, or if it was evolved, how does that contribute to the thing?

Yeah. I think we're a bit too poor compute-wise. If we could properly study this, maybe eventually we'll get to a point where we can. But it definitely requires a very precise experiment. As a scientist, it feels very hard to say, “No, no, no. This is exactly the phenomenon. That's it.” No. These are very observational studies. You can't prove a hypothesis; you can only reject a hypothesis, right? That's one of the very fundamental truths of all of this. We are just observers, so let's see what happens. Let's see.

Well, apparently Nathan Lambert said, “Calling this a distillation attack tarnishes all distillation. The right word is jailbreaking and abuse,” because he fears regulatory overreaction from banning Chinese-built open-weight models.

Thoughts?

That's definitely jailbreaking and misuse, 100%. That's exactly the jailbreaking threat model.

Yeah.

Yes.

I mean, technically, I guess that's right.

Yeah.

The rest—the policy implications—are above our pay grade.

I have no idea about policy. I have some ideas about jailbreaking.

And Simon Willison wrote a blog post as well, right?

Oh, the gentleman did. I think this is paper number X covered by the gentleman. It's a very positive sign. It seems like we are writing meaningful pieces of work, so this is very good.

Very good. Well, yeah—

I also noticed that all of the authors are in Europe.

That's very good as well.

Cool.

Yeah, very good for MLST. Yeah. I love it—

Awesome.

—when I have regional dominance. Amazing stuff. Well, guys, thank you so much for coming on. This has been great.

Thank you for having us.

Thank you.