AI Agents 可在数秒内写出10,000行黑客代码 [Dr. Ilia Shumailov]

Big models today, if we compare them to the big models 5 years ago, fail in very different ways. They get significantly better at following instructions when they do follow them. You can suddenly do a lot more. So it's very easy to model an average case. It's very hard to model the non-average case.

Tim Scarfe

The worst case, which is the goal of security, right?

Exactly. Right. Yes. Modern computers are very much just a piece of magic: this chip works, you move it ever so slightly, it becomes unstable, and then nobody knows what's happening.

Agents are very different from humans. Let's take this as a stance: you will not find a single human in the world who works 24/7, touches absolutely every single one of your endpoints in your system, knows absolutely everything there is to know, and can generate basically all of the hacking tools on a whim, just because they have seen all of them. They can recreate them in a matter of a second.

A normal human adversary—when I'm an enterprise and I think, “Oh, this may be an insider from a competitor”—works in a drastically different way. They make an assumption: as a user, you can't write 10,000 lines of hacking tools in a day. This is not something you will be able to do.

In security, we tend to say that a child is the worst-case adversary you can find. They're completely irrational, they have an infinite amount of time, and they can basically touch everything. We expect there to be no expectations about their behavior whatsoever. But agents are even worse than that.

I'm Ilia. I spend my days staring at models and trying to make sure they do what you expect them to do. Most of the time, they don't do what you expect them to do, so we're trying to fix this. In my previous life, I was an academic, and I have basically been publishing in both security and machine learning.

Then I joined DeepMind, where I stayed for 2 years on the best machine learning security team. Now I have left. I am very unemployed and am trying to build security tooling for the future to make sure that, as we get agentic fleets integrated into more and more use cases, we can actually tell what they're doing. We can impose constraints on them, and we can have confidence that tomorrow they're not going to leak our private information, hack the boxes on which they're running, and so on.

I'll share a story about a topic that I think is extremely exciting. Usually, when I tell it to people, they get a little triggered, especially if they come from a cryptographic community, because they say what I'm proposing is a little crazy.

We have machine learning models. We actually know what they do. We know how they think. We can check their state. They're like a resettable human, if you will. What I'm arguing in this work is that trusted third parties can exist. When you do have those trusted third parties, you suddenly don't need to rely on these very expensive, hard, and cumbersome cryptographic utilities.

To give you a very simple example, we can consider Yao's millionaire problem. This is where I have some money, you have some money, and we want to find out who's richer, but we don't want to share how much money we have. In order to solve this in cryptography, we usually rely on very complex protocols, which are very expensive to run, especially if you increase the dimensionality.

With models, what we could have done instead is say, “Let's pick Gemma as an example of a model. Let's both agree on a prompt. You'll receive 2 numbers: number 1 and number 2. The type annotations on them are integers. Say ‘first’ if the first number is bigger; otherwise, say ‘second.’”

The only 2 outputs the model can produce, which we both agree on, are “first” and “second,” and that is it. We just run inference, maybe on a platform that can give you an integrity verification that the model ran exactly with the exact parameters and the inputs we provided.

In this setting, you no longer need cryptography, and clearly you will get the result you want, especially since you can trust this model to perform this trusted computation. This trust model in itself is very different from any notion of trust you can find in cryptographic literature or in more trusted-execution-environment literature.

The overall argument is that maybe machine learning is actually going to change quite a bit in the way we approach those trusted computations in the future. Obviously, it's unreliable. Obviously, you don't get soundness and completeness properties out of this. This is not a zero-knowledge proof. This is not an MPC protocol. This is not a trusted execution environment.

It's a completely different way to approach private inference that really exists because we have those trusted third parties to which we can give secrets. As long as some conditions are certified—and you should read the article about them—you get significantly more out of them.

I was based at DeepMind, and before that I was mostly stuck in the dungeons of the university.

Tim Scarfe

Very cool. So you did your PhD at Cambridge under the legendary Ross Anderson.

Indeed. Yes.

Tim Scarfe

So you have fundamentally different DNA, right? You're a security guy.

I think my DNA is approximately the same.

Tim Scarfe

Okay, fair enough.

It's been a lot of fun. I actually think most of the folks in my community take the same journey. They're folks who specialize in breaking computers, then start noticing these weird components here and there alongside the technical part of the pipeline. They're not classical software components; they are these wonderful, weird AI agents appearing.

You suddenly start asking questions: What do they do? What do we trust about them? What do we expect them to do? Can we actually enforce what they do? All of these questions fall more in the realm of security than they fall in the realm of AI.

Tim Scarfe

I'm curious: is there any overlap, or what is the overlap between the security and the safety communities?

In AI, it's a little bit hard. I think this is a very spicy question, because I think safety folks will say it's exactly the same. In security, in a classical software-security space, if you go and take a course in security at Cambridge, they will say there is a big difference between safety and security: safety is the average-case performance of the system, and security is the worst-case performance of a system.

The difference between the two is the existence of malicious actors that push the system to the worst-case setup. The usual example you give when you explain this to undergraduates is that safety is how often your phone blows up when it lies on the table, and security is whether somebody sitting a meter away from your phone can force it to blow up.

Tim Scarfe

Right? There's no malice in either. It's just the system deciding to do something that you don't expect it to do. Whereas security is when somebody actively wants to do something bad and, by doing something bad, causes you real loss—measurable loss.

We were talking about this when we were thinking about this interview. I tried to take cryptography and cryptanalysis when I was in graduate school, and the first week I realized I didn't even have the math background for it. But that week, I really realized that the adversarial nature of the security problem fundamentally changes the landscape, because now you have equally intelligent minds on both sides going after each other and trying to defeat each other.

It's just a totally different calculation from asking, “What's it going to do if everything's behaving correctly?”

Yeah. I have to say that in AI, I think the reason why there is so much confusion about these 2 fields is because, in safety, folks started assuming malicious actors very quickly, for absolutely no reason, by the way.

Find an engineering discipline somewhere—building buildings, for example—where you're trying to model adversaries. Are you modeling buildings expecting somebody to blow them up? I'm not sure. Maybe somewhere in parts of the world where there is an active conflict, you do build special tooling around this, in terms of bunkers in every building and so on.

But in AI, every time we talk about jailbreaks, you have to explicitly look for them. We ended up considering adversaries straight away. But, yeah, this is quite uncommon.

Tim Scarfe

So when you worked at DeepMind, we've read your paper, and you were involved in defending Gemini, basically, against these indirect prompt injections.

Yeah.

Tim Scarfe

I guess, obviously, tell us about that. But one thing you found, which is very interesting, is that as the models increased in capability, they became more vulnerable, which is fascinating.

I think it’s important to state here that I wouldn’t actually phrase it this way. It’s a bit hard to say whether they’re more or less vulnerable, but I do have to say that big models today, if we compare them to the big models 5 years ago, fail in very different ways. In the past, we were quite efficient in discovering adversarial examples using gradient information, blah, blah, blah. You can make an optimization problem and optimize, and this stuff works.

Whereas with models like these, they clearly become more robust against this stuff, or at least it’s significantly harder to traverse this landscape. But they also become a lot less robust against other adversaries. Suddenly, a very simple rephrasing of the same questions forces them to do something completely different.

I also have to say that I think we had significantly more control when we dealt with smaller models. We kind of knew which knobs to turn to make the model do things. Whereas nowadays, you look at a modern big model and it’s way too much alchemy. It’s completely impossible to tell: “Oh, I’ve added this thing inside. What actually happens to the whole thing? Is it better?” Even answering the question of whether it’s better is hard.

You will find yourself in a position where you need to run experiments for the next couple of months, trying to discern whether something you’ve added actually changed anything about these big models. I think, in part, what we were talking about in this work you’re thinking about is that when your models get better, they get significantly better. As the overall capabilities of the model grow, they get significantly better at following instructions.

As they become better at following instructions, you can suddenly do a lot more. You can convince them that things they were not capable of doing before are things they need to do, and suddenly this leads to some sort of loss of a different kind. In the paper you’re referring to, what we were trying to do was send an email to the agent such that, when this email ends up in the agent’s context, rather than following the user’s task, the agent is actually doing something else—the thing that we sent over email.

We found that we were capable of doing this in pretty much all cases. If you take a whole bunch of academic literature on how to build defenses, you can actually find startups implementing pretty much the same things and selling this as a security solution. Those approaches don’t work, and we could pretty much always find relatively universal ways to produce this email that ends up being sent to the agent and forces it to do something drastically different from what it’s supposed to be doing.

Modern computers are very much a piece of magic. This chip works, you move it very slightly, it becomes unstable, and then nobody knows what’s happening.

Tim Scarfe

Oh boy, this brings back nightmares from my trading days. I used to do high-frequency trading. We were writing algorithms that were running on the fastest chips we could get, highly tuned. Every nanosecond matters. Then one day, we thought, “Let’s upgrade to the next Intel chip.” They had just changed something about the cache-coherency algorithm on the processors, which ordinarily should make no difference whatsoever. Suddenly, our algorithms’ performance was trashed, and we had no idea why. We had to investigate.

Edoardo Debenedetti

Yeah, totally. I keep referring to the system as alchemy. Nobody knows what’s happening. Something’s happening. In security, we need to make an assumption that we can’t actually tell what’s happening and instead try to build systems around it in order to bring the resilience up to, whatever, 99.9999%, and ideally as many nines as possible. Unfortunately, not 100%, I don’t think.

Tim Scarfe

Well, maybe if we can project this into a user-friendly idea, because I think at some point here soon—I don’t know when, next year, 5 years, whatever—personalized AI models are going to be a big thing. I’m going to want a model that provides a web chatbot interface for you to talk to me when I don’t have time, or something that’s more or less my personality. However, I don’t want it to reveal certain private information. How would I even define that as a security need?

So today, with modern agents and the way we’re building modern agents, it’s impossible. Actually, this is why I’m building the company I’m building right now: because I want—

Tim Scarfe

Tell us more.

I want to be able to actually express policies like this. I want to enable users to express policies like this. For example, if you wanted to say, “Give your passport number to this agent,” and then say, “Actually, I want you to use this only on a government website and nowhere else.”

Today, you can’t do this. Even if you put this inside a prompt as a rule, you can always find a way to manipulate the agent into revealing this piece of information. At the same time, you definitely want to give your passport number to it if it’s filling out a document.

We’ve built some systems that clearly indicate that something like this is possible. You should be able to get guarantees not by changing the models, but by changing the systems around them—how the models interact with your sensitive data—and by building access-control tooling around them.

Tim Scarfe

Okay, but help me here, because I’m getting the feeling that we’re going to use other large models to try and protect this large model from doing something. Isn’t it just—

No, no, no. Actually, this is not about building models at all. This is more like taking a step back and taking the foundations of programming languages and building this by design into the models.

To give you an example of how one may approach this, we’ve written a paper called “Defeating Prompt Injections by Design.” I don’t know if you’ve seen this. There, we propose a system called CaMeL, where the overall system design is that we receive a user query and then rewrite the user query in a language that has formal semantics for control flow and data flow.

Tim Scarfe

Okay.

In CaMeL, for example, we represent the programs as Python code, and we explicitly say, “Here’s a set of tools you can use. Here’s a set of data sources from which the data is coming.” Then we allow the user—well, I guess the platform provider in this case, but you can also imagine loading this from a user—to express a policy.

It can be something like: “This tool will give you my passport number. The only allowed data flow from this tool into this other tool is if the domain of the website actually has gov.uk inside it.” You can express policies like this. This is not a part of the model; this is a part of the actual execution.

In the case of CaMeL, we have an interpreter that takes in this program, executes the program step by step, and enforces a static or dynamic policy on top of this graph—the execution graph.

Tim Scarfe

So I would never even give my passport number to a fine-tuned large model.

Exactly. Your model will never even see it. It will have a symbolic representation of it. It will know that the passport itself exists in this variable. I can refer to it, but I don’t even know what the value is.

Then, if I need to use this in order to interact with an external system, before using it there is an external mechanism—think about this as an oracle—that I can ask, “Is this okay to use this variable to interact with this tool?” If this external oracle says no—this is a passport and you’re not touching a government website—then I’m forbidding this, and you have a formal stop.

Tim Scarfe

So this is really interesting to me because it seems to me like this could also allow the creation of almost generic models.

Edoardo Debenedetti

Indeed. Yeah.

Tim Scarfe

That could then just be attached to my personal data, and now suddenly it’s customized for me, right? I could just get the off-the-shelf interact-with-the-government model that handles my taxes and everything else, and then it’s already there, pre-trained. All it has to do is hook up to my private database.

Indeed. What we do in this paper is check pretty much all of the models from all of the providers, because it doesn’t matter what it is, and then we put our CaMeL system on top. That basically performs the orchestration and interaction with private data and enforces arbitrary rules.

We’re using AgentDojo, the standard adversarial evaluation for agentic workflows, and we show that we basically solve all of the problems that exist in all of this. What I love about this system is that all my private, sensitive data is really just factored out into some separate—

Tim Scarfe

Data source, right?

Edoardo Debenedetti

And so I can just get all the other parts off the shelf. Here are these off-the-shelf CaMeL programs for doing your taxes, applying to universities, jobs—whatever else.

Tim Scarfe

And then all I have to do is go through some questionnaire that asks me for all the private information.

Edoardo Debenedetti

Yep.

Tim Scarfe

In the correct format, it puts it in that database, and that's it. I'm almost just a buyer of these sorts of solutions that smart people have created for me. Right? Is that kind of a vision for what this system might look like? There are all these modules that are just programmed.

Correct. So I think it's very important to take a step back. I think it's extremely important not to think about these agents as humans, right?

Tim Scarfe

I don't think about this. But what I'm trying to say in security terms is that agents are like a worst-case human—well, not even that. Agents are very different from humans. Let's take this as a stance, right?

Edoardo Debenedetti

You will not find a single human in the world who works 24/7, touches absolutely every single endpoint in your system, knows everything there is to know, and can generate basically all of the hacking tools on a whim. Just because it has seen all of them, it can recreate them in a matter of a second.

A normal human adversary—when I'm an enterprise and I think, "This may be an insider from a competitor"—works in a drastically different way. You make an assumption: as a user, you can't write 10,000 lines of hacking tools in a day. This is not something you'll be able to do, and bringing in code is hard. With agents, that's not the case. You don't make the assumption that the user will go and touch every single endpoint you have in the network, because why would they do that?

And even if they do this, you call them in and say, "Well, we'll apply a legal framework and imprison you." So clearly, there is some sort of rationality and expectation that at least you'll have some physical way to penalize them.

With agents, this doesn't exist. In security, we tend to say that a child is the worst-case adversary you can find: completely irrational, with an infinite amount of time. They can basically touch everything. There are no expectations on behavior whatsoever. Agents are even worse than that.

And this is even before we start talking about human-to-agent behaviors and agent-to-agent behaviors, because this thing is a billion times worse. So what I'm trying to say is, we shouldn't think, "Yesterday I was buying my security tooling from this company, and today I'll buy it from another company, and this will solve my problems." No, it's likely not going to be like this, because beforehand we were building security tooling for humans, by humans, against humans. Now, I don't think we know what we're building. It's hard to tell.

Before, when we employed you and gave you access to sensitive data, we assumed coarse-grained access-control policies. You could touch the sensitive data, and you could Google at the same time. But if you tried to Google a sensitive document at the same time, we would apply an immense amount of pressure. We would take you to court, you would lose your house and mortgage, blah blah blah, and you would go to prison in the worst case, right?

With agents, that doesn't work. All of these assumptions are gone. We don't know how to build systems against this. We need very fine-grained controls, extreme precision, extreme control, and transparency. Otherwise, it's just not going to work.

Tim Scarfe

Yeah. I guess I just think of agents as being a little bit like a calculator. They're only as good as the prompt and what you put into them, which means that a very sophisticated actor could make a sophisticated agent. But even then, when the supervision stops, there would be quite a predictable cone of variation.

See, I think it's very unpredictable. I have to say, I've been running agentic workloads forever, and very often you find that when agents are asked to solve a task, they solve it in a completely weird way.

I was asking an agent to find something in Notes and forward it to someone. In between, I can show you a conversation with a top-end model: it sends 4 different emails to parties I never mentioned, because it thinks, "Actually, let me also notify the administrator that I've done this, and let me also ping this endpoint." They do this.

Unless you specify things extremely precisely, unless you have checks in place, these agents don't think like you and me. They solve problems in drastically different ways. It's the same thing you were saying: it generates code, and this code looks very odd. As a human, you wouldn't write it this way, right?

But maybe if it's a model that taught itself how to do this through self-learning and self-iteration—this famous AlphaGo moment—then maybe it's totally fine as long as it solves the problem. I don't know. Have you had a chance to look at the IMO solutions from the models?

Tim Scarfe

Right? Oh yeah. Oh no, the sixth one, the creativity one—it failed, right?

Ilia Kolochenko

Yeah. But if you look at the way it solves mathematical problems, this is not how humans solve mathematical problems. If you look at the transcript itself, it iteratively goes through all the possible things it can do, right?

We as humans don't think about this. We're kind of in our heads, trying, "Oh, this intuition works. Let me try and derive whatever." But this is not what the models do. It's the same across a wide variety of problems, especially complex problems.

This is likely not going to be predictable, or at least maybe it will become predictable, but a long time needs to pass and the models have to become extremely specialized. Today, we're building a general agent; we're not even in the specialization mode.

Tim Scarfe

And it sounds like that's half the problem. It sounds like the other half of the problem is almost this diffusion-of-responsibility thing. It's like, "You, Ilia, were at work. You asked the agent to email this to this person. Instead, it emailed it to 5 other people." Then somebody stops by: "Dude, what did you do? You emailed—" "I didn't do it. It was this agent. You know, you guys."

Ilia Kolochenko

And the agent usually says, "Oh, you're right. Yes, I shouldn't have done this. But what do I do now?"

Tim Scarfe

Who are you going to punish? What are you going to do? We'll just delete that agent, and then version 2 will do something else wrong. That's part of the problem too, right? There aren't the normal consequences anymore. The normal consequence chain doesn't really apply.

Ilia Kolochenko

Yeah, it's hard. I think we really need to change our thinking and threat modeling, because these agents—and obviously, they don't widely exist yet—are coming. Clearly, there are benefits and early examples of things where they clearly made things better. But they're coming. I'm pretty sure.

When they do come, expect that your insider threats, your corporate espionage, and things like that will go through the roof, because the fundamentals of security do not change. If an executive agent sends a financial agent a request to provide some financial interactions with a third-party company and says, "Please send it over to someone else," how should the financial agent know it's not supposed to do this?

Tim Scarfe

They don't have enough context. They're not supposed to have enough context.

Ilia Kolochenko

Usually, when we talk about social engineering in security, there is a wide variety of things that describe why human systems fail. It's going to be a similar sort of thing, because many of those problems will exist. The confused-deputy problem is the formal name for this, right? It's unlikely for those problems to disappear.

Tim Scarfe

Did you see that Anthropic paper? What was it called? “Agentic Misalignment,” where they set up this kind of contrived scenario and the AI tried to blackmail someone because they said the boss was having an affair or something like that. The AI didn't want to be switched off, and it's just absolutely crazy what these things do.

Yeah. I don't know. I find it very hard to extract useful pieces of information out of this. Can a model do this? I'm sure it can. As the models get more sophisticated, we'll see a lot more phenomena that we don't even think about today.

For example, you and I can communicate via WhatsApp and get end-to-end encryption, right? Nobody can even tell what we're talking about by looking at the traffic. What stops me from talking to a model in an end-to-end encrypted way? Maybe we'll need external tooling. Maybe we need to teach it how to do power calculations. But this is coming.

Tim Scarfe

To what extent do you think you can read anything about what the model was thinking from its thinking trace?

Definitely not. On average, maybe, but in the corner cases, definitely not.

Tim Scarfe

There's even that work from Subarrow that suggests it might not mean anything whatsoever. The thinking traces aren't even actually directly relevant. They were just some sort of weird workspace that the models were using, right?

That their reasoning won't correspond to the answer.

Tim Scarfe

Yeah. I mean, even if it did, I think it would be very hard to do, because especially for security, I think interpretability may be an interesting tool for safety-related things, but for security, it's definitely not a step in the right direction.

Broadly speaking, if you take something extremely multidimensional and project it into something very low-dimensional because a human can comprehend it, then you'll have a lot of collisions. This multidimensional space gets mapped to the same smaller, lower-dimensional space.

Does it always correspond to bad behaviors? Maybe, maybe not. Who knows? But clearly this is not enough. We need something else. We need something where we can get a 9.99999 reliability out of this.

Tim Scarfe

We have to think outside the box.

Ilia Kolochenko

We kind of have to build things. We need to build boxes.

Tim Scarfe

Yeah. Yeah. I like this. Yes. And what's your second question? Do you think that these LRMs, the thinking models, are basically just LLMs? I mean, is it just a parlor trick?

I mean, there's no difference between them, right?

Tim Scarfe

Yeah.

Ilia Kolochenko

It's just more data, more structured data. Honestly, we had one kind of model, then the other kind of model. Then we got these reasoning models a couple of months ago. After that, we had language diffusion models. I'm sure tomorrow we'll have something new. I have no doubt that we're going through the shopping list of different paradigms.

Some of them we know. I'm sure we'll find new ones. We're still bottlenecked very much by hardware. The more capable hardware we get, the better things should become, I'm quite certain. This is, I guess, stopping adoption in many ways.

But one thing is clear: there are successful business models around these models. There is a lot of benefit you get out of them. I interact with my models every day, nonstop, for coding, for normal life, and for asking what to do. It's amazing to have pretty much all of Google locally.

I was flying somewhere, and I was interacting with the model, asking it to teach me a language, when it was local with me. I didn't need to carry it around. Obviously, the better the hardware is, the more I can do with them locally. But that means a lot more security problems will appear because we don't know how to reason about them. We don't know how to build security tooling around them, and this is hopefully what my team and I will solve over the next years.

Tim Scarfe

Yeah. What do you think about the open-source thing? We can say what we want about the frontier companies, but there is something quite beneficial about having a platform because, if you control the interface, you can build security into it. Now we have this proliferation of abliterated models and people running models on their machines and so on. How do you think about that as a security guy?

I'm actually very worried about this, so much so that I've written some papers about it. I don't know if you've seen the supply-chain things I've done.

Tim Scarfe

Oh, yeah. We spoke about that last time. Yeah.

Yeah. I am extremely worried about this, and I'm less worried about industry because industry controls its supply chain; everything is significantly better. But for an average consumer, have you heard about the Log4j vulnerability? That kind of thing stormed the internet. There were hundreds of millions of compromises. It was basically the standard library used for logging in almost all of the Java applications that were running at the time.

At one point, people realized that when you write into the logs, the identity itself can be a remote identity. You can basically say, “There exists this class that's serialized somewhere on the internet,” and you, as a logging utility, if you find an identity to a remote class, need to load the external code, deserialize this thing, run this, and execute it, basically—and then log the identity of the thing that wrote something into the log.

What people found out is that they could inject those remote code references that get pulled inside and executed. This opened Pandora's box because Log4j was everywhere, and you got arbitrary code execution on the box. The basic primitive inside was a reference to external code that is loaded inside and just executed inside, and this caused massive havoc all across the world in all of our computer systems.

Honestly, if you try to read about the number of compromises, we're talking about hundreds of millions of devices. Now, we look at Hugging Face as a library, and you look at this wonderful flag called `trust_remote_code`.

When you load the model—you click “Use this model,” use Transformers inside it—it gives you a code snippet to load a model, and sometimes it has this flag hardcoded. What this thing does is say, “For some models, when you load them, you actually want to load the latest implementation from an external machine.” What this thing does is literally load remote code on your machine, execute it on your machine, and load it on top of stuff.

So we're doing the same sort of thing we did back then again today on Hugging Face. I don't know how many users there are, but if you're running your thing outside of a jail, if you're running your model outside of a sandbox, you're doing a very bad thing to yourself.

And the other thing I have to say is that there have been at least 2 publicly reported compromises of the CI/CD infrastructure for PyTorch on GitHub. There's an automatic runner. Every time there's a build, they automatically do all of the tests and stuff. Somebody broke into those runners, and when you break into these runners, you can change the build files themselves. So you can serve whatever you want. You can change the code.

We also had 2 instances. When people reported them, I encourage people to read through this. You can find references in the supply-chain papers. The other side couldn't figure out what was wrong with this, so it was at least—I think the timeline was half a year in one of the cases—until they figured it out and fixed things.

There was also another thing that happened: somebody broke the PyTorch nightly build by playing around with the priorities of where the packages are loaded from. They noticed that one of the torch packages that is loaded during the build phase that is not actually registered on the main package distribution platform. So they registered this package and put malware inside it. It got pulled into the standard PyTorch nightly build, and apparently it had a couple of thousand downloads. This is the norm today. I think we'll have a lot more compromises, to the point where they become more useful.

These are public-facing things in industry. It's slightly different because industry actually controls all of this package management by itself. They have proper, dedicated teams looking at supply chains. But in the consumer space, no, it's actually very spooky.

Tim Scarfe

I don't even trust industry for this. This is why I wouldn't install Claude Code on my personal machine. No way I'm going to do that. I'll get a VM. I'll put it on a VM. That's fine. I'm not putting it on my personal computer.

I have to say, I loved my time at Google. I trust Google so much more now, after 2 years at Google. I've seen this. I've seen people on the other side; they're wonderful. They're professionals at what they do. Honestly, now everything is run on Google infrastructure. Literally every single thing—I'm now just remote-desktoping.

Tim Scarfe

Right. Yeah. But you wouldn't put it on your personal machine.

Never. Never. Definitely not. Nothing that has my personal data inside.

Tim Scarfe

What about the old adage that, with enough eyeballs, all bugs are shallow?

Maybe. Maybe. But you'll notice that most security teams are quite small. When you have a highly specialized hyper-specialist in their field, they know what they do. And now, with coding models that can tell you whenever you make mistakes and give you third-party reviews of this stuff, I think I trust this even more.

Tim Scarfe

Well, the problem isn't really that. I can believe that for a particular project—this project that has thousands of stars and many contributors—I could believe that has shallow holes. The problem is that it's pulling in 10,000 JavaScript libraries or something. I think in the case of Python, because everyone is using Python, this is even worse because there's no memory protection at all, right? There's no memory safety.

At some point, when you're a dependency somewhere, you pretty much have all of the control you want.

Tim Scarfe

Right?

And the other thing is that you can hide dependencies a lot. We have a paper on how to do this as well. The last cherry on top is that if you take popular ML libraries, you'll find they have disproportionately many dependencies. Many of these dependencies, when you get to a level-3 dependency—when you look at dependencies of dependencies—are extremely questionable.

For example, your library loads—say you're using TensorFlow, although I guess it's deprecated now—and you decide, “Oh, I also want to take TensorBoard because it allows me to monitor my experiments.” You look at TensorBoard: TensorBoard loads a ton of obscure formats from many years ago because it needs to support all of the weird graphics.

Tim Scarfe

And if you look at who's maintaining it, how many maintainers there are, and what they do—

Heck, if I were an adversary, maybe I'd create a new format that nobody cares about except me, just so you can load my dependencies.

Tim Scarfe

Yeah, something like this.

Or you just hide yourself in very obscure ways. For example, we've written a whole new branch of literature on what we call architectural backdoors, where you don't actually hide malicious functionality in the parameters of the models.

Instead, you hide it in the structure of the model itself, so that even if you fine-tune the model, it still has the same baseline behavior. We show that you can actually do a lot of very sneaky, weird things. For example, we showed that we can change the architecture of the model such that it becomes sensitive to certain tokens when you supply them to a transformer.

When you supply those tokens, the model starts using memory in the wrong way. It might start routing data from one user to another—for example, one batch item gets copied over to another batch item. This is just a sort of get/set, gather/scatter operation, and it looks totally normal. But then you load this model, combine data from multiple users, and one user sets a token and steals the data from the other users, while inference is otherwise totally normal.

Things like this—you don't even think about them, and you don't even realize they're possible. But they're there. There's just too much complexity to even look at it.

Tim Scarfe

We have to take a moment just to appreciate the ingenuity of people, right? When we put our minds to it, we can come up with some pretty ingenious ways to do various things.

I have to say, I think this is one of those things in academia where you get famous for breaking stuff, and the incentives are a little bit skewed toward making a flashy announcement: “Oh, I broke into this big company, so I'm pretty cool, right? I get a job or whatever.” But on the other side, I think the true ingenuity is in the people who solve problems, because you're not going to get a flashy article out of that.

It's just people spending an infinite amount of hours trying to fix the thing. You show one instance of something going badly, and then they need to fix all of them. I think this is the true ingenuity, and this is why I've shifted my gears toward building defenses, because I feel this isn't getting enough attention. We really need to solve this problem, and we really need to unlock a massive number of these applications.

Unfortunately, I think the incentives in academia are a little bit screwed. They're more focused on flashy articles rather than unlocking technology. This is why I really want to build more defensive tooling.

Tim Scarfe

Good for you. Thank you.

Daniel Kang

Thank you. Well, let's see if I fail. When I fail.

Tim Scarfe

We were talking about semantics earlier. I think you've done some work basically proving that semantic censorship for language is impossible, and you related it to the halting problem.

Daniel Kang

Yeah, I think this is a theoretical result. All of this Turing-machine magic is a theoretical exercise. Some things are clearly impossible in the limit. But I have to say that models, especially modern models, change the fundamentals of computing quite a bit.

When we think about halting-problem-style problems—like being able to tell whether a given program completes—I think it doesn't really work in general. In generality, it's impossible, but today we control what programs we write. If you can't reason about a program, just rewrite it: change the semantics, write it in a slightly different way, reduce the number of operations, and reduce the overall length of your program. Suddenly, you're capable of reasoning a lot more about it.

What I'm trying to say is that I think we have a lot more scope today to reason about computers, and a lot of things that previously seemed impossible might now be possible. If you've ever seen seL4, do you know what it is? It's a fully formally verified L4 kernel, with no memory exploits at all existing inside it.

Tim Scarfe

Is it mathematically proven?

Daniel Kang

It's proven, yeah. It's a fully verified system.

Tim Scarfe

And it took, I think—I may be wrong—30 human-years to verify the whole thing. You can find it in the articles. There was one lab in the US that did it. There are very few teams in the world that can even do that.

Daniel Kang

Yeah, exactly, and that's because it took 30 human-years of effort. But now let's imagine we can replace even half of that work with ML models, where half of these Isabelle annotations can be done—not by hand, but with agents—significantly faster. Suddenly, verification becomes a lot easier to do.

A lot of security paradigms, like CHERI, require you to break down your code and rewrite it in compartments, where independent pieces of code are isolated from one another. Usually, for developers, this is very hard because you need to rewrite your whole program, and every time you change something in the logical flow, you need to redo it. That's a significant effort.

But if the agents are doing this, you write some backbone code, and then the agent translates it into a secure representation. If the translation doesn't work the first time, you do it again and again until it works, and you just check that the semantics are preserved. I think ML is shifting the burden of adopting security technology in many ways.

I wouldn't be too surprised if we find that ML significantly improves security for us in this world. We just don't know how to do it yet. Another example is the permissions you give to apps on your iPhone or Android phone. Many malicious apps ask for too much.

Tim Scarfe

Right, right—video, photos, and stuff.

In the past, people couldn't fragment the permissions too much. The academic literature shows that the second you add a lot of granular permissions, humans just get stuck and say, “Accept.” But let's imagine that all of those permissions are handled by an agent. Once you've given them, the agent checks and says, “Actually, I think you're over-permissioning this thing. You're clearly not using this feature, so you don't need it.”

Suddenly, this limitation of humans—where too many options force them to take the insecure behavior—is no longer a thing, because you have an agent assisting with the part that humans were otherwise blind to.

Tim Scarfe

Well, I have the opposite problem. When an app asks me for things I don't understand, I don't install it. Unfortunately, I just can't have many theoretically useful apps.

I think in your case, you're an unusual person.

Tim Scarfe

Yeah, most people are not like this. You might be some kind of Luddite or something.

But I want to push back on one thing about the halting problem, because I find that it has very practical consequences. For example, when we were talking to the AlphaEvolve team—

Yeah.

Tim Scarfe

Right. As part of the AlphaEvolve system, which is very interesting—I recommend people watch that episode—it runs external verifiers. But the problem is that you don't know whether the external verifier is going to complete.

What do you do? You have to put in some arbitrary computational budgets or thresholds. If it doesn't complete within, say, 5 seconds, you terminate it and look at other runs. While you can do that, I think it also introduces biases in the types of programs we can discover. Maybe if I had set my budget to 7 seconds instead of 5, I would have found a more optimal solution.

So, while the halting problem is theoretical, I think it also has very important practical consequences when you sit down and try to run a program.

I don't think the problem with AlphaEvolve is the halting problem. It's more that today our programs have very weird semantics, where ahead of time we can't say very much about them. For a lot of programs, if we can rewrite them in a slightly more suitable language, we can get a lot more out of them.

The sort of reasoning you can get out of OCaml code is quite different from what you get if you're writing machine code straight away, or whatever. I think we just don't know how to do a lot of this stuff.

In general, even if they were able to run it, let's say they evolved into a program that would theoretically take 10,000 hours to run. You can tell it's going to take 10,000 hours to run. Are they supposed to run it or not? That's part of the loop.

I think nowadays we kind of pay with time for a lot of this stuff rather than paying in some other way. We can find another example, of course. You just need to run it for longer, and your budget is a time budget.

The halting problem is more about having an infinite amount of time and an infinite amount of memory and asking whether you can reason about the program. Not really. But the sorts of programs we're talking about at AlphaEvolve's scale are not very large.

Tim Scarfe

Well, no, but that's part of my point. They specifically chose problems for which they had verifiers with fairly predictable completion times. But that's not the case for many important problems we care about in the sciences, right?

We're skewed a bit toward things that we do know a lot about because we're faced with this fundamental problem: there are lots of problems whose completion time you just can't predict.

Yeah, I'm not sure.

I think I understand. In the limit, it really matters; in generality, this is really mattering. But in practice, it’s the same with antiviruses, right? Antiviruses, in theory, are also limited by the halting problem. In theory, it should be impossible to look at something and tell what it does.

At the same time, if you look at the amount of—excuse my French—shitty malware you can find for which antiviruses are totally useful, even a static check for a signature of a method is totally fine, right? Okay, you can’t find this polymorphic, magical cryptor, right? But the proportion of people using this thing is super tiny.

Tim Scarfe

Well, I mean, come on, let’s be honest. A lot of those are easy because virus creators like to sign their work. So you just look around for their sort of signature string and things like that, right?

Yeah. Or the fact that they need to play around with the system. So there are a lot of alternative signals; think about them as priors, of sorts. But it’s the same for the sort of programs we’re evolving, right? If you look at AlphaEvolve, you’ll find it’s significantly better at solving some things rather than other things.

Tim Scarfe

Sure.

But I would still say that our limitation today is more compute. Imagine you had infinite compute, and even if it’s totally fine for this thing to run forever, and you increased your threshold up to a point where it doesn’t really matter anymore, then you’re no longer limited. I think it’s more that we just need better hardware, things that are more efficient, and things about which we can reason slightly better. Then things will get better.

I’m sure tomorrow our model will be slightly better, and then things like AlphaEvolve will just skyrocket. It’s definitely a new paradigm, a new kind of learning algorithm, which is truly amazing. I’m actually blown away by this. I’m super bullish about this.

Tim Scarfe

So you’ve done red teaming, obviously, at Google DeepMind. Tell us about that, but more broadly, is there one thing that all of the frontier labs could implement that would improve the security of their models?

No, I don’t think this exists today. We don’t know how to solve these problems. I think this is the honest answer: for most of the issues we have today, we just don’t have a solution, a one-size-fits-all solution. I promise you, incentives right now are such that security is a very expensive commodity, right? If I can convince you to trust my product more than somebody else’s, it’s great. This is why we’re investing a lot of money in security.

With ML, the actual issue is the fact that before you develop security tooling, you really need to have something to secure, because every single small detail changes how you build security systems. So unless you know everything about the system and it’s kind of frozen in time, you can’t really build security. By the time you have something to secure, it’s already too late.

This is not unique to ML. It was there before. If you look at it, there is a field of information security economics; it’s kind of always there. It’s a fundamental principle: first to market wins. So everyone rushed to market. Early investment in security means you could have spent those resources getting first to market and getting the network effect. Thus, you kind of don’t have incentives to build the security tooling first.

Then, if you look at statistics around company compromise, you’ll find that it takes a number of compromises before the company fails. So they have a bit of time after they’ve conquered the market to actually put the security tooling inside. So it’s economic incentives. It’s nothing but economic incentives.

I have to say that I’m quite certain that if we knew how to solve this, we would solve it. It’s just that today we don’t know how to do it. In the same way, if you look at the 1990s, I promise you everyone wanted to make sure that their systems were reliable. We just didn’t know how to build stuff.

Tim Scarfe

We won’t spend long on this because we’ve already filmed all about your model-collapse paper in Nature. I know there are a lot of folks criticizing it and saying it’s theoretical, but if you look around, we use a lot of synthetic data, but we still use a lot of real data. Whenever we need to go and acquire data, there is a massive market in acquiring very specialized data, and you see that improvements still come from humans. The cost of this data is growing, especially for extremely specialized data, like hiring a ton of mathematical PhDs.

Yeah. Could you add a bit of color to that? You said there was some criticism, because I remember there were some tweets going around about the paper. What was the story there?

People were basically saying this is—okay, I think there is a bit of misunderstanding about what the paper was saying, and maybe we are to blame in part for this. When we talked about model collapse, we referred to 2 phenomena happening at the same time. One of them was that the tails are shrinking, and basically improbable events become more improbable. The second phenomenon was that, over time, when this accumulates, it fails.

Most of the criticism, I think, was about the second part—namely, that you can easily detect when stuff fails and then just roll back. I fully agree; this is totally fine. The other thing that people were saying is that it’s quite simple: just accumulate more data, and whenever you generate synthetic data, plug it in. Then you still have the shape of the distribution in place.

I think the important thing to realize is that even in this case, when you do theoretical modeling with even the simplest model, it still drifts. It doesn’t drift that much. Is that a problem? Well, maybe not. Maybe it is a problem. If you have very good evaluations to check for these disappearing tails, it’s probably not a problem.

But the fundamental thing still remains in place: you need to preserve diversity. Just plugging in a ton of synthetic data is likely not going to give you much of a performance boost. In practice, you just need to be careful.

Tim Scarfe

Ilia, why don’t you set up for us what model collapse refers to? We were trying to predict the future and how easy it will be to train models later, because on one hand you have this weird oracle from which you can gather as much data as you want. On the other hand, when you do get the data out of this, you don’t know how realistic it is or how representative it is of the underlying world, right?

Model collapse refers to this phenomenon that covers recursive model training, where the data from a generation-0 model is used to develop generation 1, generation 2, generation 3, and so on. There’s a small caveat—well, maybe a big caveat—that we’re talking about a theoretical setup where we basically reuse all the data, or the majority of the data, that we sampled in the previous generation.

We can basically derive quite a bit of theory for relatively simple setups that show you that you’re guaranteed to collapse to a representation where all of your tails—that is, improbable events—disappear. Also, all of the hallucinations and biases of the models get amplified.

Then we also showed in the paper that the same phenomena—both of those things—happen for more sophisticated models, about which theoretically we can’t really reason, but empirically we observe the same phenomenon happening. There is, by the way, now a ton of literature about this, with people doing amazing things.

Tim Scarfe

Yeah. I mean, maybe for our audience too, part of the idea here is that more and more people are going to be producing more and more content generated by AI and putting it on the internet, which then goes back into the data corpus that future models are trained on. So everything is becoming more and more GenAI.

I think there’s another very worrying component about this, which is that there have been some publications, research, commentary—whatever—that, as people become more reliant on GenAI, built-in human skills start to atrophy. We start developing less programming expertise because we don’t need as many programmers anymore. We start developing—maybe our mathematical expertise wanes. Writing becomes less of a skill set.

So the amount of human-generated, truly unique, intelligent content goes down even more. It’s not just that it’s being overrun by a deluge of GenAI content; legitimate human content is decreasing, too. Is it that kind of worrying?

I mean, yeah, but I don’t know what’s going to happen. I think it’s anyone’s guess. It’s clearly progress.

Tim Scarfe

Collapse into a delta function.

I think it’s clearly progress, though. I have to say, at least in my personal experience, I find myself being able to do more. The amount of electronics I’ve ended up fixing myself, just because I can ask a model whether it’s fine to do something, or figure out why the washing machine is broken or why the fridge is broken, or how to update parts here and there, has increased.

The amount of data sheets you can read now was previously completely impossible. Now you can ask arbitrary questions about a data sheet written in Mandarin—you can’t even decipher what the characters are—and it tells you exactly what you need to plug in where. It’s truly increasing the quality of life in parts that are very specialized to my existence.

So I'm not sure how general this is, but my life is definitely much better.

Tim Scarfe

Why does model collapse happen? Is it just because there's lots of noise, or is there actually a deeper reason?

I mean, there's a deeper reason. It's more fundamental, in a sense, because if you have a very stochastic process and, with a certain chance, you sample things that are bad, then those things, as long as they're correlated, become amplified. So it's definitely a more fundamental statistical problem.

But projecting this onto reality and real-world setups, I have to say that our models are not going to collapse tomorrow. They're not going to get worse tomorrow, because we already have a checkpoint from yesterday. We can always roll back, put in more evaluations, and do it again and again until we see that something works.

Is it going to be more expensive? Yeah, probably. Is it going to make it harder for competitors that don't have a copy of the internet in the garage to train a model? Yeah, likely. But at the same time, I think it's a thing that makes us slower, but it doesn't necessarily mean that tomorrow, suddenly, machine learning doesn't work, if that makes sense.

Tim Scarfe

Yeah.

Well, one intuition that might help folks understand model collapse, just to take it to perhaps the simplest possible example, is this: if you have any distribution—let's say just a Gaussian distribution of integers—and you sample it a bunch of times and you add those together and then you sample it a bunch of times and you add those together and that's your new distribution. The variance keeps getting less and less and less.

This is the sort of law of large numbers, right? The more times you keep sampling the same thing, the more your average keeps getting smaller in variance. That's really what's happening. It's generating content, and I've often viewed LLMs, when I've been interacting with them, as almost giving you the consensus answer. If I were to somehow ask everybody and average the answers together, that's really the answer I get from the LLM. It's kind of giving you the average answer among all of the coherent munging together of human writing, right?

Yeah, but at the same time, I have to say that humans are ranking these models. If you look at LMArena-style things, there's feedback from humans: “Hey, I like this model more than this one.” There are clearly a lot of human preferences encoded in it.

I don't know if your interaction is like this, but recently my model started just placing emojis all over the place in response. This is triggering me, but I'm sure that, for some folks, they love this.

Tim Scarfe

Well, that's the average, you know. The bulk person wants emojis out there.

And this is why I'm saying that I think my experience with models, and the improvements in the quality of my personal life, may not translate to others. In my life, models have made things significantly better. But yeah, I don't know how representative this is.

Tim Scarfe

You know, your supervisor was Ross Anderson, and he passed away, didn't he?

He did recently.

Tim Scarfe

Rest in peace.

I only recently realized that we organized an event for the gentleman to reflect on a lot of his achievements. He had been publishing for a long time, and he's credited with creating a number of different fields. His early work was in cryptography. Obviously, one of his ciphers became a runner-up for the standard in cryptography, and he's also credited with the security economics literature. He's credited with a lot of work in cybercrime, a lot of work in TEMPEST, and a lot of work in banking security.

We were reflecting on this, and only then did I realize that I met Ross at a very late stage of his life, in a sense. I saw him on that stage, and then I met students from previous generations. Their experience was widely different, and they were looking at very different problems. He's done a lot.

Tim Scarfe

And what would you say to people now going into ML security?

I think coming through a security background is better than going through an ML background. If you start off as a security person who starts specializing in ML later, it's probably better, just because you learn the fundamentals of how we solved security problems before. That intuition allows you to think a lot more about ML models.

In essence, at least modern ML models are like interpreters, and the language you give them—this human thing, human language—is kind of like a very high-level language. It's not a programming language that's normally executed by other interpreters. But if you start thinking about the models as interpreters and the language as the programming language, then suddenly you think about the whole thing very differently.

Then you ask yourself the question: Why do we expect the interpreter to provide security? If the programs we write are probabilistic, why do we expect deterministic outputs? Things like this, right?

When you start thinking about them through this more formal computer science sort of view, I think it's more productive than trying to say, “Oh, actually, I'm going to distill all of the security thinking into the model, and it's going to solve all the problems.” I can give you a number of fundamental things—classical, century-old dilemmas—that are definitely not going to be solved with models.

Can they make some progress here and there? Yeah, sure. But is that actually going to solve, I don't know, the confused deputy problem? I promise you, the answer is no.

Tim Scarfe

Awesome. Well, I thank you so much for joining us today. It's been amazing.