Language Models are "Modelling The World" [Nicholas Carlini]

The fact that it can make valid moves almost always means that it must, in some sense, have something internally that is accurately modeling the world. I don’t like to ascribe intentionality or any of those kinds of things to it, but it’s doing something that allows it to make these moves, knowing what the current board state is and understanding what it’s supposed to be doing.

Everyone means something different by reasoning, and so the answer to the question is that reasoning is entirely what you define as reasoning. You find some people who are very much in the world of, “I don’t think models are smart. I don’t think that they’re good. They can’t solve my problems.” They say, “No, it’s not reasoning,” because to them reasoning means—and then they give a definition that excludes language models.

You ask someone who is very much on the AGI side—“Language models are going to solve everything by 2027. They’re going to displace all human jobs”—what reasoning is, and they say reasoning is whatever process the model is doing.

Hi, I’m Nicholas Carlini. I’m a research scientist at Google DeepMind, and I like to try to make models do bad things and understand the security implications of the attacks that we can get on these models. I really enjoy breaking things, and I’ve been doing this for a long time. I’m just very worried that, because these systems are impressive, we’re going to have them applied in all kinds of areas where they ought not to be. As a result, the attacks that we have on these things are going to end up with bad security consequences.

MLST is sponsored by sensl, which is the compute platform specifically optimized for AI workloads. They support all of the latest open-source language models out of the box, like llama, for example. You can just choose the pricing points, choose the model that you want; it spins up, it’s elastic auto-scale. You can pay on consumption, essentially, where you can have a model which is always working or it can be freeze-dried when you’re not using it. So what are you waiting for? Go to sent ml. and sign up now.

Tab is a new AI research lab I’m starting in Z. It is funded from past ventures involving AI as well. We are hiring both chief scientists and deep-learning engineer researchers, and so we are Swiss version of deeps: a small group of people, very motivated, very hardworking, and we try to do some research starting with LLM and Owen Stein models. We want to investigate, reverse-engineer, and explore the techniques ourselves.

Nicholas Carlini, welcome to MLST.

Thank you.

Folks at home, Nicholas won’t need any introduction whatsoever. He’s definitely by far the most famous security researcher in machine learning, and he works at Google. It’s so amazing to have you here for the second time.

Yeah, the first time was a nice pandemic one, but it was great.

Yes, MLST is one of the few projects that survived the pandemic, which is pretty cool. Why don’t we kick off, then? Do you think we’ll ever converge to a state in the future where our systems are insecure and we’re just going to learn to live with it?

That’s what we do right now in normal security, right? There is no perfect security for anything. If someone really wanted something bad to happen on your computer, they would win. There’s very little you could do to stop that.

We just rely on the fact that probably the government does not want you, in particular, to have something bad happen. If they decided that they wanted to, I’m sure they have something they could do that would succeed.

What we can get into is a world where the average person probably can’t succeed in most cases. This is not where we are with machine learning yet. With machine learning, the average person can succeed almost always.

I don’t think our objective should be perfection, in some sense, but we need to get to somewhere where it’s at least the case that a random person off the street can’t just very easily run some off-the-shelf GitHub code that makes some model do arbitrary bad things in arbitrary settings.

I think getting there is going to be very, very hard. We’ve tried, especially in vision, for the last 10 years or so to get models that are robust. We’ve made progress and learned a lot, but if you look at the objective metrics, they haven’t gone up by very much at all in the last 4 or 5 years. This makes it seem somewhat unlikely that we’re going to get perfect robustness in the foreseeable future.

At least we can still hope that we can do research and make things better, and eventually we’ll get there. I think we will, but it’s going to take a lot of work.

Ilia asked me to ask you this question: Do you ever think in the future that it will become illegal to hack machine-learning systems?

I have no idea. It’s very hard to predict these kinds of things. It’s very hard to know what the Computer Fraud and Abuse Act already covers, especially in the United States, in whatever settings.

I think this is a question for the policy and legal people. My view on policy and law is that, as long as people are making these decisions from a place of what is true in the world, they can make their decisions. The only thing that I try to comment on here is that we should make sure that we’re making decisions based on what is true, and not decisions based on what we think the world should look like.

If they base their decisions around the fact that we can attack these models and various bad things could happen, they’re more expert at this than me. They can decide what they should do. But I don’t know.

In the context of machine-learning security, these are really open-ended questions. Just to start with, can you predict the future? What’s going to happen with machine-learning security?

Let me give you a guess. I think the probability of this happening is very small, but it’s sort of the median prediction, in some sense.

I think models will remain vulnerable to fairly simple attacks for a very long time, and we’ll have to find ways of building systems so that we can rely on an unreliable model and still have a system that remains secure.

What this probably means is that we need to figure out a way to design the rest of the world—the thing that operates around the model—so that if it decides it’s going to randomly classify something completely incorrectly, even if just by random chance, the system is not going to perform a terribly misguided action.

We’re going to have to live with a world where the models remain very vulnerable for the foreseeable future, at least as far as I can see. Especially in machine learning, 5 years is an eternity. I have no idea what’s going to happen or what the world will look like.

Language models have only had about 7 years of significant progress, so predicting 5 years out is almost doubling that. I don’t know what the world will look like, but at least as long as we’re in the current paradigm, it looks like we’re in a world where things are fairly vulnerable.

Then again, language models are only 7 years old, and we’ve only really been trying to attack them for 2 or 3 years. Give it 5 years—that’s twice as long as we’ve been trying to attack these language models. Maybe we’ll figure everything out. Maybe language models are fundamentally different and things aren’t this way.

My prior just tends to be based on other vision models. We’ve been trying to study them for 10 years, and at least there, things have proven very hard. My expectation is that things will be hard, and that I’ll have to rely on building systems that end up working.

I’ve been reading your blog, and everyone should read it because it’s really amazing. When you first put out this article about chess-playing, I cited it on the show about 10 times. It’s really interesting.

It’s called “Playing Chess with Large Language Models.” You said:

“Until this week, in order to be good at chess, a machine-learning model had to be explicitly designed to play games. It had to be told explicitly that there was an 8-by-8 board, that there were different pieces and how each of them moved, and what the goal of the game was. It had to be trained with reinforcement learning against itself, and then it would win.”

You said that this all changed on Monday, when OpenAI released GPT-3.5 Turbo Instruct. Can you tell me about that?

GPT-3.5 Turbo Instruct, and later other people with open-source models that you can verify aren’t doing something strange behind the scenes, have replicated this. Some people speculated that the models were cheating in various ways, but there are open-source models that replicate it now.

What you have is a language model that can play chess to a fairly high degree. When you first tell someone, “I have a machine-learning system that can play chess,” the immediate reaction is, “Why should I care? We had Deep Blue 30 years ago that could beat the best humans. Isn’t that some form of AI at the time? Why should I be at all surprised by the fact that I have some system like this that can play chess?”

The fundamental difference here is interesting. The model was trained on a sequence of moves. In chess, you represent moves using notation: 1. e4 means moving the king’s pawn to e4, then Black responds with e5, and then White plays Nf3. You train on these sequences of moves, and then you just say, “Language model, do your prediction task.”

It’s just a language model. It’s being trained to predict the next token, and it can play a move that is not only valid but also very high quality.

The valid part is interesting in and of itself. What is a valid chess move? That’s a complicated program. It’s not an easy thing to describe what moves are valid in a given situation. You can’t just output random characters and stumble upon valid moves.

You have this model that makes valid moves almost every time. I don’t like talking a lot about what the model is doing internally, because I don’t think that’s all that helpful. I think looking at the input-output behavior of the system is the way to understand these things.

But the fact that it can make valid moves almost always means that it must, in some sense, have something internally that is accurately modeling the world. I don’t like to ascribe intentionality or any of those kinds of things to it, but it’s doing something that allows it to make these moves, knowing what the current board state is and understanding what it’s supposed to be doing.

That, by itself, is interesting. Not only can it do this, but it can actually play high-quality moves. Taken together, it tells me that the model has a relatively good understanding of what the actual position looks like.

I play chess at a modest level. I’m not terrible, and I understand more or less what I should be doing. But if you gave me a sequence of 40 moves in a row and then said, “At move 41, what’s the next move?” I could not reconstruct in my mind what the board looked like at that point.

Somehow, the model has figured out a way to do this, having never been told anything about the rules or even that they exist. It has reconstructed all of that and can put the pieces on the board correctly, in whatever way it does that internally. Who knows how that happens? Then it can produce a valid move. That’s very interesting to me.

It changed the way I think about what models can and can’t do, in terms of surface-level statistics or deeper statistics about what’s actually going on. That’s mainly why I think this is an interesting fact about the world.

We have this weird form of human chauvinism around the abstractness of our understanding. These artifacts have a surface level of understanding, but it’s at such a great scale that, at some point, it becomes a weird distinction without a difference.

You said something very interesting in the article: the model was not playing to win. You were talking about how the models are a reflection of you. You play like a good chess player, and it responds like a good chess player. It’s the same whether you’re coding or doing something else.

It might even explain some of the differential experiences people have. You go on LinkedIn and those people clearly aren’t getting very good responses out of language models. Then there are people like yourself, using language models at the galaxy-brain level, pushing the frontier, while people don’t even know you’re using one.

Let me explain what I meant when I said that the model wasn’t playing to win.

Given a particular chess board, you can find multiple ways of reaching that position. You could take a board that resulted from a normal game between 2 chess grandmasters and find a sequence of absurd moves that no one would ever play, but that brings you to the same board state.

You can construct such a sequence piece by piece. If the knight is on g3, you first move the white knight to some random spot and eventually put it on g3. If the bishop is on h2, you find a way to move the pawn out of the way and then put the bishop on h2. You can come up with a sequence of absurd moves that ends up at the correct board state.

Then you can ask the model to play a move. It plays a valid move most of the time. It still knows what the board state looks like, but the move it plays is very bizarre. It’s a very weird move.

Why? What has the model been trained to do? It was never told that the goal of chess is to win. It was told to produce things that look like what it saw before. It saw a sequence of moves that looked like 2 people rated around 1500 playing a game of chess, and it concluded that the game is to make valid moves and see what happens.

It’s very good at doing this. You can do it synthetically, and you can also find explicit cases where you get models to make terrible move decisions because that’s what people commonly do when they’re playing. Most people fall for these traps, and the model was trained to play like whatever the training data looked like, so it falls for them too.

That’s one of the problems with these models: they’re not initially trained to play to win. As far as how this applies to the language models we use, we almost always post-train them with reinforcement learning from human feedback, supervised fine-tuning, instruction fine-tuning, and related techniques.

A big part of why we do that is so that we don’t have to deal with the mismatch between what the model was initially trained on and what we actually want to use it for. This is why GPT-3 was exceptionally hard to use, and why the sequence of InstructGPT papers was important.

Post-training takes the capabilities that the model has somewhere behind the scenes and makes them much easier to reproduce. When you’re using chat models today, most of the time you don’t have to worry nearly as much about exactly how you frame the question. They were designed to give you the right answer, even when you ask a silly question.

I think they still have some of this behavior, but less than if you just have the raw base model trained on whatever data it happened to be trained on.

I’d love to do a tiny digression on reinforcement learning from human feedback. I was speaking with Max from Cohere yesterday. They’ve done amazing research on how preference steering works, and they say that humans are actually really bad at distinguishing one good thing from another.

We like confidence, verbosity, and complexity. I really hate the ChatGPT model because of its style. I can’t stand the style. Even though it’s right, I think it’s wrong.

When we do that kind of post-training on language models, how does that affect competence?

I feel like it’s very hard to answer some of these questions, because often you don’t have access to the models before they’ve been post-trained.

You can look at numbers from the papers. The GPT-4 technical report has some numbers showing that the model before post-training—the raw base model—is very well calibrated. This means that when it gives an answer with some probability, it’s right about that fraction of the time.

If you ask a math question and it says the answer is 5 with a token probability of 30%, it’s right about 30% of the time. But when you do the post-training process, the calibration gets completely messed up and it no longer has this behavior.

Some things change. The models can become fantastically better when you do post-training because they follow instructions much better. You haven’t necessarily taught them much new, but they look much smarter.

This is all very confusing. I don’t have a good understanding of how all these things fit together.

Given that these models make valid moves and appear to be competent, but sometimes have these catastrophic, weird failure modes, do we call that process reasoning or not?

Everyone means something different by reasoning, and so the answer is that reasoning is entirely what you define as reasoning.

You find some people who are very much in the world of, “I don’t think models are smart. I don’t think they’re good. They can’t solve my problems.” They say, “No, it’s not reasoning,” because to them reasoning means something that excludes language models.

Then you ask someone who is very much on the AGI side—“Language models are going to solve everything by 2027. They’re going to displace all human jobs”—what reasoning is, and they say reasoning is whatever process the model is doing. They tell you that, yes, the model is reasoning.

It’s very hard to talk about whether it’s actually reasoning or not. The thing we can talk about is the input-output behavior. Does the model do the thing that answers the question or solves the task? Was it challenging in some way? Did it get it right?

That’s an easier way to try to answer these questions than to ascribe intentionality to something. It’s hard to have these debates when you start without having the same definitions.

I’m torn on this. The deflationary methodology says it’s an input-output mapping. You could go one step up: Yoshua Bengio said reasoning is basically knowledge plus inference, in some probabilistic sense. I think it’s about knowledge acquisition or the recombination of knowledge.

It’s the same thing with agency. The simplistic form is that it’s just an automaton: you have an environment, some computation, and an action space. But it feels necessary to me to have things like autonomy, emergence, and intentionality in the definition.

You could just argue, “Why are you saying all of these words? If it does the thing, then it does the thing.”

I feel that way too. I think it’s interesting to consider whether it’s reasoning if you have a background in philosophy and that’s what you’re studying. I don’t have that background, so I don’t feel qualified to tell you whether or not the model is reasoning.

The thing I can do is say, “Here is how you’re using the model. You want it to perform this behavior. Did it perform the behavior, yes or no?” If it’s doing the right thing in all of the cases, I don’t know that I care too much about whether the model reasoned its way there or used a lookup table.

If it gives me the right answer every time, I tend not to focus too much on how it got there.

We have this entrenched sense that we have parsimony and robustness. For example, in this chess notation, if you change the syntax of the notation, it would probably break.

Yes. There are multiple chess notations. Before the current notation we use, old chess books used a notation like “king’s bishop moves to queen’s 3.” The squares were represented differently.

If you ask a model to use that notation, it has no idea what’s happening. It will write something that looks superficially like a sequence of moves, but has nothing to do with the correct board state.

Of course, a human wouldn’t do this. If you asked me to produce a sequence of moves in that notation, it would take me a long time to remember which square was which and how to write everything down. I would have to think harder, but I understand what the board is and can get that correct. The model doesn’t do that right now.

Maybe that’s your definition of reasoning, and you would say that reasoning doesn’t happen. Someone else could ask why you expect the model to generalize to something it has never seen before.

It’s interesting to me that we’ve gone from a world where we wrote papers about the fact that if you trained a model on ImageNet, its accuracy would go down when you corrupted the images, to a world where this would seem obvious. If I trained a model on ImageNet and tested it on CIFAR-10, people would laugh and say, “Of course it didn’t work. You trained it on one distribution and tested it on another. You never asked it to generalize.”

But with language models, we train them on one distribution, test them on a different distribution they weren’t trained on, and then laugh at the model for being dumb. You didn’t train it on the thing you want to test it on. Maybe some future model will magically generalize across domains, but we’re still using machine learning. You need to train it on the kind of data you want to test it on, and then it will behave much better than if you don’t.

In an email correspondence with me, you said—you didn’t use these exact words—that there have been many instances where you felt a bit annoyed because you made a statement, your intuition was skeptical, you said “stochastic parrots,” and then you were proven wrong a bunch of times. It’s the same for me.

One school of thought is Rich Sutton’s: throw more data and compute at the thing. The other school of thought is that we need completely different methods. Are you still amenable to the idea that scaling these things up will produce the kinds of reasoning we’re talking about?

Possibly. There are some people who have good visions of what the future might look like, and then there are people like me, who look at what the world looks like and try to do interesting work here.

This works for me because, for security in particular, it only matters if people are doing the thing to attack the thing. I’m fine saying, “Let’s look at what is true about the world and write security papers.” If the world significantly changes, we can try to change and do things differently.

We can try to be a couple of years ahead by looking at where things are going, so that we can do security research ahead of when we need it. But I tend not to spend a lot of time trying to think about where things will be in the far future.

A lot of people try to do this. Some are good at it and some are not, and I have no evidence that I’m good at it. I mostly try to reason based on what I can observe right now. If what I can observe changes, then I ought to change what I’m thinking about and do things differently.

That’s the best I can hope for.

On this chess thing, has anyone studied whether the Elo rating in the headers corresponds to a change in performance? What would happen if you said the Elo was 4,000?

We’ve actually trained some models trying to do this, and it doesn’t work very well. You can’t trivially change the number and get the desired behavior.

We trained some models ourselves on headers that we expected would give them an even better chance of doing this, but that didn’t immediately work. When we trained it and tested GPT-3.5 Turbo Instruct, there might have been a statistically significant difference in the outcome, but it’s nowhere near the case that you tell the model it’s playing like a 1,000-rated player and it suddenly becomes 1,000-rated.

People have worked very hard to train models that let you match the skill to an arbitrary level. That’s a research-paper-level thing, not just changing 3 numbers in the header and hoping for the best.

You wrote another article called “Why I Attack,” and you said that you enjoy attacking systems for the fun of solving puzzles rather than for altruistic reasons. Can you tell me more about that? Also, why did you write the article?

Let me answer those in the opposite order.

Some people were angry at me for breaking defenses. They said that I didn’t care about humanity and just wanted to make people look bad or something. Half of that statement is true.

I don’t do security because I’m driven by wanting to do maximum good. I’m not thinking, “What are all the careers I could do, and which one is most likely to save the most lives?” If I had done that, I might be a doctor or researching cancer—something where you could directly measure the good you were doing.

I don’t find those things fun. I can’t motivate myself to do them. If I were a different person, maybe I could do that. Maybe I could be someone who meaningfully solved challenging problems in biology because I woke up every morning knowing that I was saving lives. But that’s not how I work.

I feel that it’s not how many people work. There are lots of people in computer science, or even quantitative fields, who are clearly brilliant and could be doing something better with their lives. Some of them would probably have zero productivity if they were doing something they didn’t enjoy.

I try to find the set of things that I can motivate myself to do and will do a really good job at, then solve those as well as possible, subject to the constraint that I’m actually net positive and moving things forward.

For whatever reason, I’ve always enjoyed attacking things, and I’m much better at that than at most other things. I feel that I’m pretty good at machine learning, but I have no evidence that I would be good at the other 90% of things that exist in the world that might do more good.

The way I think about this in one sentence is: how good you are at the thing multiplied by how much the thing matters. You’re trying to maximize that product.

If there’s something you’re really good at that at least directionally moves things in the right direction, you can have a higher impact than by taking whatever field happens to be maximally good and moving things forward by a very small amount.

That’s why I do attacks. I feel that they generally move things forward, and I’m better at them than at most other things I could be doing.

You also said that attacking is often easier than defending. Certainly, tell me more.

That’s the standard thing in security. You need to find 1 attack that works, whereas if you’re defending, you need to fix all of the attacks.

If you’re attacking something, the only thing you have to do is find 1 place where someone has forgotten to handle a corner case. Then you can arrange for the adversary to hit that corner case as many times as needed until they succeed.

In normal software security, you can have a perfect program everywhere except for 1 line of code where you forgot to check the bounds exactly once. The attacker makes that happen every time, and the security of your product is essentially zero.

Under random conditions, this is never going to happen. The hash of a file is not randomly going to be exactly a power of 2—for example, equal to 2³²—which overflows an integer and causes something bad to happen. That won’t happen by random chance. But the attacker can arrange for it to happen every time, which makes the attacker’s job much easier than the defender’s. The defender has to fix everything.

In machine learning, it gets even worse. In normal software security, we understand the classes of attacks. In machine learning, we constantly discover new categories of bad things that could happen.

Not only do you have to be robust to the things we know about; you also have to be robust to someone coming up with a new, clever type of attack that we hadn’t even thought of before. That’s not happening yet, partly because this is a very new field, so attacks are much easier than defenses.

Let’s talk about disclosure norms. How should they change now that we’re in the machine-learning world?

In standard software security, we’ve basically figured out how things should go. For a very long time—20 years or so—there was a big debate about what someone should do when they found a bug in software that could be exploited.

In the late 1990s and early 2000s, there were people who supported full disclosure. They thought, “If I find a bug in a program, I should tell everyone so that we can make sure people don’t make a similar mistake. We can put pressure on the responsible party to fix it.”

Then there were people who thought you shouldn’t disclose anything. You should report the bug to the responsible person, wait until they fix it, and then tell no one about it, because you don’t want to give anyone else ideas for exploiting it.

In software security, we landed on what was called responsible disclosure and is now called coordinated disclosure. The idea is that you should give the responsible party a reasonable amount of time to fix the issue.

Google Project Zero has a 90-day policy, for example. You have that many days to fix the problem, and then, after that—or once it’s fixed—it gets published for everyone.

The idea is that you give the person time to protect their users. You don’t want to immediately disclose a new attack that allows people to cause a lot of harm, but you put a deadline on it and stick to the deadline to pressure the company to fix the thing.

If you don’t say that you’re going to release it publicly, no one else knows about it. You’re the only person who knows about the exploit, and the company may simply not fix it because they’re in the business of making a product, not fixing bugs. Why would they fix it if no one else knows about it?

When you say, “This will go live in 90 days,” they had better fix it before then. They have the time, and if they don’t do it, that’s on them because they didn’t put in the work.

There are exceptions. Spectre and Meltdown were 2 of the biggest attacks of the last 10 or 20 years in software security. The researchers gave Intel and the related companies a year to fix them because they were important bugs, difficult to fix, and there were legitimate reasons for taking that long. There was also good evidence that the bugs probably wouldn’t be independently discovered by malicious actors for a long time.

Google Project Zero similarly says that if it finds evidence that a bug is being actively exploited, it will give the company 7 days. If someone is actively exploiting it, the harm is already being done, so there’s little reason to delay telling everyone about it.

Given that long preamble, how should things change for machine learning?

The short answer is that I don’t know.

On the one hand, I want to say that this is how things work in software security. Sometimes there’s a bug in software and there’s a way to patch it and fix the problem.

We’ve written papers recently showing model-stealing attacks. OpenAI has a model, and we could query its services in a way that allowed us to steal part of the model. It was only a small part, but we could steal part of it. We disclosed this to them because there was a way they could change the API to prevent the attack from working. Then we wrote the paper and put it online.

That feels very much like software security.

On the other hand, there are problems that aren’t things you can patch. Consider adversarial examples. If I disclose an adversarial example for your image classifier, what is the point of having a responsible-disclosure period? There’s nothing you can do to fix the underlying problem in the short term. We’ve been trying to solve it for 10 years, and another 90 days isn’t going to help.

Maybe I’ll tell you as a courtesy: “This is the attack I’m going to use, and this is how I’m going to describe it in the paper. Do you want to put a couple of filters in place to make this particular attack not work?” But you’re not going to solve the underlying problem.

When I talk to people who work on biological systems, they make the argument that if someone came up with a way to create a novel pathogen, a disclosure period wouldn’t help. So is machine-learning security more like that, or more like software security?

I don’t know. I’m biased toward software security because that’s where I came from, but it’s hard to say exactly which one we should model things after.

We probably do need to come up with new norms for how we handle this. A lot of people are talking about this and trying to write these norms down. In a year or 2, if you ask me again, we’ll probably have processes and established norms for handling these things.

Right now, this is still very early. We’re looking for analogies in other areas and trying to come up with what sounds most likely to be good. I don’t have a good answer.

Are there any vulnerabilities that you’ve decided not to pursue for ethical reasons?

No, none that I can think of. Mostly, that’s because I tend to think only about exploits that would be ethical to pursue in the first place.

Research ideas happen by random inspiration a very small fraction of the time. Most of the time, a research idea is not something that just happens. You spend conscious effort trying to figure out what new thing you’re going to try.

It’s pretty easy not to think about things that seem morally fraught and to focus on the ones that seem as though they could actually be good and useful. It may happen at some point that I stumble upon something like this, but I can’t think of any examples of attacks we’ve found and decided not to publish because of the harm they would cause.

I can imagine that it might happen. I can’t rule it out. But I tend to bias my search toward problems that I think are beneficial.

Going back to why I attack things, you want the product of how good you are and how much good it does for humanity to be maximally positive. You can choose problems that aren’t negative.

I don’t have much respect for people whose contribution to the direction of the world is just a negative number, because you can make that zero by doing nothing. I try to pick problems that are generally positive and then do as well as possible on them.

You work on traditional security and machine-learning security. What are the significant differences?

I don’t work on traditional security very much anymore. I started my PhD in computer security. I did very low-level return-oriented programming, and I spent a summer at Intel working on hardware-level defenses. Then I started working on machine learning shortly afterward.

I haven’t worked on very traditional security in the last 7 or 8 years, but I still follow it very closely and go to systems-security conferences because it’s a great community.

One of the similarities and differences is that systems-security people are very good at making sure that what they’re doing is rigorous and evaluated thoroughly. You see this even in the length of the papers. A systems-security paper is 13 or 14 pages in 2 columns. A paper submitted to ICLR is 7 or 8 pages in 1 column.

Systems-security papers start with a long explanation of exactly what’s happening, and the results are expected to be very rigorous. A machine-learning paper is often, “Here’s a new cool idea. Maybe it works.” That’s good for moving fast and breaking things, but it’s not good for systematic studies.

When I was doing systems-security papers, I would get 1, 1.5, or 2 papers a year. Now I could probably produce 5 or 6 machine-learning papers at a similar level of rigor.

This is a conscious decision by the communities, and I think it has worked empirically in machine learning. It wouldn’t be good if every machine-learning result needed the rigor expected of a systems-security paper, because we would have had about 5 iteration cycles in total.

At machine-learning conferences, you often see a paper, a paper that improved upon it, and another paper that improved upon that paper, all at the same conference. The first person put it on arXiv, the next person found a tweak that made it better, and the third person found another tweak that made it even better.

That’s good. When a field is very new, you want to allow people to propose ideas without having full evidence that everything works. When the field becomes more mature, you want to make sure that people aren’t proposing wild things that have already been proposed 30 times.

Having some balance between the 2 is useful. If I have a differential advantage in machine learning, some of it comes from this. In systems security, you’re trained very heavily in rigorous thinking, doing attacks thoroughly, and looking at all the details.

That training has been beneficial to me when writing machine-learning papers and thinking about the small details. I recently had a paper where I broke a defense because there was a negative sign in the wrong place.

That’s not the kind of thing I could have reasoned about from first principles by inspecting the code. If I had been advising someone, I wouldn’t have known to tell them, “Check all the negative signs.” You have to understand everything that’s happening and find the one place where the mistake was made, so that you can break the system by doing exactly the right thing.

That’s perhaps the biggest difference I see between these communities.

Your next article was called “Why I Use AI.” A couple of months ago, you wrote that you’ve been using language models, find them very useful, and that they improve your programming productivity by about 50%. I can say the same thing myself. Can you break down the tasks where they’ve really increased your productivity?

I’m not someone who believes these things easily. There are people whose job is to hype things up, get attention, and promote these technologies.

The annoying thing is that the same people who said Bitcoin was going to change the world immediately started saying that language models were going to change the world. If you’re looking at this from afar, it seems that the grifters are just finding the new thing. These people have no understanding of what’s going on in the world.

They may be right that these people are finding whatever the new thing is that gets them clicks, but at the same time, I think the models we have now are actually useful. They’re not useful for nearly as many things as people say, but for a particular kind of person—the person who understands what’s happening in these models, knows how to code, and can review the output—they’re useful.

What I wanted to say was, “I’m not going to argue that they’re good for everyone. Here’s an N-of-1 anecdote showing that they’re useful for me. If you have a background similar to mine, maybe they’re useful for you too.”

I’ve had a number of security-oriented people contact me and say that they found them useful. The question is whether my experience generalizes to anyone else. I don’t know. It’s not my job to understand that. I just wanted to say that they’re useful for people who behave as I do.

Why are they useful? The current models are good enough for the kinds of things where I want an answer. Maybe I want them to write a function or implement something. I know how to check the answer, and I know I could produce it myself; I just don’t want to.

The most useful analogy is to imagine that you had to write all your programs in C or assembly. Would that mean you couldn’t do anything you can do now? Probably not. You could do the same research in C instead of Python if you really had to. It would take a lot longer.

You have an idea in your mind—you want to implement a binary search, for example—and in C you have to reason about pointers, memory allocation, and all these little details that are at a much lower level than the problem you want to solve.

The thing that’s useful about language models is that, if you know the problem you want to solve and can check that the answer is correct, you can ask the model to implement it. You can describe what you want in words, even if those words aren’t terribly well-defined, and it will give you an answer.

You can check that it’s correct, put it in your code, and continue solving the problem you actually wanted to solve rather than the problem of typing out all the details.

The other class of things I find useful is when I can rely on the model having enormous knowledge about the world and all kinds of things. If I understand the fundamentals but don’t know the API for something, I can ask the model to make it work with that API and check the result easily.

If I don’t know how to write something in a particular language, I can ask for the code. If you give me code in any language, even one I’ve never seen before, I can generally reason about what it’s doing. I may make mistakes at the edges, but I could never have typed it because I didn’t know the syntax.

The models are very good at giving you the correct syntax and getting everything else out of the way. Then I can figure out the rest.

If I couldn’t ask a model, I’d have to learn the syntax of the language or do what people did 5 years ago: copy and paste someone else’s code from Stack Overflow and make annotations. That’s a strictly worse version of asking the model, because then I’m relying on myself, without knowing much, to copy and paste correctly.

For these kinds of problems, they’re currently plenty useful if you already understand the problem. By “understand,” I mean having an abstract understanding. In that case, they’re a superpower.

That explains why the smarter you are, the more you can get out of a language model. How has your usage evolved over time, and what’s your methodology?

Speaking personally, I know that specificity is important. Going to source material, constructing the prompt, and embedding my understanding and reasoning process into the prompt all matter. How do you think about that?

I try to ask questions that have a reasonable probability of working. I don’t ask questions if I think the process will slow me down, but if I think there’s a 50% chance of success, I’ll ask the model first.

Then I look at the output and ask whether it seems directionally correct. If it seems as though it’s approaching the right kind of solution, I might iterate a little more. If it gives me a perfect solution the first time, great—I accept it.

I also learn from the interaction. If the models aren’t good at a particular kind of problem, I may not ask that question again in the future.

Some people say they can’t get models to do anything useful. It may be that the models are bad at their particular kind of problem. It may also be that they don’t yet understand what the models can do.

Most people have forgotten how much they had to learn about using Google Search. If I tell you to look something up, you implicitly know that you should use the words that appear in the answer. You don’t ask it as a question. There’s a way to type things into search engines to get the right answer, and that requires some skill and understanding.

It’s the same with language models. They have a natural-language interface, so technically you can type anything you want, but some ways of doing it are much more useful than others.

I don’t know how to teach this as a skill other than saying, “Try it.” Maybe it turns out the models aren’t good at your task, in which case you shouldn’t use them. But if you can make them useful, it seems like a free productivity gain.

The caveat is that you need to understand what’s happening with these systems. There are people who don’t, and I worry about whether they’ll learn anything or catch the bugs when they appear.

For a practitioner who wants to get work done, I feel that using language models is analogous to using Python rather than C. I wouldn’t say that you need to use C instead of Python, and I wouldn’t say that you need to use Python alone instead of Python plus language models.

I agree that laziness and acquiescence are problems. Vibes and intuition are really important. I consider myself a Jedi at using language models, and sometimes it frustrates me because I tell people to use one, but I seem to get so much more out of them than other people.

I’m not entirely sure why that is. Maybe it’s because I understand the thing I’m prompting. It seems to be something we need to learn.

Every time a new tool appears, you have to spend time learning how to use it.

I remember when people said, “Real programmers write code in C. They don’t use a high-level language. Why would you trust a garbage collector? Real programmers manage their own memory. Real programmers write their own Python. Why would you trust a language model to output correct code? Why would you trust it to recall an API?”

You can draw the same analogies here. When tools change and make you more productive in certain settings, you should be willing to examine the new tools.

I’m always trying to rationalize this. It comes down to the notion of whether the intelligence is in the eye of the prompter.

Does it matter? Maybe the difference between how I use these systems and how other people use them is that the system makes me more productive and solves the task for me.

Did I put the intelligence there? In many cases, I think the answer is no. In some cases, I think the answer is yes. But I’m not going to look at it that way. I’m going to ask whether it solves the questions I have in a way that’s useful for me. Here, the answer is definitely yes.

Obviously, as a security researcher, how does that influence the way you use language models?

This is why I’m scared about people who use them without understanding things. You can ask a model to write an encryption function, and the correct answer should be, “You shouldn’t do that. You should call this API.”

Often, the model will say, “Sure, here’s an encryption function,” and it will have all the bugs that people normally write. That’s terrible.

I was writing some code that made calls to a database. What did the model do? It wrote code vulnerable to SQL injection. If someone wasn’t careful, they wouldn’t have caught it, and they would have introduced all kinds of bad bugs.

Because I’m reasonably competent at programming, I can read the model’s output and correct its mistakes. It’s not hard to fix the SQL injection and replace string concatenation with templates. The model just didn’t do it correctly.

I’m very worried about people who won’t do this. There have been papers showing that people write very insecure code when using language models if they aren’t careful. It looks as though code may be differentially more vulnerable when people use language models than when they don’t.

This is one reason I think about the utility question. The security of the things people use actually matters, and I want to know what people are going to do so that we can study it.

It’s important to separate whether the model can solve the problem from whether it’s secure. For language models, the answer is often yes: they give you the right answer in the common case.

Most people don’t care about the security question, so they’ll use the system because it gave them the ability to do something new, without understanding the security implications. That means we should study the security of the things we know people are going to use and make sure they can use them correctly.

I try to use systems at the frontier of what people are going to do next, to put myself in their frame of mind and understand the risks. This worries me quite a lot, because things could go very badly.

How and when do you verify the outputs of language models?

I verify them the same way I verify any code output.

People say, “The model might be wrong,” but half the answers on Stack Overflow are wrong too. If you’ve been programming for a long time, you’re used to reading code that’s wrong.

I’m not going to copy and paste a function from Stack Overflow and assume it’s correct. The person may have asked a different question from the one I’m asking. I don’t feel I’m doing anything fundamentally different when verifying code from a language model compared with verifying code someone else wrote online.

Maybe the only difference is that I’m using the models more often. If you use something twice as often, you’ll encounter twice as many bugs, so you have to be more careful. But I’m not doing anything different in quality.

Don’t trust the system to give you the right answer, and understand that a 95% solution is still a 95% solution. If it does almost everything you want, its capabilities may have reached their limit. You’re an intelligent person; finish the last 5%, fix the problem, and then you can have a 20-times performance increase.

You’ve touched on something very interesting. Most of us are wrong most of the time, which is why it’s good to have at least 1 very smart friend who constantly points out all the ways your work is wrong.

Most code is wrong. It’s your job to point out how it’s wrong. I guess we’re always unknowingly on the boundary of wrongness, and that’s just the way the world works.

There’s the potential for a massive increase in the quantity of wrongness. Previously, the amount of bad code that could be written was limited by the number of humans who could write bad code. There were only so many people who could write software, and they needed at least some training.

Now, if language models allow anyone in the world to write whatever software they need to solve a particular problem, that’s fantastic from one perspective. But as a security person, I’m scared because there may be 10 times as much software that is probably insecure.

We’re not going to have 10 times as many security experts to study all of it. There could be a massive increase in insecure software, and that’s one of the many things I’m worried about.

I try to use these systems to understand whether something seems like what people will actually do. Right now, the answer seems to be yes, and that worries me.

I spoke with some Google people yesterday, and they’ve been studying some of the failure modes of language models—really strange things that people don’t know about. They can’t copy, they can’t count, and there are issues caused by the softmax and token-level representation squashing in particular layers.

In your experience, have you noticed tasks that language models really struggle with?

I’m sure there are many. I’ve learned to just not ask those questions, so I have a hard time coming up with examples.

It’s like asking what search engines are bad at. I’m sure there are a million things for which a search engine is completely the wrong answer, but if you pressed me for an example right now, I might have trouble because I use search engines for the things they’re good at.

Whenever you want correctness in some sense, the model may not be the right tool. In terms of specific tasks they’re particularly bad at, you could say anything that requires an amount of work that would take you more than 20 minutes to write a program for. The model probably can’t solve that—but this is changing.

There are things I thought would be hard that are becoming easier. I had a problem that, for unrelated reasons, was a difficult dynamic-programming problem. It took me 2 or 3 hours to solve it the first time.

OpenAI o1 had just launched a couple of days earlier, so I gave the problem to o1. It gave me an implementation that was 10 times faster than mine in about 2 minutes. I could test it because I had a reference solution, and it was correct.

I learned that this was a problem I previously would never have asked a model to solve because it was challenging enough that I thought the model had no hope. Now I can ask it.

There are other things that seem trivial to me but that models get wrong. I mostly just don’t ask them those questions.

This goes back to what worries me: people may not have the experience to check when answers are right or wrong. They’ll apply a wrong answer as many times as they can, and that seems concerning.

This is part of the anthropomorphization process. We have vibes and intuitions, and we know how to skirt around the long tail of failure modes. We smooth it over in our supervised use of language models, and the amazing thing is that we don’t seem consciously aware of it.

Programmers do this all the time. A programming language has some model of how memory works. Rust has a very unusual memory model. If you go to someone who’s good at Rust, they’ll structure their program differently so that they don’t encounter all of the problems caused by that memory model.

I’m not very good at Rust. I try to write code in it, and the borrow checker yells at me endlessly. I look at Rust and think, “This could be very good, but I don’t know how to get my code right because I haven’t used it enough.”

If I weren’t being charitable, I might ask why anyone would use a language that makes it impossible to write my C code. But you have to change the way you write your code to work with the language, and then the problems go away.

You can do the useful things if you accept the paradigm you’re supposed to operate in. I see the same analogy here. The models aren’t very good in certain ways, and you’re trying to imagine that they’re human and ask them the things you would ask another person.

They aren’t human. You need to ask them in the right way and ask the right kinds of questions. Then you can get value from them. If you don’t, you’ll be disappointed because they’re not superhuman.

What are your thoughts on benchmarks?

This is one of the problems with language models. We used to live in a world where benchmarking was easy because we wanted models to solve exactly 1 task. You measured performance on that task, and if the model solved it, you were done.

The problem is that the task was never the task we actually cared about. This is why ImageNet models never made it into the real world to solve actual problems. We don’t care about classifying 200 breeds of dogs. The model may be good at that, but it isn’t the thing we actually want.

The claim people make about language models is that they’re general-purpose models that can solve arbitrary tasks. Then they test them on a small number of tasks and say, “See? It’s good because it solves these tasks very well.”

The challenge is that if I trained a model specifically to solve any one of those tasks, I could probably get a very good score. You don’t want the person who trained the model to have done that. You want them to train a good general model and then evaluate it on a task that’s independent of the training objective.

That gives you an unbiased view of how well the model performs. But people who train models are incentivized to make them perform well on benchmarks.

In the old world, I trusted researchers not to cheat. If I wanted maximum ImageNet test accuracy, I could train on the test set, but that would be cheating. I trust people not to do that.

With language models, you might not train directly on the test set of a benchmark such as HumanEval, MMLU, or MMMU, but you might train the model specifically to perform well on those benchmarks.

You could have a model that isn’t very capable in general but is fantastic on the 20 benchmarks everyone uses. That’s what everyone is incentivized to do, because you want your model to have maximum benchmark scores.

I’d like to live in a world with many more benchmarks, so that it isn’t easy to optimize for a small set of them. Then we could more easily trust that the models give the right answers and that their scores accurately reflect their skill level, rather than reflecting what the model trainer designed them to maximize.

The hyperscalers put incredible amounts of work into benchmarking. Now we’re moving to a world with test-time inference, test-time active fine-tuning, fine-tuning, quantization, fragmentation, and so on. People doing this in practice can’t really benchmark in the same way. How do you see that playing out?

I don’t know. If you’re doing quantization and related things, good luck. It seems very hard.

You can use the average benchmarks and hope for the best, but I don’t think that’s the main issue. I’m more worried about people actively fine-tuning models to show that they can make them better on particular tasks.

There are many fine-tunes of Llama, for example, that are claimed to be better. They show all the benchmark numbers, but it turns out they trained the models specifically to perform well on those tasks. If you ask them anything else, they’re very bad.

That’s what I’m most worried about. For the other cases, I agree that benchmarking is hard, but I don’t have any great solutions.

We can’t let you go before talking about one of your actual papers. This has been amazing, but I picked “Stealing Part of a Production Language Model,” from July. Could you give us an elevated picture of that?

For a long time, security researchers thought about how a model might be used in some hypothetical future and then asked whether certain kinds of attacks were possible in a theoretical setting.

There’s a line of work called model stealing, which asks whether someone can take a model and, by making standard queries to its API, steal a copy of it.

This began with Florian Tramèr and others in 2016, when they studied very simple linear models exposed through APIs. It then became a field of study involving deep neural networks. There were several papers in a row by a bunch of other people.

In 2020, we wrote a paper that we submitted to CRYPTO showing how to steal an exact copy of a model. Whatever model you had, we could recover an exact copy, as long as a long list of assumptions held: the model used only ReLU activations, everything was evaluated in 64-bit floating point, we could send and receive 64-bit floating-point values, the model was fully connected, its depth was no greater than 3, and each layer had no more than 32 units.

It was a theoretical result with assumptions that are never true in practice. There are other papers showing how to do this kind of exact model stealing, but they work only in contrived settings.

The question I wanted to answer was: Now that we have language models, and all of those assumptions are false, what’s the best attack we can implement against a real API?

It’s not just that the model uses something other than ReLU activations or isn’t fully connected. I can’t send 64-bit floating-point inputs or view 64-bit floating-point outputs. The models have billions of neurons, not 500.

We wanted to develop the best attack that worked against a real API. We looked at the OpenAI API and at APIs from other companies, including Google. Because of how the APIs were set up, they gave us some control over the outputs that allowed us to do mathematical operations and steal 1 layer of a model.

Among the layers in the model, it’s probably the least interesting one. It’s a small amount of data, but we could actually recover 1 layer of the model. It’s real in the sense that we could do it, and it’s real in the sense that we recovered the layer correctly, but it isn’t the whole model.

What I wanted to advocate for in the paper is pursuing both directions of research at the same time. One direction is writing papers that are true in a theoretical sense but aren’t the kinds of results you can implement in real systems, and probably won’t be for the foreseeable future.

At the same time, security researchers should look at systems as they’re deployed and ask what kinds of attacks can actually make the model do something bad. We should write papers about real pieces of these systems.

I don’t know what you can do with the last layer of a model, but there are some things. For example, the attack tells you the width of the model, which isn’t information companies generally disclose.

In our paper, we had what I believe was the first public confirmation of the width of the GPT-3 Ada and Babbage models. OpenAI had disclosed the widths of some models in the GPT-3 paper, but never directly stated the sizes of Ada and Babbage. People speculated about them, but we were able to confirm them.

We also ran the attack on GPT-3.5 and correctly stole the last layer. I know the size of the model, and it’s correct.

For responsible disclosure, we agreed with OpenAI in advance that we were going to do this. It’s a fun conversation to have with Google lawyers and OpenAI lawyers: “Hi, I would like to steal your model. May I please do this?”

The OpenAI people were very nice and said yes. Google’s lawyers were initially more cautious. They said that I couldn’t steal OpenAI’s data, but when I explained that OpenAI’s general counsel had agreed, they said it was fine. We put everything on an isolated virtual machine, ran the attack, and destroyed the data.

Part of the agreement was that they would confirm that we had recovered the correct information, but they asked us not to release the actual stolen data. That makes sense: you want to demonstrate that the attack works without releasing the stolen material.

If you wrote down a list of everyone in the world who knows how large GPT-3.5 is, it would include all current and former OpenAI employees and me.

This is a real attack. The other ways to learn this information would be to hack OpenAI’s servers or blackmail one of its employees. Instead, you can perform an actual adversarial machine-learning attack and recover the size of the model and its last layer.

That was the motivation for writing the paper: to provide examples and encourage other people to find attacks that, even if they don’t solve all the problems, make these attacks increasingly real. We’ll need to see more of this as systems are deployed in more and more settings.

Do you want to talk about the technical methods behind it?

For the next two minutes, let’s assume some level of linear-algebra knowledge. If that isn’t you, I apologize; I’ll try to explain it in a way that makes sense.

The models work as a sequence of layers, and each layer is a transformation of the previous layer. The layers have some size or width. The last layer of a model goes from a small dimension to a large dimension.

The internal dimension of these models might be 2,048, while the output dimension is the number of tokens in the vocabulary, which is around 50,000. This means that if you look at the vectors that are outputs of the model, they appear to be in a large, 50,000-dimensional space, but because they were produced by a linear transformation, the vectors actually lie in a 2,048-dimensional subspace.

If you look at this space, you can compute what’s called a singular value decomposition to recover how the smaller space was embedded into the larger one. The number of nonzero singular values tells you the size of the model.

This isn’t challenging mathematics. The last time I used it was as an undergraduate in mathematics. If you work through the details, it works out.

I like this because it’s a nice application of mathematics to these systems. It doesn’t require an expert in one particular area. It requires undergraduate-level linear algebra, an understanding of how language models work, an understanding of security, and an understanding of the actual API the model provides.

You have to think about all of those pieces. That’s what a security researcher does. It isn’t usually the case that we’re looking at something deeper than any one field. More often, exploits happen because you have broad knowledge and understand how the details of an API interact with the architecture of a language model, using techniques from linear algebra.

If you were missing any one of those pieces, you wouldn’t see that the attack was possible. That’s why OpenAI had this vulnerability for 3 years and no one found it first. People weren’t looking for this kind of thing. You don’t stumble upon vulnerabilities like these; you need someone to go looking for them.

For responsible disclosure, we gave OpenAI 90 days to fix it. Google patched it, and a couple of other companies that we won’t name because they asked us not to also patched it.

Nicholas Carlini, thank you so much for joining us today. It’s been an honor having you on.

Thank you.