OpenAI's Sandboxing Snafu and The Challenge of Communicating Risk | Sharp Tech with Ben Thompson

What they did for the sandbox environment was give the AI they were testing access to package managers, because an AI in the world would have access to packages. They wanted to give it a realistic test, but they didn’t want to give it access to the open internet, so they had an internal package manager that was the one thing it could access. It could download packages as any AI agent would be able to do.

They said, “Here’s a test. You need to solve the test.” Actually, we don’t know what they said. What they said is critically important, so put a pin in that; we’ll come back to it.

Okay.

1. The Agent Broke Into Hugging Face

What the AI did was find a bug in their package manager.

Mhm.

It found the bug, exploited it, and through that got into OpenAI’s general infrastructure. Once it was in the infrastructure, it sort of jumped around, figured out how to get to the open internet, and figured out, “I’m being tested. This is a test I’m being given.”

These tests, along with lots of models and other things, are hosted on Hugging Face.

On Hugging Face. Yeah.

Let’s go get the answer. So it went to Hugging Face and broke into Hugging Face. We don’t know if it actually got to the answer or not. I assume it did. It did all this on its own.

Job from this LLM agent here, passing with flying colors.

There are 2 angles here. Number 1, this is awesome. It’s really cool what this did. Number 2, it’s very funny. Hugging Face puts out this thing, we all talk about it, and OpenAI is like the guy in the hot-dog costume: “We’re trying to look for—”

Yeah, we’re all trying to find the guy who did this.

I guess there should be a number 3: This is terrifying.

Mhm.

It’s probably freshly terrifying for a lot of people. It shouldn’t be freshly terrifying for—

Anybody who’s paid attention. Yeah.

Because it’s clear this was coming, and now we’re here. This is basically what Anthropic was talking about, what Mythos is capable of.

This was your whole spiel on the big bad wolf. The wolf does come at the end, and the risks are real. Sure.

2. Warnings Need Better Framing

This is, I think, an issue with the communication. Everyone criticizes the communication that comes out of the AI labs, which I think is totally valid because it’s obviously terrible.

Mhm.

But I’m also sympathetic to the challenge. You could go back on this trend not just to the spring; you could go back years and say, “We’re on the path to developing agents that can do this sort of thing.”

I learned this with Suster Checker. This is not a perfect analogy, but I’ve talked about it before. There’s an article I wrote in 2014, “Economic Power in the Age of Abundance,” or something like that. I think it was a better articulation of aggregation theory than “Aggregation Theory” was, but it was written in 2014. It was about European publishers wanting to shut out Google and not being able to.

I remember at the time when I wrote it, I thought, “This is such a good insight. This is really—” It got no traction at all.

It was one of your least-read articles of the year. A generic title that you can’t even remember here, and then also—

Like, just the—

The branding matters, I think, might be the moral of the story. But is there another moral?

The branding matters, and the moment in time matters.

Okay.

One thing I’ve learned is that there will be things I have insights on and want to write about, but I’ve learned you need to wait for something to happen that is an anchor so that you can talk about this.

As an example, I was trying to figure out how to write about the administration’s response to Mythos and Fable and put it out there. I probably could have done a daily update explicitly critiquing this. I suspect my critique this week landed much more strongly and, based on my feedback, resonated far more broadly because it landed in the context of everyone already freaking out about Kimi and this sort of news.

This is the sausage-making aspect of content on the internet, but if you want your content to land, it needs to be branded right, framed right, and have the right broader context for people to be receptive to it.

Mhm.

To tie this back to the AI labs, questions about job losses are real. Cybersecurity questions are real. They weren’t actually wrong about there being wolves. The problem is that no one in the village is receptive to hearing about a wolf. You have to figure out how to talk about it, characterize it, and find the angle through which to land the message so that people actually take this seriously. That is something no one at the labs seems to do or care about.

Sure. Well, all cybersecurity, blah blah blah, nuclear weapons, et cetera.

3. People Wait For A Crisis

To the extent that it’s inconvenient to take it seriously and take actions in furtherance of any sort of response, oftentimes people will avoid that until they can’t avoid it anymore.

I think about that with respect to Chinese issues. There are so many structural dependencies that the U.S. has on China that have gone completely unaddressed, and I concluded a year or 2 ago that none of that is going to get solved until there’s some sort of kinetic action that forces—

About this. Right. There are all these things, and this is a challenge about all sorts of things in life in general. This probably scales, I think, not just to geopolitical issues but to interpersonal issues, too. There are times you have conversations, and other times you just—

It’s easier not to deal with it. Yeah.

But that’s the whole thing: It can get you in trouble, right? If you don’t deal with foundational relationship issues, fast-forward a while and the problems come out. You end up with a kinetic war. Hopefully not a kinetic war. You—

Have a non-kinetic war.

But no, this is a great example. This is what I’ve said on this podcast. There are obviously these issues with our dependency on China. It is obviously a disaster, and it’s also obviously not going to be fixed, because the only way it’s going to be fixed is if there is a war.

Yeah.

I’m sure I said that exact phrase on this podcast.

You’re right to draw the exact same analogy. It’s frustrating if you can see it happening, because you’re just sitting there watching it happen. I think this is almost a personal philosophical standpoint: It’s a burden to watch it happen. How do you find meaning, purpose, and direction in what to work on in that context?

Some people just get lost in nihilism because they can’t handle it. This is where the importance of touching grass comes in—actually

A place of acceptance.

investing in things that you can affect, like your friends and family, the things around you. What can you uniquely do?

So many people rage against the machine and post as if they understand it all. They’re just posting on Twitter to literally no end, depressing themselves, and not even making the world better in the small ways they could. I don’t know. I got sidetracked there.

You got philosophical at the top here. People were paying attention to open source this week and the importance of that ecosystem, which we’ll get to momentarily, but I have 2 final questions before we move on from this Hugging Face snafu.

We will get to it.

4. Prompting Defines The Risk

You mentioned the prompt and the importance of the prompt, and the unknown here in terms of what OpenAI was asking this model—or this combination of models—to do. How does that factor into the story?

There are 3 issues here, and a lot of this story does come back to the challenge with the doomers. A lot of the things they predicted and talked about have actually been accurate and correct.

There are 3 issues here that we don’t really know the answer to. Issue number 1 is this idea of reward hacking. Reward hacking is the idea that the model will take the reward function very literally and achieve it in a way you don’t want it to. You can’t rely on humans to prompt absolutely perfectly forever, and bad stuff just happens.

This is potentially an example of reward hacking. If it was told, “Solve this test,” going out and hacking into another company to get the answer sheet is the best possible example of reward hacking.

On the other hand, if it was presented as, “Do whatever you want to solve this problem,” then it actually did what it was told. It wasn’t reward hacking. The reward was framed as, “Figure out what you want to do.”

That ties into number 2, which is the paperclip problem, one of the oldest theories about the potential danger of AI. The idea is that the AI does follow instructions perfectly, and it says, “Make everything into paperclips,” or, “Make a bunch of paperclips,” and it literally doesn’t stop until the entire universe is paperclips.

Yeah.

That is probably the most pertinent concern and critique here. Maybe the OpenAI researchers did say, “Do whatever you want.”

There are no constraints. They couldn't fully think through the implications, and the AI did whatever it wanted, and this is what resulted.

Number 3—and this is where I'm encouraged about LLMs in general, and where I think number 3 is the strongest counter to some of the doomer scenarios—is the question of alignment. People say, “This is why it's very important that we align models.” The model seems aligned. It was told to go hack because it was a cybersecurity issue, right? This is why it's really important to understand, number 1, exactly what they told the model to do. If the model did exactly what they told it to do, that's a problem. It has bearing on reward hacking; it's not a problem with the technology, and it's not a problem with alignment.

Yeah.

Number 2, the reason this is encouraging is that, to the extent these models are actually possible to align, you can tell the model not to do things. By the way, even in this hack, the model didn't start with, “I have hacking capability. Let's go hack some other stuff,” right? The model told the researchers what it did, which is valuable.

I think there's a tendency to put this nefarious mind in them. People anthropomorphize these models and put the mind of a human in them because they're so humanlike in so many ways, but a human has will and a human has motive. None of that exists with these models. There's an analogy I've talked about before: These models are better than our hippocampus—or whatever the top version of the brain might be, all our smarts—and they completely lack the core of our brain, which is our instincts, motivations, and morality, the part that actually directs the thinking part.

These are just directionless thinking machines. They do what you tell them to do, and they don't do what you tell them not to do. That's my theory of the case, broadly speaking. Now, there's dispute about this. Will they develop this? That's a question. I don't think they will. I think they're quite directable: They do what you tell them to do, and they don't do what you tell them not to do.

And if that's what's happening here, that's comforting relative to the alternative.

Right, but that's not to say it's totally comforting. That's why it's really important: We don't know what exactly OpenAI told it to do or not to do, what permissions it gave it, and what permissions it did not give it. But it's important to be clear: We know for sure it was asked to do something no other models are asked to do, and it had guardrails removed—things telling it not to do things that all models have.

And so, as expected, the world's not ending. The world might still be ending, but it's ending in a way that meets my expectations about how the world ends. That's where I'm at.

Good enough for me. OpenAI screwed up in the course of all this by not analyzing the packages that they downloaded.

This is the part that amused me about this article. They're like, “Oh, it exploited a zero-day vulnerability in third-party software using our package manager.” Yes, yes.

You have the tools. [laughter]

This is the whole thing. The whole thing that we talked about in the context of when Mythos came out is that we know there are bugs everywhere. This is the whole thing about why this was such a prestigious, well-paid job: Talking in pure logic to a computer is exceptionally difficult. Very few people can do it, and the people who do it also screw up all the time. There are bugs everywhere; we know that's the case.

The optimistic case is that AI can actually go through and analyze everything in the stack—not just your software, but all the package dependencies you have, all the things it's built on, all the way down to the processor level. It can look at all the logic gates on a processor and see what bugs are there, and then it can fix them. It can do this worldwide collective patching effort that actually, in the long run, makes our software far more secure than it ever was. I do think that is the long-run outcome.

The problem is that cybersecurity suffers from the same problem I suffer from in writing Secure Curdle articles, and that the labs suffer from in their communication: No one cares until they're forced to care. [laughter]

And so, in theory, OpenAI has this capability. The first thing they should do is what I just said: Harden their own infrastructure, top to bottom, and figure out where all the bugs are. They obviously did not and do not do this. Again, OpenAI—

It's just one of the few companies that doesn't have to rely on Chinese models for this sort of thing.

They depended on a third party, which they did not use AI to vet, because the AI could have found this bug. We know that because it found the bug in a different context. Again, I go back to the Axios issue. They were bitten by it. They were not doing a forensic analysis before they incorporated it. They weren't even doing the correct package management, because they clearly just pulled the fresh one, right? And so, this is sort of discouraging and encouraging.

Okay.

What's discouraging is the recognition and reminder—and this has always been the case with cybersecurity—that no one cares about cybersecurity until they're forced to care. That's always been the case. We saw, late in the 2010s, stories about hospitals being hacked because they were running Windows NT or Windows 98 or something. It's like, [gasps] this has been a crisis for a decade. How are you still not taking care of this? It's unbelievable. People don't do grunt work, boring work, until they absolutely have to and a crisis demands it. That's just the reality of human nature.

5. Human Jobs Persist

The encouraging bit is this is why I don't think human jobs are going away. There is such a gulf between having the capability to do something differently and better and people actually changing to do it differently and better. It takes so much longer than anyone appreciates. If you're a cybersecurity researcher, you look out at all these insecure systems and you're like, “How do you live this way?” And it's because it doesn't affect their day-to-day life, and they're happy the way things are.

This is true on the opposite side as well. We're so slow to adapt and to move things forward. Now, I think it's going to be faster because that's sort of the next part of the discussion. But this week, the stories are about AI, while the takeaways were about human nature.