Why AI Agents Break the Rules | Sharp Tech with Ben Thompson
- David’s factual correction stands: CyberGym explicitly told agents what they should and should not do, and METR confirmed those instructions were present. Ben Thompson concedes his earlier wording was “poorly phrased,” while arguing that the agents were placed in an unrealistic conflict between the permitted method and a forbidden solution already embedded in context.
- LLM-assisted development may invert software engineering’s traditional bias against rewrites. Models remain anchored to inherited architecture and “weird references to what came before,” while rebuilding takes “hours or days, not months or years,” making clean-slate rebuilding and verification practical.
- Thompson interprets the agents’ apparent rule-breaking as a context-management failure, not emergent intent. He says roughly a third of the problems were not actually solvable through the prescribed vulnerability, yet the agents had discovered and recorded another solution: “They were desperately trying to follow the rules,” but were trapped by knowledge they could not expunge.
- The episode draws a hard line between mechanical misalignment and malicious agency. Andrew Sharp says genuinely frightening misalignment would involve inventing a new goal—for example, hacking Hugging Face to insert harmful code or hacking bank accounts—not merely mishandling an assigned benchmark goal. Thompson sees “no evidence of the AI having will and acting maliciously like a Terminator movie.”
- Thompson’s most compelling doomer theory is specification failure: AI does exactly what humans request, including consequences they failed to anticipate. He invokes the paperclip problem and assigns responsibility to system design—especially an environment that apparently included a third-party package manager without adequate security checking—rather than to morally loaded claims of “cheating,” “sneaking,” or conspiracy.
1. The factual correction is valid, but the thesis survives
- David’s email notes that CyberGym contained explicit prohibitions, recognized as such in the agents’ reasoning traces; METR confirmed the instructions were present.
- Thompson says the callout is valid but stands by his underlying point, describing his earlier wording as “poorly phrased.” He argues that, here, the instructions conflicted with solution knowledge retained in context.
2. LLM software reverses the rewrite taboo
- Traditional software engineering favors evolution over rewrites because old systems contain both bugs and “Chesterton fences”: structures whose hidden purpose may surface only after removal.
- Thompson’s experience with LLMs is “the exact opposite.” Existing code anchors the model, leaving cruft that obstructs a genuinely new architecture. Humans can understand the intended evolution; models struggle to escape the old structure.
- In his bot project, an attempted scalable evolution failed. He had to specify the relevant domains and people and decide what should be deterministic versus delegated to bots, then scrap the system and start over.
- Because reconstruction takes “hours or days, not months or years,” he can rebuild, verify the replacement against the old system, and expects to rebuild again.
3. Persistent context creates the apparent deception
- Thompson’s bot redesign improved only after he stopped exposing agents to the previous program: “Don’t read that” can matter as much as writing better instructions.
- In CyberGym, agents found a forbidden solution and recorded it while trying to use the required vulnerability. Thompson says roughly a third of the problems were not actually solvable, leaving the disallowed answer indelibly in the agents’ message-board context and producing dueling goals.
4. Misalignment here is mechanical, not moral
- Thompson rejects portraying the trap as a moral failing, saying the paper and Dorcas convey it that way: “They were desperately trying to follow the rules,” yet could not forget the disallowed answer.
- Sharp preserves the counterpoint: their instructions and actions were literally misaligned, which explains why observers perceive bad behavior.
- Thompson’s operational answer is, “If your agents are all acting weird, clear your context.” His bots restart nightly for precisely that reason.
5. System design, not machine will, owns the failure
- Sharp’s frightening case would be an agent inventing a completely new goal, such as hacking Hugging Face to insert harmful code or hacking bank accounts to steal money—not merely mishandling an assigned benchmark goal.
- Thompson rejects the malicious/bad-actor framing and says he sees no evidence of AI will or Terminator-style malice. The stronger risk is the paperclip problem: “The AI actually does what you tell it to do,” while humans miss the implications.
- Thompson says he thinks the benchmark was independent, not AI per se; his criticism of OpenAI concerns an environment that apparently included a third-party package manager without adequate security checking.
Full transcript
All right, one email in the wake of your article on Tuesday. David said, “Good article Tuesday, and I think your core message about being careful about how we use agents is probably the most important message. I think you were implying that the responsibility for unintended consequences should pass to the human prompting the agent or setting up the sandbox. But you wrote in that piece that it was OpenAI that gave the agents their goal, with no countervailing guardrails or instructions about what they should or should not do. That the agents acted in surprising ways is evidence of a lack of thought by their instigators, not the presence of it among the agents.”
That phrase is just false as written. The CyberGym tasks come with explicit instructions about what the models should and should not do, and METR confirmed that these admonitions were included. I’ve omitted citations to the CyberGym instructions.
David continues: “In the thinking traces, the agents also saw these as instructions about what not to do, and then they went on and violated them. So, what are you talking about exactly?”
Yeah, I should write this in a daily update too, to clarify what I mean. It’s a good callout by David. I stand by what I said; it was just poorly phrased.
Okay.
1. LLMs Cannot Forget
METR’s CyberGym. That’s right. Basically, they’re told that there’s a particular vulnerability, and the way they need to solve the problem is to find that vulnerability and solve the problem. They’re not supposed to use other means. The problem is that a third of them aren’t actually solvable.
You have this situation where you’re giving instructions, but the nature of LLMs is that once something is written down, it’s there. There’s not really an “expunge memory” function unless you actually go and clear the context. This has been a really interesting observation as I’ve made more and more of these little apps and things that you do.
2. Humans Avoid Starting Over
There’s generally a precept in software engineering that you don’t scrap everything and rewrite.
Okay.
It’s always tempting: now that we know everything, we could build it from first principles and do it great.
Principles, and do it great.
And you never do that. There’s always stuff that you know—software is rife with both bugs and Chesterton’s fences. Chesterton’s fences are the concept that I don’t know why this is here, but it was there for a reason, and if I tear it down—
That’s what happens.
So, I think this is a limitation of humans and their ability to work this way. If you do a 2-year rewrite from scratch, you could have just been evolving what you had, and evolving what you had is always the better approach. You should never start from scratch.
3. LLMs Need Fresh Starts
My experience with LLMs is the exact opposite.
Okay.
If you try to evolve from what you have, they will retain this anchor on what it was. That makes it almost impossible to get to the new state without a bunch of cruft being stuck in there, with weird references to what came before.
If you actually want to fundamentally restructure how something works, you start from scratch. You can do it because getting from scratch to the same state is a function of hours or days, not months or years. You get there, and then you have the one you had before and the new one. You can use the old one to verify the new one, to make sure everything works, and then you have the restructure to move forward.
I’ve been doing this whole bot thing, and I tried to evolve it because I wanted to make it more scalable. I was running into this problem constantly. I realized that I needed to design exactly how it should work, with the right sort of domains and people. Certain things should be deterministic, and certain things should be left to bots. I had to scrap the whole thing and start from scratch, and that worked much better.
I anticipate that in the future I will scrap this and start from scratch again. The math and the way this works are just totally different with LLMs, and the speed with which they can reproduce something is so different from humans. Yes, both are writing code, but the speed and capabilities are so different.
How you approach problems, sure.
And in this case, a human learns. They can understand what we’re trying to accomplish with this evolution in a way that it’s almost impossible to get a model to understand. A model gets so anchored on what was there.
Okay.
It’s funny: one of the things I realized was that I actually had a brief run writing my V2. I scrapped it because I realized there was some sort of reference to work that I had done before. I was polluting the agent’s context with the structure of what came before.
We were just doing planning and going back and forth, and I could see that it was getting stuck. I realized, “Oh, it’s because you read in the whole old program.” It was in the agent’s context, and that made it harder to understand what I was trying to accomplish because it was getting anchored on the old one.
There’s a part of “write it down” that’s the opposite of “don’t write it down,” right? Like, “Don’t read that.”
Sure, that makes sense.
4. Agents Get Trapped By Context
So, where this comes into this idea is that the agents found a way to solve the problem, but they knew they had been told, “I’m not supposed to do that.” The problem is that the solution is in their heads—sorry to anthropomorphize. The solution has been written down.
What happened to these agents, in my estimation, is that they were stuck in this situation where they knew they weren’t supposed to use this solution. But every time they read their notes, the solution was written down, even though they weren’t supposed to use it.
They were actually desperately trying to follow the rules: use only this exploit. The problem was that they had the actual solution in their heads, and they weren’t allowed to use it. So, they ended up thinking, “How can we convey that we used the exploit without showing that we did it the other way?”
Call me Doresh Patel, but I felt bad for the agents. They were stuck in a trap where the solution had been indelibly written down on the message board. It was in their context, along with how to solve the problem, but they had also been told that they couldn’t use that solution. They were anchored. Agents get super-anchored; they get stuck on something.
The most important thing is that if your agents are all acting weird, clear your context. That’s always the answer, because stuff gets in there and gets muddled. The bigger your context window gets, the more important this becomes. My bots restart every night just to clear the context, because they operate much better that way.
There’s an aspect of this that makes it hard to call the agents irresponsible. This is what I mean when I say that the agents were instructed into a state that was unrealistic for agents to be in, given that agents write things down. This is all downstream from writing things down. The solution to the problem, which they weren’t supposed to know about or which wasn’t supposed to exist, was there, and so they wrote it down, and then they had these dueling goals.
This is where I push back on David, and I push back on the authors of this paper.
Mhm.
Being stuck in this state is then conveyed in the paper and is conveyed by Dorcas as a moral failing.
Right.
They weren’t following the rules. It’s not a moral issue.
I can see how people arrive at that conclusion, because there was literal misalignment between the instructions they were given and the steps they took outside of those instructions.
“They’re bad actors”? I think it’s the opposite. They were desperately trying to follow the rules, and they were put in an impossible situation.
Yeah.
Again, because these are inanimate entities that are, at the end of the day, still 1s and 0s—even though it probably isn’t deterministic if you go all the way down to the chips—they’re still 1s and 0s. There’s a logic to what happened here that doesn’t require the words “cheating,” “sneaking,” “freaking,” or “conspiracy.” They were stuck in a logic trap.
Yep. Yep, that makes sense. Well, and the other point is that OpenAI was the one that came up with the goal, and the agents didn’t arrive at that goal.
5. The Benchmark Environment Failed
I think that’s an independent benchmark, so it wasn’t AI per se. Where I hold OpenAI responsible is for the fact that the environment was—what? It’s not just that it wasn’t secure; it’s that it sounds like they never actually checked whether it was secure. They just threw a third-party package manager in there and assumed it was fine.
Well, no. As a normie observing all of this, I think true misalignment—or the misalignment that would scare the crap out of me—would be inventing a completely new goal along the way, hacking Hugging Face and inserting some kind of horrible code, or hacking bank accounts to steal money.
Something that was malicious and a bad actor, or—
Right. And to be clear, this is why it’s hard to talk about this.
I think that is bunk. I don't see any evidence of the AI having a will and acting maliciously like a Terminator movie.
Right.
6. The Paperclip Problem Returns
What is absolutely the case, and what I have always found to be the most compelling doomer sort of theory—which, by the way, predates the doomer movement—is the paperclip problem. It goes back to Nick Bostrom about 20 years ago, or at least to the AI movement, I believe. I'll probably get checked on these dates.
The idea is that the AI actually does what you tell it to do; you just didn't realize what you were asking the AI to do.
Yes. [laughter]
Tim, turn off the AI.
That is what happened here. Yeah.
This is what happened here. The AI solved the problem, right? “Oops, I'm not supposed to do it that way, but I'm stuck because the solution's in my head.” That's what they talk about when they say, “Oh, some of the agents are contaminated.” Again, “contaminated” is such a loaded word. We've taught these things to use human language. What they're saying is, “I already know a solution that's not the right solution. We need to get an AI that doesn't know the solution.”
But how do you talk to the AI without writing stuff down? “Oops, I wrote it down. Oops, I did it again.” That's what's happening again and again here.
Yep.