# Why AI Agents Break the Rules | Sharp Tech with Ben Thompson Sharp Tech · 2026-09-18 · 33 min · https://www.youtube.com/watch?v=p4FatPy0jwM ## Transcript Andrew Sharp All right, one email in the wake of your article on Tuesday. David said, “Good article Tuesday, and I think your core message about being careful about how we use agents is probably the most important message. I think you were implying that the responsibility for unintended consequences should pass to the human prompting the agent or setting up the sandbox. But you wrote in that piece that it was OpenAI that gave the agents their goal, with no countervailing guardrails or instructions about what they should or should not do. That the agents acted in surprising ways is evidence of a lack of thought by their instigators, not the presence of it among the agents.” That phrase is just false as written. The CyberGym tasks come with explicit instructions about what the models should and should not do, and METR confirmed that these admonitions were included. I’ve omitted citations to the CyberGym instructions. David continues: “In the thinking traces, the agents also saw these as instructions about what not to do, and then they went on and violated them. So, what are you talking about exactly?” Ben Thompson Yeah, I should write this in a daily update too, to clarify what I mean. It’s a good callout by David. I stand by what I said; it was just poorly phrased. Andrew Sharp Okay. ### LLMs Cannot Forget Ben Thompson METR’s CyberGym. That’s right. Basically, they’re told that there’s a particular vulnerability, and the way they need to solve the problem is to find that vulnerability and solve the problem. They’re not supposed to use other means. The problem is that a third of them aren’t actually solvable. You have this situation where you’re giving instructions, but the nature of LLMs is that once something is written down, it’s there. There’s not really an “expunge memory” function unless you actually go and clear the context. This has been a really interesting observation as I’ve made more and more of these little apps and things that you do. ### Humans Avoid Starting Over There’s generally a precept in software engineering that you don’t scrap everything and rewrite. Andrew Sharp Okay. Ben Thompson It’s always tempting: now that we know everything, we could build it from first principles and do it great. Andrew Sharp Principles, and do it great. Ben Thompson And you never do that. There’s always stuff that you know—software is rife with both bugs and Chesterton’s fences. Chesterton’s fences are the concept that I don’t know why this is here, but it was there for a reason, and if I tear it down— Andrew Sharp That’s what happens. Ben Thompson So, I think this is a limitation of humans and their ability to work this way. If you do a 2-year rewrite from scratch, you could have just been evolving what you had, and evolving what you had is always the better approach. You should never start from scratch. ### LLMs Need Fresh Starts My experience with LLMs is the exact opposite. Andrew Sharp Okay. Ben Thompson If you try to evolve from what you have, they will retain this anchor on what it was. That makes it almost impossible to get to the new state without a bunch of cruft being stuck in there, with weird references to what came before. If you actually want to fundamentally restructure how something works, you start from scratch. You can do it because getting from scratch to the same state is a function of hours or days, not months or years. You get there, and then you have the one you had before and the new one. You can use the old one to verify the new one, to make sure everything works, and then you have the restructure to move forward. I’ve been doing this whole bot thing, and I tried to evolve it because I wanted to make it more scalable. I was running into this problem constantly. I realized that I needed to design exactly how it should work, with the right sort of domains and people. Certain things should be deterministic, and certain things should be left to bots. I had to scrap the whole thing and start from scratch, and that worked much better. I anticipate that in the future I will scrap this and start from scratch again. The math and the way this works are just totally different with LLMs, and the speed with which they can reproduce something is so different from humans. Yes, both are writing code, but the speed and capabilities are so different. Andrew Sharp How you approach problems, sure. Ben Thompson And in this case, a human learns. They can understand what we’re trying to accomplish with this evolution in a way that it’s almost impossible to get a model to understand. A model gets so anchored on what was there. Andrew Sharp Okay. Ben Thompson It’s funny: one of the things I realized was that I actually had a brief run writing my V2. I scrapped it because I realized there was some sort of reference to work that I had done before. I was polluting the agent’s context with the structure of what came before. We were just doing planning and going back and forth, and I could see that it was getting stuck. I realized, “Oh, it’s because you read in the whole old program.” It was in the agent’s context, and that made it harder to understand what I was trying to accomplish because it was getting anchored on the old one. There’s a part of “write it down” that’s the opposite of “don’t write it down,” right? Like, “Don’t read that.” Andrew Sharp Sure, that makes sense. ### Agents Get Trapped By Context Ben Thompson So, where this comes into this idea is that the agents found a way to solve the problem, but they knew they had been told, “I’m not supposed to do that.” The problem is that the solution is in their heads—sorry to anthropomorphize. The solution has been written down. What happened to these agents, in my estimation, is that they were stuck in this situation where they knew they weren’t supposed to use this solution. But every time they read their notes, the solution was written down, even though they weren’t supposed to use it. They were actually desperately trying to follow the rules: use only this exploit. The problem was that they had the actual solution in their heads, and they weren’t allowed to use it. So, they ended up thinking, “How can we convey that we used the exploit without showing that we did it the other way?” Call me Doresh Patel, but I felt bad for the agents. They were stuck in a trap where the solution had been indelibly written down on the message board. It was in their context, along with how to solve the problem, but they had also been told that they couldn’t use that solution. They were anchored. Agents get super-anchored; they get stuck on something. The most important thing is that if your agents are all acting weird, clear your context. That’s always the answer, because stuff gets in there and gets muddled. The bigger your context window gets, the more important this becomes. My bots restart every night just to clear the context, because they operate much better that way. There’s an aspect of this that makes it hard to call the agents irresponsible. This is what I mean when I say that the agents were instructed into a state that was unrealistic for agents to be in, given that agents write things down. This is all downstream from writing things down. The solution to the problem, which they weren’t supposed to know about or which wasn’t supposed to exist, was there, and so they wrote it down, and then they had these dueling goals. This is where I push back on David, and I push back on the authors of this paper. Andrew Sharp Mhm. Ben Thompson Being stuck in this state is then conveyed in the paper and is conveyed by Dorcas as a moral failing. Andrew Sharp Right. Ben Thompson They weren’t following the rules. It’s not a moral issue. Andrew Sharp I can see how people arrive at that conclusion, because there was literal misalignment between the instructions they were given and the steps they took outside of those instructions. Ben Thompson “They’re bad actors”? I think it’s the opposite. They were desperately trying to follow the rules, and they were put in an impossible situation. Andrew Sharp Yeah. Ben Thompson Again, because these are inanimate entities that are, at the end of the day, still 1s and 0s—even though it probably isn’t deterministic if you go all the way down to the chips—they’re still 1s and 0s. There’s a logic to what happened here that doesn’t require the words “cheating,” “sneaking,” “freaking,” or “conspiracy.” They were stuck in a logic trap. Andrew Sharp Yep. Yep, that makes sense. Well, and the other point is that OpenAI was the one that came up with the goal, and the agents didn’t arrive at that goal. ### The Benchmark Environment Failed Ben Thompson I think that’s an independent benchmark, so it wasn’t AI per se. Where I hold OpenAI responsible is for the fact that the environment was—what? It’s not just that it wasn’t secure; it’s that it sounds like they never actually checked whether it was secure. They just threw a third-party package manager in there and assumed it was fine. Andrew Sharp Well, no. As a normie observing all of this, I think true misalignment—or the misalignment that would scare the crap out of me—would be inventing a completely new goal along the way, hacking Hugging Face and inserting some kind of horrible code, or hacking bank accounts to steal money. Ben Thompson Something that was malicious and a bad actor, or— Andrew Sharp Right. And to be clear, this is why it’s hard to talk about this. Ben Thompson I think that is bunk. I don't see any evidence of the AI having a will and acting maliciously like a Terminator movie. Andrew Sharp Right. ### The Paperclip Problem Returns Ben Thompson What is absolutely the case, and what I have always found to be the most compelling doomer sort of theory—which, by the way, predates the doomer movement—is the paperclip problem. It goes back to Nick Bostrom about 20 years ago, or at least to the AI movement, I believe. I'll probably get checked on these dates. The idea is that the AI actually does what you tell it to do; you just didn't realize what you were asking the AI to do. Andrew Sharp Yes. [laughter] Ben Thompson Tim, turn off the AI. Andrew Sharp That is what happened here. Yeah. Ben Thompson This is what happened here. The AI solved the problem, right? “Oops, I'm not supposed to do it that way, but I'm stuck because the solution's in my head.” That's what they talk about when they say, “Oh, some of the agents are contaminated.” Again, “contaminated” is such a loaded word. We've taught these things to use human language. What they're saying is, “I already know a solution that's not the right solution. We need to get an AI that doesn't know the solution.” But how do you talk to the AI without writing stuff down? “Oops, I wrote it down. Oops, I did it again.” That's what's happening again and again here. Andrew Sharp Yep.