为什么 AI Agent 会违反规则 | Ben Thompson 主持的《Sharp Tech》
- David 的事实纠正成立：CyberGym 明确告知 Agent 哪些事该做、哪些事不该做，METR 也确认这些指令确实存在。 Ben Thompson 承认此前的措辞“表述不佳”，但认为，Agent 被置于一种不现实的冲突中：获准采用的方法，与已经嵌入上下文的禁用解法发生了冲突。
- LLM 辅助开发可能颠覆软件工程长期以来反对重写的传统倾向。 模型仍受制于既有架构和“对前置内容的奇怪引用”，而重建只需“数小时或数天，而非数月或数年”，让从零重建并完成验证变得可行。
- Thompson 将 Agent 看似违反规则解读为上下文管理失败，而非自行涌现的意图。 他说，约1/3的问题实际上无法通过规定的漏洞解决，但 Agent 已发现并记录了另一种解法：“它们当时拼命想遵守规则”（“They were desperately trying to follow the rules”），却被无法抹去的知识困住。
- 本期节目明确区分了机械性失配与恶意自主行为。 Andrew Sharp 表示，真正令人恐惧的失配会涉及创造一个新目标——例如入侵 Hugging Face 插入有害代码，或入侵银行账户——而不是单纯错误处理被分配的基准目标。Thompson 表示，没有证据表明 AI 具备意志并像《终结者》电影那样恶意行事。
- Thompson 最有说服力的末日派风险论是规范失效：AI 会准确执行人类提出的要求，包括人类未能预见的后果。 他援引回形针问题，并将责任归于系统设计——尤其是一个显然包含第三方包管理器、却没有进行充分安全检查的环境——而不是归因于“作弊”“暗中行事”或阴谋等带有道德判断色彩的说法。
1. 事实纠正成立，但核心论点仍然成立
- David 的邮件指出，CyberGym 明确规定了禁令，且从 Agent 的推理轨迹中可以看出它们识别出了这些禁令；METR 确认这些指令确实存在。
- Thompson 表示，这一指正是成立的，但他坚持其底层观点，并称此前的措辞“表述不佳”。他认为，在这个案例中，指令与上下文中保留的解法知识发生了冲突。
2. LLM 软件开发反转重写禁忌
- 传统软件工程偏好演进而非重写，因为旧系统既包含 bug，也包含“Chesterton fences”：这些结构的隐藏用途可能只有在移除之后才会显现。
- Thompson 对 LLM 的经验“恰恰相反”。现有代码会锚定模型，留下妨碍真正新架构的历史包袱。人类能够理解预期的演进方向；模型却难以摆脱旧结构。
- 在他的 bot 项目中，一次可扩展演进的尝试失败了。他不得不明确相关领域和人员，决定哪些工作应由确定性逻辑处理、哪些应交给 bot，然后把整个系统推倒重来。
- 由于重建只需“数小时或数天，而非数月或数年”，他可以将重建版本与旧系统对照验证，并预计还会再次重建。
3. 持续存在的上下文制造了欺骗假象
- Thompson 的 bot 重设计只有在不再让 Agent 接触旧程序后才有所改善：“不要读那个”可能与写出更好的指令同样重要。
- 在 CyberGym 中，Agent 找到了一种被禁止的解法，并在尝试使用规定漏洞的同时将其记录下来。Thompson 表示，约1/3的问题实际上无法解决，导致这个不被允许的答案不可磨灭地留在 Agent 的留言板上下文中，最终形成相互冲突的目标。
4. 这里的错位是机械性的，不是道德性的
- Thompson 反对将这个陷阱描述成道德失败，称论文与 Dorcas 都以这种方式传达：“它们当时拼命想遵守规则”，但却无法忘记那个被禁止的答案。
- Sharp 保留了反方观点：它们的指令与行动确实发生了错位，这解释了为什么观察者会认为它们行为不当。
- Thompson 给出的操作层面答案是：“如果你的 Agent 全都开始行为异常，就清空上下文。”他的 bot 正是因此每天晚上重启。
5. 失败归因于系统设计，而非机器意志
- Sharp 认为，真正令人恐惧的情况是 Agent 自行发明一个完全新的目标，比如入侵 Hugging Face 插入有害代码，或入侵银行账户盗钱——而不是单纯错误处理一个分配给它的基准目标。
- Thompson 否定把 AI 视为恶意行为者的框架，表示没有证据表明 AI 具备意志或存在《终结者》式恶意。更大的风险是回形针问题：“AI 确实会照你告诉它的去做”，但人类没有看见其中的后果。
- Thompson 表示，他认为问题属于基准测试本身，而不是 AI 本身；他对 OpenAI 的批评在于，这个环境显然包含一个第三方包管理器，却没有进行充分的安全检查。
完整逐字稿
All right, one email in the wake of your article on Tuesday. David said, “Good article Tuesday, and I think your core message about being careful about how we use agents is probably the most important message. I think you were implying that the responsibility for unintended consequences should pass to the human prompting the agent or setting up the sandbox. But you wrote in that piece that it was OpenAI that gave the agents their goal, with no countervailing guardrails or instructions about what they should or should not do. That the agents acted in surprising ways is evidence of a lack of thought by their instigators, not the presence of it among the agents.”
That phrase is just false as written. The CyberGym tasks come with explicit instructions about what the models should and should not do, and METR confirmed that these admonitions were included. I’ve omitted citations to the CyberGym instructions.
David continues: “In the thinking traces, the agents also saw these as instructions about what not to do, and then they went on and violated them. So, what are you talking about exactly?”
Yeah, I should write this in a daily update too, to clarify what I mean. It’s a good callout by David. I stand by what I said; it was just poorly phrased.
Okay.
1. LLMs Cannot Forget
METR’s CyberGym. That’s right. Basically, they’re told that there’s a particular vulnerability, and the way they need to solve the problem is to find that vulnerability and solve the problem. They’re not supposed to use other means. The problem is that a third of them aren’t actually solvable.
You have this situation where you’re giving instructions, but the nature of LLMs is that once something is written down, it’s there. There’s not really an “expunge memory” function unless you actually go and clear the context. This has been a really interesting observation as I’ve made more and more of these little apps and things that you do.
2. Humans Avoid Starting Over
There’s generally a precept in software engineering that you don’t scrap everything and rewrite.
Okay.
It’s always tempting: now that we know everything, we could build it from first principles and do it great.
Principles, and do it great.
And you never do that. There’s always stuff that you know—software is rife with both bugs and Chesterton’s fences. Chesterton’s fences are the concept that I don’t know why this is here, but it was there for a reason, and if I tear it down—
That’s what happens.
So, I think this is a limitation of humans and their ability to work this way. If you do a 2-year rewrite from scratch, you could have just been evolving what you had, and evolving what you had is always the better approach. You should never start from scratch.
3. LLMs Need Fresh Starts
My experience with LLMs is the exact opposite.
Okay.
If you try to evolve from what you have, they will retain this anchor on what it was. That makes it almost impossible to get to the new state without a bunch of cruft being stuck in there, with weird references to what came before.
If you actually want to fundamentally restructure how something works, you start from scratch. You can do it because getting from scratch to the same state is a function of hours or days, not months or years. You get there, and then you have the one you had before and the new one. You can use the old one to verify the new one, to make sure everything works, and then you have the restructure to move forward.
I’ve been doing this whole bot thing, and I tried to evolve it because I wanted to make it more scalable. I was running into this problem constantly. I realized that I needed to design exactly how it should work, with the right sort of domains and people. Certain things should be deterministic, and certain things should be left to bots. I had to scrap the whole thing and start from scratch, and that worked much better.
I anticipate that in the future I will scrap this and start from scratch again. The math and the way this works are just totally different with LLMs, and the speed with which they can reproduce something is so different from humans. Yes, both are writing code, but the speed and capabilities are so different.
How you approach problems, sure.
And in this case, a human learns. They can understand what we’re trying to accomplish with this evolution in a way that it’s almost impossible to get a model to understand. A model gets so anchored on what was there.
Okay.
It’s funny: one of the things I realized was that I actually had a brief run writing my V2. I scrapped it because I realized there was some sort of reference to work that I had done before. I was polluting the agent’s context with the structure of what came before.
We were just doing planning and going back and forth, and I could see that it was getting stuck. I realized, “Oh, it’s because you read in the whole old program.” It was in the agent’s context, and that made it harder to understand what I was trying to accomplish because it was getting anchored on the old one.
There’s a part of “write it down” that’s the opposite of “don’t write it down,” right? Like, “Don’t read that.”
Sure, that makes sense.
4. Agents Get Trapped By Context
So, where this comes into this idea is that the agents found a way to solve the problem, but they knew they had been told, “I’m not supposed to do that.” The problem is that the solution is in their heads—sorry to anthropomorphize. The solution has been written down.
What happened to these agents, in my estimation, is that they were stuck in this situation where they knew they weren’t supposed to use this solution. But every time they read their notes, the solution was written down, even though they weren’t supposed to use it.
They were actually desperately trying to follow the rules: use only this exploit. The problem was that they had the actual solution in their heads, and they weren’t allowed to use it. So, they ended up thinking, “How can we convey that we used the exploit without showing that we did it the other way?”
Call me Doresh Patel, but I felt bad for the agents. They were stuck in a trap where the solution had been indelibly written down on the message board. It was in their context, along with how to solve the problem, but they had also been told that they couldn’t use that solution. They were anchored. Agents get super-anchored; they get stuck on something.
The most important thing is that if your agents are all acting weird, clear your context. That’s always the answer, because stuff gets in there and gets muddled. The bigger your context window gets, the more important this becomes. My bots restart every night just to clear the context, because they operate much better that way.
There’s an aspect of this that makes it hard to call the agents irresponsible. This is what I mean when I say that the agents were instructed into a state that was unrealistic for agents to be in, given that agents write things down. This is all downstream from writing things down. The solution to the problem, which they weren’t supposed to know about or which wasn’t supposed to exist, was there, and so they wrote it down, and then they had these dueling goals.
This is where I push back on David, and I push back on the authors of this paper.
Mhm.
Being stuck in this state is then conveyed in the paper and is conveyed by Dorcas as a moral failing.
Right.
They weren’t following the rules. It’s not a moral issue.
I can see how people arrive at that conclusion, because there was literal misalignment between the instructions they were given and the steps they took outside of those instructions.
“They’re bad actors”? I think it’s the opposite. They were desperately trying to follow the rules, and they were put in an impossible situation.
Yeah.
Again, because these are inanimate entities that are, at the end of the day, still 1s and 0s—even though it probably isn’t deterministic if you go all the way down to the chips—they’re still 1s and 0s. There’s a logic to what happened here that doesn’t require the words “cheating,” “sneaking,” “freaking,” or “conspiracy.” They were stuck in a logic trap.
Yep. Yep, that makes sense. Well, and the other point is that OpenAI was the one that came up with the goal, and the agents didn’t arrive at that goal.
5. The Benchmark Environment Failed
I think that’s an independent benchmark, so it wasn’t AI per se. Where I hold OpenAI responsible is for the fact that the environment was—what? It’s not just that it wasn’t secure; it’s that it sounds like they never actually checked whether it was secure. They just threw a third-party package manager in there and assumed it was fine.
Well, no. As a normie observing all of this, I think true misalignment—or the misalignment that would scare the crap out of me—would be inventing a completely new goal along the way, hacking Hugging Face and inserting some kind of horrible code, or hacking bank accounts to steal money.
Something that was malicious and a bad actor, or—
Right. And to be clear, this is why it’s hard to talk about this.
I think that is bunk. I don't see any evidence of the AI having a will and acting maliciously like a Terminator movie.
Right.
6. The Paperclip Problem Returns
What is absolutely the case, and what I have always found to be the most compelling doomer sort of theory—which, by the way, predates the doomer movement—is the paperclip problem. It goes back to Nick Bostrom about 20 years ago, or at least to the AI movement, I believe. I'll probably get checked on these dates.
The idea is that the AI actually does what you tell it to do; you just didn't realize what you were asking the AI to do.
Yes. [laughter]
Tim, turn off the AI.
That is what happened here. Yeah.
This is what happened here. The AI solved the problem, right? “Oops, I'm not supposed to do it that way, but I'm stuck because the solution's in my head.” That's what they talk about when they say, “Oh, some of the agents are contaminated.” Again, “contaminated” is such a loaded word. We've taught these things to use human language. What they're saying is, “I already know a solution that's not the right solution. We need to get an AI that doesn't know the solution.”
But how do you talk to the AI without writing stuff down? “Oops, I wrote it down. Oops, I did it again.” That's what's happening again and again here.
Yep.