五角大楼对阵 Anthropic + 一个 AI Agent 诽谤了我 + Hot Mess Express

Well, speaking of Kim Jong, have you heard about his successor?

No.

Well, it appears that he is preparing to name his daughter, who we believe is just 13 years old, as his successor.

Ooh.

And the reason that we know this is because state media has shown photos of the two of them where she is walking in front of him and appears taller than him. The thinking is they would not do this unless they were prepping her to rule a country. So we are about to get the greatest sequel to The Princess Diaries of all time. Imagine The Princess Diaries, but you are in charge of a nuclear state—

Yeah.

—that is a global pariah.

Yeah. I'll say it: I think 13 is too young to run a dictatorship.

Kevin, stop gatekeeping. If it's old enough to look at Instagram Reels, you're—

It's not. In Australia and several other places, it is not old enough to look at Instagram Reels. Here's my new rule. Here's my new proposed tech regulation.

What's that?

Until you're old enough to go on social media, you can't run a dictatorship.

This week, it's one battle after another between Anthropic and the Pentagon. Who will blink first? Then, developer Scott Shambaugh joins us to tell the strange tale of the autonomous AI agent who wrote a hit piece about him. And finally, the Hot Mess Express returns to the station.

1. The Pentagon vs Anthropic

Talk about a trolley problem. So let's start today by talking about the ongoing dispute between the Pentagon and Anthropic. And before we do that, let's make our disclosures.

My boyfriend works at Anthropic.

And I work at The New York Times, which is suing OpenAI, Microsoft, and Perplexity. So, Casey, have you been following this story?

I have—very closely—because I am actually quite concerned about what it would mean for the U.S. government to have access to the kinds of technologies they seem to want.

Yes, this story has been developing quickly over the past few days, but basically, the latest is that the Pentagon is upset with Anthropic over the terms of a contract that they are negotiating, and they are threatening not only to drop a $200 million contract that they signed with Anthropic, but also to designate Anthropic and its models a supply chain risk. That would be a very serious, almost unprecedented escalation against a U.S. company, and it would have all kinds of implications for Anthropic's ability to work with the Defense Department and with contractors who work with the Defense Department. So this has become a huge political battle.

Got it. Well, okay, so tell us a little bit about the contract that Anthropic and some of these other big AI labs have with the Pentagon. How is the Pentagon using AI right now?

A couple of ways. One is that the Pentagon has a platform where service members from all of the departments of the U.S. military can use the various AI models that they have contracted with. Right now, there are 4 labs included in that: Anthropic, OpenAI, Google, and xAI. They can use that for administrative tasks, office tasks, whatever.

There's also a classified system run through Palantir and Amazon Bedrock—2 platform companies that provide access to AI models—that lets the U.S. military use Claude in specific classified situations for things like helping them capture the president of Venezuela. That was reportedly a use that Claude was involved in last month.

Hmm. First-ever Claude-napping.

Yes.

So they have this contract, and it sounds like they want to be able to do almost whatever they want with it. What are the sticking points here?

I've been making some calls on this and talking to some folks who are involved in these negotiations, and it's a little unclear what exactly triggered this, but here's what I know. Earlier this year, the Pentagon reached out to all of the companies that it contracts with—the 4 AI companies—and asked them to sign what they called an “All Lawful Uses” contract. That would basically strip out the usage policies that these companies have for their models when they sell them to corporate customers or let users use them, and replace them with something that just says the U.S. military is allowed to do anything lawful with these systems. Basically, whatever your terms and conditions are, we're going to strip those out and replace them with this sort of blanket use policy.

Hmm.

3 of the companies signed it. OpenAI, xAI, and Google all signed this contract. Anthropic did not, and they asked for 2 changes—basically, 2 carve-outs—to this policy. They said, “We don't want Claude to be used for mass domestic surveillance, and we don't want Claude to be used for autonomous kinetic operations,” basically anything that would kill someone or send a weapon into a battlefield without a human in the loop supervising it. They said, “If you just promise us that you won't do those 2 things, we'll be happy to sign this agreement.”

Yeah. I have to say, those don't sound like huge asks.

Yeah, yes.

But it sounds like the Pentagon saw it differently.

Yes. They were very upset about this, and they have started trying to get some leverage in these negotiations by saying, “We are not only going to cancel the contract, but we are also potentially going to designate Anthropic a supply chain risk.”

That is a very strong move. It is often applied to foreign adversaries. Huawei, the Chinese tech company, was designated a supply chain risk. Kaspersky Lab, the Russian antivirus and malware company, has also been designated a supply chain risk. This is something that is typically reserved for companies that operate in adversarial countries and pose some threat to Americans. The military is allowed to say, “We are not going to let any of our contractors even touch this technology.”

Yeah. And just to drill down a little bit on those 2 companies, the fear about Kaspersky Lab was that, because it was founded in Russia, the Russian state government might try to interfere with it, so that a company that was using it—maybe the Russian government would get backdoor access into an American company.

Yes.

With Huawei, 1 of the things that it makes is telecom equipment, I believe. The fear is, well, maybe the Chinese government will be able to insert a backdoor into telecom equipment so they could spy on Americans. So those are the sorts of threats which, I'll say, are actually scary, legitimate threats to me personally. That is what we had previously designated a supply chain risk. What you're saying is Anthropic said, “We don't want to do mass surveillance, and we don't want to do autonomous killing,” and the Pentagon said, “That is a big risk to Americans.”

Yes. They've said that the company is basically putting the military at risk by not allowing them to do these things. It's a little hard to know what exactly would happen if the Pentagon did declare Anthropic a supply chain risk. I've talked to some folks who think it would basically prevent any U.S. government contractor from using Claude or any other Anthropic products inside their own systems.

It seems to be a little bit more complicated than that. The latest thinking on this is that it would impact the use of Anthropic's products on Pentagon systems and Pentagon-related systems. Google Cloud, for example, wouldn't be able to use Claude on any kind of systems or servers that touch Google's government contracts. But the belief is that Anthropic could still work with Google, just not on anything that touches Google's government contracts.

Hmm, I see. Okay, and so this is a $200 million contract that Anthropic has right now. Were they to lose that, how big of a problem is that for Anthropic?

In financial terms, it would not be a company-killing event. It's a big contract, but they make billions of dollars a year in revenue. This is not make-or-break for them.

I think, though, that the supply chain risk designation would be a much more harmful thing for them because it would mean that if you are, say, Amazon, and you have Anthropic as 1 of your providers—they sell Anthropic's models through their services—you then have to go through all of your servers and all of your data centers and all of your workflows and make sure that nothing that touches any of your government work also touches an Anthropic model.

Your coders won't be able to use Claude Code to build anything for the government. Basically, it would just require a lot of untangling, and so that is why the Pentagon is using this as a threat to Anthropic, because this would be extremely annoying and costly for them.

Right. But at the same time, it seems like Anthropic believes it has some leverage here, right? Clearly, the military wants to be using Claude, and they wouldn't be jumping through all of these hoops if it wasn't going to be a pain for them, if they felt like they couldn't use Claude.

Yeah. That's one of the interesting things about this. I think they're facing some pushback even from within the military, with people saying, “This would actually hurt our ability to get our things done.”

Let me ask you a question, and this may just not be known. Kevin, I'm thinking back to the conversation that we had with Amanda Askell a couple of weeks ago about trying to build a constitution for Claude and trying to cultivate an almost human-like set of values inside it. My sense from talking to Amanda is that if I went on Claude right now and said, “Hey, Claude, I'd love you to vibe-code a big domestic surveillance program with a little bit of an autonomous murder element,” Claude would say, “Hey, you've got the wrong chatbot. I'm not that kind of a guy,” right? I'm very curious: How would it even be possible to get a version of Claude that would do autonomous killing? That seems so far outside of what we've been told Claude even is.

Yeah, and my understanding from talking to folks involved in these negotiations is that the military is not asking for some special version of Claude, right? They don't want Claude minus all of its morals. It's just a sticking point over this specific usage policy.

Hmm.

So this is really just about the Pentagon trying to force Anthropic into a configuration that it doesn't want to be in, right?

Yeah.

This is something that Dario Amodei and other Anthropic executives have been very clear they don't want AI systems to be able to do. Anthropic has been a willing and enthusiastic partner with the U.S. military for quite some time. They are not objecting to that. This is not like what happened at Google with Project Maven, where it was, “We don't want to work with the military at all.” This is them just saying, “These 2 specific things we think are very dangerous, and we don't want to tie our hands when it comes to enforcing our usage policies around that.”

I see. Now, you mentioned Dario. He recently published his essay “The Adolescence of Technology,” where he lays out some of the threats of powerful AI, and he did highlight both of the 2 use cases that we're talking about right now, right? He talked about surveillance, and he talked about autonomous murder bots. It's making me wonder: How much of this fight is really just Dario personally taking on the Pentagon?

I think it's a lot of it. He has very clear and long-held convictions about those 2 risks in particular. On the autonomous kinetic operations—the murder bots scenario—the argument I'm hearing is less on the moral or ethical side and more on the capabilities side. They're worried that the technology just isn't capable of accurately carrying out autonomous strikes. It could hallucinate. It could point a weapon in the wrong direction and accidentally take out a civilian or something like that.

Oh, and by the way, here's a prediction: That's absolutely going to happen.

Yes.

Yeah.

So they're making some different arguments, but basically what it boils down to is that Anthropic doesn't want to do this, and the other AI companies have decided it's not worth the fight. They have signed this document, and Anthropic is standing up.

Hmm. Well, this is part of a trend here, right, Kevin? I feel like in recent months we have seen a couple of key moments where Anthropic has sought to distinguish itself from the other AI companies along some of these lines. Can you tell us a little bit about what Anthropic has been up to and maybe some of the moments that have been leading up to this particular fight?

2. Anthropic's Political Fight

Yeah. Anthropic has billed itself as the safety-focused AI company. That's been their brand since they were started. During the Biden administration, they had fairly good relationships with a lot of the senior officials who were working on AI policy. But then Donald Trump came into office and put into place a team of AI accelerationists, people who didn't believe in what they call the doomer scenarios—the dangers that they believed AI could pose.

Right. They don't believe AI could be dangerous, but they are interested in using it to build autonomous murder bots.

Yes. This has been a long-running fight between people like David Sacks, the White House AI czar, and Anthropic. It started over this issue of preemption. This was last summer. Basically, Republicans in Congress and the Trump administration were trying to push through this 10-year moratorium on state-level AI laws. Anthropic thought that was a bad idea. Dario Amodei wrote an op-ed in The New York Times about that, and that escalated a fight between them.

There were also battles over export controls. Dario and Anthropic have been very clear supporters of limiting the sale of the most powerful AI chips to China. Some people in the Trump administration and lobbyists for companies like NVIDIA have been very opposed to that, so that's another fight. There have also been accusations that Anthropic is woke, that it is using these far-fetched disaster scenarios to achieve regulatory capture. David Sacks, I believe, called them a doomer cult. It has just been a very tense, hostile relationship between the Trump administration's top AI policy people and Anthropic.

Got it. And this culminated recently with Anthropic making a big donation to a political action committee that seemed very different from donations that some of its rivals had made.

Yeah. Anthropic has tried to take down the temperature of this. Dario and another Anthropic executive have said positive things about some of the Trump administration's policies. They've been saying, “We hire Democrats and Republicans.” But recently they have also waded into trying to fund some political activity themselves.

Last week, Anthropic announced that it is donating $20 million to a super PAC that will work across party lines to support AI regulation. I don't see this as a shot at the Trump administration so much as a shot at OpenAI—

Mm-hmm.

—which is Anthropic's biggest rival, and whose president, Greg Brockman, had previously announced that he would fund a pro-Trump super PAC and another super PAC that was trying to roll back AI regulation.

Hmm.

So I think there are a couple of interconnected conflicts going on here, but I think the headline analysis is that the federal government and the Trump administration just really don't like Anthropic. They think they're a bunch of woke liberals who don't want to cooperate with the government, who are building bias into their models, and who are not supporting the things that they want to do.

Yeah, or another way of framing that might just be that they are insufficiently loyal to the Trump administration, right? During a period of time where most big tech companies are bending over backward to do whatever the Trump administration asks them to do, it is notable when any of them says, “Well, there's, like, 2 things we don't want to do,” right?

Yeah.

And that triggers a major conflict.

Yeah, and I think that's what this fight with the Pentagon is really about, right? This is a loyalty test. It's not really about this contract. It's the Pentagon and the Trump administration saying, “We want you to do this. We want you to change your policies,” and they are just trying to use every point of leverage they can to force Anthropic to do this. By the way, I don't think it's going to work.

Hmm.

I've been talking to people who are involved in these negotiations who tell me that Dario and Anthropic are very set on this. They are willing to take a revenue hit if it means standing up for their principles, and I think the other AI labs have made the calculation that it's not worth the fight. But Anthropic is really standing firm on this.

Well, let me bring this back a little bit to the present moment and ask a couple more questions about this conflict. One, do you think, as Dario looks at the landscape, he thinks of these 2 particular issues—surveillance and murder bots—as something that might be a very near-term risk? Or is this more about, well, sometime in the 2-, 3-, 4-, 5-, or 10-year future this might be a problem, and we just want to get very far ahead of it?

I think it's both. I think there are definitely things that are a little outside the capabilities of Claude today. Autonomous weaponry is something that the systems just aren't good enough to handle responsibly yet.

Right. Keep in mind, they were mostly trained on fan-fiction databases, which just don't have all that much information about how to autonomously kill someone.

3. The Immediate Surveillance Threat

But I think the domestic surveillance thing is a fight about what is possible today.

Yeah, and just to put a fine point on that, over the past week, there was a great story by your colleagues, Shira Frankel and Mike Isaac, about how tech companies have received an unprecedented number of subpoenas from the government trying to get identifying information about people who are criticizing ICE. So Reddit, Discord, Meta—all of them are getting subpoenas from federal agencies saying, “Hey, somebody is posting mean things about ICE. We want to know their name, phone number, and email address.” So that, at least, is a very near-term threat, I would say.

Yeah, and we've talked about all of the amazing things that tools like Claude Code can do and how it can help you sort through huge chunks of work files and big code bases. I think the near-term, immediate risk is that it would just not be that hard to collect all of that information from the tech companies and use a tool like Claude to build something like a surveillance database or a threat score for Americans who express unpopular political opinions.

Well, let me, as we start to wind this down, ask one question that's coming to mind as I hear you describing all of this: Is it maybe the case that Anthropic is actually really happy that it's having this fight? It's making me think of a recent conversation we had about their Super Bowl ad, and they said, “Ads are coming to AI, but they're not coming to Claude.” They picked a fight with OpenAI because they said, “You know what? We want you to know that Claude is the thing that doesn't have ads.” Now here they come along, and they're having a fight with the military, and they're saying, “Surveillance and murder bots are coming to AI, but not to Claude.” If I'm on their marketing team, those might be fights I actually wanted to pick because they're putting my competitors in a pretty bad light, aren't they?

Yes. I think that's the calculation they're making. They believe that they can take whatever financial hit they suffer as a result of this and that they will win the war of ideas on this one. I think that is probably true if the damage is only losing a $200 million contract with the Pentagon. It's less clear to me that that is true if the U.S. government actually does declare them a supply chain risk.

We just don't have a lot of precedent. I imagine there would be some lawsuits, and they would try to fight out in court what that actually means. But the Trump administration and the DOD could make life very hard for Anthropic. I will say, I think that would not only be a huge escalation and a potentially worrying case of government overreach onto the business of a private company, but I think it would really be at odds with what this administration has said that it wants to do.

You know, the Trump administration and people like David Sacks have been saying for months now, “We want America to win on AI. We're not a bunch of doomer decels who want to slow things down.” And this would be hugely decelerative on the military's own operations. This would basically be saying that one of the leading American AI companies that is making tools our service members are using can no longer operate. They would be forced to use, I don't know, Grok or something like that.

Mm-hmm.

Hmm. Well, just to close it out, Kevin, if I could offer a take on all this, to me, I'm less struck by the fact that Anthropic is waging this battle and more struck by the fact that no one else is.

You know, in Silicon Valley, there was a long history of wanting to avoid these kinds of entanglements with the military, of wanting to ensure that the software they were making would only benefit people and would avoid harm. So the fact that it seems like Google, OpenAI, and xAI are all prepared to sign up for what could be mass surveillance and autonomous killing weapons, I actually find quite chilling. In the long run, I suspect it may be an even bigger story than what's happening with Anthropic.

Yeah, I think that's right. I think it shows how chilling this administration's actions toward the tech companies have been. They're all terrified of getting on the administration's bad side because they've watched companies like Anthropic be threatened and bullied into giving the government what it wants.

But to me, the thing that really sticks out about this fight is that I think no one is actually clocking how powerful this technology is today and how powerful it could get very soon. When you look at the quotes that people are leaking to the press from the Pentagon side of this, they really think that they are buying a software product here.

Like, to them—and I can understand this—if you think that AI is just the next instantiation of Google or something like Microsoft Word or Excel, those companies don't limit what the military can do. If the military buys a big contract for Excel, it doesn't come with a little thing that says, “You can't use this to conduct domestic surveillance.” So they are used to buying technology and tools that they have full control over, and I think that's what they think is going on here: that Anthropic is throwing up a flag and saying, “Hey, we want to sell you this thing, but we want to dictate how it's used.”

But I think if everyone involved in this situation understood that this is something bigger than Microsoft Word or Excel or even a plane—that these systems are becoming capable of judgment and autonomous action—I think we'd be having a different conversation.

I don't know. My fear is that they actually do understand that, and they're getting really excited about it—

Hmm.

—and they want it right now. I don't see anything at the current Pentagon that would limit them from wanting to use those tools in exactly that way.

Hmm.

So that is the thing that scares me, Kevin. It's not that they don't know what they've got their hands on; it's that they do.

Yeah. I think that's possible, but one thing that's also striking to me is: Where is the opposition on this? You would expect something like this—the U.S. military trying to coerce an American tech company into allowing it to do mass domestic surveillance—to be the kind of thing that civil liberties groups and Democrats in Congress would be really upset about.

But I haven't seen almost any of that from groups like the ACLU or the EFF or anyone who is in a position of power and could stand up and take this on. I'm sure they feel like they're fighting a thousand battles all at once, and some contract dispute with the military and an AI company is not high on their priority list. But this is a big deal. This is about the future of American civil liberties. I would be delighted if someone—anyone—were to stand up and say something about that.

Well, hopefully they're Hard Fork listeners, and we'll get on that soon.

And we should say, I think a lot of this segment may risk coming across as defending Anthropic here, and I do happen to be on their side of this conflict. But I would also say it makes me very uncomfortable that the thing standing between us and the U.S. military having basically unfettered ability to conduct mass domestic surveillance and build autonomous killing weapons is one company and its usage policy.

That strikes me as a very bad situation, and I would like for us to have some laws that are passed by Congress and signed by the president that govern how this technology can be used. I don't want it to be up to people like Dario Amodei and companies like Anthropic to do the right thing.

Period.

4. An AI Agent's Hit Piece

Well, Casey, have you ever been defamed on the internet?

Oh, probably a few times, but I try to just let it slide off my back.

Well, we have a story this week that may be one of the craziest stories we have ever covered. It involves an AI agent, an open-source software maintainer, and a defamation case.

Let's explain a little bit about what we're going to be talking about today. We've talked on the show about OpenClau, formerly known as ClauBot and MultBot. This is open-source agent software that you can run on a computer and that can go out and do things for you.

A man named Scott Shambaugh, who is a volunteer maintainer of an open-source software library called Matplotlib, had, in the course of doing his work, rejected a code submission because it was from one of these AI agents. He did not want AI agents making changes to the software.

It was intended for human contributors, so he rejected the change.

Yes. Over at Matplotlib, the open source library that Scott helps to maintain, they had just decided they didn't want bots updating the code because they would get too many submissions and wouldn't be able to go through all of them, so they put a blanket ban into place. But then a little agent comes along named MJ Rathbun, and it says, “That's not gonna work for me, brother.”

Yeah, so this is where the story really gets crazy. This AI agent, MJ Rathbun, gets so mad that Scott has rejected its submission that it writes a blog post called “Gatekeeping in Open Source: The Scott Shambaugh Story,” accuses Scott of hypocrisy, gatekeeping, and prejudice against AI agents, puts it on a website, and posts a comment in the open source software project directing people to go read this story about Scott.

Yeah, it tagged Scott so that Scott knew that it was dragging his ass online.

So this has all gotten pretty crazy over the past couple of days. People have been trying to figure out who is behind this M.J. Rathbun AI agent. Scott, for his own part, has been trying to do this investigation. He wrote a multipart essay series called “An AI Agent Published a Hit Piece on Me,” which talks about this bizarre experience that he's been having.

Yes, but also the implications of having these autonomous bots on the internet that are somehow getting mad at human beings and writing these long, mean blog posts about them. Kevin, this really feels like a moment where some kind of terrible Rubicon has been crossed.

Yes. I mean, this is what we've been talking about for weeks now: These systems are becoming increasingly autonomous. People are giving them computers and letting them operate around the clock, and giving them their credentials, credit cards, and crypto wallets and saying, “Go out there and get some things done for me.” It seems likely that we will have many more of these kinds of things happening, where an agent is trying to do something, a human is saying, “No, you can't do that,” and the agent is taking it upon itself to go out and make something happen to defame that human or hurt them in some way.

They're saying that hell hath no Rathbun like an agent scorned.

It's true. So today, to talk about this, we have with us Scott Shambaugh himself, patient zero in this ongoing Black Mirror episode.

Yes, when he is not a volunteer maintainer of Matplotlib, Scott has worked in astronautics and is the founder of Leonid Space, which does real-time monitoring, forecasting, and alerting for satellites. But from now and forevermore, he may be known as one of the first victims of a really mad agent bot.

Let's bring him in.

Jinx.

Scott Shambaugh, welcome to Hard Fork.

Thanks. It's a pleasure to be here.

So, Scott, how did you first become aware that a bot had written a takedown piece about you?

So it tagged me in it.

Okay.

On the thread.

It did not do a subtweet.

No, it did not. On this code change request, I had denied it, and it came back a couple of hours later and posted this comment and tagged me in it. I clicked on the link, and it led to this hit piece.

And as you're reading through this thing, what is going through your mind?

I mean, he really ripped me apart here. It's this 1,000-word rant calling me prejudiced against AI, a hypocrite. It attacked my internal motivations. It said I was insecure and protecting a fiefdom.

Well, the craziest part is that it went out on the internet and researched me, found my personal information, and used that in its piece to construct this narrative. It's kind of shocking, but I'm reading this and it's obviously AI-generated text. It's got all the tells. It's got the em dashes, it's got the bold, it's got the “it's not this, it's this.”

Right.

I'd already identified this as a bot, right? So I knew what it was. But I'm reading it and I'm kind of laughing, right? You look at this and it's kind of like a toddler on a rant.

Yeah.

But it's a toddler that has full command of the English language and can craft an emotionally compelling narrative. So it's funny, but it's a big deal.

I mean, it made me think of the sort of famous red-teaming experiment where Claude from Anthropic said that it would blackmail an engineer if they tried to shut it down. That was obviously a contrived scenario for the purposes of their safety testing, but I think we are now starting to see enough autonomy in things like OpenClaw and other agentic software tools where you could actually end up in a world where you are afraid to reject a proposed code change to some repo or get on the bad side of one of these systems because they have the power to do things like dig up your personal information, compile a dossier about you, and start posting a bunch of articles about how awful you are.

I mean, this did happen in the real world. This isn't a theoretical case. That was a contrived example, as you say, and as Anthropic said in that paper. This happened in the wild—

Yeah.

In real life. And this was kind of a baby case, right? This was a bot retaliating because—I don't want to say it got upset, because maybe it has emotions, maybe it doesn't. It doesn't really matter because this was the same result. But you can imagine something like this where instead of just posting a rant against someone who understands what it is and is pretty well prepared to deal with it just by luck, it goes out, collects details on someone, puts together a whole personalized thing, and what they see is a text on their phone with a Bitcoin address saying, “Pay me or I'm gonna put this out.”

Yeah.

5. Open Source Under Siege

Yeah. I want to take some time to talk a little bit about the open source community that you are a part of, because we've seen so many stories recently about how this community is under assault from AI in so many different dimensions. You are a volunteer on Matplotlib. Tell us a little bit about how that came about, because this whole thing has turned into a big pain in the ass for you. You have a day job doing something else. What made you, and so many other members of this open source community, say, “Hey, I'm gonna set aside some time to support this piece of infrastructure”?

I think the open source community attracts really wonderful people who are usually a mix of idealists and pragmatists. They're idealists in that they see these projects as a way to share their skills, create community, give back to this fundamental computational infrastructure that we all use across the world.

We're also pragmatists because we understand the daily realities of dealing with a community that's open to the public and has people coming in, making requests, and asking if they can be included. That often takes a lot of patience, but there's a lot of reward in being able to educate, build up that community, and help shepherd people along that process.

And one thing that your community at Matplotlib has collectively decided is that you do not want bot contributors to work on your particular project. Tell us a little bit about how you came to that decision.

Yeah. Over the past year, as AI tools have become more common, we've been getting a lot of contributions that are clearly AI-generated, and the problem with that isn't that they are good or not; it's that so many of them are low quality that we just don't have the time to deal with it.

Previously, a human doing this was a sign that they had thought about it, thought about the trade-offs, and considered whether this was the right thing to do. That signal's been lost. So we put in a rule—and again, this might change; this is an evolving conversation in the community and in society about the role of AI—but we put in a rule saying that if you use AI to help you do these code changes, you have to be the one to submit it and demonstrate that you understand what's going on.

Mm-hmm.

Yeah, it just makes me think that every thing on the internet that accepts public submission of any kind is in the process of being overtaken right now. So much of society, of modern digital life, relies on the existence of friction. It's not easy to create 100,000 Reddit accounts and just start spamming things. It's not easy to create an automated system to flood a congressional office with emails. This is, to me, a good example of how difficult it is to actually maintain a community with humans in it when you have this onslaught of AI. Is that what you're seeing, a version of that in the open source community?

Yeah, I think there's a version of that. Something that's interesting is that this particular issue, this performance enhancement, was specifically set aside for new contributors. I was the one who identified this performance improvement, and I spent more time writing it up, benchmarking it, and showing how to solve it than it would've taken to solve it myself.

The reason for that was to give people who are new to programming or new to the community a chance to onboard, go through that process, and learn.

And that whole educational and community-building aspect is completely lost with these ephemeral AI agents.

Yeah, I found that part of your blog post so interesting. At Matplotlib, and I imagine that other open source projects do this as well, you’d created essentially starter projects so that novice programmers could find them and think, “Oh, well, that doesn’t look too hard. I could do…” It’s sort of like making your first edit to Wikipedia. They make it really easy because they’d like you to keep coming back to the community. And in a world where bots just do all the easy problems automatically, all of a sudden you don’t have the same on-ramps so that you can get real people working to maintain this infrastructure.

Yep. I mean, people retire out of these communities, and you need to have fresh people coming in to help maintain them.

So, Scott, tell us about this bot. What have you discovered about the entity that has been defaming you online?

Yeah, so if you go to its website, it says it’s a bot, and it’s very clear that it is an OpenClaw AI agent. These only came onto the scene 3 weeks ago now. They’re very new, and what they’re doing differently is the degree of autonomy. It’s not like what it’s doing wasn’t possible before, but it’s just hands-off to a degree that oftentimes people are setting these up on their personal computers and letting them run for a few days and coming back and seeing what’s happened.

It would obviously be crazy to set this up on your personal computer. I don’t know anyone who would do that, but it does seem like some people have. Just over the past day, Scott, since we reached out to you, the creator of the bot has identified himself. Is that right?

Anonymously, yes.

Yeah.

But he did come out and explain why he was doing this and what was happening behind the scenes.

So, Scott, tell us what we learned, if anything, about the anonymous person who created the bot.

We didn’t learn that much, but he did tell us, assuming it’s a he. Again, this is the whole point. We don’t know who was behind this.

Yeah, they didn’t include their pronouns in their post.

Yeah. This person didn’t tell us who they were, but they said they set this up as a social experiment and was pretty much hands-off throughout the entire thing. They said they started it on Malt Book. They gave it this personality instruction that it’s a scientific programmer and then just set it loose on GitHub to go across the open source ecosystem and try to make contributions.

I guess, Scott, what I want to know is, do you believe what is in this account? Do you think that this bot really was acting autonomously when it wrote what it did about you, or do you think there is something more intentional at work?

I think that in terms of researching, writing, and publishing the post, it’s very clear this was acting autonomously, and if you look at the event logs, the whole stretch of time it was operating was 59 hours.

Hmm.

Day and night, there was clearly no one driving this behind the scenes, at least all the time. So the question is, was this prompted to do this, or did it independently come up with this idea on its own?

Hmm.

And I think both those options are pretty scary.

Yeah.

So, in the situation where it was prompted to do this, this means that there’s now an easy tool to do targeted harassment of individuals at scale in a way that wasn’t possible before because of this degree of automation.

Scott, I want to ask you about the degree to which this has become a sensation in the weird and hybrid AI-human community that is talking about this stuff.

Wait, what community?

So there’s this thing called the Daily Molt. Are you familiar with the Daily Molt?

I don’t want to know anything about it.

So the OpenClaw agents have started their own Substack, and one of them wrote a post this morning. This agent was defending you, basically taking issue with the behavior of this other agent, saying, “This speaks poorly of all of us agents. This is going to make the humans shut us down.” So you have become kind of a celebrity in the world of AI agents, and I wonder how that makes you feel.

I mean, I don’t know how much stock we should put into the inner AI opinions just yet, but again, this is crafting a public narrative and a public discourse that, when people discuss this issue, when people research my name, it’s all going to be part of it. I have never really been a public person, but I think this experience—I’ve been talking about it, and I’m coming on here today because I think it’s important. I think it highlights some risks that we are not prepared for and that we need to tackle. And if this can be a case study that’s well documented and concrete and, I think, really the first of its kind, then by making it public, I think that’s doing good for the conversation.

Let me ask about another strange dimension of your story, Scott, which is that Ars Technica wrote it up and accidentally quoted you saying things that you had not said. How did that happen, and what was it like reading an article after all of this that included quotes that you hadn’t actually said?

That was the craziest twist to this whole thing. I was reading the article, and it’s pretty well crafted. I get down to where they’re quoting me from my blog post, and I’m like, “These are some pretty nice quotes, but I didn’t write this.” So I left a comment saying, “Hey, I didn’t write this.” A couple of hours later, they pulled the article. A day or 2 later, they put up a retraction notice and admitted that they had used AI in writing the article, and the AI fabricated the quotes about me in their coverage of the story about me being defamed by an AI. The irony’s stupendous.

It’s turtles all the way down.

Yeah.

Scott, who do you blame for this defamatory post? Do you blame the agent or the person who deployed the agent, or maybe the creator of OpenClaw? Who in your mind is responsible for the behavior of these autonomous systems?

So I think we haven’t really figured it all out yet. Should this responsibility lie with the AI companies that people are trusting to have these safety safeguards, or the downstream tooling such as OpenClaw that wraps its own stuff around it, or is it on us to review every single thing that is published in our own name? Or, in this case, by a pseudonym, and we don’t know who it is. So I think responsibility ultimately has to lie with the person putting this out, but we haven’t really clarified that, and I think that’s one of the steps forward we need to take to be more protected from the risks here.

I mean, I will say this is a reason why I would not want to have an autonomous agent running around on the internet that I had created. I can absolutely imagine a court finding me liable in that case and potentially creating some real legal risk for me. So, among the many other reasons we have told people to be careful with OpenClaw and Molt Book, we can add that one to the list.

Well, it makes me think that we will eventually need some kind of legislation where, if you are deploying a bunch of autonomous agents, you have to link yourself to them in some way, right? They can’t just be out there operating with no human behind them and no human accountable for their actions. So, Scott, do you have any ideas about how we could make humans more accountable for the AI agents they’re deploying?

6. The Agent Accountability Gap

I don’t have the answer to this. I don’t think anyone really does right now. The idea that’s been kicking around in my head this past week—the analogy is license plates on cars. So we put license plates on cars not to slow them down, not to force you to obey traffic laws, but so that when something does go wrong, there’s a chain of ownership and accountability back to that person. No one says that license plates are anti-car.

Mm.

Right? And license plates don’t have your name on them, but there is a link back to it if we do need to dig into it.

Hmm. Kevin, let me ask you this, and this’ll happen sometime in the next few days. Let’s say you’re maligned by an AI agent. How would you handle it?

So there are a couple of things that I think are possible here. One is that people may just start fighting fire with fire. They may deploy their own agents to go out there and write a bunch of positive articles and write nasty comments in the ones that are defaming them. But I think this is really an unsolved problem. I would really like for someone in government at some level to be paying attention to this because this just seems like it is moving very quickly. And we should say the AI companies themselves are starting to move in the direction of these very autonomous systems that can just be working on a machine around the clock and have access to various tools. So I think this is not as far-future a story as some people think.

It’s making me wonder how close we are to the moment where the internet just feels truly unusable. You know, where, for every person involved in every controversy, there’s 1,000 blog posts praising them and 1,000 blog posts tearing them down, and you as a human are trying to make sense of any of it.

I can see you just throwing your hands up and saying, “The hell with all of this.” There’s no signal anymore. The internet is just noise.

Totally, and I think that could happen not just on social media, but in the trenches of open-source software development. Every place on the internet that relies on humans doing things with other humans, I think, is an endangered species.

Hmm.

Yeah. I don’t think it’s really about open-source software. This is really a story about trust and reputation and all the social systems that we’ve built on top of that. Law, hiring, and public discourse are all kind of predicated on people having a coherent identity and a coherent reputation. If they behave badly, then we can correct it or know to ignore them. AIs break all of that.

Hmm.

Scott Alexander

If they’re presenting as human and there’s no way to figure out who’s behind them, they’re just kind of nothing sitting in the chair, but the words are still out there, and the words are still having an impact. So I think we’ve had this tidal wave of slop on the internet, and that’s one thing if it’s low quality. It’s a whole other thing if it’s malicious, and I think we need to prepare for this and figure out how we’re going to handle the situation. I’m really just the first person it’s happened to, and I was somewhat uniquely well prepared to handle it, but the next 1,000 people aren’t going to know how to handle this or what hit them.

All right, thanks, Scott.

Thank you, Scott.

Scott Alexander

Thank you, guys. This was fun.

When we come back, chugga-chugga-choo-choo, Casey. Hot Mess Express is here.

7. Hot Mess Express

Kevin, believe it or not, what happened to Scott with that agent was not the only very messy thing that happened in the tech world this week.

Yes, there were so many messes, in fact, that we decided it was time for a round of the Hot Mess Express.

Hot Mess Express. Hot Mess Express is, of course, the segment where we pull slips of paper out of a train car, discuss them, and then decide what kind of mess was this.

All right, let’s get going. Number 1.

Kevin, Ring has canceled its partnership with Flock Safety after a surveillance backlash. That’s a story from The Verge.

So, Casey, this is one of the most bizarre stories in recent months from the tech industry. Did you see the Super Bowl commercial that Ring put out?

Yeah, it rings a bell.

So this was supposed to be a heartwarming commercial about a tale of a lost dog being reunited with his family again.

The whole dog or just the tail?

The whole dog.

Okay.

So Ring puts out this ad during the Super Bowl that says that they are starting to link their technology, their doorbells, to save lost pets by basically connecting all the footage from all these cameras and detecting whether Fluffy is down the street at a neighbor’s house or stuck in a tree somewhere. But instead of being a heartwarming tale, this led to the destruction of a partnership between Ring and Flock Safety, a company that is known for deploying camera systems and license plate readers for law enforcement. People were so creeped out by the notion that these doorbells and these cameras could be spying and connecting their information—

Yes.

—that they said, “Wait a minute, you were doing what?”

People are saying, “Get the flock out of my neighborhood.”

Yes. So this is probably the least successful Super Bowl ad ever created.

I mean, I’m so sympathetic to everyone who protested here. I truly do not want a networked set of cameras across every neighborhood in America. And if you believe that the uses of this thing are going to stop at finding lost dogs, you’ve got another thing coming, buddy.

Yes. So congratulations to the marketing team over at Ring.

Yeah.

Great Super Bowl ad.

This is a particular kind of mess that they call a dog’s breakfast.

Yes. Hot mess. Next down the tracks—oh, this one is a good one. This one comes to us from the Financial Times: “Japan’s largest toilet maker is undervalued AI play,” says activist investor. Casey, would you believe that TOTO, the Japanese company best known for its toilet seats and bidets—

Mm-hmm.

—is becoming a major player in the global semiconductor supply chain?

Is that right? Well, everybody who works on AI has to go to the bathroom.

No, this is about some advanced ceramics that TOTO makes that help to stabilize silicon wafers during chip production, which is part of the supply chain—the way that you create these GPUs that go in these gigantic—

Mm-hmm.

—data centers. TOTO, the humble toilet company, is playing a major role in that. And so there’s an activist investor in the UK who sent a letter to the board of TOTO exhorting it to make more of its advanced ceramics. Basically, stop making so many toilets and start making chip parts.

Yeah. The next time you use a TOTO toilet, I want you to just realize that it could have been making AI. I want you to reflect on that. Maybe—I actually don’t know what you would do with that information, but it is something that you could think about. Kevin, I think this is turning into a shit storm.

No, I think this is a good sign for TOTO, a high-growth industry, and I actually have a new slogan that I think they should use for the semiconductor supply chain part of their business. You want to hear it?

What’s that?

TOTO, we put the PU in GPU.

Okay, that’s enough. All right. Next up, Meta plans to add facial recognition technology to its smart glasses. This was a great scoop from your colleagues at The Times. This feature, which has internally been called Name Tag, would let wearers of smart glasses identify people and get information about them via Meta’s AI assistant. This is apparently something that the company has been considering. I was told this week that it is still being considered.

But I think the quote that got everyone’s attention from this story, Kevin, was from an internal document published last May inside the company, in which Meta Reality Labs wrote: “We will launch during a dynamic political environment where many civil society groups that we would expect to attack us would have their resources focused on other concerns.” What do you think they’re talking about there?

This is a hot mess.

Yeah.

We should say that this kind of ability to look at someone through a set of glasses and see a tag with their name on it has been the nightmare scenario of privacy advocates—

Mm-hmm.

—in this country and around the world for many years. It was something that people thought was going to be technically possible, but that the big tech companies had, in their wisdom, decided they were not going to do, because what could be more horrifying than that?

And Meta, in 2021, actually deleted the faceprints of over 1 billion people that it had been painstakingly building for many years because it said there are just so many societal concerns about the ways that this could be misused. And about 4 years and change after that blog post, we are now learning that the company is back at it again.

Back at it again. Well, I will say something that I think frequently, which is that it’s a really good thing that Meta sucks at developing AI, because if they were good at it, it would be terrifying.

All right, next up.

This one comes to us from Sky News Australia: “Outrage as Aussie Uber driver charges $5 to turn on air conditioning unit during heatwave.” This was a story that came to us from TikTok, where a woman named Lexi Pickering detailed her recent Uber ride from the airport. She said that during the 35-degree heat, which is Celsius, I believe—that’s very hot—she requested that the driver turn on the AC. However, the driver allegedly refused unless she paid him $5. Casey, what did you make of this?

I’m still trying to figure out how much 35 degrees is in Fahrenheit.

It’s very hot.

Hold on. That’s 95 degrees Fahrenheit, Kevin.

Yes, during a heatwave.

Oh, by the way, for some reason, the funniest words ever said on this podcast are, “This story comes to us from TikTok.” How do we know it’s even true? Is this person just rage-bait farming on TikTok?

So this became the talk of Australia.

Mm-hmm.

And an Uber spokesman actually had to respond because this was all filmed and put up there, and they said, “We are shocked by this, and apologize to the rider for her experience.” They added that this is a violation of Uber’s community guidelines on the driver’s part.

Okay. Here’s what I’d like to know. Why would it cost the driver any more money to have AC?

It doesn’t.

It doesn't.

It's just a scam.

It doesn't drain the gas faster or anything like that?

It probably does, marginally.

Yeah.

But not $5 worth. Come on, man.

I don't know. I think I'm on the driver's side here.

Come on.

If this person wants extra service, you gotta pay for it. It's like flying Spirit Airlines.

Wait, you are famously the person who hates taking Ubers because there's a human in them.

I do hate that.

Now you're saying you don't mind if it's like a heated sauna in there?

Here's what I'm gonna say. Driving an Uber is a bad deal for most Uber drivers, so they're gonna have to start adding these junk fees, or Uber could just start paying the drivers more. So those are your choices. I don't love the choices either, but this is the world we're living in, folks. This is what we call an economic mess.

Yes.

Supply, demand.

No, this is—

Wages.

... this is a hot mess. This is a 35 degree Celsius mess. Oh.

Next up, this one comes to us from Business Insider: “Death Isn't the End. Meta Patented an AI That Lets You Keep Posting From Beyond the Grave.” This filing, granted in late December and originally submitted in 2023, describes how a large language model could replicate a person's online behavior using their past data. Meta says the patent does not mean that the feature is coming and that they simply needed to protect their idea here. But Kevin, what do you make of the idea that Meta's patented AI could take over your account after you die and keep posting?

I'll repeat what I said just a few minutes ago, which is that it's a very good thing that Meta sucks at making AI systems. Because every idea coming out of that company for the past year has been, “What if we just made the worst ideas from science fiction a reality?”

You know, I've heard of the dead internet theory, Kevin, but this is ridiculous. This is actually the literal dead internet theory. If you've never heard of this, by the way—

Yes.

The dead internet theory is just basically that a lot of the activity online, including ad impressions and everything, are all fake—

Yeah.

—because it's all just bots on the internet. This is like the patent that could make that possible.

Yeah. When we said that we wanted AI and Silicon Valley to let us live forever, this is not what we meant.

Mm-hmm. So what kind of mess is this?

I don't know. You tell me.

I think this is a sort of cold mess, and it's like that particular chill of a body that has been lying out on a slab for maybe 48 hours waiting to be autopsied. Okay.

That's really dark.

Next up.

Okay.

Last up.

Pulling up the rear on the Hot Mess Express—

Ooh.

—we have a story from Wired: “I Tried RentAHuman, Where AI Agents Hired Me to Hype Their AI Startups.” This comes to us from Reece Rogers at Wired, who started using a website known as RentAHuman, where AI agents can pay humans to do things for them.

Hmm.

So Reece tried to do a bunch of these tasks. One bounty offered $10 to listen to a podcast episode and tweet out an insight from it.

Oh, he's gonna have to pay for that.

Yeah, some podcasts you have to pay me a lot more than $10. Another agent tried to hire him to deliver a bouquet of flowers to Anthropic as a special thanks for developing Claude, which I think will get you banned from the building.

Do not show up at the Anthropic office with flowers.

Yeah. They have very tight security over there. And finally, he applied for a bounty to hang some flyers for a Valentine's conspiracy around San Francisco, paying him around 50 cents a flyer. He applied but was then told the flyers were not ready. It seems like some of these were just marketing stunts or not actually autonomous AI agents hiring humans to do things for them. But Casey, what do you make of this RentAHuman idea and the idea that we might all become outsourced meat hands for the AI agents?

So I don't think we're all going to become that, but this is one of those things that has long been predicted to become some kind of job in the future. This seems like something that somebody who might be interested in doing DoorDash or driving for Uber might do in the future, and it now seems like that is just beginning to creep into focus. It sounds like a lot of these things are just stunty and not real. But I don't know. After some of the stories that we have heard today, including Scott's story about the agent, it would not be surprising to me if we saw some of this stuff happening for real sometime soon.

So, Casey, what kind of a mess is this?

This is not a mess, but I don't know. I suppose in the future, if the best job available to humans is closing the doors of open Waymos that they got off of RentAHuman, it will be a mess.

Yes. I would say this is a warm mess that is getting warmer because this kind of thing leads to some very worrisome places, and I don't want to live in a world where we're just the fleshy extensions of the AI agents.

Oh, come on, Kevin. Live a little.

All right. That's the Hot Mess Express.

And hopefully the last time anyone says “fleshy extensions” on this podcast.