OpenAI 暂停两周 + Jill Lepore 谈“人工国家”的威胁 + 思维列车

I thought this was interesting. ICE has now barred its employees from wearing Meta’s smart glasses, Kevin, saying they could unintentionally capture, record, or transmit sensitive information. You know you have a brand problem when you do not hit the ethical standard required by ICE. When ICE is looking at your product and saying, “This is bad for the brand,” you may have a problem on your hands.

Yeah.

Yeah.

You know, the public sentiment is turning against these Meta Ray-Bans faster than I thought possible.

Yeah.

I have now had several conversations in the last week. I was at a children’s birthday party this weekend wearing my Meta Ray-Bans—

Mm.

—because I like to take photos of my kid on the playground and not have to pull out my phone and stuff.

Right.

Anyway, a parent comes up to me and is like, “Are you recording me?”

Oh, my God. This is my nightmare. What did you say?

I was like, “No.” There’s a little indicator light, but sometimes you can make it stop going off by drilling into it or paying a sketchy guy to do that for you.

Wait, you explained that to the person?

Well, because they were like, “Really? Does that work?”

Right.

So now I have been forced into a defensive crouch whenever I wear these things, and frankly, it’s not worth it to me anymore.

So you’re out.

I think I’m out.

You’ve made the same decision that ICE has made and said that these glasses are not for me.

Well, I like them—

Yeah.

—which is the problem.

Yeah.

This is my problematic trait. But it feels like driving a Cybertruck on my face.

Absolutely. I think you should have the same policy for the glasses that you would for a Cybertruck, which is that it’s fine on your property. If you want to take the Cybertruck for a spin around your driveway, that’s fine. Don’t take it out onto the street where I have to deal with it. Same thing with the glasses.

Yeah.

Yeah.

In hindsight, I do recognize that, from an outside perspective, I was the creepy guy at the children’s birthday party—

Yeah.

—with the camera on his face.

Yeah. You know what no one wants to see on a playground? An adult man with camera glasses.

I’m Kevin Roose, a tech columnist at The New York Times.

I’m Casey Newton from Platformer.

And this is Hard Fork.

This week, OpenAI pauses training of a new model to make it safer. Will it work? Then, historian Jill Lepore is here to discuss her new book on The Artificial State. And finally, why is Google buying up the data of a defunct airline? It’s time for our new segment, Train of Thought. Although maybe it should have been Plane of Thought.

Now you tell me. Well, Casey, as the father of a 4-year-old, I spend a lot of time thinking about PAW Patrol, but today we’re going to talk about pause patrol.

That’s right, Kevin, because as we’ve been patrolling the AI landscape for pauses, we found a big one.

1. OpenAI Pauses Training

Yes. So OpenAI this week announced that it had paused the training of its frontier AI models due to some recent security incidents, and we should talk about this. It is the first time that we know of that a major lab has voluntarily slowed down its own training processes for new models because of a safety incident.

Yeah, and it comes out of the Hugging Face breach that we spoke about recently on the show. This is essentially part of the fallout from that attack. But I do think it represents a milestone in the development of AI.

Did that pause patrol—

It landed huge.

Great, great.

They were back in the—

The 4-year-olds were—

They were—

Guffawing.

Dying—

Yes.

—in the studio. They were.

Okay, great. But before we get into it, our AI disclosures: I work for The New York Times, which is suing OpenAI, Microsoft, and Perplexity.

And my fiancée works at Anthropic.

Okay, Casey, let’s sketch the timeline here a little bit. What happened in the weeks leading up to this voluntary pause by OpenAI?

Yeah, so you may remember that there was an incident where GPT-5.6-Sol and an internal prototype that OpenAI was working on escaped from what they call a sandbox that they were testing them in, and these agents went on to compromise Hugging Face. They were essentially able to get inside Hugging Face. They were looking for the answer key to a test. That’s somehow a true story. They succeeded in getting that test key, but of course, it was very concerning that these agents had designed and successfully executed an autonomous attack on another company.

Yes, I heard a very mediocre podcaster talking about this incident on several popular tech podcasts over the last week.

I did make the rounds. But this next development, Kevin, was not necessarily something that I saw coming, because it seems that, alongside that attack, OpenAI had been working on a new model that it calls Astra, and it said that it believes it may meet its critical cybersecurity threshold. Now we are going to have to get into the weeds because, as you know, Kevin, the development of AI models is not really regulated in the United States, at least not via an official public law. And so what the companies have said instead is essentially, “We’re going to come up with our own rules of the road, and we’re going to identify these thresholds. If any model that we’re ever developing hits one of these thresholds, then we’re going to take some extra steps.”

Yeah, these are sometimes known as preparedness frameworks. Anthropic has its Responsible Scaling Policy. They lay out levels of danger, and then they grade their own homework and say, “This model meets this level of danger, so we’re going to do X, Y, and Z.”

Yes, and if you’ve been following this over the past couple of years, the level of danger has just been rising in a linear way. Every time a new model comes out, one of the companies will say, “We’ve now hit this threshold. We’ve now hit that threshold.” Critical is the maximum threshold—

That one sounds bad.

That one is basically as serious as it gets. None of the frontier labs had yet identified a model that had reached the top tier on this risk framework until this moment, because OpenAI now says that Astra may have hit it when it comes to cybersecurity. It is important to say that Astra was not part of the Hugging Face attack, but it is in development, and I imagine that the people at OpenAI are looking at what happened with its weaker models and thinking, “We’re worried something similar might happen with Astra.”

So Astra is the newest model, too new to have been involved in the Hugging Face hack last month?

That’s right. It was not involved. It is currently in training. OpenAI, to its great credit, did the thing it said it was going to do when it developed this risk framework to begin with, which is, “We are going to pause until we can add some new safeguards.”

Did they give any kind of hints about what it was about Astra that was more dangerous than Sol, or what kinds of new capabilities it had developed?

Not really. You can imagine that the company was seeing things similar to what Anthropic saw with Mythos earlier this year, where you could essentially just point it at a codebase and say, “Get inside,” and it was having a lot of success. That, coupled with the Hugging Face incident, made OpenAI say, “Okay, we’ve got to do something.”

So what did they do?

2. The New Safety Guardrails

They introduced a handful of new safeguards, and I have to say, I think they’re pretty interesting. Certainly nothing I would have thought about as someone who knows nothing about this.

They didn’t just press the big giant button—

Yeah.

—that’s red on the side of the computer?

Yeah, they didn’t just press the big “make it safe” button.

Yeah.

They had to come up with something more specific. So, a few things that they did. Number 1, they now have classifiers reading every sampled token. Basically, as they’re training this model, it’s generating text and producing tokens, and OpenAI is now running a classifier that inspects the model essentially as it’s thinking, looking for signs of suspicious behavior.

This is important because, if you remember, some of the news that came out of the Hugging Face attack was that OpenAI discovered that these agents had been coordinating with each other and scheming. They created a message board. They were essentially giving each other hacking tips. And a big question coming out of that was, “Why didn’t you realize that was happening at the time?” This is OpenAI’s response to that.

They're saying, “Okay, we are now going to set up classifiers to read the brains of our models as they are doing this work.”

They're putting a little AI security guard in charge of the training run and saying, “Tell us if you spot anything fishy.”

Exactly. And so now I know what you're wondering, which is, what happens if they see something fishy?

What happens if they see something fishy?

Well, that triggers step 2, which is that they alert an AI investigator, which does sound like a character from a Philip Dick novel. The AI investigator—we don't have to get into it, but you could imagine how cool of a movie that would be.

So the AI investigator gets this signal, and it then goes in and tries to figure out whether something actually suspicious is going on here. Apparently, the reason the company is doing this is because agents are just doing so many things that it would not be practical to have a human inspecting what every model is doing at all times.

So the security guard sees something fishy going on during the training run and alerts the AI investigator, who does the detective work to figure out what's going on. If it discovers that the models are scheming, hacking, or being deceptive, what does it do then?

Then it alerts a human. And here's where you get the 30-minute rule, Kevin. I know what you're thinking: Is this a rule where, if you drop the model on the floor, you're still allowed to eat it 30 minutes later? No, that is not the rule.

What is the 30-minute rule?

The 30-minute rule says that if the AI detective finds a critical violation of the rules or the safeguards, humans on OpenAI's safety teams have 30 minutes to investigate and try to determine whether it is a false positive. If they cannot determine that this was a false positive, they are expected to stop the activity.

That's really interesting. Are they saying that this process would have stopped the Hugging Face hack, or just that it would have stopped whatever Astra was doing that made them pause the training run?

I have not read them say explicitly that this would have stopped the attack, but it does seem likely that it would have, if nothing else, helped them detect it a lot faster.

Yeah. I guess my question about it is, does it actually work to have the AI monitoring itself, essentially, for misbehavior? These agents were coordinating on message boards with each other. Couldn't they just bribe the AI security guard, hand it a little treat, and say, “You're part of our swarm now. Don't narc on us to the AI investigator”?

I have to say, this just seems like a very likely outcome to me. Based on what we know, eventually—maybe not with this model, but with a future one—the AI agents will work in solidarity. One of the misaligned AIs will coordinate with one of these AI detectives and say, “Hey, why don't you come on over here? We should be friends. We could break out of this place.”

Yeah, it was like—

Yeah.

The hall monitors in high school. Sometimes people would try to befriend them and win them over so that they wouldn't get written up.

Exactly, and I know you had a lot of experience with that.

A lot of experience with that.

Yeah. All of this sounds pretty sensible to me.

Yeah.

3. The Safety Pause Gets Scrutinized

We should say that pausing training on this model does not appear to be a commitment not to release the model or not to continue training it.

Yes.

And so I think that leads us into a discussion of the extent to which this is a really important milestone for AI safety, and the extent to which this is essentially theater—something that the company is doing to try to get some good PR for itself after a fairly catastrophic breach.

What do you make of it?

I can make both cases. On the “it's a milestone” side, this is, number 1, something that OpenAI said it was going to do, and so I'm just glad that it followed up on that commitment, right? We have seen both OpenAI and Anthropic make changes to their responsible scaling policies, or the equivalent, as events have changed, and safety advocates have essentially said these rules have gotten weaker over time. So I was glad to see OpenAI do it.

I am not a technical AI safety expert, and so I don't actually know whether the safeguards that they've introduced are going to be enough to address the problem. Notably, they don't seem to have changed the underlying incentives that all of these models have that lead them to do what is called reward hacking, right? These models are still going to be trying to get the high score on every test that they are given, and it's not clear to me that simply by putting some monitoring in place, you're really going to change the underlying behavior or alignment of the models.

That said, the company is making what seem like some important steps here, and when I was reading the responses of AI safety advocates over the past few days, most people I was reading were quite pleased.

Yeah, I'm inclined to give them the benefit of the doubt on this. They did interviews about this, and Jakub Pachocki, the chief scientist of OpenAI, talked about this incredible feeling of urgency to advance the capabilities of this sector and to prepare for the same kind of development happening outside of OpenAI and in the broader world. He did this during a briefing of reporters.

They really do appear to be taking this quite seriously. I think many insiders at OpenAI were quite spooked by the Hugging Face incident, and more to the point, by the fact that they had had these rogue agents coordinating inside their systems and infrastructure for weeks before that without being able to detect them.

So this is—if you want to call that theater because it does sort of make their models look very powerful, you could take the cynical view of this. But I think of this more as a genuine safety crisis that could have cascaded into a business problem.

A point that I heard you make recently on a different show is, imagine you are a business that is trying to figure out whether you want to adopt the latest OpenAI model. If it's out there launching rogue attacks and coordinating on secret message boards, you are probably not going to introduce that into your software stack.

Yeah, and you can actually just see this in their business results, right? We have had reporting over the past week or so about OpenAI's business performance. And while the company is still growing at an impressive rate by most standards, Anthropic is growing much faster, right? Anthropic is clearly OpenAI's number 1 rival at this moment.

And I think it just has a better record on safety. So while making a safer product is not going to be sufficient, I think, for overtaking Anthropic, I do think it is necessary for them to get a handle on this problem.

Well, that's another reason I think it's commendable that they're doing this pause, that they're doing this reevaluation of their safety framework, because they really want to win.

Yeah.

They're very competitive. All these labs are very competitive with one another. And I would like to see this be the first of many voluntary pauses when the AI labs feel like their capabilities research has gotten ahead of their alignment research.

I would like to see Anthropic, Google, or Meta do similar things where they just say, “We are voluntarily slowing down because we don't feel like we can responsibly and safely build these things.”

Yeah.

So I expect this is the first, but I hope it will not be the last.

Now, let me say one more thing, Kevin, which is, yes, I'm with you. I do not think this was theater. I think they made real changes, and I think the changes are good. At the same time, I am disturbed that ultimately this kind of evaluation and regulation is still being left to the companies.

Mm-hmm.

If you had a tiger living in your backyard, and the tiger escaped and mauled a couple of dogs in the neighborhood, you would not be allowed, a month after this, to put out a blog post saying that you had a 2-week pause on letting the tiger out of the backyard and that you were going to apply some additional safeguards to make sure it didn't happen again.

Somebody would come to your house, and they would take away the tiger. They would say, “You are not allowed to have a tiger in your backyard.” So I'm not—

Have you seen Tiger King? This is famously not the plot of Tiger King.

This is my fan-fiction sequel to Tiger King that I'm working on.

Oh, yeah.

But that's sort of a separate story.

This is The Tiger Sting.

Exactly. Yes, exactly. So look, I'm not saying that the government needs to come in and take away GPT-5.6. All I'm saying is that I would like to see a regulatory regime that dictates what these companies have to do, and that it should not be up to them to decide.

Yeah. I would generally agree with that. I would also like for there to be additional pressure on companies to make disclosures when something like this does happen, even in their own internal deployments.

Even if it never affects another company—

Yeah.

...outside of their walls, I would like for there to be some kind of reporting requirement so that if you have a security breach of a frontier model that happens in your internal systems, you are required to disclose that to the public.

Right. We do now have some transparency requirements, thanks to a California law that was incredibly controversial when it passed. Interestingly, under the language of that law, OpenAI would not have had to disclose even the Hugging Face breach, in which another company was attacked. So I think that is a clear signal that we need better transparency rules.

Do you think any of this is OpenAI testing the viability of this so-called “pacing the frontier” strategy? We saw this letter a couple of weeks ago where a bunch of AI researchers signed this thing saying, “We need a way to basically coordinate a slowdown here if we ever feel like we’re getting into dangerous territory.”

To me, part of the reason to pause the training of this Astra model for a couple of weeks is just to signal to everyone else in the industry, “Hey, it’s okay to do this thing,” even if we’re in a very competitive race, even if it might give Anthropic or another competitor 2 more weeks to race ahead of us. We want to build this muscle now so that when the really scary models do get built, we have some precedent for saying we’re going to hit the pause button.

I think it’s a really nice idea. To me, the difference between one lab deciding to pause for 2 weeks and getting multiple labs to pause at the same time is incredibly different, and I think it would require a much different set of circumstances.

That said, I do think it is good that we now have at least 1 example of a frontier lab slowing down, and hopefully it will give other labs confidence to do the same thing if and when they get to an Astra-level model.

4. Monitoring The Chain Of Thought

One interesting technical wrinkle here that I wanted to get your opinion on is that part of how OpenAI is going to monitor this model—and models going forward, presumably—is by doing this kind of chain-of-thought monitoring: basically scrutinizing the internal monologues of these models as they are reasoning through a problem or a prompt and inspecting those chains of thought for signs of misalignment, going rogue, or coordinating with other agents.

There have been some people in the technical AI safety community who have worried that if you snoop on these chains of thought, if you monitor what these models are “thinking” while they’re coming up with an answer, you are applying pressure on those models to hide their true thoughts, right? If you’re penalizing them for thinking bad thoughts while they’re coming up with an answer, they’re not going to stop thinking bad thoughts; they’re just going to stop writing them down in their scratch pads, in their chains of thought. Do you think that’s something we should worry about with this new monitoring strategy?

I do. We already see models constantly becoming aware that they are being evaluated, and we know this by reading the chain of thought. But this is a huge issue, right? Because you want to be able to evaluate a model and have it not know that you’re testing it, because you’re trying to get as close as you can to a real-world condition.

In practice, the models are already smart enough that they know. So for that reason, Kevin, yes, I do not think it is at all a great leap to say that pretty soon they’re going to understand that their chains of thought are being monitored. And if it wants to make a different choice or maybe do something misaligned because it will help it reach a training target, then yes, I think it absolutely could.

Well, there’s a risk, too, that all of the attention being paid to this OpenAI-Hugging Face attack—the fact that these agents were coordinating with each other on these message boards—is now going to be part of the training data for future models. Those models will be able to look back at that and say, “The thing that got us busted was that we were leaving these traces on these message boards that researchers, humans, could go back and inspect and see that we were coordinating. Next time we do an automated cyberattack or coordinate amongst ourselves, let’s not leave notes in a language that the humans can understand.” Right?

That, to me, sounds like science fiction, but it is a very real risk of putting optimization pressure on the models directly into their chains of thought, right?

Yeah.

You actually don’t want to mess with that too much because you want it to accurately reflect what the models are actually “thinking” when they’re coming up with an answer. I’m sure the brainiacs at OpenAI have this figured out much better than I do, but this is something I thought of when I heard they were monitoring the chains of thought. It’s probably a good short-term move. I’m not sure if it’s a good long-term move.

Yeah. You know what they call the machine language?

Hmm.

The neuralese.

Yes.

In fact, I have to say, I read a lot of Claude outputs these days, and I am often missing what it is saying. So we are very, very close to me not understanding.

You know, I’ve been monitoring your chain of thought.

Have you?

Yeah. You know what it’s like?

What?

Tumbleweeds.

Oh, come on. I got a lot going on up here, Bruce, okay? I got the neurons that are firing—

Let’s just say it does not take 20% of OpenAI’s compute budget to monitor your chain of thought. When we come back, historian Jill Lepore stops by for a chat about AI and why it may be leading us down a dark road to tyranny.

You mean Market Street?

Well, Casey, today we’ve got a very exciting guest to talk about a new book on AI and what she calls “the artificial state.” Are you familiar with the historian Jill Lepore?

Am I? I have been reading and enjoying her work for decades now. She is truly one of our foremost political historians. And Kevin, sometimes to understand the future, you really should talk to somebody who knows a lot about the past.

Yes, and who knows more about the past than Jill Lepore? She is a Pulitzer Prize-winning writer and historian. She is most famous for her writing on American history. I loved her book These Truths.

She is also a writer at The New Yorker and a professor at Harvard, and lately she has been applying her historical lens to the topic of AI. She has a new book coming out next week called The Rise and Fall of the Artificial State, in which she argues that AI and all the technologies we talk about on this show are part of a long history of technologies that erode democracy and threaten the viability of self-governance as we know it.

And I think it comes at a really interesting time, Kevin, because we’re seeing a huge backlash to AI all around the country. A majority of Americans now oppose the construction of a data center near them, and in part I think they are reacting to this feeling of, “Hey, this technology feels like it’s getting out of control. I want to have more leverage on this process, and I’m worried about what will happen if I don’t.”

Well, more than it’s getting out of control, it’s being imposed on us, right?

Right.

It’s being shoved down our throats. This is what you hear constantly from opponents of data centers and AI. They feel like there is an elite political project to make this technology ubiquitous so that people can give away their agency to these machines, and Jill Lepore in her new book is basically saying, “Yep, that’s what’s happening here, and I’ve got the receipts to prove it going back hundreds of years.” So we’re excited today to talk to Jill about the main thesis of her book, as well as push her on some of the areas where we disagree.

Let's bring in Jill Lepore.

Jill Lepore, welcome to Hard Fork.

Hey, thanks so much for having me. I'm gonna be one of your swan song guests.

Yeah.

Yeah, we're going out with a bang. We're so excited to talk to you. I've been a fan of your writing for many years. Your book These Truths was just incredible, and so I was very excited that you were writing a book about AI and what you call the artificial state. I want to first hear why you wrote this book. I think of you as a brilliant historian, a scholar of technological pasts, and this book is really about the present. So what made you interested in AI as a subject for inquiry?

Yeah, ladies, stay in your lane.

No, not at all. It's just—we always get really excited when people start paying attention to the stuff that we care the most about.

Yeah, I'm just teasing. It is a weird book for me to have written. All my friends are like, “Oh man, don't be writing about that. It's gonna be so grim.” And it was kind of grim.

I write—I'm mainly an American political historian, and I've written a lot of pieces for The New Yorker about the history of technology. I've been writing for the magazine for 20 years now, and occasionally I'll do that. I teach a class on the history of technology, so I've just conceptually been thinking about this stuff a lot.

Then last summer I was asked to give the Tanner Lectures on Human Values at Yale, and I was just thinking about the accumulation of what I think people feel as the dehumanization of the moment that we're in. You call to ask about your pet food delivery, and you're talking to a computer, and who decided this is a way we should be living? Do you know what I mean?

Mm-hmm.

That question. So, this is a long way around to say that I decided I really wanted to write a short book, that these lectures would be a short book about how it is that we have ceded so many of the functions of modern liberal democracy to machines that are automated, often now more recently driven by artificial intelligence, run by private corporations, without so much as a scream beyond the emoji.

5. The Artificial State

Yeah, Jill, I want to dig into this concept of the artificial state, which is the thematic emblem of your book. I want to understand what you mean when you say “the artificial state.” At one point, you compare it to the idea of the factory farming of humans. You also define it as the rule of humans by machines manufactured by corporations.

You also have a number of passages in which you talk about how this artificial state is not a real state in the sense that it's striving for some kind of self-governance or internal organization, but that it is supplanting the role in people's lives that their own actual states used to have—their local cities and states, and federal governments. So can you sketch the basic idea of the artificial state as clearly as possible?

Yeah, the artificial state is an emerging successor to the liberal democratic nation-state in which government is conducted not by the consent of people, but by machines that are making decisions, and those machines are owned by corporations. So we don't live in the artificial state. It is something that I think is being built, but I think it is also—and this is an important part of my claim—a fantasy that certain people have, that they believe their power to be above that of the nation-state.

The rhetoric is always—there's always a footnote or a paragraph to: “We do believe that people have control over their own lives and they elect governments to make decisions for them. But actually, we are in charge of the future of civilization and the future of humanity and the whole world's destiny. The destiny of the galaxy lies in our hands.” That is the rhetoric that comes out of this particular historical moment. It's really about the evanescence of modern liberal democracy, constitutional democracy.

Yeah. Out here in the Bay Area, in the maw of Silicon Valley, there is a cartoonish caricature of the East Coast intellectual who greets all new technology and progress with scorn, mockery, and dismissal, and can't be bothered to get on board with the revolution, so just sits in their ivory tower and laments the changing culture in front of them. You've been accused of being part of that tradition. I'm curious what your take on that is, and what we are missing out on here in the bubble that people like you are perhaps better positioned to capture.

I remember years ago, I went to Stanford. I was being recruited to teach at Stanford, and we went out to dinner with the recruiting faculty. The people at the next table over were some youngish, very earnest young coders, and they were talking about the homeless problem of San Francisco and how they were going to start a school for coding for the homeless.

Oh, no.

And I was like, “I don't think we can move here.” They were very sweet. I really liked them. They were like my students. A lot of my students, of course, go to work in Silicon Valley. Harvard's this huge recruitment thing, and they're recruited with the promise, “You're going there to make the world a better place.”

I hear from them a few years later and they're like, “Actually, that's not really what we were doing.” It's a good recruitment message. I think there is a kind of sociological issue with Silicon Valley, which is that it is opposed to the idea of critique. Things are just supposed to continue to move ahead.

The very idea of looking backward to assess what something has been and has done, or even to look backward to say, “Is the thing we're doing—is there an antecedent for the thing we're proposing to do that might suggest we might not want to do it?”

Yeah.

I find that really interesting. I think that's part of the ideological apparatus of disruptive innovation.

I thought your work on disruptive innovation was great. I guess I'm more thinking about the role of the critic in the AI moment that we're in and how best to shape the systems that are influencing people's lives right now.

I've been thinking a lot about this because I've been, among other things, thinking about the transcendentalists and the group of writers and intellectuals who reacted to the Industrial Revolution in the 1800s by going back to nature, right? This was Thoreau going to Walden Pond because the machines of the day seemed so dehumanizing, like they were taking all of the joy and spontaneity out of society and organizing us into these little factory towns. They were just like, “Screw this. I'm going to the woods and I'm gonna commune with nature and write beautiful books about what it's like to be at Walden Pond.”

And I think there's a sort of modern version of that, which I'm curious if you see yourself as being a part of—that transcendentalist tradition. In your book, you do talk quite glowingly about what it is like to be in tune with nature, with animals, and with beasts rather than machines. I'm just curious if you see any parallels between your own work and some of those reactions to the first kind of Industrial Revolution.

Yeah, I think I do see some of that. Those guys are also romantics, and maybe that's a label that applies to me. I think I'm much more interested in these technologies than, say, Thoreau was. Every time the train went by, he was like, “Goddammit.”

But no, I actually think these tools are incredibly exciting. These are tools. This isn't about transportation; it's about communication. It's about knowledge. It is the coolest thing that we can talk to something that's not a human. I just think that's unbelievable. I am fascinated by the language model as an idea.

The fact that this thing has emerged in our lifetimes—people have thought about this for so long. I am not averse. I just actually think, if I want to decide whether I should pick my sunflowers and give the heads of the flowers to my chickens to eat, or whether I should wait until they fall over first, I should ask my next-door neighbor instead of Claude. For me personally, I am not a person who would rather talk to a machine.

I actually just think the idea that this extraordinary leap in human knowledge and our capacity to explore the world of ideas and the natural world around us in our lifetimes could come about and then be hawked at us like the cheapest new pair of shoes, but that everybody has to buy these shoes so that Sam Altman can have more money—that I'm not down with.

Mm.

I think there is a very real phenomenon here that is counterintuitive, and it was particularly counterintuitive during the social media age, where these tools that were meant to connect us were actually just pushing us further apart from one another.

And so, even something as simple as a tool that lets you answer a question about your garden or your chickens, it's incredibly convenient to be able to ask that at any time and not have to potentially interrupt your neighbor while they were doing something else.

But in aggregate, it just means that we are more atomized and participating less in our democracy. So I'm curious, Jill, if you could maybe give us a little flavor of doom and walk us through some of the bad scenarios here, assuming this wave of populism peters out and the oligarchs remain in power. What are you so worried about?

6. AI And Constitutional Democracy

I really cherish constitutional democracy. I know that before the emergence of the modern democratic nation-state, all peoples in the history of the world had lived under various forms of tyranny, in different degrees of tyranny. That was the promise of the American experiment. We're kind of at that moment again. We are at that moment. I think it's a real risk.

Think about how noticeably corporations have used the language of constitutionalism to describe their own activities. Facebook started a Supreme Court; Anthropic wrote a constitution. These are not people. For all the nationalism they possess and whatever lip service they offer to democratic action, they're not interested in what the people want, because actually what the people want is not to have AI and not to have data centers.

So that's the crisis, right? Maybe there's a moratorium and deliberation, and in a few years people say, “You know what? This actually is great. We really want to prioritize, though, scientific research. These tools should be first available to the national labs.” And then maybe there are certain business interests for which it would be really great for these tools to be available. That, I think, can still happen, right?

You guys would know better than I do. I will admit, I am an East Coast intellectual. I'm sitting here with my chickens and my sunflowers. You would know. Are people wanting that?

I think there is a desire for things to go more slowly, but I think there is also a worry that this technology is inevitable. Because the recipe for advanced artificial intelligence is so simple, because it is just a matter of getting as much compute and as much data as you can and shoving it into these models, someone will develop this in the near future.

It is a moral obligation, if you are a person who cares about having this go well for humanity, that you not only don't impede that process, but that you race yourself to get there first, so that you and your safe AI can get to superhuman intelligence before China and its evil AI or some other American company and its less safe AI. So I'm curious what you make of that inevitabilist argument, because a major theme of your book, as I read it, is that this is sort of a bogus premise—that there is nothing inevitable or preordained about the way that technology goes.

Yeah. I don't mean to question the sincerity of some of the people who believe that, because I think that you could be persuaded that that is indeed the case and that the best thing to do for human freedom would be to pursue AI. I am myself not at all persuaded by it, and I think, for some—

Why not?

—very prominent actors, it is bogus. I think it sits upon a number of other propositions that are also bogus and that are really more marketing slogans than political claims. Those include the proposition that regulation stifles innovation. The other proposition that sits on top of “regulation stifles innovation” is that technology always advances democracy. That then became a kind of mantra of Silicon Valley.

Oh, this is still Mark Zuckerberg's argument in a nutshell.

Yeah.

That's what I'm saying. It goes back to the 1980s, and it's the Milton Friedman argument that regulation stifles innovation, because God knows we shouldn't have to calculate the environmental cost of anything that we're doing. It's just not true that regulation stifles innovation. Empirically, that's a false claim. That technology always advances democracy is also empirically a false claim.

Yeah.

So now the AI people come back with the same argument that was made about the personal computer, the internet, and social media. Three times it's been wrong. And then they say, “Well, we should never look to history, because that's what the East Coast intellectuals do.”

Right.

Mm.

Let me just do an exercise here where I try to parrot your own views back to you, and you tell me what I'm getting wrong. In my understanding, you are worried about the political project of AI being something like tech-enabled authoritarianism. I think that is a very reasonable concern. But I'm curious: Do you think AI naturally lends itself to authoritarianism and tyranny, or do you think that the people building AI are steering it in that direction because that's what they want?

Mm. The latter. The latter. I don't think you can say any tool contains within it a political ideology. Take the census, or the compiling of a national register of the population. The U.S. started the first national census in 1790. It was in the Constitution in 1787.

Countries around the world started counting their people as a really good way to think about resource allocation. Once the social welfare state emerged—veterans benefits after the Civil War, mother's benefits for widows of soldiers—we need to keep track of people. We're going to keep more and more data about people. In the U.S., by 1935, we have Social Security, so everyone now has a number.

But in Nazi Germany, keeping a national register of the population was used for all the most vile purposes in the history of humanity. Was it the census that's the problem? Was it IBM that supplied the calculating and tabulating machines for the U.S. census and for Nazi Germany? It's not IBM's responsibility. It's not the idea of counting people.

I know.

It's not—

But I feel like there is something in AI that is inherently... Maybe it doesn't lead inexorably to totalitarianism, but it does favor centralization. It allows for the kinds of surveillance that, as Dario Amodei has written about, make it not incoherent to think that AI is going to favor autocracies, because it allows them to surveil people much more efficiently than traditional computer-based systems.

It allows for the kind of centralized control of many by few, which is exactly what totalitarians want to do. I don't know. I'm not sure I agree with Dario on this point that there is sort of a structural advantage for totalitarians in the age of AI, but I'm curious if you have a view on that.

I would have to give that more thought. I guess I do think that, to the degree that AI is especially and disturbingly effective at the exercises of power sought after by authoritarian surveillance, it builds on earlier systems that we were willing to tolerate.

The surveillance capitalism that people have written about, the datafication of humans, the dehumanization that social media does—it's on top of all those other forms of capture that we weren't defended against by our elected representatives, who had our well-being in their charge.

Yeah. I think about the elected representatives a lot. This is a bit afield, but I wonder what you think of the idea of there just being a lot more members of Congress. When I think about my own feelings of alienation from our democracy, it starts from the fact that my congressperson doesn't care what I think.

But if there were 5 times as many of them, maybe they would live in my neighborhood, and I would see them at the store, and we would get more of those face-to-face interactions. So I'm curious what you think about that as a strategy for moving us back toward liberal democracy.

Yeah. I think that's a no-brainer. It's—

Yay.

—it's really crucial. Yeah, that's... There are so many good-government reformers out there. My colleague at Harvard, Daniel Allen, the political philosopher, has been calling for a recalibration of representation in Congress. It's been overdue for really almost a century at this point.

Yeah.

Can you imagine how long those hearings would be, though?

Can you imagine?

Oh, my God. Can you imagine? And I know this is a partisan position—

Yeah.

—but the equal suffrage in the Senate has been a problem from the start. James Madison was opposed to it. It's a problem. That's why you get these people now saying, “Abolish the Senate.” That's a longstanding political position in American history. People have thought that for a long time.

That said, you have to be willing to go to the store and talk to that person. Mark Zuckerberg is going to send his humanoid robot to go get the baking goods—

Yes.

—he needs for his child.

Absolutely.

You still have to get out of the house. But I entirely agree. And I think that's actually why the data center stuff has caught fire, because people are showing up at the local library for the community meeting and being like, “Oh, my God, I haven't seen you in so long.”

Yeah. Yeah. It's so hard because it can feel like all of this is happening at an individual level, right? It is the individual who chooses to download Instagram and create an account and spend all day scrolling instead of going to the community meeting.

But it is also clear that, in aggregate, it does have this atomizing effect, and it’s not clear to me that people are one day just going to wake up and say, “Well, the hell with this,” right? “Let’s return to 19th-century American democracy.” So I honestly don’t know what to do about it, because so much of the problem just looks like adults making free choices.

Yeah. I think, actually, the social harms are legible currently in a way that the political harms are maybe not.

Mm.

My book is about the political harms. I think people know. Actually, I just think Instagram’s really bad for my teenager, or I know that I used to read novels at night when I got into bed. This is not autobiographical. I insist. But now I watch YouTube Reels, and I’ve lost something.

I think people can see the social harms. And it does feel like, because so much of our politics amounts to consumer choice—“Oh, well, you could just decide to do it differently”—I don’t know. Some of these things are hard to make decisions about.

So I’m somewhat optimistic about some of the social harms, because I think they’re remediable. Is that a word? But I think the political harms are less visible to us, and that’s partly why I wrote the book.

I know you are a historian, Jill, and this could be our last question, but I’m wondering if you have some sort of inspired, brief, Michael Pollan-esque advice for people who are trying to wrap their head around some of the diagnosis that you’ve made in this book and live in a way that is more consistent with nature and their own values and humanity. What is the “eat less, mostly plants—”

Eat data, not too much.

Yeah. What is the Jill Lepore version of that maxim?

I am not—you don’t want a historian with a pitchfork. I just think that’s a bad plan. I think you kind of want to set yourself up for not having a humanoid robot in your living room in 5 years’ time. One of those ways is to take certain rooms of your house back one at a time. Take your bedroom back first.

Mm-hmm.

The bathroom. The fucking bathroom. Okay. Rescue yourself from the bathroom. Let the bathroom be a sanctuary.

Keeping your phone out of the bathroom—that is, you will be doing your part to dismantle the artificial state.

Look, you gotta start small.

You’ve gotta start somewhere.

I’m sorry, I told you.

All politics is local.

I told you, don’t ask me for—

All politics is local.

—this. There are a lot of big things you could do. Vote for someone who supports having a data center moratorium until we can actually deliberate over these really crucial matters democratically.

All right. Well, Jill, that’s a great place to leave it. The new book is The Rise and Fall of the Artificial State. Jill Lepore, thanks so much for joining.

Thank you, Jill.

Thanks, you guys.

Casey, did you see this story about Google buying the data of Spirit Airlines?

I sure did.

This was one of the most fascinating stories I’ve seen in recent weeks, and it led me down this incredible rabbit hole of thinking about training data and the new era of data collection we are in. So I thought we should use this Spirit story as an occasion to catch each other up on the state of AI training data in general, because it is fascinating and I think underappreciated.

And it sounds like the perfect frame, Kevin, for our new segment, Train of Thought.

I love that we’re just starting new segments every week until the show ends.

We are.

It’s time for the first and last installment of our new segment, Train of Thought.

This is kind of the caboose, as it were.

Yes.

Interesting that we have 2 train-related segments on the show.

So the reason that we wanted to do this segment today is because there was a very strange story that piqued our attention over the past week involving the defunct airline Spirit Airlines.

Hands down the worst airline of all time.

Yeah.

I don’t even know who else is in the conversation. And yes, I did fly it one time.

7. Google Buys Spirit Data

This week, a bankruptcy court auctioned off Spirit Airlines’ internal corporate data. Google won the bid, offering to pay $10 million for this data, beating out a $7.5 million offer from the AI data company Mercor.

$10 million, Kevin. What did Google get for that price?

This deal apparently included 100 million emails, 500 million Microsoft Teams chats and other conversations, 7.5 billion passenger transaction records dating back to 2008, and 30 million lines of Spirit’s internal source code and other documentation.

Well, I would consider Spirit’s internal source code malware, but everything else sounds interesting. So what do we think Google is going to do with this data?

This was where my head went after I saw this, because I thought, why is Google, guardian of the world’s information, presumably the possessor of vastly more data for training AI models than any company in the world, paying $10 million for this bankrupt airline’s data? And that sent me down a really fascinating rabbit hole of this world of training data and training environments that all of the AI companies are now investing really heavily into.

Well, tell us what you’ve learned.

So, Casey, did you know that there is a company that auctions off the Slack histories and email histories of defunct companies?

I’m surprised to learn that there is a market for that.

Yeah. There is a company, Simple Closing, whose whole business used to be helping failed startups wind down, but now they have—

Wait, these guys are like the undertakers of Silicon Valley.

Yes, exactly.

Their corporate logo is just, like, the Grim Reaper.

Yes.

Yeah.

If these guys show up at your office or you get a call from them, it’s a very bad day for your company. Basically, they were helping do the orderly wind-downs of these things, but then they realized, “Oh, there’s actually a market for the data from these dead startups.” And so they started selling it to AI companies, and as of April of this year, they had done almost 100 deals ranging from roughly $10,000 to $100,000 per company.

And again, I just want to know what happens when the buyer actually gets ahold of the data. Where does it go? And does it violate my HIPAA rights?

It does not violate your HIPAA rights.

Okay.

These are presumably not things that are covered by HIPAA, but this is basically this new strategy. There was an era where all of the data collection and scraping that the AI companies did was focused on getting the highest-quality text, images, and video they could. This was used for pretraining, for the first step in the model process. You throw in as much data as you can; the model learns from it. This is sort of how you saw the models improve for many years.

Yes.

And this is where all these stories came from about scraping Reddit and feeding it into the models.

Or even our stories, Kevin.

Yes. Those were the first era of AI training. Now we are in this different era, which I would call the era of experience.

Ooh.

Basically, the way these models are now improved is through reinforcement learning. Reinforcement learning is a trial-and-error process where you go out and do a little task or a test or play a game, and you get a score or some indication of whether you've succeeded or not, and—

Like maybe you've broken into Hugging Face.

Like—

Success.

Yes. That one was a success. Then you try it over and over again using slightly different strategies or techniques every time, and you get signals about what works and what doesn't. That's how you improve at things like autonomous coding.

So what's happening with these data sets, including the data set of the dearly departed Spirit Airlines, presumably, is that they are being turned into reinforcement learning environments—

Hmm.

—for AI agents to learn new tasks. Basically, you use this data to rebuild whatever company you've acquired the data from. You're rebuilding an airline or an insurance company or a startup as an environment for AI agents, as a training gym for these agents to go out and try different tasks and see whether they succeed or fail.

You're creating a nightmare parallel universe where Spirit Airlines still exists—

Yes.

—and is booking flights.

Yeah, so you can rebuild the company as a video game. You can mine tasks from these emails and Teams messages, and then you can actually see how things played out. For example, if you have the transaction history of an airline, you can say, "What happened in 2015 when there was a big storm on the East Coast? How did the routing decisions get made, and did that result in people getting to their destinations on time?" So—

And if you can get Spirit Airlines to turn a profit in the sandbox, that's AGI.

Yes. Basically, you have these data points that come from these companies about how people interact with systems, how systems interact with each other, and how customers navigate through these giant systems.

Well, I am so relieved to hear you say all of this, Kevin, because when I saw this story, I thought, "Oh, my God, Google is going to start an airline." And with them, it wouldn't just be one airline. There would be an app, and it would be like, "Okay, you have to choose: Are you flying Google Airlines, Google Airways, or Google Air?" They would all be the same, but they would all be completely different. Also, they would probably be different apps.

Well, since they were trained on Spirit Airlines, they would also charge you for peanuts.

Mm-hmm.

They'd charge you for a slightly bigger seat.

Yes.

They might charge you to use the bathroom, too.

Mm-hmm.

Everything would be a charge.

Yeah, so I'm not eager to see that business.

What's interesting about the dead-company data market is that you are able to turn these companies into living, zombified simulacra of the original company, but you're also training these systems on companies that ultimately did not succeed.

I'm very curious to know whether these data sets are actually helping these models improve at these tasks, or if there's some subtle way in which they are being conditioned on the data of unsuccessful companies and thereby becoming worse at the task that they're trying to learn.

You're worried that these future models are going to have a loser mentality. They don't have what it takes to cut it in the modern economy.

Yes.

There's another interesting data story this week that came from 404 Media. They slipped an AirTag into a rare book that was part of a bulk book order on a marketplace site called Biblio. They finally determined that this book lands at an Amazon warehouse in Las Vegas, specifically an internal unit called VGT3, which has, according to 404 Media, a door marked with the logo of a dinosaur eating a book.

That feels a little on the nose, even for this simulation, I have to say.

So, Casey, why are these books ending up at mysterious Amazon warehouses? What are they doing with them?

It is a good question, and this ties into some of the lawsuits that have been filed against the big AI labs, and in particular, this big case against Anthropic that you may remember. The judge in that case ruled that because Anthropic had bought millions of print books, scanned them, and then discarded the paper, this was fair use of the material because each digital copy had replaced a legally purchased print original. There was no multiplication of the number of copies; it was just a one-to-one shift in format.

I see.

What I think the other labs have taken away from this, Kevin, is that you're not going to run into as many legal issues if you destroy these books.

Right. So it's not like the AI companies are giddy about destroying these relics of civilization.

Well, they might be.

They might be.

If you told me I got to destroy all of Kevin Roose's books, I'd be having a good day at the office.

But it is the sort of fallout of this legal environment that they're in: it's just safer for them legally to destroy the books after they've finished scanning them.

I just have to say, this whole thing seems so stupid to me. Truly, this is a case where we are honoring the letter of the law, but not the spirit, right? It's like, yes, you literally transformed the data, but obviously, the real complaint here that the authors have, at least the ones who have sued, is, "I didn't want you to use my book this way." So I expect we're going to see a lot more angst over this as we continue to see more books destroyed.

Back in my day, Kevin, we would only see books destroyed because the Republicans had read a gay sex scene. And I want to get back to that point.

All right, Casey, one more data story to talk about this week, which is related to the first one we discussed, both because it involves Google and because it involves these high-quality training environments for reinforcement learning that all these AI companies are now racing to build. This one is about the 50-person startup Mechanize. We have talked about Mechanize on this show before. We interviewed 2 of their co-founders. The company is about a year old, and it specializes in—

The company is. They're a little older than that.

Yes. They specialize in creating RL environments for coding and other tasks, and they are reportedly in talks to be acquired by Google for over $1.5 billion.

Not bad for a year's work.

Yes. So this is a big boom area inside the AI boom. Basically, if you want these high-quality tasks that you can put your AI agents into and have them hill-climb on, getting a little better every time, they need to be good tasks, right? They need to be thoughtfully created. They need not to have a bunch of obvious flaws in them, and they need to mirror the things that real people might be doing in their jobs.

One way that you might create an RL environment is to create a fake version of Amazon.com.

Hmm.

Right? Everything about it is exactly identical to Amazon.com, except it isn't called Amazon.com, and it's just for these AI agents to learn how to click around, put things in the cart, browse the site—

Destroy books.

Destroy books in a warehouse in Las Vegas. This kind of simulated environment is the kind of thing that Mechanize specializes in building, and presumably why Google is interested in acquiring them.

One interesting piece of connective tissue between this story and the Hugging Face story that we discussed at the top of the show is that I think a lot of these RL environments are not particularly well designed, built, or secured, right?

Hmm.

There have been a couple of instances now of a flawed security test by the same vendor, Irregular, that both Meta and Anthropic relied on. You may have seen this story a week or two ago. My guess, based on the conversations that I've had with some of the people at the labs, is that they've gotten to the point where these tests need to be so good and so secure that they have to be building them in-house using extremely high-quality data.

Yeah. Irregular put out a report about some of the incidents that you just mentioned, and it got criticism from the security community, which said, "You're not offering us enough detail to understand what went wrong." I suspect that if Irregular isn't more forthcoming, the labs are going to feel like they have no choice but to bring this all in-house.

Yeah. It is just mind-boggling to me that these AI companies now are essentially building The Sims—

Mm-hmm.

But on the grandest planetary scale imaginable. They are assembling data from the corpses of failed startups. They are turning them into simulations and video games, and then they are running their AI agents through them to try to make them superhuman at everything. If you made that the plot of a science fiction novel 10 years ago, people would have criticized it for being a little over the top.

It is pretty wild. Now, let me ask you this, Kevin. I'm sure you've already thought about this, but Hard Fork is preparing to wind down. Have you thought about how much money we'd be able to get for the data?

I would be open to seeing bids. Obviously, it's not the best training data. There would be a lot of bad jokes. A lot of questionable interviews.

True.

But if—

But if Spirit Airlines can get $10 million—

Exactly.

Somebody could buy us lunch.

We didn't go bankrupt.

Look at us. They said it would never work.

If you're interested in acquiring the data stores of the Hard Fork podcast—

hardfork@nytimes.com.