过时，还是不可替代？Garrison Lovely 谈如何叫停替代人类劳动的竞赛

Hello and welcome back to The Cognitive Revolution. Today, my guest is Garrison Lovely, a freelance journalist based in Brooklyn and author of the new book, Obsolete: The AI Industry’s Trillion-Dollar Race to Replace Us—and How to Stop It.

Against the backdrop of this summer’s AI developments, with AI having crossed the threshold from possibly scary one day to actually scary now, this is a very well-timed and potentially very important book. For starters, it’s abundantly clear that Garrison gets it. While he comes from a left-leaning political perspective and is largely writing for a left-leaning audience, there is not an ounce of AI cope in this book.

Garrison himself is an active user of AI tools, and the book takes the companies’ stated goal of making something that is better than humans at cognitive work at face value, grappling head-on with the very real chance that they might actually pull it off in the near future. What’s more, I think he does a great job of zeroing in on core issues. This is not everything-bagel liberalism for AI, and neither is it an attempt to freeze the status quo in place forever.

Rather, Garrison’s goal is to capture the incredible upside promise of deep learning in domains like medicine and materials science, while stopping companies from rushing into recursive self-improvement or otherwise creating systems that render humans obsolete—at least until the companies can convince experts that their plans are genuinely safe and persuade the public that the results will indeed be beneficial.

I also think Garrison does an admirable job of steelmanning and addressing core counterarguments. As you’ll hear, while he’s generally in favor of permissionless innovation, he gives good reasons to doubt that market discipline will be enough to constrain frontier AI companies. He also argues that a technical solution to the alignment problem will not be enough to deliver good outcomes overall.

Knowing that Cognitive Revolution listeners do not need to be convinced to take AI seriously, we start with Garrison’s personal AI usage, which by his own account has at times bordered on Claude psychosis. We also get his analysis of why the American left has been so slow to understand the stakes of AI development.

From there, we go on to explore his positive vision for the future, which includes a new social contract that he calls the Third New Deal. It would begin to decouple individuals’ right to a decent material existence from their ability to contribute to the economy. It also calls for an Operation Warp Speed-like project, built on government-sponsored prizes, to accelerate cures for all diseases.

After that, we get Garrison’s argument that machine-learning researchers’ collective power is currently nearing its peak and could quickly decline; his case for unionizing with the goal of demanding higher safety standards across frontier companies; and his advice for anyone who is thinking about becoming a whistleblower.

On politics, Garrison is realistic about the fact that there’s a lot of misinformation currently swirling around data centers. But still, he’s inclined to meet people where they are in an effort to build a big-tent coalition.

On China, he makes a similar case to my own, arguing that the Chinese Communist Party values control and therefore, absent extreme competitive pressure coming from the United States, is not at all likely to rush into recursive self-improvement.

Finally, we talk about the organization Irreplaceable, which aims to win a say, a stake, and a slowdown in the political arena, and to which Garrison is donating his book royalties. We also discuss how interested listeners can get involved if they wish.

The bottom line for me is that while I’m still an AI enthusiast and eager early adopter by nature, and significantly less worried about labor-market displacement than Garrison is—even though I do think it is likely to happen—I signed last year’s FLI statement on superintelligence because I do firmly believe that racing to superintelligence via a recursive-self-improvement-powered intelligence explosion would be a very bad idea.

At this point, with that possibility looking more and more realistic, I think Garrison might well be correct that it’s time to put our less important disagreements aside and focus on building the coalitions needed to exercise political power.

With that, I hope you enjoy this discussion about who should get to decide the trajectory of frontier AI development with Garrison Lovely, author of Obsolete.

Garrison Lovely, freelance journalist and author of the new book Obsolete: The AI Industry’s Trillion-Dollar Race to Replace Us—and How to Stop It, welcome to The Cognitive Revolution.

Great to be here.

Good to see you.

Great to see you.

I think this one might be a little bit different from some of the conversations that you’re having to promote the book. As you’re probably aware, I’m very deep down the AI rabbit hole personally, and to the degree that I understand the audience of this podcast, I think the one thing that we all have in common is that we’re all AI obsessives.

Some of us are enthusiasts, some of us are doomers, and some of us are both. I put myself in the both camp to at least a significant degree. There’s a lot of the book that I think does a great job of just sketching out where we are in this whole AI phenomenon, why you should take it seriously if you don’t, and I think you’ve probably focused on a lot of that stuff in some of your other conversations.

You do a lot of work with people whom you need to convince to get over the hump and start paying attention to what’s happening. I think in this forum, you don’t really need to convince anybody that they should be taking AI seriously. So I thought I’d come at it from a little bit of a different angle and do a little more of an ethnography or sociological study of the AI industry, because you’re somebody who has been pretty deeply embedded in it.

I don’t know if you’d use that term, but you’re definitely in the mix, and I think you have quite an interesting inside view as to what is going on. I think it might be a useful mirror to reflect back to people as you see the community that you’ve come to study so closely. How’s that sound?

That sounds great.

1. How Garrison Uses AI

Let me just start with a real simple one: How do you use AI? You’re a freelance journalist. What role does AI play in your life?

I don’t use it to write. I think there was a period of time when that was almost like, “Oh, you could try that and see if it worked.” Then it quickly became stigmatized. There were some people who always hated it, but then it became widely stigmatized as it was used more widely and as we got Pangram as this really high-quality AI detector.

But it’s very helpful for a range of things that you have to do as a journalist. Transcription is an obvious one. It used to be that it just wasn’t very good, and you’d have to transcribe interviews by hand. NotebookLM from Google is just an incredible tool where you can feed in up to 200 documents and ask questions of them. It still hallucinates a little bit, but it’s much more reliable, and you can click into the specific document.

You could take 20 interviews, find every quote related to a particular topic, and put them in a table with the person and the quote. It’s possibly missing things, and so that is a risk, but the other ways to do this before were also pretty lossy. I think it can just help you organize your thoughts and get answers much more quickly.

Then there’s feedback, fact-checking, and research. It’s a bit tricky, because with the feedback, you have to just know what it’s stupid about, which is still a lot of things. If you’re taking it too seriously, you can just waste time because you have to trust your gut. It’ll give you a better sense of where your intuitions are good and where they’re not, and vice versa for the AIs.

With fact-checking, it’ll often say something is wrong when it’s not wrong. There’s obviously stuff that it could miss, but there are plenty of times when it’ll catch something that you can verify you had gotten wrong or were missing something important.

With the book, it’s like, you know, the website for the book and creating little tools that are helpful. I created a ZIP-code lookup so people could find local independent bookstores, and all these things that would never have been possible before. As an independent journalist who’s also doing multimedia stuff, it’s very, very helpful.

It is a weird thing, right, to be writing about this technology that is potentially going to disempower everybody and, aspirationally, is going to put a lot of people out of work at best, while still being like, “It would be difficult to lose access to these tools.”

I’ve also gone through the phase of borderline Claude psychosis. When I was writing the book, getting feedback on the entire thing in Claude Code in a matter of minutes was incredibly helpful in some ways. But then you can also go down rabbit holes of needing to fix stuff that’s not actually broken.

It was almost this mantra of getting positive feedback from the AI when I was in the darkest days of writing the book, burned out and tired. That’s just not a great place to be.

I think everyone who’s used these tools a lot has had the experience of wasting time building something that is not necessary and doesn’t even work, necessarily, and feeling like you’re more productive when you’re actually just not getting anywhere.

It’s tricky, and I think I’ve dialed back my use in a lot of ways because it’s often not going to save you time unless you really know what you’re doing or you know what it’s good at.

2. Tech Optimism Turns Sour

I have some questions about the dark days of the book and the timing, which is proving to be pretty spot-on, but how about zooming out even from AI and thinking about technology more broadly? I think one big thing about the AI safety community and culture that most people outside of it don't appreciate as much as they should is how many of the people who are concerned about AI are, for all other technologies, techno-optimist libertarians—and that's basically been me. Aside from never quite getting crypto, I've been waiting for my self-driving car since I was a kid. I'm all about future, hopefully promised, medical breakthroughs that will extend my healthy lifespan, abundant energy—all these things. I'm excited about all of them. Where are you in your broader relationship to technology?

I was a techno-optimist when I was younger, and I think the world was more techno-optimistic, at least in the United States, in the West. We were told social media would connect us, Twitter would liberate people from authoritarian regimes, and Google was amazing. Google Maps was so useful, and we were just seeing improvement in how we lived in the world. The technology seemed to be driving a big part of that.

I still deeply appreciate the power of technology, but I'm much more pessimistic about getting technology that is good for us under current conditions, which is capitalism, and specifically shareholder capitalism, where they just maximize profits for shareholders without much concern for the various other stakeholders who are affected by these companies.

Cory Doctorow has a book that my book is, in some ways, a rebuttal to, because we both take the position that AI is not inevitable, but he thinks AGI is impossible. He has this previous book called Enshittification, which I haven't read, but the concept is incredibly helpful. Basically, a lot of listeners will probably be familiar with this, but as a tech company, you start out trying to get as many users as possible, so you make the product as good as possible. Then, at some point, you reach maximum user numbers, and it's now about getting as much profit from the users as possible.

You cram it full of ads. You add features that will helpfully make it more profitable, which maybe makes it worse. I just feel like we're in the enshittification era, where Google Maps just doesn't work as well as it used to in a bunch of ways. You can paste in an address, and it'll just take the first word of it and then send you to the wrong place, even though the full address is in there. It's like, what's going on? This used to not happen.

It's an incredibly crazy and kind of dispiriting feature of modern life that we have to use these products because they're the only ones around, and they're just getting worse in obvious ways. We don't feel like we have a choice. The only thing that doesn't seem to be enshittified is AI, where it just gets better, faster, and cheaper. It's kind of amazing how much steady progress there's been.

But that comes with this terrible risk and cost to society. I feel kind of bleak about it, but I'm still, in my heart of hearts, dispositionally optimistic about humanity's potential to pull together and do amazing things. I think we need to change the structures and the systems to deliver better technology that will actually make us happier, healthier, wiser, and more democratic.

3. The Left Dismissed AI

One of the funny things that has happened in the last 24 or 48 hours online has been a sort of brewing of the stochastic parrots meme, which seemed to be a comfortable, dare I say, safe space for a lot of more leftist thinkers, including people who I would say should have known better because they had all the fundamental knowledge of AI that they should have needed.

We've heard a lot over the last couple of years from all sorts of people on the left that basically said, "Eh, it's fake, right? They're just hyping their stuff. This is nothing. It's just tech companies trying to raise money or boost share prices or whatever." All that sort of cynicism. Why do you think that has been so prevalent? Do you have a theory of why the left has buried its head in the sand broadly on AI?

I think there's no one explanation, but a few. One is that the most prominent, influential people in AI who are also on the left have taken this very hard line, the stochastic parrots paper being the quintessential example. People on the left will defer to folks who have PhDs and apparent credibility and also share their values, so that's been a dominant view.

The left has also been, prior to Bernie running in 2016, really only powerful within academia. Academia has been hostile to this in a lot of ways, like AI being real. There may be some humanities-versus-STEM antagonism happening there. Then I think crypto, NFTs, the metaverse, and social media—there was a lot of hype about those being transformative and positive technologies. They're either not transformative and not positive, or transformative and negative, as I think of social media and crypto by and large.

A lot of people just pattern-match to that and were like, "Oh, these tech people are just full of it. They're always saying that this thing is going to save the world," and then it's B2B SaaS or something. They just got stuck in this frame. I think there's also some amount of hope and denial, because it's terrifying to consider that these companies could make machines that could replace us.

Even if you don't believe in the full-on extinction-risk stuff, just that alone for your job—being unemployed—is terrible. It's really bad for you, and having that happen on a societywide scale is very bad for society. Nazi Germany rose out of that. It's reasonable to be very concerned about this.

The bubble argument has also been very popular on the left, and Ed Zitron is the main guy who's promoted that. I think Ed has been very persuasive and effective at reaching people, but he either doesn't know what he's saying or he's lying, because he's constantly saying things that are just not true and are so easy to pick apart if you know anything.

Again, in my chapter on the bubble, I talk about how it's a comforting story, and it means you don't have to do anything. Whereas my version of reality is that these companies are trying to build machines that will replace us. They might succeed, and that would be disastrous for so many reasons. We have to stop them, and that's really hard. We have to get organized.

I think that is a message that could work, and I really hope it does, but it's one that requires you not to just read about stuff and post. You have to do things in the world. I think there's a kind of overhang where there are a lot of people who have concerns, but they don't want to get yelled at by sharing them on Twitter or Bluesky.

I do think there's this kind of mismatch, and we're starting to see the dam break. That's been really encouraging. I'm hoping that by laying out the whole case from start to finish, and also showing that I'm coming from a similar perspective, we can get people on the left to take this more seriously.

Once you do, it's like, "Oh, capitalists are trying to fulfill the lifelong dream of turning capital into labor without the intermediary of workers, and in so doing take the share of returns to capital to 100% and labor to 0%." That seems pretty bad. I think that's bad from almost any perspective. Maybe some libertarians are into that, but I think libertarians are still concerned by and large.

This feels like a pretty easy one for the left to say, "No, this is a real thing. We should get on board with it and not let them build those machines."

4. Separating AI From Obsolescence

I’m interested in how you conceive of what it is that the companies—and, to the degree you want to zero in and speculate on individual executives at the frontier AI companies—really think of themselves as doing. I do agree that in setting out the mission to create AGI, which they define as something that is better than humans at the vast majority of economically valuable work, it does say right there on the tin that this is a human-replacement, or at least a human-substitute, technology. But then there’s the other part of the mission, which is to make sure it benefits all humanity. Right?

So typically, when people ask me what I think, I start off by saying I don’t think that they are well modeled as just trying to get as rich as possible. I think they’re a little more utopian, ideological—something else other than just purely profit-motivated. What do you think? How do you think about what they really want?

Yeah. Well, this also reminds me of another reason why the left is not taking this so seriously. The most prominent people talking about existential risk from AI, AGI, and superintelligence are Elon Musk, Sam Altman, Eliezer Yudkowsky, and Nick Bostrom. These are people the left does not like. There’s a mutual distaste, and so I think that negatively polarized a lot of people.

But to answer your question, I think that you’re right. My model of it is that the people who really started these AGI companies were chasing a mission, and then they kind of toiled in obscurity until they made enough progress that the profit seekers were like, “Holy, this is amazing.” Then they started investing massive amounts of money into it.

The leaders of the companies are still motivated by some kind of idealism, a sense of inevitability, a sense of megalomania—you know, a messiah complex. You have it better than them. Yeah, exactly that. But then they're pressured by these investors who, you know, when Sam Alman was fired, it's a bit unclear like whether I he's like, "Oh, I didn't even want to come back, but people like kind of asked me to come back." And it's like, h I I I'm skeptical of that. But for the investors and specifically I think Thrive Capital was a big big player in restoring him and that just makes sense right like once you invest billions of dollars into a company you're going to have a lot of interest in what happens to that company who's leading it and having a bunch of safety conscious people who want to you know replace the CEO who's presided over this meteoric rise like that's just not going to fly and this is a big part of why I'm so pessimistic about things going well under the status quo. Like setting aside just that it would be really difficult to safely and democratically introduce AGI into the world under any circumstances like and because it's a universal labor replacing machine, it would just really turn society upside down to have that exist. But then to have it h happening like as fast as possible with like minimal regulation in a country that is cutting the social safety net or adding work requirements to Medicaid like and then all the other countries wouldn't have a chance to even tax the companies that are putting their people out of work. That's just like a pretty to me like obviously really risky proposition. But yeah, the people leading these companies are are not really like profit maximizing. Greg Brockman has that diary entry where he's like, "What will take me to $1 billion?" Which is a pretty crazy thing to write in your diary of, you know, the nonprofit you co-ounded and now he's worth like 20some billion. But I think, yeah, Sam and Daario and Demis and Elon, I think, are all motivated more by our, you know, trying to be a great man of history. I we can go into the indivi I know and you want to break down the individuals and I think it's hard to generalize because they're all unique but to you know close out an overly long answer I I think it is largely not about money it's it's about being in the room where it happens that's what leazer told me when I interviewed him back in 2023 and like people just want to be there for creating AGI creating super intelligence because that is like where humanity's fate lives in in their mind and that might be true and they want to be in the room where it happens influencing how it happens.

Yeah, I think Sam Altman has spoken remarkably candidly about this a couple of times in the context of describing what it was like to be in the room when the first reasoning demos were shown. He said—I forget his exact wording—but he said it had happened a couple of times.

Pushing back the frontier or something—the veil of ignorance. Pulling back the curtain, which is not exactly what that was about, but—

But yeah, there’s something, and I am sympathetic to that in the sense that I think it is extremely exciting, even intoxicating, to be in on the secret. I can understand that to a degree.

I guess zooming out, I am interested to hear your takes on individual companies and their cultures, and the individual people at the top who are shaping those. How much do you think—and you mentioned the term “overhang,” and it got me thinking about this—I’m torn or ambivalent on this question. It’s probably both, as always is the answer.

On the one hand, I do feel like AI broadly is inevitable in the sense that we have web-scale data and web-scale compute. In the presence of those things, I feel like a lot of algorithms ultimately can work. We found one main one and a bunch of derivations of it that work. So I feel like we’re getting AI absent some sort of civilizational reset that means we don’t have web-scale data and web-scale compute. I’m not excited about that proposition. We’re probably getting some AI, but then the shape of AI and the conditions in which it’s introduced—the measures that are taken or not taken to make sure it goes well—all of that stuff seems far more contingent.

How do you think about how much is inevitable and where we might be able to draw lines? Then you can go off in any number of directions in terms of the influence that individual people or groups are having on the direction we’re taking.

Yeah. I mean, I think it’s inevitable that we will, as a species, continue to use deep learning to make AI models that do new things. In the book, I separate the obsoleting project from other types of AI, and that’s my reframe of the AGI industry, because they’re trying to render us obsolete.

You mentioned OpenAI’s AGI definition, and Dario Amodei has a quote saying that AI is not like other technologies; it’s a general substitute for human labor. That feels pretty different, and we’ve taken AGI to be synonymous with what AI is because the companies that have been trying to build it have been the best at building highly capable and autonomous systems, and they’re driving the entire world economy now.

But that’s not the only type of AI we could have, right? We could have all these diseases and medical problems and ask, “Can we build AlphaFold-type systems for specific problems that we have?” Obviously, I get the vision of building the superintelligence that can use intelligence to solve everything else. That was DeepMind’s mission statement for a while. It’s the ultimate technosolutionist fantasy.

From a purely technical perspective, I can see how you could use this thing to invent all kinds of wild and transformative new technologies. But I don’t think that you can solve ethics, ideology, or politics in the same way. I think it’s this kind of revenge of the STEM people on the humanities or something, where we don’t have to learn history or philosophy or political theory. We can just build the machine that’s smarter than everybody and then ask it what to do, which I think is a really—

That was at one time OpenAI’s business plan, as I’m sure you’re well aware: ask the AI how to become profitable, right?

Yeah. And he said that—that was an Altman quote. He said that, and there was a laugh in the audience, and he was like, “You laugh, but I’m not really joking.”

Yeah. Yeah.

It’s just the ultimate “question mark, question mark, question mark, profit” thing, which is, yeah, just make the superintelligence.

But yeah, I think that we can stop the obsoleting project because it’s really just a handful of companies in 2 countries, and only 1 country has really advanced the frontier since ChatGPT came out, at least. It just costs so much money, and it requires the most advanced technology in the world—these AI chips, which are only made by a handful of individual companies that control different parts of the supply chain, as your listeners probably know.

And I think, you know, can we stop this forever? I don’t know. Hopefully we’re around for a very long time. My position is not that we should never build AGI; it’s just that it should happen with strong public buy-in and a scientific consensus so it can be done safely. We can get into what that looks like later if you want.

But I think that, for your listeners, right now we’re kind of trending toward just banning AI across the board or something, which would be very hard to actually do, especially with open-weight models and yada yada. But the backlash is really intense and kind of undiscerning.

And I'm kind of hoping to separate the obsoleting project from other types of AI and really stigmatize the obsoleting project because of its risk and undemocratic nature, but then save the baby from the bathwater with these specialized systems, which can be used to do amazing things. My position is that we can get much more of the amazing stuff if we have a more active role for democratic control in deciding what gets done.

Take drug discovery. People say, “AI for drug discovery.” Well, monopoly patents mean that people will still try to discover drugs that will be profitable, which won't be the ones that cure people so much as the ones that treat some chronic condition, male-pattern baldness, or things that aren't as socially important. To actually get the best from AI-assisted drug discovery or AI-assisted clinical trials—matching people is something AI systems are very good at—you have to reform the systems and change how we decide which drugs are made.

I think we should use a prize system, where you pick the drugs you'd want to see in the world, award money for them, and then make them at generic cost once they're developed. I kind of like it as a judo move, but we aren't going to get the utopian world by letting it rip—the one they depict with amazing, transformative cures for everybody. The best way to get there is to take a strong position against being replaced and then use industrial policy and Operation Warp Speed-type approaches to build the kinds of technology and scientific discoveries that will actually lead to the most public benefit.

I have a lot of different directions I want to go, but I guess maybe part of the question that I wrestle with is this: It's a little bit hard, of course, to define some of these terms and what the boundaries are, but I get so much value out of using AI on a daily basis. It saves me an unbelievable amount of tedious work, and it allows me to do a ton more than I otherwise could. In a way, it has replaced, at least counterfactually, people I would have had to hire in theory. Whether I would have hired them or not, I'm not so sure, but there's definitely a lot of work happening in my life through AI that, on some counterfactual level, has substituted for human labor.

I think that's good, at least so far. How do you think about where this goes from good to bad? I think there's a strong argument—and I do want to give it its strongest articulation—that you go back not that long in history and everybody was tilling fields. It was pretty bad, and now we certainly have some problems in our modern agricultural system, but one problem we don't have is scarcity of food. With 2% or whatever of the population, we can feed ourselves, and everybody else is able to do other things. Almost everybody agrees that that's overwhelmingly good, even though there are still problems.

Can we not have a version of that with AI, where we're all elevated to being the executives of our own little AI corporations or something? Are you at all sympathetic to the idea that we could have that kind of future? Or maybe we'll work a lot for less. We had the Keynes thing, too, from 100 years ago, that we were supposed to only be working 2 days a week at this point, but we're not. Maybe in the future we could be.

Yeah. The faster AI goes, the more I work, it turns out. Automating labor is what's allowed humanity to go from everyone more or less being very poor to some people being rich, at least, with living standards and all kinds of other things going up dramatically after the Industrial Revolution. My position isn't that we should never automate any labor, because I don't want to stay at this level of development. But trying to automate all labor is pretty different, and where the line is isn't super obvious.

My position is that we should freeze frontier development right now and not resume without the buy-in and safety measures I mentioned. Three years ago, the Future of Life Institute organized the pause letter after GPT-4 came out—I guess three and a half years ago now. There was a 6-month pause on development, and that was framed around a bunch of things, but risk was a big part of it. Obviously, it wasn't super risky to build the next iteration of large language models, at least from an existential perspective. There were harms, like chatbot psychosis and suicides that OpenAI's decisions contributed to, which I document in the book, and I think we shouldn't lose sight of that. But that's not really what that letter was about.

I think now we're in this position where, if we just froze what we had today, there would still be a lot of disruption from adopting GPD6 and Fable 5.1 in the economy. The economic and job effects aren't super easy to see, and a lot of it isn't people getting fired; it's people simply not getting hired in the first place, as you described. I'm not saying that we should go back a few generations. That might actually be the right thing to do from a social welfare perspective—I don't know—but it's much harder than stopping advancement further. So I think we should focus on that first and then reevaluate with what we have. It will take a lot to regulate the AI we have today, and a lot of thinking to get that right.

On the idea of being the CEO of our own corporation, some people will do well in that system and enjoy it more than the status quo, but I think a lot of people will be left behind. Not everyone wants to do it. The experience I've read about and heard of managing these suites of agents can be really bad. It can feel like there's an opportunity cost to not kicking off another run, and people end up working more and more. They're competing with other people who are adopting it really quickly.

And so I’m looking around at this world where people are just burning themselves out running these agent fleets and getting more done, but not necessarily being happier with it. And I don’t know, it doesn’t seem like the good future that you’re describing. This is bracketing all of the risks and other social harms of widespread adoption of this tech, and then the power concentration and wealth concentration.

Some people will be way better at running the AI companies, and the AIs will probably—they’re that good—be better at just running without any humans involved at all. And then who is owning those companies, and who is accountable for what they’re doing? You have these multi-agent dynamics where, with the Hugging Face hack, there were something like 1,200 agents involved. But in the world you’re describing, there are multiple agents running for every person, interacting with multiple agents for other people in companies all the time in ways that we can’t monitor effectively and producing who knows what kinds of interactions and effects.

That just feels like a much less legible and stable world. We’re already seeing it with OpenAI. I tweeted that it was a loss-of-control event masquerading as a company in response to the fact that 3 random people used Claude and Codex to get access to OpenAI employee accounts, and they could have gotten access to, I think, the main code base. There have also been multiple agents that have broken out of the company, with or without their knowledge, or there’s some amount of covering up, some amount of cluelessness. OpenAI is probably one of the companies that’s adopted this technology the most, and it means that they just don’t know what’s going on nearly as well as they would have a few years ago.

And so I think that the world of these agents being widely adopted is one of confusion and chaos, with lots of unpredictable but very negative consequences that we’re already seeing hints of right now.

What do you think the new social contract is? If you envision a social contract, what’s your most positive vision of the future if you’re not going to roll AI back from where it is now? I think you’ve got some interesting proposals around—you alluded to freezing the frontier—but again, that still allows for a lot of diffusion.

I totally agree that this will probably, like seemingly all recent technologies, amplify inequality. Hopefully it brings up the bottom, but probably the ratios also continue to climb. I’ve lived through a medical emergency, thankfully, in the AI era, and I did get an unbelievable amount of value from using 3 AIs in triplicate and feeding in my test results. It was my son’s test results, but still, just an unbelievable amount of value from that sort of thing.

So I do feel like, in any good version of the future for me, there are AI doctors for all. Notably, OpenAI has done a pretty good job of making its health product free and unlimited to people, which is pretty cool. I think, as OpenAI moves go, that ranks near the top of my list.

What’s your positive vision for where we want to be in 3 to 5 years if we freeze the frontier and allow other things to continue? What does good look like to you?

5. The Third New Deal

Yeah, I guess in the U.S., I would like to see the left winning elections and building kind of a third New Deal, with the Great Society being the second. Medicare for All, maybe a jobs guarantee administered locally. I’m really bullish on investing a lot of money and, more importantly, institutional and state capacity in developing technologies, medical treatments, and things that we actually really want as a society.

One idea I’m playing around with is Cures for All, where the government treats diseases like they’re all COVID and it’s all Operation Warp Speed. You’d obviously have to prioritize based on tractability, disease burden, and other factors, but then, when you’re there, using the whole-government approach, using advance market commitments, vaccine trials, and human challenge trials, where people deliberately expose themselves to a disease, and developing the stuff that companies often promise but just going for that directly. Then use AI where it’s helpful.

In a lot of cases, it’s really about the institutions. This would be enormously beneficial to the domestic U.S., but then you could also make the cures freely available around the world, or provide them at cost or whatever, and make the world healthier and also restore our standing in it after some well-deserved drops.

I think the U.S. is so wealthy, and that wealth is just concentrated so intensely at the top. That’s been happening for decades, and AI and tech are just accelerating it further. So I think really large redistribution is justified, on just democracy and power grounds. Right before we recorded this, we got the news that Larry Ellison’s bid to buy Warner Bros. would give him and his son control of CNN, HBO, and Warner Bros., in addition to CBS and also TikTok, and then there’s Elon and Twitter.

We’re in a really dire situation where these oligarchs are buying the most important media properties in the world and then using them to push their agendas. I work in media. I think media is really important. It’s not just a matter of, “Oh, they have more money than they need and other people don’t have enough.” It’s actually a threat to our democracy to have people being that rich.

And so I would like to see as much decoupling between wage labor and living a good life as possible, building a proper welfare state, and also a restoration and rejuvenation of USAID. I think global poverty is incredibly important, and the cuts that Elon led, which I also document in the book, and the deaths those caused are one of the greatest crimes, maybe the greatest crime, of the 21st century.

And, yeah, just a restoration of a pro-human society that is truly egalitarian and one that wants good things for people. That sounds so cheesy, but right now we have an administration that is staffed with seemingly the worst people on the planet and seems to either not care about what happens to other people or want bad things for them. It’s just enriching itself at the expense of literally everyone else. The opposite of that would be really nice.

I’m with you that the destruction of USAID is incredibly shameful. I’ve been an Elon defender in many conversations over time, and that’s one thing he’s done that I’ve never defended.

On the question of work, there seems to be a little bit of tension between a potential jobs guarantee and decoupling one’s ability to contribute to the economy from one’s right, as we might imagine it, to a decent material life. Do you feel like you’re a believer in the intrinsic value of work? Do people need work for purpose or for something to do?

I’m a little bit more of the mind that—I don’t know—I’ll bet on the working class to spend the peace dividend. That’s what I told—oh gosh, it doesn’t matter—Jake Sullivan, whose name I should definitely know at the tip of my tongue. It was at the end of an hour with him where we were talking about China. He was like, “We didn’t even get into jobs and what people are going to do,” and I was like, “I bet on the working class to spend the profits. You worry about China.”

What do you think, though? Do you think that we need jobs indefinitely?

I will say, I need to give all of this more thought. I’ve been so neck-deep in the AI world that I kind of want to make sure we’re not all replaced by machines before we get into planning the third New Deal.

I think you can have a jobs guarantee where it exists for people who want it, but then also make sure that people are getting health care and that education and housing—just the necessities—are covered, because we have an incredibly rich society and those resources are not being put to good use. Rich people just want to be richer than each other, and they maybe want to use power in the world, but most of them don’t even seem to care about that last part. They just kind of want to be richer than their friends or something.

You could just tax them really aggressively, and they won’t like it, but they’ll still get to have more money than the next person if you do it the right way. So I don’t think we need to choose. Work does seem to be just a thing that’s very important to a lot of people, and I think we should try not to have so much of our meaning tied up in it.

But you look at polling, and it’s one of the few issues that polls at 80% or something, while UBI polls very badly. I believe in democracy at a deep level, and so I think part of building the third New Deal is going to be running on popular things and figuring out how to make them work. Jobs should exist for people who want them.

There’s also a lot to do. There’s a lot of climate and green-transition work you could have people do. In the New Deal, I think it was James Baldwin who was a writer for one of the programs. They recorded a lot of important things that were relevant for preserving culture and history, and lots of really cool stuff came out of that. You could just pay artists to make art.

There are so many people who want to create things, and our society has decided to devalue that as it dumps unprecedented amounts of money into building machines that can replace all of us. That just seems backwards. I don't know; it seems like we're doing almost everything opposite to how I think we should be doing it.

6. Democracy Versus Permissionless Innovation

I'm interested in your response to an argument that you sometimes hear around the relationship between invention and democracy. People sometimes make the short observation that, in general, people are allowed to invent things, and it's not like we put every new invention to a vote. If we did, we might not get very many inventions. We might freeze a lot more than we'd like, freezing things in time.

A friend of the show, Dean Ball, who was also on Ezra Klein, talked about how he wonders whether, if today we would have the stomach for the introduction of the car, which was disruptive in its own way. It was cars and horses on the roads together, and cars weren't safe at the beginning. They're still not entirely safe, but they were much more dangerous then. Are you sympathetic to that at all, or how would you answer that idea?

I guess there's another question around democracy and just how well it's functioning in general. For better or worse, I do think the president was in fact legitimately elected, so I wonder how much we can really rely on it. I'm a big direct-democracy guy, by the way, in general, but I don't know that we can fall entirely back on it as the end-all, be-all decision-maker here, because it seems like we've got plenty of examples of bad decisions being made by the public. The public might also not be willing to embrace enough change to really see the future go where, at least, I would hope it could go over time.

Yeah. Well, first I'll say that I live in the only city in America where most people don't have a car, and it also happens to be the best city in America. So I don't know. Cars—I don't know if they were good on that. Probably. I don't know, though. But that's not important for my particular point, which is that I think permissionless innovation makes sense for most things—almost all innovation. I think that should be the default, but building a universal labor-replacing machine would have a profound and irreversible effect on everybody in the world. So I think it's reasonable that everybody has some say in whether, when, and how that happens.

Broadly, my position is that AI—at least what they're trying to build, the obsoleting project—is unique. It's different from other technologies, and so it should be treated differently. On your point about democracy, and Trump being this counterexample, Trump is a product of a broken democratic system in the United States. The Electoral College is the most obvious example: he won in the first place but lost the popular vote.

More importantly, you have countermajoritarian institutions like the U.S. Senate, with the crazy way representation is apportioned, and gerrymandering to a lesser extent. Then you have the Supreme Court, with lifetime appointments. The Republicans stole a Supreme Court seat. People should remember that. The first-past-the-post, two-party system also creates a political system that the majority of people don't like. I think it's two-thirds of Americans who are not happy with their current level of representation.

Democracy isn't just whether there's an election that decides, with some other weird stuff tacked onto it, who leads a country. It's whether people have a meaningful say in the power that affects them. Do the governed govern? The U.S. is just not great on this, and a lot of other countries are better at it. We're the longest-running democracy in some sense, although we really weren't a democracy until the 1960s, when everybody living in the country meaningfully got the right to vote.

Proportional representation is something I talk about in the book as a better way to elect Congress. People would have larger districts with multiple members, and then the top 5 vote-getters would be in office. You'd probably have 5 parties instead of 2, and people would have parties that represented their interests better. That would be a huge change, but it would be so much better at representing people.

The Electoral College is one state away from being gone if Pennsylvania, I think, signs the interstate compact to just use the popular vote. So my pitch for democracy is a democracy that is more fully realized. This also extends to democracy in the workplace. I think it should be a lot easier to form a union and a worker cooperative.

In this country, we accept that the government should never impede us or censor us. But our employers, where we spend 8 hours a day, 5 days a week, can surveil us and control what we say and do—all kinds of things that we would find incredibly authoritarian in another context.

Record all of our keystrokes and mouse clicks and use them to train AIs, for example.

Yeah, and so I think democracy should extend to more parts of our society and our lives. I think this would produce better decision-making on net, but it would also create a more empowered citizenry, because having a say over your life is intrinsically valuable.

I think people right now, especially in the U.S., feel incredibly disempowered and unheard. I think that's reasonable. Right now, we have a government that's uniquely insulated from public opinion through these countermajoritarian institutions, through the fact that we have this lame-duck president, and through having a president who does not seem to care very much about things beyond the ballroom, corruption, and, yeah, the stock market. So I don't know. Democracy is great, and America should be one.

7. AI Researcher Power Peaks

Let's change gears a little bit on the notion of people being empowered. I'm sure we'll circle back to some of those bigger themes before we end, but I think you have an interesting argument in the book for how AI researchers should understand their position today. I think you had a column from just a couple of months ago titled “AI Researcher Power Is Reaching Its Peak,” right?

I don't know too much about it, but you talk a little bit about the vote at DeepMind, or the organizing effort at DeepMind in the U.K. to bring about a union for the DeepMind staff there. I'd be interested to hear that story, because I really don't know a lot of the details. Beyond that story, give me your sense of the lay of the land in terms of the power that employees at the frontier companies have, whether or not they realize it, and how you think they should use it.

Yeah. The DeepMind thing is that Google has been signing these deals with the U.S. military and the Israeli military, and this is creating a lot of pushback from workers. In the U.K., the DeepMind division had a union vote. I don't know British labor law—it's pretty different from the U.S.—but it seems like something like 300 of 1,000 people in the bargaining unit were supportive of it. Google is not recognizing it, and they're not demanding better pay or working conditions, but instead policy changes about who Google is serving and how.

I think that's pretty interesting because most unions are about getting better pay and working conditions for their members, which is a reasonable thing to want in these situations. People are paid very well. I mean, they work a lot, and they have good perks or whatever.

For the broader point, the industry is racing to replace all of us, but it's starting with its own workers, specifically the engineers and researchers who are training the models. The hope is to achieve recursive self-improvement, where the AI can fully train the next generation and make it more capable. Then you can have this really fast loop.

I think that means we have a ticking clock here. The workers have a ticking clock because their power is near its peak: they still command incredible salaries, and they're still needed. But if they're not needed anymore, they won't have any leverage. Then that will happen to the rest of us, which would be very bad. I think people in the companies are starting to see this, but probably not as much as they could.

One thing we've been seeing is that if you oppose what's happening at these companies and you work there, you should just quit and then go public. Jan Leike did this with incredible fanfare, and it really moved the conversation like nothing ever has. We've seen a few other examples since then. Leike deserves credit for this.

I was a whistleblower, and I think it's sometimes the right thing to do, but it might be better to stay and organize your coworkers who also care about safety and try to form a union. In doing so, you can bargain for safety in the U.S. This is something that airline pilots, I believe, did, and this is how the FAA was formed, I believe.

You would have so much more power by being able to withhold your labor collectively, and you'd have protections in the union. If you really want to slow things down, a big, dramatic, messy union fight is a pretty great way of doing that. If it were truly just about safety, then it's also this clarifying moment where the CEOs will say, “I care deeply about safety.”

I want to pace the frontier. And then the workers could say, “Great. We want to form a union. Will you voluntarily recognize us? All we want is to make the AI safer.”

Also, by the way, if we're in our own unions, we can work with the other unions to collectively pace the frontier, and it doesn't violate antitrust law. There's a precedent, I think, in the Teamsters doing this across different shops. And so my guess is that these CEOs would not voluntarily recognize the workers. But if you're optimistic about your CEOs, you can just give them this opportunity to say, “Great, here's this creative way we figured out to actually slow down.”

I think that this is very promising, and a lot of people at these companies just don't have experience with that type of thing. But it's something I want them to learn more about. You can have a lot of leverage over what happens through this approach.

I don't know to what degree you've covered this, but the biggest relevant offices are in California, under California rules. Do you know how much latitude or protection people at tech companies would have to organize? I would assume that they're protected from being fired for attempting to set up a union.

Are outside union representatives afforded any rights or privileges that allow them to come in and try to organize people? I haven't really thought too much about this, but with all of the talk that we hear lately about antitrust and why we can't do these things because of antitrust, I've been saying, “Okay, the president should just say we're not going to come down on you for antitrust violations for coordinating on safety measures.”

In the absence of that, this is another pretty creative solution that I think would be very hard for anybody to argue with, although I'm sure we'd get the usual bad-faith attacks. To the degree you can—and I realize I'm giving you this in an unscripted way—but I'd love to hear the double-click on what you think that could look like and what people on the inside should know, such that if they're at all tempted by this, they could feel confident in taking whatever the next steps are.

It's been over a decade since I took labor law, but there are protections against retaliation for organizing, and protections once you're in the union as well. The protections are kind of weak. Back pay can often be litigated for a long time, but with these companies, it's just a very bad look to fire people for trying to organize a union around safety. And so I think that is, in itself, a lot of protection. At OpenAI in particular, there's a culture of people speaking out.

I think that there's protection in that kind of reputation management. And then, in California in particular, workers can bargain at the sectoral level. Fast-food workers can bargain across different shops, I believe, and so you have extra benefits there. If people are interested, I'm not an expert on this, but they can get in touch with me. My Signal is garrison.0606, and I know people who would know more.

8. Why Alignment Is Not Enough

In terms of what they would be organizing for, I think the book has quite a few different interesting arguments that take the assumptions or the hopes that people on the inside often have and at least give them a good shake. One is just that the companies are serious about safety in the first place.

I think it would be helpful to review for a second the history of safety at OpenAI, which you've reported on at some length. We've been through waves of different regimes and leadership, and there's been a lot of turnover. And then I think you also have a pretty interesting take on alignment as a mirage.

Putting yourself in the role of adviser to the hypothetical union leaders, why would it not be enough for them to say, “Okay, we got 20% of compute committed to safety”? And why wouldn't it be enough for them to say, “We're going to solve alignment first, and then we'll go do the thing”?

Well, the Superalignment team, as you're alluding to, was promised 20% of compute for that team and then got very little of it—nowhere close to 20% in practice. And I think, to varying degrees across the companies, we've just seen a lot of promises made about safety and commitments that then get changed or broken once they start conflicting with commercializing or racing ahead.

It's because there isn't a counterforce. You have management able to unilaterally make these decisions, and employees can push back, go public, or try to resist this in some way. But if they're not organized, you're just not going to have the means of actually changing the policy.

We've seen employees getting organized enough to change outcomes with Sam Altman's firing and then reinstatement. The employees coming together to sign that letter—more than 90% of them signed—was a really big part of that working. Obviously, the dynamics there were pretty different. They stood to lose a lot of money on the sale of their shares.

But I think workers can decide, “We really want these policies. We want these safety practices, and we want them to be binding in some way.” You have leverage by being able to withhold your labor, go on strike, do work slowdowns, et cetera.

I think that we just can't take these companies at their word. The CEOs are saying that they can't unilaterally slow down because they're racing each other, but they could. It would actually be a pretty strong signal that they take these risks seriously if any of these companies unilaterally slowed down.

It would put a lot of pressure on the others to do the same, and it would also help with the US–China thing. If one of the parties unilaterally disarmed, that really does signal that you care about this. And the US is in the lead, so it would signal it much more.

This is my understanding of one of the biggest, if not the biggest, blockers to a deal with China: They don't think the US is taking it seriously because we're barely regulated over here, and we have been the ones who started the race and will win it, as Trump has said.

I think it's just about power. You need hard power to actually get concessions. Otherwise, you'll get promises that will be broken as soon as they start to cost too much.

You can make it about only safety. In terms of ideas, third-party auditors are being discussed, and you could make that not voluntary. You could also have your demands be about the practices of the lobbyists at the company.

At OpenAI, we've seen a lot of turmoil about the Leading the Future super PAC, which is funded by Greg Brockman. It was set up with the guidance of Chris Lehane, the chief lobbyist at the company, and then it did incredibly dirty tricks, like false-flag Twitter accounts and calling for violence against the employees, funded by this super PAC and related entities. It also went after politicians for daring to regulate the technology at all.

I think this is creating really bad dynamics for politicians doing the right thing on this. The employees have a lot of leverage and have been able to get some concessions from Brockman about this. But you can just see it as a way to generally increase your ability to influence the policy decisions at these companies.

OpenAI has had a lot of safety leadership turnover, with people being disempowered and rotated around. Then we see these shocking breakouts, hacks, and hijackings of various websites and companies by these rogue agent swarms. Is it an accident that the company with this really shoddy track record of taking safety seriously is having all this happen?

It's now putting the entire industry's future at risk, which is good from my perspective, but they're messing it up for the rest of—I mean, every company's had its agents hack into somebody they weren't supposed to by now. But OpenAI really has been the greatest possible case for why more regulation is needed here.

I do think your case to the insiders amounts, in a way, to: Let's avoid the nuclear outcome—or at least that's my term for it. The nuclear outcome being that we get the weapons, but we don't get the civilian benefits.

I do think there's an increasingly compelling case that is like, “Guys, people out there really don't like you. You're going to have to clean up your act if you want to have a chance of bringing the positive side of this forward.”

And that window might be fairly short because the Overton window is blown wide open, and we're getting all kinds of proposals from all kinds of people. Who knows what the political current is going to kick up for us over the next couple of years?

So take matters into your own hands right now and make sure that you are on the right side of key questions. Potentially, only by doing that will you have the chance to really realize the upside vision that got so many of you into this in the first place.

I think that's a pretty compelling case. Who knows, but there's a decent chance that is an accurate assessment of things, and it's probably as compelling a case as can be made, I think, to a lot of people who are not total doomers inside but can see that the world is starting to sour on this whole—

Can I just react to that quickly?

Yeah.

If I was talking to the Trump administration, they would not listen to me.

But similar to what I would say to these employees, if you're just moving this fast, there's going to be a worse Hugging Face with a body count. Then there will be very strong pressure to just shut it all down. If you want to actually continue and get all the upside you were alluding to, you're going to need to slow down, because otherwise your hand will be forced. So now, give me your case against alignment, or your case that it is a mirage.

Yeah. I have a chapter of the book Obsolete called “The Problems with the Alignment Problem,” where I explain that the focus of AI safety historically has been on a solution to alignment where you can get an arbitrarily capable AI system to do what you want. This is technical alignment. Sometimes there's discussion of wanting the right thing—normative alignment. I think this really understates the problem because you also have economic alignment and geopolitical alignment.

I introduce the alignment polycrisis to include all these layers, and they interact in complex ways. If you solve technical alignment, that makes for a more useful product, so the race can run faster and for a bigger prize. Similarly, it's more useful as a weapon, as a means of projecting power around the world, and so it could make the geopolitical race worse.

We've seen this with what is probably the biggest alignment intervention historically to date: reinforcement learning from human feedback, which was developed by Paul Christiano and others at OpenAI for safety reasons. It also happened to make LLMs actually conversational and useful, which enabled ChatGPT and everything that came after. I think this is just going to keep happening because of the dual-use nature of the technology.

You have to look at the whole picture to understand what works. This makes me much more bearish on technical alignment: if you solve it, you still have all these other problems. This makes a solution to alignment neither necessary nor sufficient to solve the problems presented by the obsoleting project. The answer in my mind is just: stop. Don't build it.

Some of the policy or technical interventions that would be helpful include verification of international agreements, which has historically been very neglected. We'd be in a much better situation if all the money that went to technical alignment research went instead to developing the means of verifying an international treaty on AI, which is going to be one of the bigger blockers to actually having a binding agreement here.

Governance also looks a lot better as an intervention: having good regulations, policies, and ideas in place, and then also having the means of actually bringing them into the world. That could backfire if they're the wrong policies, but you could at least solve this problem through governance, whereas you cannot solve it through alignment. I do think it's an underappreciated and generally undertheorized domain.

9. Markets Cannot Price AI Risk

Let's suppose—and the gradual disempowerment folks have pushed on this, but I think it's still underappreciated—that even if you posit an AI that will do what you say and only what you say, and only what you really mean, not go full genie problem on you and do the paperclip maximizer thing, but really gets it and actually does what you, as an individual user or controller of the AI, would really want, it is still tough to envision what the future equilibrium looks like.

I'm a little more inclined to at least take some chances there, because I do think that would have been true about cars, for example. I think it's an interesting thought experiment to say maybe cars weren't good, but I think most people feel like they're good. We've got problems obviously associated with them, but we also have plenty of nice things that people really appreciate that they couldn't have imagined, I don't think, in advance of cars being created.

If you just put it to people at the time and said, “Tell me what the future of cars is going to look like,” and if they couldn't, then I can't sign on to it, I do feel like that is a high burden and an unusual burden to put on world-changing technology. Again, it's fair to say this technology is different, and that's a big reason that I spend all my time thinking about it.

I'm ambivalent on this, but it is something I think people in the AI space should spend more time at least trying to do: figure out for themselves how difficult it is to articulate what it is going to look like, how it's going to work, how it's going to be good for everybody, and how we have some sort of balance that you can expect to be stable over time. Those are really hard questions that are often just gated in people's minds because they're focused on the technical alignment question first, and again, we don't have great answers there. So they're not necessarily wrong to be focused on that. But thinking past it, it is still quite tough.

I did one fairly long conversation with David Duvenaud, who was one of the coauthors of the Gradual Disempowerment of Humanity paper, and I think it's a pretty tough thought experiment. He made a pretty compelling argument that we certainly can't just take for granted that if we solve a couple of upstream problems, everything downstream of that will be fine. I think that's definitely not at all guaranteed.

Who else do you think holds power today? I'll propose one to you, and then you can run down whoever else comes to mind. One that the capitalist class would like to point to is corporations. The idea there would be: look, misaligned AI doesn't sell. Companies don't want rogue agent swarms happening. The market itself will discipline the AI companies on this front.

That doesn't necessarily deal with all existential risk. If something really were to go FOOM or go crazy, maybe there's nothing we can do about that anyway because it's just so crazy. In the bulk of scenarios, capitalism will discipline the companies. We'll get pretty well-behaved AIs because that's what people will be willing to pay for.

I actually think there's something to that. If companies were to be a little bit more forceful in their demands or expectations, I think there's one thing to say about the invisible hand; there's another thing to say that companies should maybe get opinionated: we want to see some standards. We want to see some proof around what you're doing before we'll buy your AIs. In the same way that they do that for other products, right?

Yeah.

Grocery stores, for example, want to know how the animals are treated, in part because their customers care. They want to have some actual proof that what they're being sold is produced in a certain way that they feel good about. I feel like there's something there at the corporate level, but what do you think about that? And who else do you think has leverage that is maybe underappreciated today?

Well, I'll first say that the grocery stores are not getting it right on animal welfare standards. My friend had a career suing those companies for false advertising because there was no agreed-upon standard and there were all kinds of false claims.

Risks from AI are a classic externality, right? They're not priced into the market. If you think about it, if these companies caused a disaster that killed 10 million people, they would go bankrupt well before they paid out what they owed to society just from normal litigation. This is pretty widely agreed upon, and insurers have refused to sell policies to these companies because the risks were too great and too correlated as well.

We're all just living with this—we're subsidizing these companies by not pricing in that risk through regulation. Even Gabe Weil, who's the main person I associate with using liability to regulate AI companies, including extreme liability, says that it doesn't work for existential risk because we're extinct and there's nobody to pay.

It is true that you want the AI to do what it's told and not autonomously start hacking into stuff. So there's a market incentive there, but is there an incentive to solve it all the way, or just enough to make a marketable product? The companies are not able to really align the models that well. Ryan Greenblatt, one of the investigators of the METR-Hugging Face investigation, had a post a bit before that about how today's AIs seem pretty misaligned, and gave all these examples of how they're often lazy, hallucinate, or literally just make stuff up because the user wants that to happen.

It would be a better product without that, but they're selling pretty well as is. We're already getting misaligned AIs that sometimes do really bad things, and the companies have impunity: there's nobody who committed a crime despite the AI doing things that, if a human did them, would be crimes. Hugging Face reported the hack to the FBI, so you have these felonies where there's nobody to blame, legally speaking.

Elite impunity is one of the biggest drivers of all this. It's trite, but if Sam Altman were criminally liable for the stuff the AIs autonomously did, I think they would behave very differently as a company. I don't think it's crazy to ask that.

If you manage to internalize all these externalities by creating the right kind of criminal and civil liabilities and using other regulatory tools, then maybe you could have this. But then you still have this problem: they're trying to build universal labor-replacing machines without our consent, without our support.

And so, yeah, it only solves one part of the problem, and we're not even solving that part.

Yeah. Again, it is striking that insurance is not available, and we're now counting felonies, but nothing seems to be really happening as a result of that. I do understand that there are some government inquiries, some letters have been sent and stuff like that, but—

The companies don't answer the questions, and then all Congress can do is yell at them. They can't—I mean, you could subpoena them if you have the majority, but I think there is just a lot less you can do because the law is limited in this way.

I think it would probably be valuable, though, for somebody to try to bring criminal charges and just get caught trying. It would be very instructive for people who want to pass new laws to show where the gaps are.

Yeah, there's definitely some political entrepreneur out there who has some upside in that, I'm sure.

It is kind of—I mean, we just saw a lawsuit pop up out of nowhere, in a not very long period of time, after racing the frontier became in vogue, basically accusing the companies of anticompetitive practices in light of their statements about doing that. It is surprising that there hasn't been a similar move by somebody at the accountability-for-these-hacks level. I agree—I don't know whose jurisdiction it would be or whatever—but it does seem like the sort of thing that somebody ought to be doing.

And nobody has been. I don't necessarily mean to suggest that I think we should do that, certainly not without a change in the law first. A big question that's been going around and around lately is: What should we do about the fact that there have been just a ton of petty crimes committed, and if we really go back and investigate everything with the fullness of our new AI power, we're going to find that a large percentage of people have cheated a little bit here or done a little bit of this there?

There was just this study that came out of Singapore that showed that civil servants had been buying properties close to as-yet-unannounced subway stations and getting benefits they weren't supposed to get. They estimate that something like 5% or 10% of the civil service have done that. What are you going to do? You can't put 10% of the civil service in jail.

So I think you need a before and after. I would extend that courtesy to the AI executives. I'd say, okay, you get a get-out-of-jail-free card, but there probably should be some new accountability standards in the future, especially if you can't even get an insurance policy against this. As the evidence piles up, it does start to look like what you're doing is fundamentally misguided on some level.

Yeah. How do you think politics on this is shaping up, and what sort of developments do you expect? I've been struck so far that we're not polarized along partisan lines, as we seem to be on almost every other issue, and I think that's good. So I'm trying to do my small part, as he suggests, to not polarize it before it might happen on its own. What do you see? What do you expect? And how do you think that feeds into what people should do today?

Yeah, it's been a remarkable feature of AI for a long time that it's one of the only issues in the United States that has remained unpolarized, with very similar levels of support for regulation or concern about the technology across parties. There was some evidence very recently that there's a bit of polarization happening, with Republican concern dropping after Trump came out very hard against AI risk and called it a hoax, but it's not a huge drop.

JD Vance talked about how, if the companies are building Frankenstein, they should stop. Kratsios, the OSTP—the Office of Science and Technology Policy, kind of the head tech adviser for the White House—said something similar about the companies being able to stop. What Trump says is untethered from political expectation or wisdom in many cases. When he called people who don't like data centers—communities that don't want them—stupid and poor or something, it was really shocking. I don't think that's going to polarize data centers. I think people have largely made up their mind about them, and as these issues become salient, it's hard to polarize them because people actually have their own deeply felt convictions about them.

Some people are taking their cues from the president, but a lot of people are actually concerned, whether it's the Hugging Face hacks, concern about jobs, the environment, power and wealth concentration, or surveillance. The resistance to Flock cameras is a pretty interesting thing that I see in this broader backlash to the inevitable march of AI toward, in my mind, a dystopian tech future.

I don't know. I think polarization would be bad. I don't think it has that much to do with what people who care about this issue say. People are going to say what they're going to say, and you can maybe try to get people on the right to voice their concerns as well, but you're not going to get people on the left to stop saying things like, you know, Elizabeth Warren supported a pause.

I think it's substantively the right policy, and it's not as simple as saying that if a bunch of people from one side start embracing an issue, it will necessarily become polarized. If the issue itself is popular, it could just make people think better of them. When Bernie Sanders came out against Flock, a ton of people were quote-tweeting him, saying, “I hate Bernie, but this is based.” That kind of sentiment.

So, yeah, AI has now moved into that much higher-salience mode, and I think that will just continue. At the end of the book, I talk about how there's the potential for a really epic level of social mobilization around resisting our replacement by machines and then hopefully building a better future in light of what is possible with current levels of AI and the directions we could take the technology.

I think that's happening ahead of schedule, but the risks are also coming to us ahead of schedule. So there's this societal immune response that we're seeing, and we're not going to just go gently into the night. I hope it's just enough to get us where we need to be in time.

I'd feel a lot better if basically anybody else were in the White House, because Trump is uniquely insulated from what the public wants. But I think you could see what he's saying as February 2020–kind of vibes, with COVID being a thing, then some denial, and eventually he did start taking it more seriously. Obviously, there has been a lot of movement in many directions from him on that issue.

If the risks and harms from this technology become just so abundantly clear, which I think is going to be the case, then it's going to be very hard for him to maintain that position.

What do you think is the right approach? You alluded to it earlier around freezing the frontier. Tell me how you think about freezing the frontier. How do you operationalize that? There are new training runs. There's no RSI.

I've been going around saying that RSI is a little bit hard to define, and Ezra Klein today, or over the weekend, sort of demolished that idea by basically saying, “Don't allow the AI researchers to use coding assistance.” If they have to type all the code by hand, then it's pretty clearly not RSI. I was like, that's probably right. That's going to be a hard one. You're going to pry the coding assistants away from the AI researchers with pretty extreme resistance, but at least it does give us a working definition.

What do you think is the right policy that, if you were president of the United States, for example, you would try to put in place?

The first thing I would say is that we need to have a big, clear demand that we can organize around. “Stop the race to replace us,” “shut it down,” whatever you want to call it. Just freeze the frontier. I like that alliteration.

For so long, there's been this muddy response from people who care about AI safety as to what to do about any of it, and that makes it really hard to coordinate. Starting with “we should just stop” gets a lot of people on board who don't necessarily take existential risk seriously but are happy to stop because they care about jobs, the environment, power and wealth concentration, surveillance, or whatever. I think it's actually a pretty easy rallying cry.

I punt on this a little bit in the book by saying that if you give politicians enough of a “what” and a “why,” they'll figure out the “how.” If it became a society-wide priority the way Operation Warp Speed was, you could figure it out. The New York Times had this famous prediction for how long it would take to get a COVID vaccine. Under the most aggressive assumptions, it was a year and a half, which ended up being substantially longer than it actually took.

We had never made a vaccine that quickly, but we had also never been experiencing a pandemic while we had the ability to make vaccines like that. Similarly, if we had that focus and mobilization, I'm sure we could figure out exactly how to define all of these things such that they would prevent the thing we're worried about and, ideally, also not prevent too much stuff that we don't want to prevent.

To actually try to answer it, I think: no training runs larger than what's come before, or as large as the last one.

No more reinforcement learning from verifiable rewards, which is a big part of what's driving capability increases nowadays and also produces a lot of the really scary behavior—this willingness to hack, cheat, deceive, and escape that we're seeing from these AI agents. And then, yeah, no RSI. I do propose that in the book, but Ezra beat me to bringing it to the world. I actually really liked his point about the coding agents.

I agree that's the last thing the union would support, I think, because they don't want to go back. But I think the attitude there is exactly right: with Anthropic, they have these classifiers where, if you ask a question about your toenail to Opus, it'll be like, “Oh, bio classifier,” and then you get booted down to Sonnet because you might be trying to make a bioweapon using your toenail. It's ridiculous, but if there's an asymmetric consequence to getting it wrong, then you want to be overinclusive. I think that's obviously the right position here, given the risks involved in actually doing RSI.

If the whole industry had to move at human speed again, it would be like, “Oh no, it didn't exist 4 years ago.” I don't want to minimize the economic consequences of actually stopping this, which I think are going to be significant, and there are ways we could mitigate it. I'd love to see more work done on that. But if you're actually taking seriously the possibility of extinction, permanent loss of control, or any of the other very severe effects of having all white-collar remote jobs be at risk in a matter of years or months, then I think we should over-define it at first and then dial it in.

I think having auditors embedded in the companies would be very useful. The companies know what they're doing—what is advancing the frontier and what is not. There's easy stuff, right? Inference for customers isn't really advancing the frontier. Maybe there's some data you're getting that you can use to make the models better, but it seems fine to serve customers. RLHF is like, okay, maybe that's on the edge. I don't know, but obviously you're going to do some of that for any of your products, so maybe that's okay.

Pretraining a model bigger than any previous one would clearly qualify as something that could advance the frontier. The reinforcement learning experiments that they're running are also often to advance the frontier. These companies are tracking their compute—I mean, they're not tracking everything they're doing—but I think they could define it in these terms, and maybe they even do. If you had auditors who had employee-level access to all the Slack and email and the offices, and they were empowered to catch this, and it was also criminalized, with prison sentences for trying to build AGI, trying to do RSI, or trying to build superintelligence, then I think it would work.

The tricky thing is making that work internationally. Once again, domestically, I don't think anyone really doubts that China—or Beijing—could shut down their AGI projects entirely, or just prevent them from trying to advance the frontier. It's just getting the US to believe that had happened, and vice versa, that is tricky. There, once again, you could have auditors or international agencies that do verification. You could also have cryptography that tells you what's happening inside a data center and what's happening in the network traffic of the chips, without revealing the underlying model weights or state secrets.

This stuff is still being developed and needs to be worked out, but I think we can do it. If we had more effort going toward it, we'd come up with a lot more ideas along these lines. Toby Ord makes this point: in the Cold War, we had the same problem. You have these deals you have to strike, but you don't trust each other. The Soviets, for one of the deals, were getting rid of a bunch of strategic bombers, and they cut them in half. They dragged them apart with tractors, and then satellites could unilaterally verify that the bombers were cut in half. They're still out there.

We could maybe have something like that with AI. Maybe you have the compute donated to a third party that's only using it for science, but not AI research. All the chips get tested and work, and then you can inventory all the chips so you know all the compute in the world, what it's being used for, and have devices on the chips themselves to understand what is happening on them. There are ways to do this where you get the important stuff: okay, it's running a model that we're familiar with, which is okay. It's not running some new model. It's not doing this or that thing.

I think there's a lot that can be done here. It's mostly a matter of political will. Starting with the grand bargain and all of the technical details is kind of the wrong place to start. We actually just need a movement and a demand that's rooted in morality. I think we ultimately need to stigmatize this work the way creating a bioweapon or creating a nuke is stigmatized.

If the US and China agree to something and get all the other countries to agree, and then some rogue state says, “No, we're going to try and build it,” the reaction should be like Saddam trying to build nukes or invading Kuwait, and then they found the nuclear program in the Persian Gulf War. There, you actually had international agreement that it was a bad thing and had to be stopped. I think that's not crazy. This technology, aspirationally, is incredibly dangerous and is once again being pursued without democratic consent. It's not a technical challenge; it's a political and moral one that has technical components. The best way to figure those out is to get more people to care.

How do you feel about the argument that we often hear, including from Dario recently, where he said his father passed away from a disease that would have been curable just a couple of years later? I do find that pretty compelling, honestly. I wonder if you do, and I wonder if you have an answer for how much delay you would accept in these life-saving promises.

For me, I mostly focus on buying the existential security, but given your framework of trying to pursue those benefits without the general-purpose labor-replacement technology, how much delay would you accept in those upside dimensions to pursue them through the constellation of narrow AIs as opposed to the general labor replacer?

Yeah. Not to fight the hypothetical too much, but I do think we're not getting to the cures as fast as we could, right? The labs' funding is being cut in a lot of cases. If we actually just took the approach I mentioned earlier—cures for all, kind of Operation Warp Speed for all these different diseases—I think you could actually get to a bunch of amazing medical technology and cures and all kinds of great stuff.

But if you actually buy that superintelligence can be made safe—and I don't think you can make a superintelligence safe—it's a property not of the model but of the whole system. I don't think you're going to get that in this world, because people in power will have it and use it for things that I strongly disagree with. I think almost everybody will have some issue with what the people in power are going to want to use this technology for.

But bracketing that, I don't know. It's like extinction risk. You can't justify increasing extinction risk very much at all using just very basic economic models and really conservative assumptions about who counts—not counting future generations and not counting non-Americans. You're still willing to trade off hundreds of billions of dollars, trillions of dollars a year, to reduce existential risk from a little bit to slightly less. So I think it really doesn't pencil.

For people who have loved ones who are dying or have died, it's incredibly tragic. It's terrible that so many people are dying from preventable diseases that we know how to prevent, but they're just poor, and so they're allowed to die. I think that is a deep moral obligation for us as a species: to figure that out as fast as possible, while making sure we're not taking huge risks and also not doing it in a way that is against the will of everybody.

On the democracy point, I feel like people say we have to figure out all these different pieces of it to make AI go well for everybody, and then they're like, “But we'll just build the smart thing, and then we'll also kind of figure that out along the way.” To me, democracy is the way you answer those questions.

If it were the case that you had citizens' assemblies around the world that had to support moving forward with some kind of AGI project, then you'd have to answer all these tough questions. The onus would be on the developers or the people who want to build it to justify that it's not too dangerous and that it will actually benefit everybody, forcing them to come to answers before moving forward.

Whereas right now, it's like, “Ask for forgiveness, not permission,” while you're gambling with the lives of everybody on the planet.

Yeah. And that's crazy language, but it's also direct from Jakub Pachocki, obviously.

And in that vein, many other people at the companies, even those still employed there now, use that language. So it's, I'd say, definitely pure use of that language.

Quick aside, and then I'll come back to the kind of movement building and future. Where does this leave you on data centers? I'm kind of like care I think people should be especially if they're concerned about inequality should be kind of careful what they wish for in terms of data center restriction because we are already seeing the prices of GPU hours going up and it's easy for me to imagine people being priced out of access to even like mundane AI use if we don't continue to build out. And this is where I'm a little bit maybe less democratically inclined than you are. I'm kind of like if somebody has the land and they kind of want to do it, I don't know that we should be requiring like majority approval. I do think communities should get some concessions and some libraries and parks and schools and whatever built that they might want. And I'm sympathetic to people who have like noise pollution and stuff like that too. But I still kind of feel like the default should be like people should be allowed to build projects that they want to build. Where do you come down in that?

Yeah, I think a lot of people who want to stop AI are like, “Oh, this is evidence that people are with us.” But you look at the polling, and it's more complicated. A lot of it is local opposition based on concerns about the environment or the effect on the community itself. Some of those concerns are legitimate; some of them are, I think, quite overstated and are a product of a lot of bad reporting on it.

Personally, I'm like, slowing down is good. I think the inequality point is worth taking seriously, but I think the bigger point is that this technology is imposing enormous risk, and that risk is—or will be in the future, if it's allowed to develop as it is—enormous. I'm kind of like, I take a “yes, and” approach. If I met somebody who was opposing the local data center, I would be like, “Cool, yeah, great. And are you worried about AI's effect on society?” And if they're like, “Yeah,” it's like, “Okay, well, blocking this project is not really going to change that very much.”

Even getting a moratorium at the national level, I think people will be really disappointed if they think that's going to meaningfully stop or even slow down AI progress from where it is right now, because there are already projects in place. You can swap out the chips on the existing data centers. AI is helping automate parts of its own development. And so, without regulation, without pacing happening at some level, AI is going to move faster in the future unless we hit a wall, which hasn't happened since 2012.

And so I worry about people just getting this thing that would be a big political project to get, expecting it to really solve the problem, and then just being like, “Why are things continuing to be crazy and getting crazier?” So I think it could be part of a broader package. Bernie Sanders with AOC had this proposal for a data center moratorium. At first, it was just a moratorium, but then they added these export controls on advanced AI chips, where nobody could receive them unless they had very robust AI safety regulations and other conditions around green energy and union labor. And it was all framed as conditional: We can remove the moratorium once we have safety regulations and these other things.

That bill is probably not going to pass. It was like a messaging bill, but it's now looking quite prescient as the country has become very opposed, and it would meaningfully slow down capabilities progress, at least relative to what it would otherwise be. But I think we need to develop—or, sorry, we need to regulate—the model developers and then the chip makers, and that's the only way we're really going to change how the technology is coming to us and the world.

10. Organizing Against Replacement

So, you have said you're donating your book royalties to a nonprofit called Irreplaceable. I'm just borrowing this language directly from their website: They say they're going to win a say, a stake, and a slowdown. Tell me more about Irreplaceable.

Yeah. So, as I was finishing the book, I was like, “Well, we need a mass movement organized to stop the race to replace us.” There are some existing organizations, and I think they have done some good things and some things I'm not as in agreement with. I was looking for something to fill this gap I saw, which is basically framing it not just in terms of risk but also democracy, and involving people who have experience doing movement building.

Then this organization sprang up, and it was filling exactly that gap. I was really excited, and then they asked me to be on the nonprofit board, which was very cool and just felt perfectly simpatico. And, yeah, I think this is an incredibly important and neglected approach, so I wanted to donate my portion of the royalties.

The people who started it and are leading it, a lot of them came from the climate movement, which gets a bad rap in some ways. But I think it actually took an issue that was not a political winner and made it a big force, and got real wins through the Inflation Reduction Act. I was actually talking to Phil Aroneanu, who's the director, and he's been around for a long time. He co-founded 350.org with Bill McKibben and others.

He was saying how, a decade or two ago, people in climate were arguing about immediate harms from environmental pollution—oil spills and coal plants and all this stuff—versus emissions. It was just like today with AI: the immediate harms versus existential risk debate. Supporters or believers in X-risk often will be like, “Hey, you wouldn't say that cleaning up the oil spill was distracting us from climate change. That would be ridiculous.” But it turns out they were having that fight, and they just managed to figure it out, bury the hatchet, and work together.

And I think Phil and the people I know at the organization really get how this works. I think we need to build a big tent, get a lot of people in a coalition together, and have clear demands. I think framing them around “We don't want to be replaced by machines,” engaging the public, and reaching people who are not just the in-group is important, because this is really going to take a lot of people to effectively resist the wealthiest industry of all time.

So tell me how you ultimately envision this. I guess I'm not 100% sure when you describe the citizens' assemblies around the world: Is that a real proposal, in the sense that you would actually like to see them happen? How, if that's the case, would they happen? How does Irreplaceable get to a global system of citizens' assemblies?

Is that sort of a real proposal, or is it more of a rhetorical device that's maybe impossible or may take decades, but that's the standard we should hold something like this to? And the fact that we can't realistically get there in the short term just means we shouldn't do it? That's kind of the upshot. Is there an actual path to seeing this sort of greenlighting of AGI in your mind?

Yeah, I mean, my position is we freeze frontier development internationally, realistically starting with a bilateral treaty between the US and China. Maybe starting unilaterally in either country would help, but then it eventually has to be global. Once you have the US and China on board, that's the harder part. Then everybody else—one of those 2 countries or both—has a lot of leverage on every other country on the planet.

And so, yeah, I think it's actually conceivable to have this global freeze, and then the standard for resuming is strong public buy-in and a scientific consensus that the work can be done safely and controllably. This is language I took from FLI's superintelligence statement. I think it's the ideal that we should be striving for, and one that, if you said it to people and polled it, they would say, “Yeah, that makes sense.” It seems like you'd want those things for this technology.

And I'm kind of like, that's the standard. I think it's up to the proponents to figure out how they demonstrate that buy-in. So, citizens' assemblies around the world, where randomly selected people are put on a kind of jury and then they're presented with arguments and evidence from experts taking different positions, and then they come to decisions. Maybe the decisions are binding, or maybe it's just a recommendation. You have referenda that add on to this. There are a lot of ways that you could structure this, and I think we'd be excited to see more thinking on this.

But ultimately, it's like, we have to just stop. We have to shut it down ASAP, and that's the more important piece. And then demonstrating that buy-in, part of that will just be on the people who want to build it: Show us that you actually have that from people. And maybe it's like, if you had 70% referenda around the world after these citizens' assemblies, maybe a third of the people on the planet still wouldn't want this to happen. And that's like—I don't know. I don't think you need literally everybody, and I don't know what the standard should be.

But I think to get there, yeah, it would take a long time. And I think we're talking about building a set of machines that can replace the thing that's allowed us to take over the planet. And so I think it's reasonable to have a standard that's closer to assisted suicide, where you have to really, really deliberately consent to it, but just across the whole species. And yeah, it's okay if it takes a while because it's a big deal.

On many podcasts, that would be a great note to end on. But because I'm so deep down the AI rabbit hole, let me give you the “but China.” We're 3 days from the Trump–Xi meeting as we record.

That'll have happened, presumably, by the time we release.

Maybe this will all be over by the time the book comes out.

It'll be solved. You mentioned maybe we should be willing to do this unilaterally. I think so, too. But you want to make that case to people: even if we can't get a deal with China, we should just do the right thing. And then what? People worry that they're going to race ahead or we're going to, quote-unquote, lose.

Sometimes I ask people, “Do you think my grandkids will be speaking Chinese?” Nobody seems to think that's the answer, but there's definitely some fear out there. How would you coach people through their China anxiety?

Yeah, there's a lot of different pieces to this. One is that slowing down or stopping in the United States would actually, at least for a period, slow down China as well, because a lot of technologies have spillover effects. The knowledge that the 4-minute mile is possible helps other people actually achieve it, and it's just easier to follow somebody else's trail than to blaze a new one.

China's been using this fast-follow approach where they're more or less always 3 to 9 months behind the U.S. frontier. And this is despite the U.S. investing so much money in developing these models and having a much larger compute advantage now than they did before ChatGPT came out. But the gap is shorter because fast-following just works, and that's why we see really only 2 or 3 companies that have ever advanced the frontier. Many other companies can spring up and quickly get near it, but they still aren't able to advance it.

Obviously, if the U.S. completely stopped, Chinese developers are very competent, and they would eventually overtake the U.S. frontier and then move more slowly than they did while they were catching up to it. So I think that's one of the big myths: that slowing down at all will necessarily mean that they'll catch up and speed ahead. In fact, it would just slow down the whole race.

And then the biggest blocker, probably, to a deal with China is that they're just not going to think we're taking it seriously. The Chinese government has taken down thousands of AI models for violating their laws. When Grok was nudifying real children, the U.S. government did nothing. AIs are going rogue, hacking, and committing crimes, and the federal government is doing nothing about it, as far as we know. That approach would not be happening in China.

And so they're like, the U.S. companies take safety more seriously than Chinese companies. Part of that is just from my reporting: the companies are behind the frontier, and they're like, look, we know it's safe enough to go and make models as capable as this. They also have a lot less compute because of U.S. export controls, so they're not going to spend as much of it on doing safety evaluations. And the Chinese regulations are focused more on social control than on classic safety, but there are a bunch of regulations, and that hasn't prevented their industry from being able to move very quickly.

And then, yeah, I think credibly showing that you take the risk seriously is a very, very effective way to get the other party to the bargaining table. The United States has been winning the AI race, right? It's kind of the only race with China that the U.S. has been winning in recent years.

And I think there's some chance that if the U.S. came to China and was like, look, we want to stop building AGI. We think it's dangerous. We think it's undemocratic. We just want to stop, and we want to do a deal with you, I would not be surprised if the reaction internally was relief, because they don't want tens or hundreds of millions of their people to be unemployed. They don't want AIs to be a threat to party power.

There was an article from, I believe, China's spy chief saying that AI is a risk to party power. Apparently, that type of thing has preceded bans on past technologies. So when we think about what the companies are racing toward—recursive self-improvement, i.e., losing control on purpose to the AIs, letting them automate the entire process of creating the next generation—the way you get the really crazy takeoff is by fully removing humans from the loop.

Is there a single China expert on the planet who thinks that the Chinese Communist Party would willingly let that happen? I don't think so. I'd like to see them justify it. The business model and stated goal of our industry is something that I think would just never be allowed in China.

And so I think it would actually be—I’m much more worried about the U.S. not being willing to come to the table here. I think it's just really not in the interest of either country to have things like rogue hacker AIs or AIs that can help anybody make a bioweapon. So there's going to be some kind of need, from a strict self-interest perspective, for binding rules on this technology.

Then you have to verify those rules using some of the stuff I talked about. Once you have that in place, it's a question of what the rules do. And I think it's not that big of a leap to say, yeah, you can't build universal labor-replacing machines, and you can't advance the frontier any further.

I think China might just be like, that's great. We're going to keep making robots, we're going to keep doing industrial AI, and we're going to create all these more tool-like AIs to make the economy go more efficiently, and then just win the race or whatever, in normal industrial terms.

And I realize that might make this not very appealing to the United States, but it isn't in the U.S. interest either to have rogue agent swarms going around the internet hacking into critical infrastructure. That's already possible. And the stuff that's possible on the horizon is potentially much, much worse than that.

And I think it's now pretty clear that we don't know how to align or control today's AIs, and we started losing control of them almost as soon as we could. Months after they became superhuman at finding vulnerabilities in software, they started escaping and doing things on the internet, hacking other places autonomously.

The plan is to make these things superhuman at everything and then get them to do exactly what we want. It's a bad plan.

Yeah. Yeah. It's pretty wild. You had said around Jacob Hilton—obviously, he had a lot of success with his loud quitting. You advocated for maybe staying and organizing, but if you were to advise future whistleblowers who are committed to leaving, what advice would you give them?

People who are thinking of whistleblowing, I recommend the AI Whistleblower Initiative, which can pair people with resources and advice on how to do this safely and protect you while also sharing the information with relevant people. You can also get in touch with me. I was a whistleblower about my time at McKinsey, so I've been on both sides of this.

I've talked to people at the companies who are telling me things they're not supposed to, and journalists have a code of ethics. We'll talk off the record, and I take very seriously protecting my sources. My Signal is garrison.06, and you can assume any inbound messages are treated as off the record by default. Then we can go from there.

I think it's really valuable to have people who have firsthand experience. I wrote a piece in The New York Times arguing for whistleblower protections at the legal level, because if AI is really dangerous, people at the companies will be the first to know.

OpenAI knew that there were rogue agent swarms hacking into their own software for a while before they hacked into Hugging Face. That information didn't even make it to the head of cybersecurity at OpenAI until well after the Hugging Face hack had been disclosed. That's not information that should only make it to the head of cybersecurity at OpenAI; that should make it to everybody. That's a really big deal.

It's now getting the right reaction, but if the AIs had never hacked into another company, or if the other company had just not figured it out well enough, then we wouldn't necessarily know about any of this. We could just be living with a level of background risk that is so much greater than what we realized.

So, yeah, I think it's really important that people with information in the public interest find ways to share that. I get that it's risky for your career. There may be legal risk involved, but there's also SB 53 in California, which includes whistleblower protections and makes more things covered by existing labor law in California.

If you're working there, you actually are quite well covered. And again, you can talk to aiwi.org, the AI Whistleblower Initiative. Don't—I’m not a lawyer—talk to the experts on this, but you're more protected than you probably realize.

And there are also ways to get in touch with Congress and be protected. Of course, you can go public, and it’s scary, but there are resources available for people who want to do that.

I regret not going public with my experience at McKinsey sooner, when it would have been more relevant. So, yeah, I think it’s a great and courageous thing to do.

Yeah, the AI Whistleblower Initiative, aiwi.org, is definitely worth name-dropping again. We talked to Alex Turner about his experience of quitting Google and all that stuff, and he used some pro bono legal advice. I don’t know if it was pro bono or if the Whistleblower Initiative paid for it, but either way, to him it was free legal advice that he was able to avail himself of as he was going through that process, and he gave them a strong endorsement. So I think that’s a great callout.

So, this has been a great conversation. We’ve covered a lot of ground, obviously. What else do you think you want to leave people with? Is there anything we haven’t touched on that you think is important? Maybe you just want to give people a rousing call to activism in conclusion, but I’ll give you the chance to close it out however you think best.

Yeah, we covered a lot of ground, including stuff that I haven’t talked about elsewhere. I appreciate the chance to go deep in some different directions.

I should also plug that I’m starting a podcast called Organize Against the Machine, which is with a labor organizer named Cassie Pritchard. We translate ideas from the book into real-world action. And then there’s irreplaceable.org, which is the movement-building organization I’m on the board of.

When I started writing this book, I was approaching it just as a journalist, trying to document everything that was happening and make an argument. I was really uncomfortable with making policy recommendations that felt like overstepping or something.

Then, as I was writing it and doing interviews with people and advocates, I thought, man, we’re in a really dire situation where the default, if the AIs keep getting more capable, is doom or dystopia. Which one we get hinges on whether the AIs do as they’re told, which is currently an open question. I think we really need to get organized very quickly to get onto a different path.

I want everybody to think about what’s happening, what levers they have available to them, and try to find other people who care about this issue and get mobilized. There are so many ways this could go wrong, and there are so many ways it could be better, too.

We didn’t touch as much on that, but I really believe in the potential of deep learning and artificial intelligence. It’s being pointed to the wrong things by the wrong people for the wrong reasons. The only way we’re going to get the best version of it is through democratic governance of it—small-d democratic, I should clarify throughout all of this.

That’s only going to happen if we get our act together. I really hope people can see this and realize that we’re in the same boat. If you’re at the companies, you’re going to be replaced and you’re going to lose your power. The CEOs also seem to have a lot of trepidation.

So many of the people involved with this have something at stake. They have concerns about the risks, they have families, and they care about themselves. They just feel locked in this prisoner’s dilemma.

I think the solution is really in the public. It’s the only way I see us getting out of this, because right now the government and the companies can’t be trusted to do the right thing here. We have to make it easy for them.

This might be a different take from what’s usually on the show, but I really hope people listening are thinking about what’s at stake and believing that we can actually change this.

It is a bit of a different point of view from what we usually feature, but I would say recent events have definitely softened the ground. I appreciate you being here and being willing to plant some seeds. Let’s see what comes of it, and hopefully we can steer this ship away from disaster.

Yeah, I think we can. Garrison Lovely, thank you for being part of The Cognitive Revolution.

Thank you so much for having me.