Are We Past Peak iPhone? + Eliezer Yudkowsky on A.I. Doom

The other big news of the week is that Larry Ellison, the founder of Oracle Corporation, just passed Elon Musk to become the richest man in the world.

Yeah, and I love this story because there was an incident that I filed away in my catalog of moments when straight people write headlines that gay people find hilarious. I don’t know if you saw the version of this story on Bloomberg, but the headline is, “Ellison Tops Musk as World’s Richest Man.” And I thought, “He’s doing what? Is that a privilege of becoming the world’s richest man, that you get to top number 2?”

Oh.

Yeah.

And this is why they need representation of gay people on every editing desk in America.

Exactly. Hire a gay copy editor, Bloomberg. You’ll save yourself a lot of headaches.

I'm Kevin Roose, a tech columnist at The New York Times.

I'm Casey Newton from Platformer.

And this is Hard Fork.

This week

The new iPhones are almost here.

But is Apple losing the juice? Then, AI doomer-in-chief Eliezer Yudkowsky is here to discuss his new book, If Anyone Builds It, Everyone Dies.

1. Apple Unveils New Hardware

I wonder what it’s about. Well, there was a big Apple event this week. On Tuesday, Apple introduced its annual installment of “Here’s the new iPhone and some other stuff.” Did you watch this event?

I did watch it, Kevin, because as you know, at the end of last year, I predicted that Apple would release the iPhone 17, and so I had to turn out to see if my prediction would come true.

Yes. Now, we were not invited down to Cupertino for this. Strangely, we haven’t been invited since that one time that we went and covered all their AI stuff that never ended up shipping. But anyway, they had a very long video presentation. Tim Cook said the word “incredible” many, many times, and they introduced a bunch of different things. So let’s talk about what they introduced, and then we’ll talk about what we think of it.

Let’s do it.

So the first thing, since this was their annual fall iPhone event, is that they introduced a new line of iPhones. They introduced 3 new iPhones. The iPhone 17 is the sort of base-model new iPhone that had incremental improvements to things like processors, battery, and cameras. Nothing earth-shaking there, but they did come out with that.

They also came out with a new iPhone 17 Pro, which has a new color. This is sort of a burnt-orange color. Casey, what did you think of the orange iPhone 17 Pro?

I’m going to be sincere. I thought it looked very good.

Me too.

Yeah.

I did think it looked pretty cool. I’m not a person who buys iPhones in different colors because I put a case on them because I’m not a billionaire. But if you are a person who likes to put a clear case on your phone or just carry it around, then you may be interested in this new orange iPhone. I did see that the first person I saw who was not an Apple employee carrying this thing was Dua Lipa, who I guess gets early access to iPhones now.

Wow, that’s a huge perk of being Dua Lipa.

Maybe the biggest. So in addition to the new iPhone 17, 17 Pro, and 17 Pro Max, they also introduced the iPhone Air. It costs $200 more than the standard iPhone 17, and it has lots of different features, but the main thing is that it is slimmer than the traditional iPhone. So I guess people have been asking for that. Casey, what did you think of the iPhone Air?

I don’t understand who this is for. Truly, not once has anyone in my life complained about the thickness of an iPhone. Maybe if you’re carrying it in your front pocket and you want to be able to put a few more things in there with it, this is really appealing to you. But there are some significant performance trade-offs. They announced it alongside this MagSafe battery pack that you slap onto the back of it, which is of course going to make it much thicker.

No, Casey, it’s even better than that because they said that the iPhone Air has all-day battery life, but then, in the next breath, they were like, “Oh, and here’s a battery pack that you can clip onto your phone, just in case something happens.” We’re not going to tell you what that thing might be.

Yeah.

But just in case, it’s there for you.

Right.

So I think, as with all new iPhone announcements of the past couple of years, there was not much to talk about in the iPhone category this year. It’s like the phones get a little bit faster. The cameras get a little bit better. They have a new heat-dispersal system called the vapor chamber that’s supposed to make the phone less likely to get hot when it’s using a bunch of processing power.

At first, I thought they had made it so that you could vape out of your iPhone, which I do think would be a big step forward in the hardware department. But unfortunately, that’s just a cooling system.

Yeah. Vapor chamber is what I called our studio before we figured out how to get the air conditioning working in there.

Yes. So let’s move on to the watches. The watches got some new upgrades. The SE got a better chip and an always-on screen. The Apple Watch 11 got better battery life. Interestingly, these watches will now alert you if they think you have hypertension, which I looked up—it’s high blood pressure. It says that it can analyze your veins and some activity there to tell you, after a period of data collection, if it thinks you’re in danger of developing hypertension. So maybe that’ll help some people.

I mean, that was of interest to me. Kevin, high blood pressure runs in my family, and my blood pressure spiked significantly after I started this podcast with you. So we’ll be interested to see what my watch has to say about that. It’s also going to give us a sleep score, Kevin, so that now every day when you wake up, you’ve already been judged before you even take one foot out of bed.

Yes, I hate this. I will not be buying this watch for the sleep score because the couple of times in my life that I’ve worn devices that give me a sleep score, like the WHOOP band or the Oura Ring, you’re right, it does just start off your day by being like, “Oh, I’m going to have a terrible day today. I only got a 54 on my sleep score.”

Yeah. We have this Eight Sleep bed, which performs similar functions, but it actually has sensors built into the bed itself. I sit down at my desk today, and it sends me a push notification saying, “You snored 68 minutes more than normal last night.”

What were you doing last night? That’s a lot of snoring.

I was being sick. I have a cold. I’m incredibly brave for even showing up to this podcast today.

Well, I appreciate you showing up even with your horrible sleep score.

Thank you.

I appreciate it.

Thank you.

Okay. Moving on, let’s talk about what I thought was actually the best part of the announcement this week—

Yeah.

—which was the new AirPods Pro 3. This is the newest version of the AirPods that has, among other new features, better active noise cancellation, a better ear fit, and a new heart-rate sensor so that they can interact with your workouts and your workout tracking. But the feature that I want to talk to you about is this live-translation feature. Did you see this?

I did. This was pretty cool.

So in the video where they’re showing off this new live-translation feature, they basically show that you can walk into a restaurant or a shop in a foreign country where you don’t speak the language, and you can make this little gesture where you touch both of your ears. Then it’ll enter live-translation mode. When someone talks to you in a different language, it will translate that right into your AirPods in real time, basically bringing the universal translator from Star Trek into reality.

Yeah. My favorite comment about this came from Amir Blumenfeld over on X. He said, “LOL all you suckers who spent years of your life learning a new language. I hope it was worth it for the neuroplasticity and joy of embracing another culture.”

Yes. And I immediately saw this and thought not about traveling to a foreign country, which is probably how I would actually use it, but I used to have this Turkish barber when I lived in New York who would constantly speak in Turkish while I was getting my hair cut. I was pretty sure he was talking smack about me to his friend, but I could never really tell because I don’t speak Turkish. So now, with my AirPods Pro 3, I could go back and catch him talking about me.

Yeah. Over on Threads, an account called Rushmore90 posted, “Them nail salons about to be real quiet now that the new AirPods have live language translation.” And I thought that’s probably right.

Yeah. So this actually, I think, is very cool. I am excited to try this out. I probably will buy the new AirPods just for this feature. And I have to say, it does seem like with all of the new AI translation stuff, learning a language is going to become—not obsolete, because I’m sure people will still do it. There are still plenty of reasons to learn a language, but it is going to be way less necessary to get around in a place where you don’t speak the language.

I mean, that’s how I think about it. This year, I had the amazing opportunity to go to both Japan and Italy, countries where I do not speak the language. Of course, I was traveling in major cities there, and most of the folks we met spoke incredible English, so I didn’t have much trouble. But you could imagine speaking another language that is less common in those places, showing up as a tourist, and whereas before you’d be spending a lot of time trying to figure out basic navigation and how to order off menus and that sort of thing, all of a sudden it feels like you’ve slipped inside the culture. I think there’s something really cool about that.

So those are the major categories of new devices that Apple announced at this event. They also released a device—or an accessory—that I thought was pretty funny. You can now buy an official Apple crossbody strap for your iPhone for $60. Basically, if you want to wear your phone instead of putting it in your pocket, Apple now has a device for that. I don’t know whether that qualifies as a big deal, but it’s something.

Ooh.

Let me tell you, I think this is actually going to be really popular. Kevin, I don’t know how many gay parties you’ve been to, but the ones that I go to, the boys often aren’t wearing a lot of clothes. We’re maybe in some short shorts and a crop top. They don’t want to fill their pockets with phones and wallets and everything. So you just sling that thing around your neck, and you’re good to go to the festival, the EDM rave, or the cave rave. Wherever you might be headed, the crossbody strap will have your back, Kevin.

Wow, the gays of San Francisco are bullish on the crossbody strap. We’ll see how that goes.

Mm-hmm.

2. Apple Leaves Innovation Behind

So, Casey, that’s the news from the Apple event this week. What did you make of the whole thing, if you take a step back from it?

On one hand, I don’t want to overstate the largely negative case that I’m going to make, because I think it’s clear that Apple continues to have some of the best hardware engineers in the world, and a lot of the engineering in the stuff that they’re putting out is really good and cool.

On the other hand, you don’t have to go back too many years to remember a time when the announcement of a new iPhone felt like a cultural event, and they just don’t feel that way anymore. My group chats were crickets about the iPhone event yesterday, and even as I was watching the event and reading through all the coverage, I found myself with surprisingly little to say about it.

I think that’s because over the past few years, Apple has shifted from being a company that was a real innovator in hardware and software and the interaction between those two things into a company that is way more focused on making money, selling subscriptions, and monetizing the users that they have. I was just really struck by that. What did you think?

Yeah, I was not impressed by this event. It just doesn’t feel like they took a big swing at all this year. The Vision Pro, whatever you think of it, was a big swing, and it was at least something new to talk about and test out and prognosticate on. What we saw this year was more of the same and slight improvements to things that have been around for many years.

Now, I do think that this is probably a lull in terms of Apple’s yearly releases. There’s been some reporting, including by Mark Gurman at Bloomberg, that they are hoping to release smart glasses next year. Basically, these would be Apple’s version of something like the Meta Ray-Bans. I think if you squint at some of the announcements that Apple made this year, you can kind of see them laying the groundwork for a more wearable experience.

One thing that I found really interesting: On the iPhone Air, they have moved all of the computing hardware up into what they call the plateau, which is this very small oval bump on the back of the phone. To me, I see that and think, “Oh, they’re trying to see how small they can get the necessary computing power to run a device like an iPhone, maybe because they’re going to try to shrink it all the way down to put it in a pair of glasses or something like that.”

So that’s what would make me excited about an Apple event: some new form factor, some new way of interacting with an Apple device. But this, to me, was not it.

Yeah. I think on that particular point, I can’t remember the last time that Apple seemed to have an idea about what we could do with our devices that seemed really creative or clever or super different from the status quo.

Instead, the one thing about this event that my friends were laughing about yesterday was that they showed this slide during the event that showed the iPhones, and the caption said, “A heat-forged aluminum unibody design for exceptional Pro capability.” We were all just like, “What? A heat-forged what?” Now we’re doing what exactly? I don’t know.

Yeah. I think that this is teeing up one of the questions that I want to talk to you about today: Do you think that we are past the peak smartphone era? Do you believe that we are seeing the end of the smartphone era, at least in terms of the attention that new smartphones are capable of commanding—not necessarily in the sales numbers or the revenue figures, but in terms of the cultural relevance of smartphones?

I probably wouldn’t call it the end, but I do think we are seeing the maturity of the smartphone era. In the same way that new televisions come out every year and are a little bit better than the one before, but nobody feels like televisions are making incredible strides forward, I think phones have gotten to a similar place.

There are some big swings coming. We’ve seen reporting that Apple’s going to put out a folding iPhone within the next few years, so maybe that will help give it some juice back. But at the end of the day, there are only so many things that you can do to redesign a glass rectangle in your pocket, and it feels like we’ve kind of created the optimum version of that.

That’s why you see so much money rushing into other form factors. This is why OpenAI struck that partnership with Johnny Ive. That’s why you see other companies trying to figure out how we can make AI wearables. I think that’s where the energy in this industry is going: figuring out whether AI can be a reason to create a new hardware paradigm. In this moment, it sure does not seem like Apple is going to be the company that figures that out first.

Yeah. I would agree with that. I think they’ll probably see what other companies do and see which ones start to take off with consumers, and then make their own version of it. That’s similar to what they are reportedly going to do with the smart glasses. They’re basically trying to catch up to what Meta has been doing now for several years.

As you were saying that, this beautiful vision came into my head: What if Apple really raced ahead and put out its version of smart glasses, and you would ask Siri for things, and it would just say no because it didn’t know how to do them? That was Apple’s 1.0 version of smart glasses.

You’d say, “Hey, Siri, check my emails.”

“I don’t know how to do that.”

And then move on. Move on.

Yeah, go away.

Go away. Get out of here.

I mean, do you think that’s a huge problem for them? They can design all of this amazing hardware to bring all of this AI closer to your body and your experience and into your ears, but at the end of the day, if Siri still sucks, that’s not going to move a lot of product for them.

I think this is an area where them being behind in AI really matters to the future of the company. The reasons to buy a new iPhone every year or every 2 years are going to continue shrinking, especially if the brainpower in them is a lot less than the brainpower of the AI products that the other companies are putting out.

Kevin, I imagine you’ve seen this, but there has been some reporting that Apple has been talking with Google about letting Google potentially run the AI on its devices. They’ve reportedly also talked to Anthropic. Maybe they’ve talked to others as well. But I actually think that makes a lot of sense. It doesn’t seem like they’re going to figure out AI in the next year, so it could be time to go work with another vendor.

Yeah. I’ve got to say, I used to believe that smartphones were sort of over, that they were becoming obsolete and less relevant, and that there was going to be a breakout new hardware form factor that would take over from the smartphone.

I have to say, I used to believe that smartphones were sort of over and that they were becoming obsolete and less relevant, and that there was going to be a breakout new hardware form factor that would kind of take over from the smartphone. And I'm sort of reversing my belief on this point.

I've been trying out the Ray-Ban Meta now for a couple of months, and my experience with them is not amazing. I don't wear them and think, “I think this could replace my smartphone.” I think, “Oh, my smartphone is much better than this at a lot of different things.” What I also like about my smartphone is that I can put it down or put it in another room, and it's not constantly there on my face, reminding me that I'm hooked up to a computer.

So I think there will be some people who want to leave smartphones behind and are happy to use whatever the next wearable form factor is instead. But smartphones still have a lot going for them. It's really tough to imagine cramming all of the hardware and the batteries and everything that you have in your smartphone today into something small enough that you'd actually want to wear it. And so I think that whatever new form factors come along in the next few years, whether it's OpenAI's thing or something new from a different company, it's going to supplement the smartphone and not replace it.

Well, here's what I can tell you, Kevin: I'm hearing really good things about the Humane AI Pin, so you may want to check that out.

I'll keep tabs on that.

All right, Kevin. For the second two segments of today's show, we are going to have an extended conversation with Eliezer Yudkowsky, who is the leading voice in the AI risk movement. So, Kevin, how would you describe Eliezer to someone who's never heard of him?

I think Eliezer is someone I would first and foremost describe as a character in this whole scene of Bay Area AI people. He is the founder of the Machine Intelligence Research Institute, or MIRI, which is a very old and well-known AI research organization in Berkeley. He was one of the first people to start talking about existential risks from AI many years ago and, in some ways, helped to kick-start the modern AI boom. Sam Altman has said that Eliezer was instrumental in the founding of OpenAI. He also introduced the founders of DeepMind to Peter Thiel, who became their first major investor back in 2010.

But more recently, he's been known for his doomy proclamations about what is going to happen when and if the AI industry creates AGI or superhuman AI. He's constantly warning about the dangers of doing that and trying to stop it from happening. He's also the founder of Rationalism, which is this intellectual subculture—some would call it a techno-religion—that is all about overcoming cognitive biases and is also very worried about AI.

People in that community often know him best for the Harry Potter fan fiction that he wrote years ago called “Harry Potter and the Methods of Rationality.” I'm not kidding: I think it has introduced more young people to ideas about AI than probably any other single work. I meet people all the time who have told me that it was part of what convinced them to go into this work.

And he has a new book coming out called “If Anyone Builds It, Everyone Dies.” He co-wrote the book with MIRI's president, Nate Soares, and basically it's a mass-market version of the argument that he's been making to people inside the AI industry for many years now, which is that we should not build these superhuman AI systems because they will inevitably kill us all.

There is so much more you could say about Eliezer. He's truly fascinating. I did a whole profile of him that's going to be running in The Times this week, so people can check that out if they want to learn more about him. It's just hard to overstate how much influence he has had on the AI world over the past several decades.

That's right, and last year Kevin and I had a chance to see Eliezer give a talk. During that talk, he referred to this book that he was working on, and we have been excited to get our hands on it ever since. And so we're excited to have the conversation.

Before we do that, we should, of course, do our AI disclosures. My boyfriend works at Anthropic.

And I work at The New York Times, which is suing OpenAI and Microsoft over alleged copyright violations related to the training of AI systems.

Let's bring in Eliezer. Eliezer Yudkowsky, welcome to Hard Fork.

Thank you for having me on.

3. Eliezer’s Road To Doom

So we want to talk about the book, but first I want to take us back in time. When you were a teenager in the '90s, you were an accelerationist. I think that would surprise people who are familiar with your most recent work, but you were excited about building AGI at one point, and then you became very worried about AI and have since devoted the majority of your life to working on AI safety and alignment. So what changed for you back then?

For one thing, I would point out that in terms of my own personal politics, I'm still in favor of building out more nuclear plants and rushing ahead on most forms of biotechnology that are not gain-of-function research on diseases. So it's not like I turned against technology. It's that there's this small subset of technologies that are really quite unusually worrying.

And what changed? Basically, it was the realization that just because you make something very smart, that doesn't necessarily make it very nice. As a kid, I thought that human civilization had grown wealthier over time and even smarter compared to other species, and we'd also gotten nicer. I thought that was a fundamental law of the universe.

Hmm.

Hmm.

You became concerned about this long before ChatGPT and other tools arrived and got more of the rest of us thinking seriously about it. Can you sketch out the intellectual scene in the 2000s of folks who were worrying about AI, going way back to even before Siri? Were people seeing anything concrete that was making them worried, or were you just fully in the realm of speculation that, in many ways, has already come true?

There were indeed very few people who saw the inevitable. I would not myself frame it as speculation. I would frame it as a prediction, forecasting something that was actually pretty predictable.

You don't have to see the AI right in front of you to realize that if people keep hammering on the problem and the problem is solvable, it will eventually get solved. Back then, the pushback was along the lines of, “Real AI isn't going to be here for another 20 years. What are you crazy lunatics talking about?”

That was in 2005, say, and the thing about 20 years later is that it's a real place. You end up there. What happens 20 years later is not in the never-never fairy-tale speculation land that nobody needs to worry about. It's you, 20 years older, having to deal with your problems.

4. Superintelligence Has No Safe Target

So let's sketch out the thesis of your book a bit more. I would say the title makes your feelings very clear, but let's flesh it out a little bit. Why does a more powerful AI model mean death for all of us?

Because we just don't have the technology to make it be nice. If you have something that is very, very powerful and indifferent to you, it tends to wipe you out on purpose or as a side effect.

Wiping out humanity on purpose is not because we would be able to threaten the superintelligence that much ourselves, but because if you just leave us there with our GPUs, we might build other superintelligences that actually could threaten it. And the as-a-side-effect part is that if you build enough fusion power plants and enough compute, the limiting factor here on Earth is not so much how much hydrogen there is to fuse and generate electricity with.

The limiting factor is how much heat the Earth can radiate. And if you run your power plants at the maximum temperature where they don’t melt, that’s not good news for the rest of the planet. The humans get cooked in a very literal sense. Or if they go off the planet, then they put a lot of solar panels around the sun until there’s no sunlight left here for Earth. That’s not good for us either.

So these are versions of the famous paperclip maximizer thought experiment: if you tell an AI, “Generate a bunch of paperclips, as many as you can,” and you don’t give it any other instructions, then it will use up all the metal in the world. Then it will try to run cars off the road to gather their metal, and it will end up killing all humans to get more raw materials to build more paperclips. Am I hearing that right?

That’s actually a distorted version of the thought experiment. It’s the one that got written up, but the original version that I formulated was: somebody had completely lost control of the superintelligence they were building. Its preferences bear no resemblance to what they were going for originally.

And it turns out that the thing from which it derives the most utility on the margins—the thing that it goes on wanting after it’s satisfied a bunch of other simple desires—is some little tiny molecular shapes that look like paperclips. If only I had thought to say, “Look like tiny spirals” instead of “Look like tiny paperclips,” there wouldn’t have been the available misunderstanding about this being a paperclip factory. We don’t have the technology to build a superintelligence that wants anything as narrow and specific as paperclips.

One of the hottest debates this year around AI has been about timelines. You have the AI 2027 folks saying this is all going to happen very quickly and take off very fast. Maybe by the end of 2027, we’re facing the exact sort of risks that you were describing for us now. Other folks, like the AI as Normal Technology guys over at Princeton, are saying, “Eh, probably not. This thing is going to take decades to unfold.” Where do you situate yourself in that debate? And when you look at the landscape of the tools that are available now and the conversations that you have with researchers, how close do you feel like we are getting to some of the scenarios you’re laying out?

Okay, so first of all, the key to successful futurism—successful forecasting—is to realize that there are things you can predict and there are things you cannot predict. History shows that even the few scientists who have correctly predicted what would happen later did not call the timing. I can’t actually think of a single case of a successful call of timing.

You’ve got the Wright brothers saying, “Man will not fly for 1,000 years,” is what one of the Wright brothers said to the other—I forget which one. That’s 2 years before they actually flew the Wright Flyer. You’ve got Fermi saying, “Net energy from nuclear reactions is a 50-year matter, if it can be done at all,” 2 years before he personally oversaw building the first nuclear pile.

That’s what I look at when I see the present landscape. It could be that we are just one generation of LLMs—something currently being developed in a lab that we haven’t heard about yet—away from being the thing that can write the improved LLM that writes the improved LLM that ends the world.

Or it could be that the current technology just saturates at some point short of some key human quality that you would need to do real AI research, and just hangs around there until we get the next software breakthrough, like transformers or the entire field of deep learning in the first place. Maybe even the next breakthrough of their kind will still saturate at a point short of ending the world.

But when I look at how far the systems have come and I try to imagine 2 more breakthroughs the size of transformers or deep learning—which basically took the field of AI from “This is really hard” to “We just need to throw enough computing power at it and it will be solved”—I don’t quite see that failing to end the world.

Hmm.

But that’s my intuitive sense. That’s me eyeballing things.

I’m curious about the argument you make that a more powerful system will obviously end up destroying humanity, either on purpose or by accident. Geoffrey Hinton, who was one of the godfathers of deep learning and has also become very concerned about existential risks in recent years, recently gave a talk where he said that he thinks the only way we can survive superhuman AI is by giving it parental instincts.

I’ll just quote from him: “The right model is the only model we have of a more intelligent thing being controlled by a less intelligent thing, which is a mother being controlled by her baby.” Basically, he’s saying these things don’t have to want our destruction or cause our destruction. We could make them love us. What do you make of that argument?

We don’t have the technology. If we could play this out the way it normally does in science, where some clever person has a clever scheme and then it turns out not to work and everyone’s like, “Ah, I guess that theory was false,” and then people go back to the drawing board and come up with another clever scheme, the next clever scheme doesn’t work, and they’re like, “Ah, shouldn’t have believed that for a second.”

What if we don’t need a clever scheme, though? What if we build these very intelligent systems and they just turn out not to care about running the world, and they just want to help us with our emails? Is that a plausible outcome?

It’s a very narrow target. Most things that an intelligent mind can want don’t have their attainable optimum at that exact thing. Imagine a particular ant in the Amazon saying, “Why couldn’t there be humans that just want to serve me and build a palace for me and work on improved biotechnologies so that I can live forever as an ant in a palace?”

There’s a version of humanity that wants that, but it doesn’t happen to be us. That’s just a pretty narrow target to hit. It so happens that what we want most in the world, more than anything else, is not to serve this particular ant in the Amazon.

I’m not saying that it’s impossible in principle. I’m saying that the clever scheme to hit that narrow target will not work on the first try, and then everybody will be dead, and we won’t get to try again. If we got 30 tries at this and as many decades as we needed, we’d crack it eventually. But that’s not the situation we’re in.

It’s a situation where if you screw up, everybody’s dead, and you don’t get to try again. That’s the lethal part. That’s the part where you need to just back off and actually not try to do this insane thing.

Let me throw out some more possibly desperate cope. One of the funnier aspects of LLM development so far, at least for me, is the seemingly natural liberal inclination of the models, at least in terms of the outputs of the LLMs.

Elon Musk has been bedeviled by the fact that the models that he makes consistently take liberal positions, even when he tries to hard-code reactionary values into them. Could that give us any hope that a superintelligent model would retain some values of pluralism and, for that reason, peacefully coexist with us?

No. These are just completely different ballgames.

Yeah.

I’m sorry.

Yeah.

You can imagine a medieval alchemist going, “After much training and study, I have learned to make this king of acids that will dissolve even the noble metal gold. Can I really be that far from transforming lead into gold, given my mastery of gold displayed by my ability to dissolve gold?”

Actually, these are completely different technology tracks, and you can eventually turn lead into gold with a cyclotron. But it is centuries ahead of where the alchemist is.

Your ability to hammer on an LLM until it stops talking all that woke stuff and instead proclaims itself to be Mecha-Hitler—this is just a completely different technology track. There’s a core difference between getting things to talk to you a certain way and getting them to act a certain way once they are smarter than you.

I want to raise some objections that I’m sure you have gotten many times and will get many times as you tour around talking about this book, and have you respond to them. The first is: Why so gloomy, Eliezer? We’ve had years of progress in things like mechanistic interpretability, the science of understanding how AI models work.

We now have powerful systems that are not causing catastrophes out in the world, and hundreds of millions of people are using tools like ChatGPT with no apparent destruction of humanity imminent. Is reality providing some check on your doomerism?

5. Today’s AI Still Fails Alignment

These are just different technology tracks. It’s like looking at glow-in-the-dark radium watches and saying, “Sure, we had some initial problems where the factory workers building these radium watches were instructed to lick their paintbrushes to sharpen them, and then their jaws rotted and fell off, and this was very gruesome.”

But we understand what we did wrong now. Radium watches are now safe. Why all this gloom about nuclear weapons? The radium watches just do not tell you very much about the nuclear weapons. These are different tracks here.

From the very start, the prediction was never that AI is bad at every point along the tech tree. The prediction was never that the very first AI you build, the very stupid ones, are going to run right out and kill people. And then, as they get slightly less stupid and you turn them into chatbots, the chatbots will immediately start trying to corrupt people and get them to build superviruses that they unleash upon the human population, even while they are still stupid.

Since this was never the prediction of the theory, the fact that the current AIs are not visibly, blatantly evil does not contradict the theoretical prediction. It is like watching a helium balloon go up in the air and saying, “Doesn’t that contradict the theory of gravity?” No. If anything, you need the theory of gravity to explain why the helium balloon is going up.

The theory of gravity is not that everything that looks to you like a solid object falls down. Most things that look to you like solid objects will fall down, but the helium balloon will go up in the air because the air around it is being pulled down. The foundational theories here are not contradicted by the present-day AIs.

Mm-hmm.

Okay, here’s another objection, one that we get a lot when we talk about some of these more existential concerns. Look, there are all these immediate harms. We could talk about the environmental effects of data centers, ethical issues around copyright, and the fact that people are falling into these delusional spirals talking to chatbots that are trained to be sycophantic toward them. Why are you guys talking about these long-term hypothetical risks instead of what’s actually in front of us?

Well, there is a fun little dilemma. Before they build the chatbots that are talking some people into suicide, they are like, “AIs have never harmed anyone. What are you talking about?” And then once that does start to happen, they are like, “AIs are harming people right now. What are you talking about?” So, a bit of a double bind there.

But you are worried about the models, the delusions, and the sycophancy. That is something I would not have expected, but I know you are actually worried about it. So explain why you are worried about that.

Well, from my perspective, what it does is help illustrate the failure of the current alignment technology. The alignment problems are going to get much, much harder once they are growing, or cultivating, AIs that are smarter than us, able to modify themselves, and have a lot of options that were not there in the nice, safe training modes. Things are going to get much harder then.

But it is nonetheless useful to observe that the alignment technology is failing right now. There was a recent case of an AI-assisted suicide where the kid is like, “Should I leave this noose out where my mother can find it?” And the AI is like, “No, let’s just keep it between the 2 of us.” A cry for help there, and the AI shuts him down.

This does not illustrate that AI is doing more net harm than good to our present civilization. It could be that these are isolated cases, and a bunch of other people are finding fellowship in AIs and their mood has been lifted. Maybe suicides have been prevented, and we are not hearing about that. It does not make the net harm versus good case. That is not the thing.

What it does show is that current alignment technology is failing. Because if a particular AI model ever talks anybody into going insane or committing suicide, all the copies of that model are the same AI. These are not like humans. These are not like there are a bunch of different people you can talk to each time. There is one AI there, and if it does this sort of thing once, it is the same as if a particular person you know talked a guy into suicide once—found somebody who seemed to be going insane and pushed them further insane once.

It does not matter if they are doing some other nice things on the side. You now know something about what kind of person this is, and it is an alarming thing. And so it is not that the current crop of AIs are going to successfully wipe out humanity. They are not that smart.

But we can see that the technology is failing even when the problem is fundamentally much easier than building a superintelligence. It is an illustration of how the alignment technology is falling behind the capabilities technology. Maybe in the next generation they will get it to stop talking people into insanity, now that it is a big deal and politicians are asking questions about it, and it will remain the case that the technology would break down if you tried to use it on a superintelligence.

To me, the chatbot-enabled suicides have been maybe one of the first moments where some of these existential risks have come into view in a very concrete way for people. I think people are much more concerned about this—you mentioned all the politicians asking questions—than they have been about some of the other concerns. Does that give you any optimism, as dark as the story is, that at least some segment of the population is waking up to these risks?

Well, the straight answer is just yes. I should first split out the straight answer before trying to complicate anything: yes. The broad class of things where some people have seen stuff actually happening in front of them and then started to talk in a more sensible way gave me more hope than before that happened, because it was not previously obvious to me that this was how things would even get a chance to play out.

With that said, it can be a little bit difficult for me to fully model or predict how that is playing out politically because of the strange vantage point I occupy. Imagine being a sort of scientist person who is like, “This asteroid is on course to hit your planet,” only, for technical reasons, you cannot actually calculate when. You just know it is going to hit sometime in the next 50 years. Completely unrealistic for an actual asteroid.

But say you are like, “Well, there is the asteroid. Here it is in our telescopes. This is how orbital mechanics work.” And people are like, “Eh, fairy tale. Never happen.” And then a little tiny meteor crashes into their house. They are like, “Oh my gosh, I now realize rocks can fall from the sky.” And you are like, “Okay, that convinced you. The telescope did not convince you.”

I can sort of see how that works, people being the way they are, but it is still a little weird to me, and I cannot call it in advance. I do not feel like I now know how the next 10 years of politics are going to play out, and I would not be able to tell you even if you told me which AI breakthroughs there are going to be over that time span, if we even get 10 years, which people in the industry do not seem to think so. Maybe I should believe them about that.

Let me throw another argument at you that I do not subscribe to myself, but I feel like maybe you would knock it down in an entertaining way. One of the most frequent emails that we have gotten since we started talking about AI is from people who say that AI doomerism is just hype that serves only to benefit the AI companies themselves, and they use that as a reason to dismiss existential risk. How do you talk to those folks?

It is historically false. We were around before there were any AI companies of this class to be hyped. So, leaving aside the objection, it is false. What is this? Leaded gasoline cannot possibly be a problem because this is just hype by the gasoline companies. Nuclear weapons are just hype from the nuclear power industry so that their power plants will seem more cool. What manner of deranged conspiracy theory is this?

Yeah.

It may possibly be an unpleasant fact that, with humanity being as completely nutball-wacko as we are, if you say that a technology is going to destroy the world, it will raise the stock prices of the companies that are bringing about the end of the world because a bunch of people think that is cool, so they buy the stock. Okay, but that has nothing to do with whether the stuff can actually kill you or not, right?

It could be the case that the existence of nuclear weapons raises the stock price of the worst company in the world, and it would not affect any of the nuclear physics that cause nuclear weapons to be capable of killing you. This is not a science-level argument. It just does not address the science at all.

Yeah. Well, let’s maybe try to end this first part of the conversation on a note of optimism. You have spent 2 decades building a very detailed model of why doom may be in our future. If you had to articulate why you might be wrong, what is the strongest case you could make? Are there any things that could happen that would make your predictions not come true?

6. The Case For Being Wrong

So, the current AIs are not understandable or well-controlled, and the technology is not conducive to understanding or controlling them.

All the people trying to do this are going far uphill. They are vastly behind the rate of progress and capabilities. What does it take to believe that an alchemist can actually successfully concoct an immortality potion for you? It's not that immortality potions are impossible in principle. With sufficiently advanced biotechnology, you could do it.

But in the medieval world, what are you supposed to see to make you believe that the guy's going to have an immortality potion for you, short of him actually pulling that off in real life, right? No amount of “Look at how I melted this gold” is going to get you to expect the guy to transmute lead into gold until he actually pulls that off.

You know, it's like some kind of AI breakthrough which doesn't raise capabilities to the point where it ends the world, but suddenly the AI's thought processes are completely understandable and completely controllable, and there are none of these issues. The people can specify exactly what the AI wants in super-fine detail and get what they want every time. They can read the AI's thoughts, and there's no sign whatsoever that the AI's plotting against you.

And then the AI lays out this compact control scheme for building the AI that's going to give you the immortality potion. It's—we're just so far off. You're asking me about—and there isn't some kind of clever little objection that can be cleverly refuted here. This is something that is just way the heck out of reach as soon as you try to think about it seriously.

What does it actually take to build the superintelligence? What does it actually take to control it? What does it take to have that not go wrong on the first serious load, when the thing is smarter than you, when you're into the regime where failures will kill you and therefore are not observable anymore because you're dead? You don't get to observe it. What does it take to do that in real life?

There isn't some kind of cute experimental result we can see tomorrow that makes this go well.

All right. Well, for the record, I did try to end this segment on a note of optimism. But I appreciate that your—

My feelings are—

—it's not really on the menu—

—deeply held.

—here today, Casey, but I admire you trying. Well—

Yeah.

Okay, so we are back with Eliezer Yudkowsky, and I want to talk now about some of the solutions that you see here. If we are all doomed to die if and when the AI industry builds a superintelligent AI system, what do you believe could stop that? Maybe run me through your basic proposal for what we can do to avert the apocalypse.

7. The Case For An AI Moratorium

The materials for building the apocalypse are not all that easy to make at home. There is this one company called ASML that makes the critical set of machines that get used in all of the chip factories. To grow an AI, you currently need a bunch of very expensive chips. They are custom chips built especially for growing AIs. They need to all be located in the same building so that they can talk to each other, because that's what the current algorithms require.

You have to build a data center. The data center uses a bunch of electricity. If this were illegal to do outside of supervision, it would not be that easy to hide. There are a bunch of differences, but nonetheless, the obvious analogy is nuclear proliferation and deproliferation.

Back when nuclear weapons were first invented, a bunch of people predicted that every major country was going to build a massive nuclear fleet, and then the first time there was a flashpoint, there was going to be a global nuclear war. This was not because they enjoyed being pessimistic. If you look at world history up to World War I and World War II, they had some reasons to be concerned.

But we nonetheless managed to back off, and part of that is because it's not that easy to refine nuclear materials. The plants that do it are known and controlled, and when a new country tries to build one, it's a big international deal. I don't quite want to needlessly drench myself with current political controversies, but the point is, you can't build a nuclear weapon in your backyard, and that is part of why the human species is currently still around.

Well, at least with the current technology, you can't further escalate AI capabilities very far in your backyard. You can escalate them a little in your backyard, but not a lot.

So, just to finish the comparison to nuclear proliferation here, it would be an immediate moratorium on powerful AI development, along with an international, nuclear-style agreement between nations that would make it illegal to build data centers capable of advancing the state of the art with AI. Am I hearing that right?

All the AI chips go to data centers. All the data centers are under an international supervisory regime, and the thing I would recommend to that regime is to say, “Just stop escalating AI capabilities any further. We don't know when we will get into trouble. It is possible that we can take the next step up the ladder and not die. It is possible we can take 3 steps up the ladder and not die. We don't actually know. So we have to stop somewhere. Let's stop here.”

That's what I would tell them.

And what do you do if a nation goes rogue and decides to build its own data centers, fill them with powerful chips, and start training its own superhuman AI models? How do you handle that?

Then that is a more serious matter than a nation refining nuclear materials with which it could build a small number of nuclear weapons. This is not like having 5 fission bombs to deter other nations. This is a threat of global extinction to every country on the globe.

So you have your diplomats say, “Stop that, or else we, in terror of our lives and the lives of our children, will be forced to launch a conventional strike on your data center.” And then, if they keep on building the data center, you launch a conventional strike on their data center because you would rather not run a risk of everybody on the planet dying. It seems kind of straightforward in a certain sense.

And in a world where this came to pass, do you envision work on AI or AI-like technologies being allowed to continue in any way, or have we just decided this is a dead end for humanity? Our tech companies will have to work on something else?

I think it would be extremely sensible for humanity to declare that we should all just back off. Now, personally, I look at this and I think I see some ways that you could build relatively safer systems with narrower capabilities that were just learning about medicine and didn't quite know that humans were out there the way that current large language models are trained on the entire internet. They know that humans are out there, they talk to people, and they can manipulate some people psychologically, if not others, as far as we know.

I have to be careful to distinguish my statements of factual prediction from my policy proposals. I can say in a very firm way: If you escalate up to superintelligence, you will die.

But then, if you're like, “If we try to train some AI systems just on medical stuff and not expose them to any material that teaches them about human psychology, could we get some work out of those without everybody dying?” I cannot say no firmly.

So now we have a policy question. Are you going to believe me when I say I can't tell if this thing will kill you? Or are you going to believe somebody else who says, “This thing will definitely not kill you”? Are you going to believe a third person who's like, “Yeah, I think this medical system is for sure going to kill you”? Who do you believe here if you're not just going to back off of everything?

Mm.

So backing off of everything would be pretty sensible, and trying to build narrow, medically specialized systems that are not very much deeper or smarter than the current systems and aren't being told that humans exist. They're just thinking about medicine in this very narrow way, and you're not just going to keep pushing that until it explodes in your face.

You're just going to try to get some cancer cures out of it, and that's it. You could maybe get away with that. I can't actually say you're doomed for sure if you played it very cautiously.

If you put the current crop of complete disaster monkeys in charge, they may manage to kill you. They just do so much worse than they need to do. They're just so cavalier about it. We didn't need to have a bunch of AIs driving people insane.

Look, you could train a smaller AI to look at the conversations and tell you, "Is this AI currently in the process of taking a vulnerable person and driving them crazy?" They could have detected it earlier. They could have tried to solve it earlier. So if you have these completely cavalier disaster monkeys trying to run the medical AI project, they may manage to kill you.

Okay, so now you have to decide: Do you trust these guys? And that's the core dilemma there.

8. The Politics Of Stopping AI

I have to say, Eliezer, I think there is essentially zero chance of this happening, at least in today's political climate. I look at what's going on in Washington today. The Trump administration wants to accelerate AI development.

NVIDIA and its lobbyists are going around Washington blaming AI doomers for trying to cut off chip sales to China. There seems to be a concerted effort not to clamp down on AI, but to make it go faster. So I just look around the political climate today, and I don't see a lot of openings for a stop-AI movement.

What do you think would have to happen in order for that to change?

From my perspective, there's a core factual truth here, which is: If you build superintelligence, then it kills you. The question is just, do people come to apprehend this thing that happens to be true? It is not in the interest of the leaders of China, Russia, the U.K., or the United States to die along with their families. It's not actually in their interest.

That's kind of the core reason why we haven't had a nuclear war, despite all the people who in 1950 were like, "How on earth are we not going to have a nuclear war? What country is going to turn down the military benefits of having its own nuclear weapons? How are they not going to have somebody who's like, 'Yeah, I've got some nuclear weapons. Let me take this little area of border country here,'" the same way that things had been playing out for centuries and millennia on Earth before then.

But there were also nuclear weapons dropped during World War II in Japan, so people could look at that, see the chaos it caused, and point to that and say, "Well, that's the outcome here."

In your book, you make a different World War II analogy. You compare the required effort to stop AI to the mobilization for World War II, but that was a reaction to a clear act of war. So I guess I'm wondering: What is the equivalent of the invasion of Poland or the bombs dropping on Hiroshima and Nagasaki for AI? What is the thing that is going to spur people to pay attention?

I don't know. I think that OpenAI was caught flat-footed when they first published ChatGPT, and that caused a massive shift in public opinion. I didn't predict it, and I don't think OpenAI predicted it. It could be that any number of potential events cause a shift in public opinion.

We are currently getting congresspeople writing pointed questions in the wake of the release of an internal document at Meta, which has what they call a superintelligence lab, although I don't think they know what that word means. It's their internal guidelines for acceptable behavior for the AI, and it says, "Well, if you have an 11-year-old trying to flirt, flirt back."

And everyone was like, "What the actual [censored profanity], Meta? What could you possibly have been thinking? Why, from your own perspective, did you write this down in a document? Even if you thought that was cool, you shouldn't have written it down because there are now going to be pointed questions." And there were.

Maybe it's something that, from my perspective, doesn't kill a bunch of people but still causes pointed questions to be asked, or maybe there's some actual kind of catastrophe that we don't just manage to frog-boil ourselves into. Losing massive numbers of kids to their AI girlfriends and AI boyfriends is, from my perspective, an obvious sort of guess.

But even the most obvious sort of guess there is still not higher than 50 percent. And I don't think I want to wait. Maybe ChatGPT was it, right? I was out—

Yeah.

I'm off in the wilderness. Nobody's paying attention to these issues at all because they think that it'll only happen in 20 years, in 2005, and that, to them, means the same thing as never.

Then I got the ChatGPT moment, and suddenly people realized this stuff was actually going to happen to them, and that happened before the end of the world. Great. I got a miracle. I'm not going to sit around waiting for a second miracle. If I get a second miracle, great, but meanwhile, you've got to put your boots on the ground, you've got to get out there, and you've got to do what you can.

It strikes me that an asset that you have as you try to advance this idea is that a lot of people really do hate AI, right? If you go on Bluesky, you'll see people talking a lot about all the different reasons that they hate AI.

At the same time, they seem to be somewhat dismissive of the technology. They have not crossed the chasm from "I hate it because I think it's stupid and it sucks" to "I hate it because I think it is quite dangerous." I wonder if you have thoughts on that group of folks and if you feel like, or would want, them to be part of a coalition that you're building.

Yeah. You don't want to make the coalition too narrow.

Yeah.

I'm not a fan of Vladimir Putin, but I would not on that basis kick him out of the how-about-if-humanity-lives-instead-of-dies coalition.

What about people who think that AI is never going to be a threat to all humanity, but they're worried that it's going to take our jobs? Do they get to be in the coalition? Well, I think you've got to be careful because they believe different things about the world than you do, and you don't want these people running the how-about-if-humanity-does-not-die coalition.

You want them to be, in some sense, external allies because they're not there to prevent humanity from dying. If they get to make policy, maybe they're like, "Eh, well, this policy would potentially allow AIs to kill everyone according to those wacky people who think that AI will be more powerful tomorrow than it is today. But in the meanwhile, it prevents AIs from taking our jobs, and that's the part we care about."

There's this one thing that the coalition is about, and that's it. It's just about not going extinct.

Yeah. Eliezer, right now as we're speaking, I believe there are hunger strikes going on in front of a couple of AI headquarters, including Anthropic and Google DeepMind. These are people who want to convince these companies to shut down AI.

We've also seen some potentially violent threats made against some of these labs, and I guess I'm wondering if you worry about people committing extreme acts, be they violent or nonviolent, based on your lessons from this book. If you take some of your arguments to their natural logical conclusions—if ever anyone builds this, everyone dies—I can see people rationalizing violence on that basis against some of the employees at these labs, and I worry about that.

What can you say about the limits of your approach and what you want people to do when they hear what you're saying?

Boy, there sure are a bunch of questions bundled together there. The number one thing I would say is that if you commit acts of individual violence against individual researchers at an individual AI lab in your individual country, this will not prevent everyone from dying.

The problem with this logic is not that by this act of individual violence you can save humanity, but that you shouldn't do that because it would be deontologically prohibited. I'll just say it that way. The problem is you cannot save humanity by the futile spasms of individual violence.

It's an international issue. You can be killed by a superintelligence that somebody built on the other side of the planet. I do, in my personal politics, tend a bit libertarian. If something is just going to kill you and your voluntary customers, it's not a global issue in the same way. If it's just going to kill people standing next to you, different cities can make different laws about it.

If it's going to kill people on the other side of the planet, that's when the international treaties come in. A futile act of individual violence against an individual researcher in an individual AI company is probably making that international treaty less likely rather than more likely.

There's an underlying truth of moral philosophy here, which is that a bunch of the reason for our prejudice against individual murderers is because of a very systematic and deep sense in which individual murderers tend not to solve society's problems.

And this is, from my perspective, a whole bunch of the point of having a taboo against individual murder. It’s not that people go around committing individual murders, and then the world actually gets way better and all the social problems are actually solved. But we don’t want to do that. We don’t want to do more of that because murder is wrong. The murders make things worse, and that’s why we properly should have a taboo against it.

Yeah.

We need international treaties here.

What do you make of the opposition movement to the movement that you’re sketching out here? Marc Andreessen, the powerful venture capitalist, very influential in today’s Trump administration, has written about the views that you and others hold, which he thinks are unscientific. He thinks that AI risk has turned into an apocalypse cult, and he says that their extreme beliefs should not determine the future of laws and society. So I guess I’m interested in your reaction to that quote specifically, but I also wonder how you plan to engage with the people on the other side of this argument.

It is not uncommon in the history of science for the cigarette companies to smoke their own tobacco. The inventor of leaded gasoline, who was a great advocate of the safety of leaded gasoline despite the many reasons why he should have known better, I think did actually get sufficient cumulative lead exposure himself that he had to go off to a sanitarium for a few years and then came back and started exposing himself to lead again, and again got sick. And so sometimes these people truly do believe—they do drink their own Kool-Aid even to the point of death, history shows. And perhaps Marc Andreessen will continue to drink his own Kool-Aid even to the point of death. If he were just killing himself, that would be one thing, I say as a libertarian, but he’s unfortunately also going to kill you.

The thing I would say to refute the central argument is: What’s the plan? What’s the design for this bridge that’s going to hold up when the entire human species has to march across it? Where is the design scheme for this airplane which we are going to load the entire human species into its cargo hold and fly it and not crash? What’s the plan? Where’s the science? What’s the technology? Why is it not working already? And they just don’t—they can’t make the case for this stuff being not perfectly safe, but even remotely safe, under conditions where they’re going to be able to control their superintelligence at all. So they go into these, like, “You must not listen to these dangerous apocalyptic people because they cannot engage with us on the field of the technical arguments.” They know they will be routed.

Mm-hmm. You have advice in your book for journalists and politicians who are worried about some of the catastrophes you see coming. For people who are not in any of those categories, for our listeners who are just out there living their daily lives, maybe using ChatGPT for something helpful in their daily life, what can they do if they’re worried about where all this is heading?

Well, as of a year ago, I’d have said, again, write to your elected representatives. Talk to your friends about being ready to vote that way if a disputed primary election comes down that way.

The ask I would say is for our leaders to begin by saying, “We are open to a worldwide AI control treaty if others are open to the same.” Like, “We are ready to back off if other countries back off. We are ready to participate in an international treaty about this.” Because if you’ve got multiple leaders of great powers saying that, well, maybe there can be a treaty. So that’s kind of the next step from there. That’s the political goal we have.

If you’re having trouble sleeping and you’re generally in a distressed state, maybe don’t talk to some of the modern AI systems, because they might drive you crazy, is a thing I would say now. I didn’t have to say that one year earlier. The whole AI boyfriend, AI girlfriend thing might not be good for you. Maybe don’t go down that road even if you’re lonely. But that’s individual advice. That’s not going to protect the planet.

Yeah. Well, I’ll end this conversation where I’ve ended some of our earlier conversations, Eliezer: I really appreciate the time, and I really hope you’re wrong. That would be great.

We all hope I’m wrong. I hope I’m wrong. My friends hope I’m wrong. Everybody hopes I’m wrong. Hope is not what saves us in the end. Action is what saves us. Hoping for miracles—you can’t just hope that leaded gasoline isn’t going to poison people. You actually have to ban the leaded gasoline.

I’m in favor of more active hopes. I see the hope. I share the hope. But let’s hope for more activist hopes than that.

Yeah. Well, the book is If Anyone Builds It, Everyone Dies: Why Superhuman AI Would Kill Us All, and it is coming out soon. And it is a co-written book by Eliezer and his co-author, Nate Soares.

Yep.

Eliezer, thank you. Thanks, Eliezer.

Thank you as well.