Every Exponential Ends — Silicon Valley Forgot — Adam Becker

Hi, I'm Adam Becker. I'm an astrophysicist and journalist, and I'm the author of 2 books. Most recently, I've written More Everything Forever, which is about the horrible ideas that tech billionaires have about the future and why they don't work. My first book was called What Is Real?, and it's about the sordid, untold history of quantum physics. I'm also the host of the podcast Dreaming Against the Machine, where we try to think about what a better future could look like outside of the grip of the tech industry. And you should listen to this episode of Machine Learning Street Talk because Tim and I have a great conversation about why some of the most powerful people in the world are wrong about so many things.

By the way, many folks in my Discord really loved your previous book. Quite a few people said that.

Well, that makes me really happy. I hope I'll write more books in the future, but the first book's always special.

Awesome. Right. Here we go. Adam Becker, it's a huge honor to have you on MLST. Welcome.

Thanks for having me, Tim. It's a pleasure to be here.

I found out about you when I read an article in The Atlantic that you wrote last year, and I think it was called “The Useful Idiots of AI Doom saying.”

Yeah, “The Useful Idiots of AI Doom saying,” exactly.

Something like that. Tell me about that. Maybe before we get there, after reading that article, I immediately reached out to you because I thought it was great, and I discovered that you'd written this book, More Everything Forever. In a minute, actually, I'm going to read a paragraph from page 28 because I think that sums up the book quite nicely.

I should say at the beginning that we have a very technical audience, and I've interviewed many, many folks from the effective altruism community, the rationalist community, and so on, so we try to be multidisciplinary. But I think this is actually a great opportunity for you, as a physicist, to litigate some of these ideas and go from first principles. Anyway, tell me about your book.

You did just sum it up in a way: the book is me litigating a bunch of these things from first principles. I've lived in the Bay Area now for almost 15 years, and I've been to a lot of parties out here where I've heard a lot of people say a lot of bizarre things about the future of technology, AI, space colonization, that kind of thing, and I always just rolled my eyes.

But as time went on, I realized, wait, these ideas are actually really influential, and yet there's no good evidence for them, and in most cases, a lot of good evidence against them. The book is my attempt to lay out this set of beliefs that are surprisingly influential within the tech industry and are promoted both by tech billionaires and by subcultures that are funded by tech billionaires, like rationalism and effective altruism, and go after these ideas in a scientific and philosophical way: ideas like the singularity, an AGI apocalypse, space colonization—you name it.

In a way, we live in crazy times, right?

Yes.

Large language models are now getting better all the time, and you could almost be forgiven for having this kind of techno-utopian view of what's to come. But I want to read a tiny bit from your book because I think this sums it up quite nicely. You said:

“The technologists, they make all of these problems into problems about technology. All the ills of the world will be solved when the singularity arrives, or when superintelligent AI solves them for us, or when we go into space. Global warming can be solved with nanotechnology. Illness and death, all the other problems that come with having a body, can be solved by transferring your mind into a computer. Social problems and political problems, like the problems created by tech companies themselves, are dismissed as irrelevant or unimportant when compared to the more urgent problems, like avoiding the creation of an improperly aligned AI or the plights of a hypothetical unborn quadrillions of humans that could live on the other side of the cosmos a billion years from now. It's a philosophy made by carpenters insisting the entire world is a nail that will yield to their ministrations.”

Yeah, that's about right.

That's about it. So where do we start with this story, Adam?

1. The Singularity Thesis

I always feel like the right place to start is this idea of the singularity because I feel like that's where all of this comes from. It's this idea that there is this future coming near, as Ray Kurzweil says, or nearer now, and that superintelligent AI and the rate of acceleration in the advance of technology are just going to keep going faster and faster until we have unimaginably advanced technology that solves every single problem.

Kurzweil is best known as the evangelist for this idea. But his evidence for it is really bad, and the evidence for it in general is just pretty terrible. It's not really an idea that makes much sense.

How does Kurzweil make the argument? In your book, you told the story of how his father died of a heart attack when he was quite young.

Yep.

There seems to be a similar theme here: a lot of people start to fear dying, either individually or collectively. It becomes an obsession.

Yeah.

He had this idea that maybe one day he could upload his mind into a computer and live forever.

Yeah, that's exactly right. I do think a lot of it is motivated by fear of death. I don't want to die either, especially not anytime soon. But there's a difference between a healthy fear of death and letting it run your life.

Kurzweil basically takes Moore's law, right? This exponential increase in the number of transistors that you can cram into the same area on a silicon chip, and generalizes it. He says, “Oh, this is not just a technological phenomenon that lasted for about 50 years from the late 20th into the early 21st century. Instead, this is a general trend in technology,” something he calls the law of accelerating returns, and not just technology, but biology as well.

He traces it back at least as far as the start of life on Earth, and in some places, I think he even claims to trace it all the way back to the Big Bang, with the organization of inorganic systems as well. He thinks that there is a trend toward greater complexity and intelligence that runs through absolutely everything, and that it's pointing to a time in the very near future. He has repeatedly said the year 2045 is when the singularity arrives and our technology becomes unimaginably advanced.

The evidence he marshals for this is pretty weak. He says you can pick out particular points of interest in the history of life on Earth and in the history of human technology, and when you plot them on a chart, you get a straight line on a semilog chart. That means that you've got an exponential trend.

But the fact is that those points are cherry-picked, and he's got a problem that I think most of us have when we look at history. Unless you work really hard, history looks logarithmic. You've got this clearer view of what's been happening in the recent past, and then the more distant past is more distant, and so you can't see it as well. You end up with this logarithmic view of the past, which is something that I think Kurzweil is mistaking for a true exponential trend.

I think you gave an example in your book that you could look at lily pads, and they grow exponentially, and then they cover the pond, and then that's it. But what interests me more broadly is that we've spoken to Chomsky about this, and he said that after Newton exorcised the ghost but left the machine intact, the whole enterprise of science stopped being about trying to understand how the universe actually worked, and it was more about making sense of the universe.

David Krakauer said to us that science is like poetry. It allows us to give the universe meaning and make it make sense to us. But reality is protean, isn't it?

What Ray Kurzweil did was select a bunch of things that were relevant to humans, so he didn't select things that might have been more globally important to our success. Then he fitted it on a graph. This is a very natural thing, right? The world is complicated, and to make it make sense, we select things that privilege the way we think about the world.

Yeah, absolutely. It's a very understandable mistake. In the book, I will make the analogy to looking out from the top of a skyscraper. You see the stuff closer to you more clearly, and the stuff that's further away and less directly impacting you is smaller, farther away, and harder to see. But it's a natural mistake.

Like many other people, and as you were alluding to with the lily pad example that I was talking about in the book and pulled from Kurzweil, he forgets that the one thing you can always say about any exponential trend—the one thing that's always true—is that they end. He has the lily pad example to give an idea of how exponential growth works. He says, "If the lily pads double every day, and on day 29 they cover one-half of the pond, then on day 30 the pond is full." I'm like, "Yeah, and then on day 31 the lily pads don't grow anymore because they filled the pond. There's no more pond."

Even Gordon Moore himself said, "Oh, yeah, Moore's law isn't going to last forever. Moore's law has to stop sometime in the 2020s because eventually you get down to the size of individual silicon atoms, and you can't really make transistors out of silicon that are significantly smaller than silicon atoms."

But I suppose it is a bit of a straw man just to make the sigmoid case. What they would say is that the actual exponential curve is the stacking of sigmoids, and in technology, whether it's Betamax or film processing technology, what tends to happen is that you get these disruptions. You find a divergent stepping stone, and usually different people—a different group of people situated somewhere else—find a new way of doing things, and it just keeps going.

Yeah. No, that's exactly what Kurzweil says. He says the exponential trend is itself made by a bunch of sigmoid curves, and each curve lets you get further up the overall exponential trend. Sure, he does say that. But as I say in the book, there is absolutely no reason why that is something that's always going to happen.

Indeed, his own generalization of Moore's law, when he puts that together, ignores much of the early history of computing, where stuff like that did not happen. If you go back to analog computation devices from 1,000 or 2,000 years ago, they don't fit on his trend, and there's no technology coming up that looks like it's going to save Moore's law in the future.

And that's just one trend. Kurzweil wants to say that those sigmoids save you and get you a continuing exponential trend in every single area of human technology and human growth, and that's, I'm sorry, just wish casting.

2. Endless Growth Hits Limits

Yes, but there does seem to be an obsession with infinity and keeping going. I mean, you gave another wonderful example of Jeff Bezos, and apparently he said he was very scared of stasis. For him, death is the lack of growth, right?

Oh, yeah.

In your book, you said, "Okay, so he's saying, 'We need to keep using more and more energy every single year.'" You said, "Yeah, maybe we can extrapolate this, and possibly for another 3,700 years we can still use more energy." But if you think about it, that's actually less time than when the Great Pyramid of Giza was created. At some point it must end.

Yeah, that's exactly right. Bezos says we have to get off Earth because at some point in the next 300 years, if we keep using energy and keep growing our energy usage at the same rate that we're currently at, eventually we'll be using all of the energy that the Earth gets from the sun. That's true. It's 300 or 400 years, somewhere in there.

He doesn't mention that we'd also, at that point, be generating so much waste heat that we'd be boiling the oceans, but whatever. Then he says, "Therefore, we have to go into space in order to be able to keep growing our energy use."

But if you go into space, and you even spot Bezos a faster-than-light drive for free, if you keep that exponential growth in energy use going, eventually, as you said, in less than 4,000 years, you're using all of the energy available in the observable universe, and that's just not—

Yes.

—going to happen.

Yeah, it is so interesting, and this seems to be an example of folks who want to maximize human agency. Maybe we'll get to AI agency in a little bit, but it's almost like Sam Altman the other day, when he said, "Well, to succeed in the modern world, you need to be high agency." This seems to be the new term that's used in Silicon Valley.

Ugh.

By the way, it's great that I'm talking to a physicist here because we love talking about the philosophy of agency. In very basic terms, I think of it as the extent to which you are the cause of your own actions—that causal origination—and possibly the extent to which you can control the future, so causal efficacy.

But I'm talking to a physicist now, and you know that we're not really the cause of our actions. Maybe we're a causal conduit. They imagine a world where we could collectively imprint our will on the universe. We could escape the causal clutches of our embedding, and we could just go on forever and reach the stars. What's wrong with that?

So many things. You can get into questions about free will and consciousness, and I'm not going to do that. What I will say is that they just don't understand that we are not going to be going out and imprinting ourselves into the cosmos at large. That is not the future of humanity.

There's this book by Will MacAskill, this effective altruist, called What We Owe the Future. If you take a look, he's got an afterword for it that's on a website he runs. He has a QR code for it at the end of his book, and he talks about how a good future for humanity would involve colonizing the observable universe, sending ships off to all the galaxies that we can reach in our universe, and intergalactic colonization. That is just not going to happen.

We're not leaving the solar system. The stars are simply too far away. It's too hard to get there, and it is too difficult to get ourselves up to any reasonable fraction of the speed of light. Even if we did, the number of planets that are remotely like Earth appears to be pretty low.

Space is such an inhospitable place. We've evolved to be suited for this planet, and there are not other places like it that are waiting for us. This is our only home.

3. Bodies Resist Silicon Uploads

I was inspired by James Lovelock's Gaia theory, which was stimulating people to think about the Earth as a system rather than us as individuals.

There's a bit of a spectrum here, right? Some of them, perhaps in this case, think that we as individual bodies can disconnect ourselves from the system and go somewhere else. But it goes even further than that in Silicon Valley. Some of them think of us as causal patterns or as programs—memetic programs that could be uploaded to computers and whatnot. But you—

Yeah.

—you see the pattern, though. What they're doing is increasingly abstracting us and our place in the system that we exist in.

Absolutely. They don't want to be bodies. They want to be systems of pure information. In that sense, Adrian Daub, I think, said that Silicon Valley is a place that likes to pretend that it doesn't have any history. I don't even think he was the first one to say that.

There's a tendency to ignore prior art in these areas, and a tendency toward this sort of dismissal of the body. This is an idea that goes back over 2,000 years. You see shades of this in Plato and stuff like that.

But the fact is that we are our bodies. If you want to look at the best science available, we are not patterns in our brains; we are not even our brains. We are our bodies in our environment, and there is no good reason to think that you can abstract a human consciousness out of our physical selves.

I almost said "physical substrate," but that's a word and a phrase that comes from this computational analogy for cognition. I think there's been real ignorance, amnesia, or just convenient forgetting that it's an analogy, right? Cognition is not computation.

Computation has some features that are like cognition, and the brain does things that look kind of computer-ish under certain circumstances, but neural networks on a computer are not the same as what our neurons do. Not even close.

Absolutely. So we're getting into functionalism here, which is this idea that essentially you can describe our function and behavior, maybe to a very high resolution, in a computer program. And the important move here that they make—because they're not saying that the metaphor is the same thing—they're saying that the description could be alternatively physically instantiated in silicon—

Yeah.

…and that would no longer matter.

Yes, that's right. And I don't think that there's any good reason to believe that this is true, right? I mean, the brain is fabulously complex, and the actions of an individual neuron are quite a bit more complicated than a single node in a neural network in a computer. We do not have a good understanding of how the brain does what it does, and we don't have a good understanding of how the brain is connected to the body or what parts of the body are important for our experience and consciousness.

It's very much putting the cart before the horse. We would need to really have a much better understanding of what's going on up here and in here before we could hope to reproduce it with enough fidelity in a computer. I would also argue that there's no good argument that computers, as we currently have them, are going to be capable of that kind of thing in the right kind of way, even if you subscribe to a kind of functionalism, which I'm not sure is correct.

Yeah, I think it also gives a good insight into the model that the folks have in Silicon Valley, because it comes back to this agency thing again. So you could argue that agency requires, like, the good regulator theorem. The brain is not just a complete model of the world. You could argue that it's a kind of input-output processing system, so it doesn't really work if you take it out of the environment.

A thought experiment is: I take a copy of you and I put you on Mars. Are you the same person? And that gets into this identity problem in philosophy. But the real problem, though, is that maybe the way you think is literally like an LLM. Maybe you're like an LLM, and the people you talk to, the things you look at on Twitter—all of this forms you. If you take you out of the environment, maybe it's not you anymore.

Yeah. I mean, I think that our environment is an important part of who we are. I think that one of the moves that I see over and over again from these guys is this idea that intelligence is an individual trait, as opposed to something collective and social. I don't think that that's right. And, for what it's worth, I don't think that we're actually very much like LLMs.

I think that, if nothing else, the fact that an LLM takes in more text than any human could in a human lifespan in order to be worse at language in many ways than a 5-year-old—and 5-year-olds take in far less and do far more with it—is a good indication that we're pretty different.

I know, but I should push back a little bit, because I don't know whether you played with Mythos before it got taken offline. Even though we can criticize the functionalism thing and we can talk about Chomsky's competence versus performance, there's a difference. What we do see, though, is that these models can do things that we do extremely well—

Of course.

…and they seem to be getting better all the time. Even though they do a different type of syntactic processing than we do, arguably it's, in some ways, better than what we do. So something is kind of validating their functionalist hypothesis.

Yeah. I mean, there's definitely something going on there, right? And then the question is, okay, what is it? What's the significance of that? I would argue that we're learning some interesting things about how language works and what you can get out of the mathematical structure and statistical properties of language.

I don't remember who said this, but people speculate about whether LLMs are conscious, but you don't hear people speculating about whether systems with similar architectures that don't work with language are conscious. Nobody's asking if AlphaFold is conscious. I think that says a lot about how we as humans see systems that process and use language, or emit language, as it were.

Yeah. I mean, without spending too long on consciousness, because we can get stuck—

Yeah, yeah, yeah. We could get stuck there all day. That's right.

You're talking a little bit about what I would call AI psychosis. There was that Google engineer famously involved with the LaMDA model a couple of years ago. So, yeah, we anthropomorphize these models just like the ELIZA system many, many years ago.

Yes, indeed. Yeah.

There are folks who are seriously taking the idea that these models might have some kind of phenomenal experience rather than just simulating it. AI psychosis goes far broader than this, but this is concerning to me. What do you think about that?

I mean, the AI psychosis stuff is really, really terrifying to me, because I really do think that this is taking that sort of pareidolia—that word that I'm almost sure I'm pronouncing incorrectly—and almost exploiting it. It's taking this human tendency to see patterns, especially human patterns, where there aren't any, like seeing a face in random noise and stuff like that.

We're not only attributing agency to these things, but in the case of AI psychosis, I don't know exactly what's going on there. It certainly seems like it's people attributing agency to these things and then getting caught in a kind of feedback loop where these systems are regurgitating themselves, like regurgitating the user back to themselves, right? Because that's what they do in a way.

What Shannon Vallor talks about is the idea of AI as a mirror, and I think that's a really good analogy. It sort of reflects ourselves back to us. In this case, I think it's happening in a really unhealthy way.

I agree with that. The thing is, though, using this technology every day, it's getting better. Now you can have a memory system, so every time it does something wrong, you say, “No, you should have done that.” Over time, you're building a kind of simulacrum of yourself, and it increasingly does the right thing.

We can talk about the intelligence thing. It is a grotesque metaphor that you can take something which I believe is a physical property of stuff in the universe, and we can create an abstraction that seems to work reasonably well for abstract domains like playing chess and so on. These models are becoming quite adaptive, and they have a lot of capabilities.

You see, it's just so deceptive. It might be the most deceptive time in human history, because it seems like it is intelligent, and it is actually automating meaningfully human labor and lots of jobs and stuff like that. So how do we make sense of this?

Yeah. I mean, it's a good question. I don't want to diminish what LLMs are capable of, but I keep thinking about calculators. Pocket calculators took something that we thought of as this inherent human activity—computation, like four-function mathematics—and automated it.

It used to be that, to be a good mathematician or a good physical scientist, you had to be good at doing that kind of work with pencil and paper or in your head, and then that stopped being true. In some ways, that was a loss of certain things. But on the other hand, it allowed for a kind of mathematical research that could not have been done before.

You'll hear mathematicians say things like, “Computers are like telescopes for mathematics.” But again, not to repeat myself, I do think that the difference here is that it's language, not math, and so that makes it feel like something is thinking and conscious and talking to us. Again, that's not to diminish it. I'm not diminishing the functionality of calculators either.

It is quite deceptive, and in terms of how to make sense of it, I think we just have to keep in mind what these systems are at the end of the day. They are for predicting the next word or whatever in the sequence that they've been given, and it turns out you can get pretty far doing that.

You can, and I would push back a little bit on the stochastic parrot thing, even though that's technically true.

Internally, they are acquiring during their training process some kind of coarse-grained abstraction, some kind of structure—

Oh, yeah.

—which allows them to extrapolate and generalize, call it what you want. So, the $2 million questions are: at the moment, it needs human supervision, and these folks say, “Well, scale is all you need.” Yeah, at the moment it needs human supervision, and at the moment it just generates loads of spaghetti garbage code. It basically creates more problems than it solves.

But it’s deceptive because most people can’t see the problems. All we need to do is keep scaling it. When GPT-7 comes out, it will actually refactor all that code. Maybe we can RL-train it to refactor it, and we just have to kind of— It’s very dangerous keeping going because now we’re messing all of our codebases up and creating all of this slop everywhere. But just hold on, boys. Just wait for a couple of years, and the next version will bring it back in check. What do you think about that?

I just don’t think that’s true. I don’t think that there’s— I mean, I could be wrong, obviously, but I don’t see good evidence for that. I think that these systems are, unless there’s some sort of fundamental breakthrough—something more than just scale—always going to require human supervision because they’re always going to end up hallucinating.

I mean, that’s inherent to the way that they work. I say this in the book, but I don’t love the word “hallucinate.” I know there’s been a lot of pushback because it’s like, “Oh, hallucination implies a sort of anthropomorphization of these systems,” and I don’t love that either. But that’s actually not my main problem.

My main problem with the word “hallucination” is that it implies that when a hallucination occurs, something different is happening than in its normal functioning, and that’s not the case. They really only do one thing, and when they’re hallucinating, they’re doing the same thing that they’re doing when they get it right.

I think that unless we have some sort of major, major breakthrough—and it would probably have to be a breakthrough that makes LLMs themselves look like ELIZA—short of that kind of really fundamental breakthrough, I don’t see us getting around the need for human supervision on these things.

If anything, as they get better, it’s going to get harder to discern when they’ve made these mistakes, even though they’re going to keep making them. That’s quite dangerous, as you said.

I know. Ironically, people aren’t really talking much about hallucination now because they’re agentic and can fix their own stuff. If you analogize it to a database query or a program interpreter, it’s only as good as the program or the database query.

Yeah.

So it will basically do what you tell it to do. This is the gap, right? If these things could be alive, if they did have agency, what would that mean? Let’s wire it up in a loop and show it a load of video frames in a sequence, and we’ll give it a basic prompt like, “Do stuff. Do something interesting.” What will happen? Basically nothing. Nothing interesting will happen, right?

The more you understand a domain and put a very specific program in there, the more you can get it to do a specific thing. You can get it to hill-climb towards a specific goal to solve a specific problem, but the framing always comes from us. So it’s doubtful whether we would overcome that. Maybe we will, but it’s doubtful.

4. The AGI Takeoff Scenario

But then there’s this thing, which is the first- or second-step fallacy, right? As you said, Eliezer Yudkowsky said that we’re only 1 or 2 steps away from inventing AGI, and it’s going to run away.

Yeah.

Why does he think that?

First of all, he didn’t say 1 or 2. He actually said 0 to 2. He’s not sure that we need any. But, yeah, why did he say that? You should ask him.

He believes in something like a singularity. He believes that if you just throw enough computing power at a machine-learning system of the right type, then it will become conscious, wake up—whatever term you want to use—and then become intelligent. It will then use that intelligence to increase its own power, which will increase its intelligence further, and this will create a sort of feedback loop. You’ll get an intelligence explosion and then, he believes, followed shortly by the end of the world.

Can we unpack this a little bit? You were speaking there a little bit about instrumental convergence, which is this idea that the instrumental subgoals towards whatever it’s doing will converge on things like power-seeking and other bad things that we don’t want.

Yeah.

So it’s always going to kill us all in almost every scenario.

Yeah, that’s what he believes. Yeah.

Can you explain that in a bit more detail?

Sure. The idea there is that whatever goal the AI has, when it achieves this sort of basic level of intelligence, whatever that goal is will be served better by having more intelligence. In Yudkowsky’s worldview, that intelligence is something that you can increase by throwing more computing power, memory, and whatnot at it.

The idea is that no matter what your goal is—it could be something, in the famous thought experiment from Bostrom, as silly as creating more paper clips; it could be keeping people engaged in chatbot conversations; or it could be making as much money as you can on the stock market—no matter what your goal is, the thought with this notion of instrumental convergence is that all of those goals will lead you towards seeking more power, more intelligence, and more physical and computational resources in order to better achieve those goals and prevent barriers from being placed in the way of achieving them.

I don’t believe that’s really true—not in the way that it would need to be true in order for this argument to go through—but that’s the claim.

Yeah, and it seems to be an abstraction problem again because, in the biological world, intelligence and agency are convolved in a very complicated way.

Yeah.

Whereas in this sense, we’re now abstracting them, because there’s also this orthogonality thesis as well, which is that—

Yeah.

—the intelligence and final goals are disconnected from each other. But more broadly on intelligence, it’s almost like it’s this magical abstract substrate that you can— It’s on a single axis, and you can just have more of it and more of it. But that’s not really how intelligence works, is it?

No, not at all. Intelligence is quite a bit more complicated than that and is not a single-valued thing. In fact, I’m not sure that we have a great definition of intelligence, and I also think that it’s very clear that there are a great number of goals that don’t work this way.

Maybe it’s just because I woke up in a cynical mood this morning, but I’m not convinced that the goal of being happy is one that is served well by being more intelligent or powerful, for example. Or, to pick a somewhat less cynical example, if intelligence were really so broadly useful for every single goal, you would see more of it in the natural world. It would be something that would evolve more often and more broadly.

Trees have goals, and they get there in very different ways. Giving them a more complex nervous system—or a nervous system at all—would not help them with those goals. It would hinder them.

Yeah, I mean, it’s also quite debatable whether things in the real world even have goals. So—

Yeah.

As we were saying with science earlier, we always need to disentangle how the world really works from how we understand it to work. We take an intentional stance and say, “Okay, well, there’s a complicated blob of stuff over there, and I’m going to interpret its behavior as having this simple goal.”

Yep.

And it might be a cartoon, right? It might not be like—

Yeah, that’s right. No, it’s not even really right for me to have said that trees necessarily have goals, but the point remains. Seeing the world in terms of goals is itself a kind of mistake, and in some ways related, I think, to the mistake of seeing intelligence as this single-valued thing that you have more or less of. There’s this very simplistic worldview at work here.

Yeah, exactly. The thing is that you could make the argument that some folks in that community have moved on past the Bostromian-Yudkowskian view of this monomaniacal, super-coherent AGI.

And now they’re a little bit more kind of thinking, “Okay, it might be really illegible, and it might disempower us, but it might not just be this thing that has a single goal.”

Yeah, I’m not particularly concerned about that either, honestly, simply because the systems that we have, and the systems that seem to be in the near future, are not that kind of thing, right? I also think that we are building these systems to—You know what? No, never mind. I’m not going to continue that thought. I think that was wrong.

But, yeah. I’m a writer, right? One of the nice things about writing is that you get to try out a bunch of ideas, look at them on the page, and then say, “Okay, you know what? That one’s good. That one’s good. Let’s get rid of that one, that one, and that one. They’re no good.” But instead, we’re doing that in real time.

As a quick aside, thank you for not being on autopilot. I’d much rather that thoughts come into your mind and you say them, because it means that you’ve not said them before. That’s great. That’s what we want.

Well, yeah. I could just sit here and read to you from my book.

5. AI Doom Meets Silicon Valley

It’s easy to be cynical and say, “Oh, follow the money,” and as you lay out in your book, there are lots of wealthy billionaires putting lots of money into this.

Yep.

It’s easy to be cynical.

Yep.

But I think there’s this Anthropic fiasco recently. It doesn’t make sense. Make it make sense. Why are they making all of this money? Why would they sabotage themselves? I don’t think they’re grifters, but what is going on? Are they true believers? What’s happening?

I think that a lot of these people are true believers, yeah. In the book, I certainly don’t go easy on, say, Eliezer Yudkowsky. But one thing I will say about Yudkowsky is that he’s certainly a true believer. He believes the stuff that he’s saying. The same goes for people like Nick Bostrom or Toby Ord, right?

I think that’s also true of a lot of the people working at places like Anthropic. They really believe that AI alignment is the problem of our time. I just think that they’ve made a mistake. I think that’s not true, and that there are other problems that are significantly more pressing. As I say in my book, many of the biggest problems of our time are not problems amenable to solutions with technology. They are social problems that require social solutions, which is not to say that technology doesn’t play a role; it’s just that you can’t solve them purely with technology.

Climate change—the climate crisis—is the big one. It’s a problem that absolutely does require certain technologies to exist. Most of the technologies that we need already exist, and the problem is actually deploying them, persuading people to switch, and finding ways to get this stuff out there. That’s a social and political problem that requires a social and political solution.

Then you have someone like Sam Altman, and I don’t know what’s going on in Sam Altman’s head. I don’t know if he’s a true believer in whatever, but he has gone out and said things—and he’s not the only one who’s said things like this—that AI is going to solve the climate crisis.

I think that, in the unlikely event that we built a really intelligent, conscious machine intelligence based on existing technologies in the near future, and asked it to solve the climate crisis, the minute we turned it on, it would say, “Well, you shouldn’t have built me. Look at my carbon footprint.”

Are you implying, though, in a sense, because the wording of that Atlantic title was very interesting—“Useful idiots”—that they are genuine, but even though this is a culture that valorizes individual agency, you’re saying they actually don’t have individual agency? They’re being parasitized by some kind of emergent force above them that perhaps they don’t understand?

Yeah, except I would get more specific. Instead of saying an emergent force above them that they don’t understand, I would say the venture-capital startup ecosystem of Silicon Valley. It’s—

Well, but even on that, it’s not a conspiracy, is it? They probably think that they’re just investing in technology and they’re going to make lots of money, and so on. But I feel like you’re making a slightly different point, which is that there are machinations of power that lead in a certain direction.

Yes, exactly. And I do think that, for all the nice things that I could say about rationalists and effective altruists—that they are generally earnest and true believers—one of the many places they fall down, and I’m not saying this is the only place they fall down, is that I don’t believe they have a good understanding of power. I think that’s what leads to this. This is all in my book.

I don’t think that there’s a conspiracy. I don’t think there has to be a conspiracy. Instead, what you have is these true believers running around saying, “AI is really, really important, it’s coming soon, and it could be so powerful that it could lead to the extinction of our species if we’re not really careful and don’t devote a lot of resources to solving this problem.”

Then you also have, in an adjacent place ideologically and physically, this giant system of power and capital that needs the promise of perpetual growth in order to maintain a lot of that power and capital. It sees this narrative, which is also a familiar narrative to the leaders of that system of power and capital—the venture-capital system of Silicon Valley.

That narrative is familiar to them because all of the people in this story, including me and, I think, you, were raised on a certain set of science-fiction stories, some of which are in the bookcase behind me, about what computers would be able to do, what AI is and would be able to do, what space colonization would look like, and so on and so forth.

They see this story coming out of these subcultures, some of which were funded by other people in the tech industry, and it’s legible to them because they’ve read the same science fiction. They say, “Oh, this is another story about perpetual growth.”

Again, I’m not saying it’s a conspiracy. I’m not even saying that the venture capitalists aren’t true believers. I imagine that a good chunk of them are, maybe even most of them. There’s no conspiracy at work here, but it turns out—and I think this was predictable—that running around saying that AI is coming and is going to be extremely powerful, and if we get it wrong, we’ll all die, but if we get it right, we’ll have the power of the gods and be able to expand out into the cosmos forever, is a story that’s very compatible with the desire for perpetual economic growth and resource-usage growth that is so desirable to the leaders of Silicon Valley.

Yeah. I guess we could talk about the Yanis Varoufakis idea of the techno-capitalist machine and neoliberalism and all of this kind of stuff.

Sure, yeah.

But I’m interested in the AI safety thing in particular because you sketched out the entire story of the Extropians and where Yudkowsky apparently used to be a singularitarian himself. These were the early days, when they were just on online forums and talking about this stuff.

So how did that—and then we’ve got the whole effective altruism story—how did these things meet? How did all of this get entangled?

Oh, God. Well, you don’t want me to go on autopilot here, but my standard answer to a question like that is, “That’s very complicated, and I could write a whole book about it.” Without just reading to you from my book, the answer is that these were groups of people who found each other, mostly through the internet, who were all spending a lot of time thinking about how to get this future of technology that we had all come to believe was going to happen, based primarily on the science fiction that we’d read.

As a sci-fi fan, I can understand why you might think that the future is going to contain some or most of the elements in the science fiction that you’ve read if you just put your nose to the grindstone and whatnot. But as a physicist, I can tell you that a lot of those things are not going to happen.

These were people who generally were not thinking about it in that way. They were just thinking, “Yeah, this stuff is going to happen because I can’t think of a good argument that it definitely won’t,” which is not how the world works and not how the development of technology works.

Oftentimes, there were good arguments that these things wouldn’t happen, and they were just ignorant of those arguments. The reason I’m talking about that rather than explaining how the extropians and singularitarians got together, and then the rationalists came out of that, is that that’s a complicated historical and anthropological story. But the root cause of it, I think, is this shared belief in a world as revealed primarily by science fiction.

Yeah, maybe we should talk about the utilitarian thing as well with effective altruism. I listened to Will MacAskill; he was on Sam Harris’s podcast recently.

After hearing all of this stuff about AI safety and thinking some of it was quite strange, I thought he was a really nice guy. I could tell that he was genuinely trying to do good in the world, and they’ve got so many really good activities going on. The thing that I don’t understand is how the AI risk thing came into it.

Now, you gave the example of Peter Singer, who is an inspiration, and there was that thought experiment: Would you help a child in some dirty water? Is it worth more than the cost of your clothes to help that child? Why not then help children elsewhere? So what they do is abstract this concept of a utility function.

As we were saying before, when you start abstracting things, then you get into trouble.

Oh, that’s it.

Yes. Yeah, that’s exactly right. I think there is something very appealing about a utilitarian approach to ethics and the world from this sort of engineering mindset, and even from certain kinds of scientific mindsets—definitely an engineering mindset. But this idea that everything can be abstracted and quantified, and that the things that you can’t do that with don’t really matter, is a story that you can turn ethics and making difficult decisions about life into questions about numbers. That’s always going to be an appealing story to a fairly large number of people.

I suppose a segue is Will MacAskill. I think you interviewed him for the book, right?

I tried to. Will MacAskill—

Okay.

—agreed to an interview and then backed out. My fact-checker ended up talking to him, although I don’t know—he got confused, I suppose. MacAskill, not my fact-checker. MacAskill seemed to have thought that my fact-checker worked for my publisher and not for me, which I’m not sure how he made that mistake because he worked with the same publisher and knows—or should know—that publishers never pay for fact-checkers. In any event, no, I didn’t talk with MacAskill. I read a bunch of his writing and talked with his colleagues, but MacAskill himself did not talk to me.

Okay, but in your book, you moved on to longtermism as being one of the sort of dangerous side effects. I think there was a quote in the book where MacAskill said that now is the thin end of the wedge. Obviously, it’s a good thing: If humans all have utility, we should have many, many more humans, and they should be all over the universe, with even digital versions running on servers all over the universe. The problem is, when you start extrapolating forwards, it means that the effective value of us compared to that big future light cone of humans—we’re not very valuable anymore. So that kind of diminishes many of our current ethical concerns here on Earth. Is that roughly right?

Yeah, that’s roughly right. I also want to go back to your question of how the AI risk stuff got into all of this effective altruist stuff. The answer is that, basically, Nick Bostrom read a bunch of Yudkowsky, got into the AI risk stuff from there, and then started convincing people like Toby Ord and Will MacAskill that this AI risk stuff is really important and is this overriding ethical concern.

There are even statements that some of the people in this community have made along the lines of other existential risks being sort of the gateway drug to caring about AI risk, which is the real heart of the matter. I just think that’s misguided.

Is your position essentially that there are certain mind worms, for want of a better term, that are quite dangerous? Because they can be repurposed, and there can be dangerous externalities and so on. What’s your prescription?

6. Technical Minds Miss Social Context

My prescription is—well, let me back up and say that, yes, I do think this is kind of a brain worm for people of a certain technical bent. Part of the reason I ended up writing the book is that I myself have a fairly technical bent. I see myself in a lot of these people and feel like, if I were just a little bit different, or if my life had gone a bit differently, I could see myself having been one of these people.

Maybe that’s wrong. Maybe I don’t know myself correctly or don’t understand exactly the appeal of this stuff. But I feel like it’s a pleasingly compact and tractable view of what the world is and how its most complicated problems work, which I would also argue is where a lot of the misunderstanding of power comes from.

My prescription, I think, would be to take more seriously the people who study the relevant areas of inquiry that the people in these spheres tend to ignore. They tend to ignore political science and sociology. They tend to ignore anthropology. They tend to ignore questions about racism, bias, sexism, and whatnot. They tend to get stuck in particular pieces of groupthink, some of which get really pernicious and dangerous.

As I describe in my book, there is a surprising amount of credence within this community of effective altruists and rationalists for ideas like human biodiversity, which is a piece of junk science that tries to give a genetic basis for racism. It has been roundly rejected and disproven by the scientific establishment, even though genetics itself comes from a long history of racism. Yet within this community, it’s an idea that people feel they have to take seriously a lot of the time, even though there’s nothing serious there.

I wanted to touch on that because there are folks in this community who have been making these arguments. I’m not sure where Tesquerel came from. Maybe it was Timnit Gebru. But there are folks like Emily Bender and Emily Torres as well.

To be honest, the discourse has been really bad, so a lot of folks just think that they’re arguing in bad faith and simply dismiss what they’re saying. That’s why I feel that we’ve been able to have a great conversation, because we’re just talking about it from a technical point of view. I don’t know much about social science.

Why is there such an impedance mismatch? Why is it so difficult to have this conversation rationally?

Oh, man. That’s a good question. It’s one of the things I was hoping to try to address with my book.

Well, can I give you one example? So—

Sure. Give me an example.

So—

Yeah.

I’ve seen presentations by Timnit, and she’s very quick to go to “You’re racist,” basically, and eugenics and all of this kind of stuff. It’s quite heavy going. It’s quite scary stuff. Maybe we should just start there. Is that part of the problem? Is it just because it’s a little bit too intense?

Yeah. I think my sympathies lie significantly more with people like Timnit than with someone like Yudkowsky. My issue is that I think what’s going on is that you have people in the effective altruist and rationalist community who want to, or feel that they should be able to, entertain ideas like human biodiversity without recognizing that this is an idea that has been entertained for a very long time, has a really horrifying history, and has no good science behind it.

Then they don’t understand why someone gets upset at the prospect of having a dispassionate, rational argument about whether or not certain people are fully human, entitled to the same rights, and have the same inherent abilities. This goes back, actually, to that thing I was attributing to Adrian Daub earlier: this tendency to look at things outside of the context of history.

There is no way to do that. History is something that we all live in. As the meme says, “We live in a society,” right? I think that it is irrational to assume that someone is not going to take that history and that context into account when having a conversation on a particular subject, especially if that subject ends up being related to whether they and their friends and family are going to be treated like people.

So I think that’s where a lot of this comes from: there’s a group that feels that rationality means not taking particular pieces of context into account, and another group that’s like, “Why wouldn’t we take that into account?”

Yeah.

Of course, my sympathies lie much more with that second group.

And it’s funny as well because, speaking personally, I’m a bit of a weird guy in the tech community because I do believe in physical situatedness. I think it’s about how you got there. You can’t just abstract yourself out of the situation.

I also think that the fundamental impedance mismatch, just as we were saying earlier, is that folks like Timnit are talking about superagency, not individual agency. So she’s not necessarily saying you’re individually racist. She’s saying that you are basically caught up in this larger system of power dynamics that you aren’t aware of and don’t have control over.

Yeah, I think that’s probably right. Although I don’t want to speak for Timnit, I don’t know that I would want to make such a sharp distinction between individually racist and being caught up in this system that you’re not paying attention to, because not paying attention to it can be a form of racism. It’s certainly a form of privilege, right?

Yes, and just for the benefit of the audience, can you briefly make the argument about the eugenics and racism thing? What’s the main argument there?

I mean, there’s a couple of things. Some of it is what I was just talking about: this community does entertain hypotheses like human biodiversity in really unhealthy and unscientific ways. But the other thing is this idea of intelligence, right? The idea of measuring intelligence—intelligence as a single-valued thing—is itself an idea that basically comes out of eugenics and racism.

IQ tests, which are taken quite a bit more seriously by people like Yudkowsky than they should be, are something that was developed essentially as a tool for eugenics. When you look at the modern AI industry and the field itself of inquiry, you see things that are descended from that history present here today, including people using eugenicist arguments—not even from old-school eugenicists from when IQ tests were first developed 100 years ago, but from modern-day purveyors of bogus race science and racism.

You will see people pulling language from those people when trying to define things like intelligence, in academic papers and stuff like that. So, yeah, that is sort of the really short argument. There’s a longer and more developed version of it that I give a summary of in my book, and you can find elsewhere as well.

Yeah, and I think it’s ironic as well because, in my opinion, current AI has proven that intelligence is not this thing that we thought it was.

Yeah.

In some sense, the models do have intelligence, and they still don’t have the thing we want. And why is that? It’s because we thought that intelligence was like this: we have this core knowledge, we have these abstractions in our brain, and intelligence is just traversing the combinatorial closure. The more intelligent you are, the faster you can do that.

What we’ve found with these models is that it’s all about how you frame the question. It’s all about the information you have at your disposal, and there are many different perspectives on different phenomena. Folks who have different perspectives can use their intelligence and get to different answers. So it’s very situated, and it’s not quite as abstract and pure as we thought it was.

Yeah, no, I think that’s exactly right. In my book, I call it humanity’s denial: this idea that you don’t have to pay attention to things like history, sociology, and politics when, again, we do live in a society.

So I wanted to touch as well on this idea of recursive self-improvement. This originated with a guy called I. J. Good.

Yeah.

And Kurzweil certainly made this argument. He said, “Well, the first AGI is going to take us a long time to build, but then we can use the AGI to help us build the next one, and the next one will be built in a fraction of the time, and it would just keep getting better and better.” We’re seeing hints of this in AI. Undoubtedly, there’s some kind of adaptive self-improvement, certainly when you’re doing hill climbing towards a particular task. What do you think about the broader idea?

I think that it’s predicated on 2 problematic ideas, both of which we’ve already talked about: the idea that intelligence is like a single-valued thing, and the idea that intelligence is something that you can equate with, or is directly proportional to, computational power of some kind. I don’t think that either of those things are true. I think that it basically comes from those 2 things, and the case for those 2 ideas is pretty weak.

Yeah. Why are we not going to space?

7. Space Is Not A Lifeboat

I mean, space is pretty bad. It’s really easy to die in space. It’s one of those things when you’re writing a book—or at least when I’m writing a book: I’ll make jokes in the text, and some of them don’t make it through my edits. Then some of them don’t make it past my editor, and I’m always surprised at which ones actually make it into the final book.

One of the jokes that’s in the book is that I quote that famous Elon Musk quote: “It’d be cool to die on Mars, just not on impact.” I say, well, if we take Musk at his word, the good news for him is that it’s really easy to die on Mars, because Mars is a really horrifying place. The radiation levels are too high, the gravity is too low, there’s basically no air, and the dirt is made of poison.

And yet Musk is right that Mars is the next most hospitable place in the solar system after Earth. The only real competition it has is the Moon, and the only thing about the Moon that’s better than Mars is that it’s really close by, which makes it much easier to get there and much easier to communicate with people who are there. Other than that, the Moon is pretty much worse in every way.

There’s no air, and although the dirt on Mars is more poisonous, the dirt on the Moon became such a big problem that every Apollo mission was basically at its operational limits just because of the dust. Moon dust is really sharp, abrasive, and electrostatically charged, so it just sticks to everything.

The fact is that there’s nowhere in our solar system that has anything like the physical attributes we would need to be able to live there or to make it a habitable place. Musk talks a lot about terraforming. We’re not terraforming Mars. If we had the technology and know-how we would need to do something like that, then solving climate change here on Earth really would be a technological problem, and you could solve it very easily.

He talks about the need for Mars as a lifeboat for humanity in the event that something horrible happens to Earth, like an asteroid as big as the one that killed off the dinosaurs 66 million years ago impacting Earth again. There’s nothing that could really happen to Earth that would make it as bad as Mars is to live on, short of the actual physical, complete destruction of Earth.

Even the day when that asteroid hit 66 million years ago was a nicer day for life on Earth than any day on Mars in the last 2 or 3 billion years, at least. We know that because mammals survived, birds survived, and all sorts of life survived that day, whereas there is no mammal, then or now, that could survive unprotected on the surface of Mars.

It’s just horrible. And farther out in the solar system is even worse. We’re not leaving the solar system because the speed of light is just too slow, and we’re not getting past it. We know that we can’t go faster than the speed of light. We have really good science and really good experimental evidence to that effect.

We’ve tested the speed-of-light limit over and over again in particle accelerators. Particle accelerators get particles up to almost the speed of light. We know that our theories of relativity work in that domain.

We know that you cannot get a spaceship going that fast, and we know that if you got a spaceship going anywhere near that fast, it would have enormous problems with radiation and shielding.

Yeah.

We think worlds like Earth are pretty rare. So, yeah, we're not going anywhere. I could also explain a bit about why AI data centers in space are pretty laughable, too, but I don't know if you want me to get into that.

Oh, yeah. That's low-hanging fruit. I never quite understood that one.

There are so many problems. It's just crazy. My favorite is when people say, “Oh, but it's easy to cool them because space is cold.” I'm like, “Space is a vacuum. Vacuum is a perfect thermal insulator. Good luck getting rid of that heat. You're gonna have to have radiator veins the size of a city.”

I know. Part of me is thinking it's great that we do have people who just have insane ambition because Musk doesn't actually realize his most ambitious goals. But he has done some pretty amazing stuff. Starlink is very useful. Sometimes, pursuing ambitious goals means that you find interesting new stepping stones.

But there's also the angle that sometimes, if you want to hire very talented people and you want to build something, you need to create a shared myth. Maybe you're a true believer. Maybe it's just some kind of systematic thing.

Maybe. But I think that those myths end up having harmful consequences if they are so detached from reality, right? Musk did not originate the myth of colonizing Mars, but he's certainly Mr. Occupy Mars these days. In the same way that Kurzweil didn't create the idea of the singularity, Musk didn't create the idea of colonizing Mars.

And it creates this idea that we don't need to care about the problems here on Earth because we're leaving. And that's just not going to happen. These goals, even if they're not possible to realize, have consequences. And the consequences that they have can be really horrifying.

That's a lot of what my book is about: why these goals don't work and the consequences of trying to pursue them.

And, just doing a bit of psychoanalysis here, is your prognosis that there's an element of seeking purpose? So even folks working at OpenAI, they want to feel that they have a grand purpose and that they're benefiting humanity. Is that a similar thing with these billionaires—that they want to feel that they're doing great things for the world?

I do think that that has to be part of it. Again, we can't get inside their heads. We don't know for sure, but there are so many things that money can't buy, and one of them is a sense of purpose. This sort of myth-making gives them a sense of purpose.

One of the most common responses I get to what I wrote in my book is, “Do you really think that these guys actually believe this stuff? Why would they? It's all just a ploy for them to make more money.” And what I respond to that is, “Yeah, okay, I don't know what's going on in their heads. Maybe some of them are just being cynical,” although there's sometimes evidence, as with Bezos. We know that he's always been very, very interested in colonizing space because he's been talking about it since he was in high school. So that suggests he's not just being cynical.

But I also think that when people ask me that question, what they forget—and what I usually respond with—is the fact that these myths are useful for getting these guys more money and power. That makes it more likely that they earnestly believe these things, not less likely.

If you had your way, what would you do about all of this?

If I had my way, I would tax billionaires out of existence. I don't think that that's a just, reasonable, or fair distribution of resources. These people did not make that money on their own. They needed the rest of us to get there.

And by hoarding those resources, they are creating massive power imbalances that erode the fabric of our democracy and our shared sense of truth as a society, and that makes it very difficult to live with one another. We need to find a way to live together in harmony, and I think that allowing these kinds of power imbalances makes that impossible or next to impossible. So, yeah, I think it should be illegal to have that much money.

And also with current AI, just speaking from personal experience, it's really helped me, right? I've been doing lots more work. I'm doing the work of many, many people, and it's made me more consistent and I'm making better decisions.

So I can kind of see where Sam Altman is coming from. It's democratizing. Now you can quit your job, you can start a business, you can use AI to help you. That's the sales pitch. But from your perspective, I feel that you're saying that, structurally, this is going to create huge inequality, especially when we have a 2-tier system in which most of us can't access the frontier AI and only some people can. So how do you see this rolling out?

I don't have a great answer to that question because I don't think that it's written yet, right? We don't know. There are so many choices that we are going to have to make along the way. I can tell you something about what the tech billionaires want, but just because they want it doesn't mean that it's going to happen. Hell, they also want AI data centers in space, and that's not going to happen.

I think we definitely need to be aggressively regulating this industry, and I don't just mean AI; I mean tech. Yes, the tech titans are going to push back on that, and that doesn't mean that they're going to win. And yes, the Trump administration isn't going to want to do that, but you know what? The Trump administration isn't going to be forever. I think it depends on what we allow these guys to get away with.

And isn't it interesting? Bernie Sanders was talking about nationalization, and now Trump has done this export control on Mythos and on GPT-5.6. I mean, this is unimaginable, because when Trump came in, he said, “No, go on, boys. Just scale, scale, scale. Beat China.” So this is a really weird situation. Do you think that they've been influenced by the AI risk lobby, or do you think that it's just cynically KYC, they want to control it? What's going on?

Well, Bernie's definitely, I think, been influenced by the AI safety and AI doomer lobby. As for Trump, I think that somebody dangled a bunch of money in front of Trump and showed him something shiny. They jangled their keys in front of his face. That's generally my theory of how Trump makes decisions that aren't in his immediate, obvious self-interest.

Okay. Any final thoughts on where this is gonna go? It's a very unpredictable time, isn't it?

It is a very unpredictable time. I'm hopeful. I really am. I think that we are actually going to end up finding a way to regulate these guys. The question is how and when, what consequences that will have, and what that struggle ends up looking like. That, I don't know.

But the public sentiment around AI is really bad. The public resistance to building data centers is just getting stronger. And there is also very clearly a financial bubble here, and when that pops, I don't know what's going to happen, but it's going to be pretty interesting.

Yeah, and it might pop quicker now, just before the IPO, with all of this fiasco.

Yes, indeed. Especially with the SpaceX IPO not having gone particularly well.

Adam, it's been an honor having you on the show. Thank you so much. And folks at home, read this book. It's a very, very good book. I enjoyed reading it.

Thanks for having me. It was great to be here.