AGI 仍需30年——Ege Erdil 与 Tamay Besiroglu

Today, I'm chatting with Tamay Besiroglu and Ege Erdil. They were previously running Epoch AI and are now launching Mechanize, which is a company dedicated to automating all work. One of the interesting points you made recently, Tamay, is that the whole idea of the intelligence explosion is mistaken or misleading. Why don't you explain what you're talking about there?

Yeah, I think it's not a very useful concept. It's kind of like calling the Industrial Revolution a horsepower explosion. Sure, during the Industrial Revolution, we saw this drastic acceleration in raw physical power, but there are many other things that were maybe equally important in explaining the acceleration of growth and technological change that we saw during the Industrial Revolution.

What is a way to characterize the broader set of things that the horsepower perspective would miss about the Industrial Revolution?

So I think in the case of the Industrial Revolution, it was a bunch of these complementary changes to many different sectors in the economy. So you had agriculture, transportation, law and finance, urbanization, and moving from rural areas into cities. There were just many different innovations that happened simultaneously that gave rise to this change in the way of economically organizing our society. It wasn't just that we had more horsepower. That was part of it, but that's not the central thing to focus on when thinking about the Industrial Revolution.

And I think similarly, for the development of AI, sure, we'll get a lot of very smart AI systems, but that will be one part among very many different moving parts that explain why we expect to get this transition and this acceleration in growth and technological change.

I want to better understand how you think about that broader transformation. Before we do, the other really interesting part of your worldview is that you have longer timelines to get to AGI than most of the people in San Francisco who think about AI. When do you expect a drop-in remote worker replacement?

Maybe for me, that would be around 2045.

Wow. Wait, and you?

Again, I'm a little more bullish. It depends what you mean by “drop-in remote worker” and whether it's able to do literally everything that can be done remotely, or do most things.

I'm saying literally everything.

For literally everything, just shave Ege's predictions by 5 years or by 20% or something.

Why? Because we've seen so much progress over even the last few years. We've gone from ChatGPT 2 years ago to now we have models that can literally do reasoning and are better coders than me, and I studied software engineering in college. I did become a podcaster; I'm not saying I'm the best coder in the world. But if you made this much progress in the last 2 years, why would it take another 30 to get to full automation of remote work?

So I think that a lot of people have this intuition that progress has been very fast. They look at the trend lines and just extrapolate: obviously, it's going to happen in, I don't know, 2027 or 2030 or whatever. They're just very bullish. And obviously, that's not a thing you can literally do. There isn't a trend you can literally extrapolate of “when do we get to full automation?” because if you look at the fraction of the economy that is actually automated by AI, it's very small. So if you just extrapolate that trend, which is something Robin Hanson likes to do, you're going to say, “Well, it's going to take centuries,” or something.

Now, we don't agree with that view. But I think one way of thinking about this is: How many big things are there? How many core capabilities, competences are there that the AI systems need to be good at in order to have this very broad economic impact, maybe 10x acceleration in growth or something? How many things have you gotten over the past 10 years, 15 years? And we also have this compute-centric view.

So just to double-click on that, I think what Ege is referring to is, if you look at the past 10 years of AI progress, we've gone through about 9 or 10 orders of magnitude of compute, and we got various capabilities that were unlocked. In the early period, people were solving gameplay on specific, very complex games. That happened from 2015 to 2020: Go, chess, Dota, and other games.

Then you had maybe sophisticated language capabilities that were unlocked with these large language models, and maybe advanced abstract reasoning, coding, and math. That was maybe another big capability that got unlocked. So maybe there are a couple of these big unlocks that happened over the past 10 years, but that happened on the order of once every 3 years or so, or maybe 1 every 3 orders of magnitude of compute scaling.

And then you might ask the question: How many more such competencies might we need to unlock in order to be able to have an AI system that can match the capabilities of humans across the board? Maybe specifically just on remote work tasks. And so then you might ask, well, maybe you need coherence over very long horizons, or you need agency and autonomy, or maybe you need full multimodal understanding, just like a human would. And then you ask the question, “Okay, how long might that take?”

And so you can think about, just in terms of calendar years, the previous unlocks took about 1 every 3 years or so. But of course, that previous period coincided with this rapid scale-up of the amount of compute that we use for training. So we went through maybe 9 or 10 orders of magnitude since AlexNet compared to the biggest models we have today.

And we're getting to a level where it's becoming harder and harder to scale up compute. We've done some extrapolations and some analysis looking at specific constraints, like energy or GPU production. Based on that, it looks like we might have maybe 3 or 4 orders of magnitude of scaling left. And then you're really spending a pretty sizable fraction, or a nontrivial fraction, of world output on just building up data centers, energy infrastructure, fabs, and so on.

Which is already like 2% of GDP, right?

I mean, currently it's less than 2%.

Yeah, but also currently most of it is actually not going toward AI chips.

But even most TSMC capacity currently is going toward mobile phone chips or something like that, right? Even leading edge.

It's like 5% of leading edge.

Yeah, even leading edge is pretty small.

But yeah, so that suggests that we might need a lot more compute scaling to get these additional capabilities unlocked. And then there's a question of whether we really have that in us as an economy to be able to sustain that scaling.

But it seems like you have this intuition that there's just a lot left to intelligence. When you play with these models, they're almost there. You forget you're often talking to an AI.

What do you mean they're almost there?

I don't know. I can't ask Claude to pick up this cup and put it over there. Remote work, you know?

Okay.

But even for remote work, I think the current computer-use systems can't even book a flight properly. How much of an update would it be if, by the end of 2026, they could book a flight?

I probably think by the end of this year, they're going to be able to do that.

I think some people do.

Nobody gets a job where they're paid to book flights.

But I think that's an important point, because a lot of people look at jobs in the economy, and then they say, “Oh, that person's job is to just do X.” But then that's not true. That's something they do in their job. But if you look at the fraction of their time on the job that they spend doing that, it's a very small fraction of what they actually do.

It's just this popular conception people have. Or travel agents: they just book hotels and flights. But that's not actually most of their job. So automating that actually wouldn't automate their job, and it wouldn't have that much of an impact on the economy.

So our friend Leopold has this perspective of, quote unquote, “unhobblings,” where the way to characterize it might be that they're basically baby AGIs already. And then, because of the constraints we artificially impose upon them by, for example, only training them on text and not giving them the training data that is necessary for them to understand a Slack environment or a Gmail environment, or previously, before inference-time scaling, not giving them the chance to meditate upon what they're saying and really think it through, and not giving them the context about what is actually involved in this job, only giving them this piecemeal, couple-of-minutes-worth of context in the prompt, we're holding back what is fundamentally a little intelligence from being as productive as it could be. Which implies that unhobblings just seem easier to solve for than entirely new capabilities of intelligence. What do you make of that framework?

I guess you could have made similar points 5 years ago and said, “You look at AlphaZero and there's this mini-AGI there, and if only you unhobbled it by training it on text and giving it all your context and so on, that just wouldn't really have worked.”

I think you do really need to rethink how you train these models in order to get these capabilities. But I think the surprising thing over the last few years has been that you can start off with this pretrained corpus of the internet, and it's actually quite easy. ChatGPT is an example of this unhobbling, where 1% of additional compute spent on getting it to talk in a chatbot-like fashion with post-training is enough to make it competent—really competent—at that capability.

Reasoning is another example where it seems like the amount of compute spent on RL right now in these models is a small fraction of total compute. Again, reasoning seems complicated, and then you just do 1% of the compute and it gets you that. Why not think that computer use, or long-term agency in computer use, is a similar thing?

So when you say “reasoning is easy,” and “it only took this much compute,” and “it wasn’t very much,” and maybe, “You look at the sheer number of tokens and it wasn’t very much, so it looks easy,” well, that’s true from our position today. But I think if you asked someone to build a reasoning model in 2015, it would have looked insurmountable. You would have had to train a model on tens of thousands of GPUs, you would have had to solve that problem, and each order of magnitude of scaling from where they were would pose new challenges that they would need to solve.

You would need to produce internet-scale, or tens of trillions of tokens of data, in order to actually train a model that has the knowledge that you can then unlock and access by training it to be a reasoning model. You need to maybe make the model more efficient at doing inference and maybe distill it, because if it’s very slow, then you have a reasoning model that’s not particularly useful. So you also need to make various innovations to get the model distilled so that you can train it more quickly, because these rollouts take a very long time. It actually becomes a product that’s valuable if it’s a couple of tokens per second. A reasoning model that was very slow would have been very difficult to work with.

In some sense, it looks easy from our point of view, standing on this huge stack of technology that we’ve built up over the past 5 years or so, but at the time it would have been very hard. My claim would be something like: I think the agency part might be easy in a similar sense. In 5 years, or 3 years’ time, or whatever, we will look at what unlocked agency and it’ll look fairly simple. But the amount of work, in terms of these complementary innovations that enable the model to learn how to become a competent agent, might have just been very difficult and taken years of innovation, along with a bunch of improvements in hardware, scaling, and various other things.

Yeah, I feel like what’s dissimilar between 2015 and now is that in 2015, if you were trying to solve reasoning, you just didn’t have a base to start on. Maybe if you tried formal proof methods or something, but there was no leg to stand on. Whereas now you actually have the thing: you have the pre-trained base model, and you have these techniques of scaffolding, post-training, and RL.

So it seems like you think that those will look to the future the way AlphaGo looks to us now, in terms of the basis of a broader intelligence. I’m curious if you have intuitions on why not think that language models as we have them now are like—we got the big missing piece right, and now we’re just plugging things on top of it?

Well, I mean, I guess, what is the reason for believing that? You could have looked at AlphaGo or AlphaZero. Those seemed very impressive at the time. You’re just learning to play this game with no human knowledge; you’re just learning to play it from scratch. I think at the time it did impress a lot of people. But then people tried to apply it to math, they tried to apply it to other domains, and it didn’t work very well. They weren’t able to get competent agents at math.

So it’s very possible that these models, at least the way we have them right now, are going to have the same thing happen that people saw with reasoning, but for agency. It’s not going to work very well.

I’m sorry, you’re saying that by the end of 2026, we will have agentic computer use? I think Ege Erdil said you’d be able to book a flight, which is very different from having full agentic computer use.

I mean, the other things you need to do on a computer are just made up of things like booking a flight.

I mean, sure, but they’re not disconnected tasks. That’s like saying everything you do in the world is just moving parts of your body, and then moving your mouth and your tongue, and then rolling your head.

Yeah, individually those things are simple, but then how do you put them together, right?

Yeah.

Okay. So there are 2 pieces of evidence that you can have that are quite dissimilar. One, the METR eval, which we’ve been talking about privately, shows that the task length over certain kinds of tasks—I can already see you getting ready. AI’s ability to do the kind of thing that it takes a human 10 minutes to do, or an hour to do, or 4 hours to do—the length of time for corresponding human tasks—seems to be doubling every 7 months.

The idea is that by 2030, if you extrapolate this curve, they could be doing tasks that take humans 1 month to do, or 1 year to do. And then this long-term coherency in executing on tasks is fundamentally what intelligence is. So this curve suggests that we’re getting there.

The other piece of evidence is that I kind of feel like my own mind works this way. I get distracted easily, and it’s hard to keep a long-term plan in my head at the same time. I’m slightly better at it than these models, but they don’t seem that dissimilar to me. I would have guessed reasoning was just a really complicated thing, and then it seems like, “Oh, it’s just something like learning 10 tokens’ worth of MCTS: ‘Wait, let’s go back. Let’s think about this another way.’” Chain-of-thought alone just gets you this boost. It just seems like intelligence is simpler than we thought. Maybe agency is also simpler in this way.

Yeah. I think there’s a reason to expect complex reasoning not to be as difficult as people might have thought, even in advance, because a lot of the tasks that AI solved very early on were tasks involving various kinds of complex reasoning. It wasn’t the kind of reasoning that goes into a human solving a math problem, but if you look at the major AI milestones since 1950, a lot of them are for complex reasoning.

Chess is, you could say, a complex reasoning task. Go is, you could say, a complex reasoning task. But I think there are also examples of long-term agency. Winning at StarCraft is an example of being agentic over a meaningful period of time.

That’s right.

So the problem in that case is that it’s a very specific, narrow environment. You can say that playing Go or playing chess also requires a certain amount of agency, and that’s true, but it’s a very narrow task.

That’s like saying if you construct a software system that’s able to react to a very specific, very particular kind of image, or very specific video feeds, or whatever, then you’re getting close to general sensorimotor skill automation. But the general skill is something that’s very different. And I think we’re seeing that.

We still are very far, it seems like, from an AI model that can take a generic game off Steam. Let’s say you just download a game released this year. You don’t know how to play this game, and then you just have to play it. Most games are actually not that difficult for a human.

I mean, what about Claude Plays Pokémon? I don’t think it was trained on Pokémon.

Right, so that’s an interesting example. First of all, I find the example very interesting because it was not trained explicitly. They didn’t do some RL on playing Pokémon Red. But obviously, the model knows how it’s supposed to play Pokémon Red, because there’s tons of material about Pokémon Red on the internet.

In fact, if you were playing Pokémon Red and you got stuck somewhere, you didn’t know what to do, you could probably go to Claude and ask, “I’m stuck in Mount Moon. What am I supposed to do?” It’s probably able to give you a fairly decent answer. But that doesn’t stop it from getting stuck in Mount Moon for 48 hours.

So that’s a very interesting thing, where it has explicit knowledge, but then when it’s actually playing the game, it doesn’t behave in a way that reflects that it has that knowledge. All it’s got to do is plug the explicit knowledge into its actions.

Yeah, but is that easy? If you can leverage your knowledge from pre-training about these games in order to be somewhat competent at them, they’re going to be leveraging a different base of skills. But with that same leverage, they’re going to have a similar repertoire of abilities, if you’ve read everything about whatever skill that every human has ever seen.

A lot of the skills that people have, they don’t have very good training data for them.

That’s right. What would you want to see over the next few years that would make you think, “Oh, no, I’m actually wrong, and this was the last unlock. It was now just a matter of ironing out the kinks”? And then we get the thing that will kick off the, dare I say, intelligence explosion.

I think something that would reveal its ability to do very long-context things, use multimodal capabilities in a meaningful way, and integrate that with reasoning and other types of systems.

Also, agency, and being able to take action over a long horizon and accomplish some tasks that take a very long time for humans to do—not just in specific software environments, but just very broadly. Say, downloading an arbitrary game from Steam, something that it’s never seen before. It doesn’t really have much training data; maybe it was released after a training cutoff, and so there are no tutorials, or maybe there are no earlier versions of the game that have been discussed on the internet. And then accomplishing that game, actually playing it to the end, and accomplishing these various milestones that are challenging for humans. That would be a substantial update.

I mean, there are other things that would update me, too, like OpenAI making a lot more revenue than it’s currently doing.

Is the $100 billion in revenue that would, according to their contract, mark them as AGI enough?

I think that's not a huge update to me if that were to happen. So I think the update would come if it was, in fact, $500 billion in revenue or something like that. But then I would certainly update quite a lot. A hundred billion seems pretty likely to me. I would assign that maybe a 40 percent chance or something.

If you've got a system that is, in producer-surplus terms, worth 100 billion dollars, then what is intelligence? The difference between this and AlphaZero is that AlphaZero is never going to make 100 billion dollars in the marketplace. It's like something able to usefully accomplish its goals, or your goals. If people are willing to pay 100 billion dollars for it, that's pretty good evidence that it's accomplishing some goals.

People pay 100 billion dollars for all sorts of things. That itself is not a very strong piece of evidence that it's going to be transformative, I think. People pay trillions of dollars for oil. It seems like a very basic point, but the fact that people pay a lot of money for something doesn't mean it's going to transform the world economy if only we manage to unhobble it. That's a very different claim.

So then this brings us to the intelligence explosion, because what people will say is, we don't need to automate literally everything that is needed for automating remote work, let alone all human labor in general. We just need to automate the things which are necessary to fully close the R&D cycle needed to make smarter intelligences. And if you do this, you get a very rapid intelligence explosion.

The end product of that explosion is not only an AGI, but something that is potentially superhuman. These things are extremely good at coding and reasoning. It seems like the kinds of things that would be necessary to automate R&D at AI labs. What do you make of that logic?

I think if you look at their capability profile, if you compare it to a random job in the economy, I agree they are better at doing coding tasks that will be involved in R&D compared to a random job in the economy. But in absolute terms, I don't think they're that good. I think they are good at things that maybe impress us about human coders. If you wanted to see what makes a person a really impressive coder, you might look at their competitive programming performance.

In fact, companies often hire people, if they're relatively junior, based on their performance on these kinds of problems. But that is just impressive in the human distribution. So if you look in absolute terms at what skills you need to actually automate the process of being a researcher, then what fraction of those skills do the AI systems actually have?

Even in coding, a lot of coding involves working with a very large code base, and the instructions are very vague. For example, you mentioned the METR eval, in which, because they needed to make it an eval, all the tasks have to be compact and closed and have clear evaluation metrics: “Here's a model; get its loss on this data set as low as possible,” or “Here's another model, and its embedding matrix has been scrambled. Just fix it to recover most of its original performance.” Those are not problems that you actually work on in AI R&D. They're very artificial problems.

Now, if a human was good at doing those problems, you would infer, I think logically, that that human is likely to actually be a good researcher. But if an AI is able to do them, the AI lacks so many other competences that a human would have—not just the researcher, just an ordinary human—that we don't think about in the process of research.

So our view would be that automating research is, first of all, more difficult than people give it credit for. I think you need more skills to do it, and definitely more than models are displaying right now. On top of that, even if you did automate the process of research, we think a lot of the software progress has been driven not by cognitive effort—that has played a part—but by compute scaling.

We just have more GPUs. You can do more experiments to figure out more things, and your experiments can be done at larger scales. That is just a very important driver. If, 10 or 15 years ago, you were trying to figure out what software innovations were going to be important in 10 or 15 years, you would have had a very difficult time.

In fact, you probably wouldn't even have conceived of the right kinds of innovations to be looking at, because you would be so far removed from the context of that time, with much more abundant compute and all the things that people would have learned by that point. So these are 2 components of our view: research is harder than people think, and it depends a lot on compute scale.

Can you put a finer point on an example of the kind of task which is very dissimilar from “train a classifier” or “debug a classifier” that is relevant to AI R&D?

Examples might be introducing novel innovations that are very useful for unlocking innovations in the future. So that might be introducing some novel way of thinking about a problem. A good example might be in mathematics, where we have these reasoning models that are extremely good at solving math problems.

Very short horizon. Sure.

Maybe not extremely good, but certainly better than I can and better than maybe most undergrads can. And so they can do that very well, but they're not very good at coming up with novel conceptual schemes that are useful for making progress in mathematics.

They're able to solve these problems that you can neatly excise out of some very messy context, and they're able to make a lot of progress there. But within some much messier context, they're not very good at figuring out what directions are especially useful for you to build things or make incremental progress on that enables you to have a big innovation later down the line.

So thinking about both this larger context, as well as maybe a much longer horizon and much fuzzier things that you're optimizing for, I think they're much worse at those types of things.

Right. So I think one interesting thing is, if you just look at these reasoning models, they know so much, especially the larger ones, because they know, in literal terms, more than any human does in some sense. And we have unlocked these reasoning capabilities on top of that knowledge, and I think that is actually what's enabling them to solve a lot of these problems.

But if you actually look at the way they approach problems, the reason what they do looks impressive to us is that we have so much less knowledge. The model is approaching the problems in a fundamentally different way compared to how a human would. A human would have much more limited knowledge, and they would usually have to be much more creative in solving problems because they have this lack of knowledge, while the model knows so much.

But you'd ask it some obscure math question where you need some specific theorem from 1850 or something, and then it would just know that, if it's a large model. So that makes the difficulty profile very different.

And if you look at the way they approach problems, the reasoning models are usually not creative. They are very effectively able to leverage the knowledge they have, which is extremely vast. And that makes them very effective in a bunch of ways.

But you might ask the question: Has a reasoning model ever come up with a math concept that even seems slightly interesting to a human mathematician? I've never seen that. They've been around for all of 6 months—I mean, that's a long time. One mathematician might have been able to do a bunch of work over that time, and they have produced orders of magnitude fewer tokens on math.

I just want to emphasize it, because just think about the sheer scale of knowledge that these models have. It's enormous from a human point of view. So it is actually quite remarkable that there is no interesting recombination, no interesting “Oh, this thing in this field looks kind of like this thing in this other field.” There's no innovation that comes out of that.

And it doesn't have to be a big math concept. It could be just a small thing that maybe you could add to a Sunday magazine on math that people used to have. But there isn't even an example of that.

I think it's useful for us to explain a very important framework for our thinking about what AI is good at and what AI is lagging in, which is this idea of Moravec's paradox: Things that seem very hard for humans, AI systems tend to make much faster progress on, whereas things that look a bunch easier for us, AI systems totally struggle with or are often totally incapable of doing.

AI systems tend to make much faster progress on this kind of abstract reasoning: playing chess, playing Go, playing Jeopardy, doing advanced math, and solving math problems. There are even stronger examples, like multiplying 100-digit numbers in your head, which is just the one that got solved first out of almost any other problem.

Or following very complex symbolic logic arguments, like deduction arguments, which people actually struggle with a lot. People have a very hard time with how conclusions logically follow from premises. That's very easy for formal proof systems.

An insight that is related and is quite important here is that the tasks that humans seem to struggle with and AI systems seem to make much faster progress on are things that have emerged fairly recently in evolutionary time. Advanced language use emerged in humans maybe 100,000 years ago, and certainly playing chess and Go and so on are very recent innovations.

So evolution has had much less time to optimize for them, in part because they're very new, but also in part because when they emerged, there was a lot less pressure: They conferred small fitness gains on humans, and so evolution didn't optimize for these things very strongly.

[Speaker?]

And so it’s not surprising that on these specific tasks that humans find very impressive when other humans are able to do them, AI systems are able to make a lot of fast progress. In humans, these things are often very strongly correlated with other competencies, like being good at achieving your goals. Being a good coder is often very strongly correlated with solving coding problems, or being a good engineer is often correlated with solving competitive programming problems.

But in AI systems, the correlation isn’t quite as strong. Even within AI systems, the strongest systems on competitive programming aren’t even the ones that are best at actually helping you code. So o3-mini-high seems to be maybe the best at solving competitive coding problems, but it isn’t the best at actually helping you write code. And it isn’t getting most of the enterprise revenue from places like Cursor or whatever—that’s just Claude, right?

But an important insight here is that the things that we find very impressive when humans are able to do them, we should expect AI systems to make a lot more progress on. But we shouldn’t update too strongly about their general competence, because we should recognize that this is a very narrow subset of relevant tasks that humans do in order to be competent, economically valuable agents.

Yeah. First of all, I really appreciate that there is an AI organization out there where people take the compute perspective seriously and try to think empirically about scaling laws and data. Taking that perspective seriously leads people to just be like, “Okay, 2027 AGI,” which might be correct, but it is interesting to get: “No, we’ve also looked at the exact same arguments, the same papers, the same numbers, and we’ve come to a totally different conclusion.”

So I asked Dario Amodei this exact question 2 years ago, when I interviewed him, and it went viral. Didn’t he say AGI in 2 years? But Dario’s always had short timelines. Okay, but we are 2 years later. Did he say 2 years? I think he actually did say 2 years. Did he say 3 years? So we have 1 more year. 1 more year. Better work hard.

But he’s, I think, in particular, not been that well calibrated. I remember talking to a very senior person who’s now at Anthropic in 2017. He told various people that they shouldn’t do a PhD because by the time they completed it, everyone would be automated.

So anyway, I asked him this exact same question because he has short timelines: if a human knew the amount of things these models know, they would be finding all these different connections. In fact, I was asking Scott about this the other day, when I interviewed him—Scott Alexander—and he said, “Look, humans also don’t have this kind of logical omniscience.” I’m not saying we’re omniscient, but we have examples of humans finding these kinds of connections. This is not an uncommon thing, right?

I think his response was that these things are just not trained in order to find these kinds of connections, but their view is that it would not take that much extra compute to build some RL environment in which they’re incentivized to find these connections. Next-token prediction just isn’t incentivizing them to do this, but the RL required to do this would not be that much extra compute; you could set up some sort of scaffolding. I think Google DeepMind actually did do a similar scaffold to make new discoveries. I didn’t look into how impressive the new discovery was, but they claim that some new discovery was made by an LLM as a result.

On the Moravec paradox, this is actually a super interesting way to think about AI progress. But I would also say that if you compare animals to humans, long-term intelligent planning—an animal isn’t going to help you book a flight either. An animal isn’t going to do remote work for you. I think what separates humans from other animals is that we can hold long-term goals, come up with a plan, and execute on it.

Whereas other animals often have to go by instinct, or within the kinds of environments that they have evolutionary knowledge of, rather than, “I’m put in the middle of the savanna, or I’m put in the middle of the desert, or I’m put in the middle of the tundra, and I’ll learn how to make use of the tools and whatever there.” I actually think there’s a huge discontinuity between humans and animals in their ability to survive in different environments, just based on their knowledge.

And so it’s a recently optimized thing as well. Then I’d be like, “Okay, well, we got it. Soon, AIs will optimize for it fast.” Right. So I would say if you’re comparing animals to humans, it’s kind of a different thing.

[Speaker?]

I think if you could put the competencies that the animals have into AI systems, that might just already get you to AGI. I think the reason why there is such a big discontinuity between animals and humans is because animals have to rely entirely on natural-world data, basically, to train themselves.

Imagine that the only thing, as a human, that you saw was that nobody talked to you, you didn’t read anything, you just had to learn by experience—maybe to some extent by imitating other people—but you had no explicit communication. It would be very inefficient. What’s actually happening is that evolution is sort of this outer optimizer that’s improving the software efficiency of the brain in a bunch of ways. I think some other people have made this point as well.

There’s some genetic knowledge that you inherit, not that much because there isn’t that much space in the genome. And then you have this lifetime learning, where you don’t actually see that much data during lifetime learning. A lot of this is redundant and so on.

So what seems to have changed with humans compared to other animals is that humans became able to have culture and language, which enables them to have a much more efficient training-data modality compared to animals. They also have, I think, stronger ways in which they tend to imitate other humans and learn from their skills, so that also enables this knowledge to be passed on.

I think animals are pretty bad at that compared to humans. So basically, as a human, you’re just being trained on much more efficient data, and that creates further insights to then be efficient at learning from it. That creates this feedback loop where the selection pressure gets much more intense. So I think that’s roughly what happened with humans.

But a lot of the capabilities that you need to be a good worker in the human economy, animals already have. So they have quite sophisticated sensorimotor skills. I think they are actually able to pursue long-term goals, but ones that have been instilled by evolution. I think a lion will find a gazelle, and that is a complicated thing to do and requires stalking.

But when you say it’s been instilled by evolution, there isn’t that much information in the genome. I think if you put the lion in the Sahara and you’re like, “Go find lizards instead.”

Okay. So suppose you put a human in the Sahara and they haven’t seen the relevant training data. I think they’d be slightly better.

Slightly better, but not that much better. Again, didn’t you recently have an interview with Joseph Henrich?

[Speaker?]

Yeah. So he would probably tell you that. I think what you’re making is actually a very interesting and subtle point that has an interesting implication.

So often people say that ASI will be this huge discontinuity because we have this huge discontinuity in the animal-to-human transition. Not that much changed between pre-human primates and humans genetically, but it resulted in this humongous change in capabilities. And so they say, “Well, why not expect something similar between human-level intelligence and superhuman intelligence?”

One implication of the point you’re making is that it wasn’t that we just gained this incredible intelligence. Because of biological constraints, animals have just been held back in this really weird way that no AI system has been arbitrarily held back from—not being able to communicate with other copies or with other knowledge sources.

And so, since AIs aren’t held back artificially in this way, there isn’t going to be a point where we should take away that hobbling and then they explode.

Now, actually, I think I would disagree with that. The implication that I made, I would actually disagree with. I’m sort of an unsteerable chain of thought.

We wrote a blog post together about AI corporations where we discussed that there will actually be a similar unhobbling with future AIs. It’s not about the intelligence, but about a similar level of bandwidth, communication, and collaboration with other AIs.

That is a similar-magnitude change from non-human animals to humans in terms of social collaboration—something that AIs will have with each other because of their ability to copy all their knowledge exactly, merge, and distill themselves.

And I think part of our disagreement is that intelligence is important, but just having a lot more intelligence and reasoning and good reasoning isn't something that will accelerate technological change and economic growth very substantially. It isn't the case that the world today is totally bottlenecked by not having enough good reasoning. That's not really what's bottlenecking the world's ability to grow much more substantially.

I think that we might have some disagreement about this particular argument, but I think what's also really important is just that we have a different view as to how this acceleration happens: it's not just having a bunch of really good reasoners that give you this technology that then accelerates things very drastically. Because that alone is not sufficient.

You need complementary innovations in other industries. You need the economy as a whole growing and supporting the development of these various technologies. You need the various supply chains to be upgraded. You might need demand for the various products that are being built.

And so we have this view where, actually, this very broad upgrading of your technology and your economy is important, rather than just having very good reasoners and very, very, very good reasoning tokens that give us this acceleration.

1. Intelligence explosion

All right. So this brings us back to the intelligence explosion. Here's the argument for the intelligence explosion: you're right that certain kinds of things might take longer to come about, but this core loop of software R&D that's required—if you just look at what kinds of progress is needed to make a more general intelligence—you might be right that it needs more experimental compute. But as you guys have documented, we're just getting a shit-ton more compute every single year for the next few years.

So you can imagine an intelligence explosion in the next few years where, in 2027, there'll be like 10× more compute than there is now for AI. And you'll have this effect where the AIs that are doing software R&D are finding ways to make running copies of them more efficient, which has 2 effects. 1: You're increasing the population of AIs who are doing this research, so more of that in parallel can find these different optimizations.

And a subtle point that they'd often make here is that software R&D in AI is not just an Ilya Sutskever–type researcher coming up with new Transformer-like architectures. To your point, it actually is a lot of—I mean, I'm not an AI researcher, but I assume there's, from the lowest-level libraries to the kernels, to making RL environments, to finding the best optimizer, just so much to do. In parallel, you can be doing all these things or finding optimizations across them.

And so you have 2 effects, going back to this. 1: If you look at the original GPT-4 compared to the current GPT-4o, I think it's—what, how much cheaper is it to run? It's like maybe 100 times cheaper for the same capability or something.

Right.

So they're finding ways in which to run more copies of them at 100× cheaper or something, which means that the population of them is increasing, and the higher populations are helping you find more efficiencies. And not only does that mean you have more researchers, but to the extent that the complementary input is experimental compute, it's not the compute itself, it's the experiments.

And the more efficient it is to run a copy or to develop a copy, the more parallel experiments you can run, because now you can do a GPT-4-scale training run for much cheaper than you could do it in 2024 or 2023. And so for that reason, also, this software-only singularity sees more researcher copies who can run experiments for cheaper, dot, dot, dot.

They initially are maybe handicapped in certain ways that you mentioned, but through this process, they are rapidly becoming much more capable. What is wrong with this logic?

So I think the logic seems fine. I think this is a decent way to think about this problem, but I think that it's useful to draw on a bunch of work that, say, economists have done for studying the returns to R&D and what happens if you 10× your inputs—the number of researchers—what happens to innovation or the rate of innovation.

There, they point out these 2 effects: as you do more innovation and you get to stand on top of the shoulders of giants, you get the benefit from past discoveries, and it makes you as a scientist more productive. But then there's also diminishing returns: the low-hanging fruit has been picked, and it becomes harder to make progress.

And overall, you can summarize those estimates as thinking about the kind of returns to research effort. And we've looked into the returns to research effort in software specifically. We look at a bunch of domains in traditional software, linear/integer solvers, or SAT solvers, but also in AI: computer vision, RL, and language modeling.

And there, if this model is true—that all you need is just cognitive effort—it seems like the estimates are a bit ambiguous about whether this results in this acceleration or whether it results in merely exponential growth. And then you might also think about, well, it isn't just your research effort that you have to scale up to make these innovations, because you might have complementary inputs.

So, as you mentioned, experiments are the thing that might bottleneck you. And I think there's a lot of evidence that, in fact, these experiments and scaling up hardware are just very important for getting progress in the algorithms and the architecture and so on.

In AI—this is true for software in general—if you look at progress in software, it often matches very closely the rate of progress we see in hardware. For traditional software, we see a roughly 30% increase per year, which basically matches Moore's law. And in AI, we've seen the same until you get to the deep learning era. Then you get this acceleration, which in fact coincides with the acceleration we see in compute scaling, which gives you a hint that, actually, compute scaling might have been very important.

Other pieces of evidence, besides this coincidental rate of progress, are the fact that innovation in algorithms and architectures is often concentrated in GPU-rich labs and not in the GPU-poor parts of the world, like academia or maybe smaller research institutes. That also suggests that having a lot of hardware is very important.

If you look at specific innovations that seem very important—the big innovations over the past 5 years—many of them have some kind of scaling or hardware-related motivation. So you might look at how the Transformer itself was about harnessing more parallel compute. Things like FlashAttention were literally about how to implement the attention mechanism more efficiently, or things like the Chinchilla scaling law.

And so many of these big innovations were just about how to harness your compute more effectively. That also tells you that, actually, the scaling of compute might be very important. And I think there are many pieces of evidence that point toward this complementarity picture.

So I would say that even if you assume that experiments are not particularly important, the evidence we have, both from estimates of AI and other software—although the data is not great—suggests that maybe you don't get this kind of hyperbolic, faster-than-exponential super-growth in the overall algorithmic efficiency of systems.

I'm not sure I buy the argument that because these 2 things—compute and AI progress—have risen so concomitantly, this is a sort of causal relationship. So broadly, the industry as a whole has been getting more compute and, as a result, making more progress.

But if you look at the top players, there have been multiple examples of a company with much less compute, but a more coherent vision and more concentrated research effort, being able to beat an incumbent that has much more compute. OpenAI initially beating Google DeepMind.

And if you remember, there were these emails that were released between Elon and Sam and so forth, like, “We’ve got to start this company because they've got this bottleneck on the compute,” and, “Look how much more compute Google DeepMind has.” And then OpenAI made a lot of progress.

Similarly now with OpenAI versus Anthropic and so forth. And then I think, just generally, your argument is just too “outside view.” We actually do know a lot about this very macroeconomic argument. I'm like, well, why don't we just ask the AI researchers?

I mean, AI researchers will often overstate the extent to which just cognitive effort and doing research is important for driving these innovations, because that's often convenient or useful. They will say the insight was derived from some nice idea about statistical mechanics or some nice equation in physics that says that we should do it this way. But often that's an ad hoc story that they tell to make it a bit more compelling to reviewers.

Daniel Kokotajlo mentioned this survey he did where he asked a bunch of AI researchers, “If you had 1/30th the amount of compute”—and he did 1/30th because AIs will be, they suppose, 30 times faster—“if you had 1/30th the amount of compute, how much would your progress slow down?” And they say, “I make 1/3 of the amount of progress I normally do.”

So that's just a pretty good substitution effect: you get 1/10th the compute, and your progress only goes down 1/3. And then I was talking to an AI researcher the other day, one of these cracked people who probably gets paid tens of millions of dollars a year. And we asked him, “How much do AI models help you in AI research?”

And he said, “In domains that I'm already quite familiar with, where it's closer to autocomplete, it saves me 4 to 8 hours a week.” And then he said, “But in domains where I'm actually less familiar, where I need to drive new connections, I need to understand how these different parts relate to each other, and so forth…”

“It saves me close to 24 to 36 hours a week.” And that's the current models. And I'm just like, “He didn't get more compute, but it still saved him a shit ton more time.” Just draw that forward. That's a crazy implication or crazy trend, right?

I'm skeptical of the claims that we have actually seen that much of an acceleration in the process of R&D. These claims seem to me like they're not borne out by the actual data I'm seeing. So I'm not sure how much to trust them.

On the general intuition that cognitive effort alone can give you a lot of AI progress, it seems like a big, important thing the labs do is this science of deep learning. Scaling laws—I mean, it ultimately netted out in an experiment, but the experiment is motivated by cognitive effort. So, for what it's worth, when you say that A and B are complementary, you're saying that just as A can bottleneck you, B can also bottleneck you. When you say you need compute and experiments and data, but you also need cognitive effort, that doesn't mean the lab that has the most compute is going to win, right? That's a very simple point: either one can be the bottleneck.

If you just have a really dysfunctional culture and you don't actually prioritize using your compute very well, and you just waste it, then you're not going to make a lot of progress, right? So it doesn't contradict the picture that someone with a much better vision, a much better team, and much better prioritization can make better use of their compute if someone else was just bottlenecked heavily on that part of the equation.

The question here is, once you get these automated AI researchers and you start this software singularity, your software efficiency is going to improve by many orders of magnitude, while your compute stock, at least in the short run, is going to remain fairly fixed. So how many OOMs of improvement can you get before you become bottlenecked by the second part of the equation? Once you actually factor that in—how much progress should you expect? That's the kind of question I think people don't have.

I think it's hard for people to have good intuitions about this because people usually don't run the experiments. So you don't get to see, at a company level or at an industry level, what would have happened if the entire industry had 30 times less compute. Maybe, as an individual, what would happen if you had 30 times less compute? You might have a better idea about that, but that's a very local experiment, and you might be benefiting a lot from spillovers from other people who actually have more compute. So because this experiment was never run, it's sort of hard to get direct evidence about the strength of complementarity.

Actually, what is your probability that, if we live in a world where we get AGI in 2027, there is a software-only singularity?

Quite high, because you're conditioning on compute not being very large. So it must be that you get a bunch of software progress.

Yeah, right, right. You just have a bunch of leverage from algorithmic progress in that world. Okay, that's right. I was thinking these are independent questions.

I think a point I want to make is that I know some labs do have multiple pre-training teams, and they give people different amounts of resources for doing the training, different amounts of cognitive effort, and different-sized teams. But none of that, I think, has been published. I'd love to see the results of some of those experiments.

I think even that won't update you very strongly, just because it is often very inefficient to do this very imbalanced scaling of your factor inputs. In order to really get an estimate of how strong these complementarities are, you need to observe these very imbalanced scale-ups. That rarely happens, and so I think the data that bears on this is just really quite poor.

The intuitions that people have also don't seem clearly relevant to the thing that matters: what happens if you do this very imbalanced scaling, and where does this net out?

One question I have, and it would be really interesting if somebody could provide an example of this: maybe, through history, there was some point at which, because of a war or some other kind of supply shock, you had to ramp up production or ramp up some key output that people really cared about, while, for some weird historical reason, many of the key inputs were not accessible to a ramp-up, but you could ramp up one key input. I'm talking in very abstract terms. You see what I'm saying, right? You need to make more bombers, but you ran out of aluminum and you need to figure out something else to do. How successful have these efforts been, or do you just keep getting bottlenecked?

Well, I think that is not quite the right way to do it. Because I think if you're talking about materials, then I think there's a lot of sense in which different materials can be substitutable for one another in different ways. You can use aluminum. I mean, aluminum is a great metal for making aircraft because it's light and durable and so on. But you can imagine that you make aircraft with worse metals, and then it just takes more fuel and it's less efficient to fly. So there's a sense in which you can compensate and it just costs more.

I think it's much harder if you're talking about something like complementarity between labor and capital, complementarity between remote work and in-person work, or between skilled and unskilled work. There are input pairs for which I would expect it to be much more difficult.

For example, you're looking at the complementarity between the quality of leadership of an army and its number of soldiers. There is some effect there, but if you just scale up, you have excellent leadership, but your army only has 100 people. You're not going to get very far.

King Leonidas and Thermopylae?

Well, they lost, right? It would be funny if we're building models in a software-only singularity and we're like, “What exactly happened in Thermopylae?” It's somehow relevant. I can actually talk about that, but we probably shouldn't.

2. Ege & Tamay's story

Okay, sure. So the audience should know: my most popular guest by far is Sarah Paine. Not only is she my most popular guest, she's my 4 most popular guests, because all 4 of those episodes that I've done with her are, on a viewer-minute-adjusted basis, my most popular episodes. I host the Sarah Paine Podcast, where I occasionally talk about AI.

Anyway, we did this 3-part lecture series. One of them was about India-Pakistan wars through history, one of them was about Japanese culture before World War II, and the third one was about the Chinese Civil War. And for all of them, my history tutor was Ege. Why does he know so much about random 20th-century conflicts? But he did, and he suggested a bunch of the good questions I asked her. Ege, what's going on there?

I don't know. I don't really have a good answer. I read a bunch of stuff, but it's a kind of boring answer. Imagine you ask a top AI researcher, “What's going on? How are you so good?” And then they will probably give you a boring answer, like, “I don't know.” That itself is interesting: often, these kinds of questions elicit boring answers. It tells you about the nature of the skill.

How'd you find him?

We connected on a Discord for Metaculus, which is this forecasting platform. I was a graduate student at Cambridge at the time, doing research in economics, and I was having conversations with my peers there. I was occasionally having conversations with Ege, and I was like, “This guy knows a lot more about economics.” At the time, he was a computer science undergrad in Ankara, and he knew more about economics and about these big trends in economic growth and economic history than almost any of my peers at the university. So I thought, what the hell is up with that?

Then we started collaborating frequently and ended up hiring Ege for Epoch, because it clearly makes sense for him to work on these types of questions.

And it seems like at Epoch, you've just collected this group of internet misfits and weirdos. How did you start Epoch? And then how did you accomplish this?

I was at MIT doing more research, and I was pretty unhappy with the bureaucracy there, where it was very hard for me to scale projects up and hire people. I was pretty excited about a bunch of work that my PI wasn't excited about because it was maybe hard to publish or it didn't confer the same prestige.

So I was chatting with Jaime Sevilla, one of the co-founders. We just collaborated on projects and then thought we should start our own org, because we could just hire people and work on the projects we were excited about. Then I hired a bunch of the insightful misfits.

But was the thesis like, “Oh, there's a bunch of underutilized internet misfits, and therefore this org was successful”? Or you started the org and then you were like—

I think it's more of the latter. It was more like, we can make a bunch of progress because clearly academia and industry are kind of dropping the ball on a bunch of important questions that academia is unable to publish interesting papers on. Industry is not really focused on producing useful insights, and so it seemed very good for us to just do that.

Also, the timing was very good. We started just before ChatGPT, and we wanted to have much more grounded discussions of the future of AI. I was frustrated with the quality of discussion that was happening on the internet about the future of AI, and to some extent—or to a very large extent—I still am. That's a large part of what motivates me to do this. It's just born out of frustration with bad thinking and arguments about where AI is going to go.

3. Explosive economic growth

Okay, so let me ask you about this. Just to set the scene for the audience, we're going to talk about the possibility of this explosive economic growth and greater-than-30% economic growth rates.

So I want to poke you from both perspectives: maybe suggest that this isn't aggressive enough in the right kind of way because maybe it's too broad, and then poke you from the more normal perspective that, hey, this is fucking crazy. I imagine it would be difficult for you to do the second thing.

No, I think it might be fucking crazy. Let's see.

The big question I have about this broad automation: I get what you're saying about the Industrial Revolution, but in this case, we can just make the argument that you get this intelligence, and then what you do next is you go to the desert and build this Shenzhen of robot factories, which are building more robot factories, which are building more robot factories. If you need to do experiments, then you build bio labs and chemistry labs and whatever.

Or you can build Shenzhen in the desert. I agree that looks much more plausible than a software-only singularity. But the way you're framing it, it sounds like McDonald's and Home Depot and fucking whatever are growing at 30% a year as well. The alien's-eye view of the economy is that there's a robot economy in the desert that's growing at 10,000% a year and everything else is the same old, same old—or is it like—

[Speaker?]

There is a question about what would be possible, or physically possible, and what would actually be efficient. Once you're scaling up the hardware part of the equation as well as the software part, I think the case for this feedback loop gets a lot stronger. If you scale up data collection as well, I think it gets even stronger—real-world data collection by deployment and so on.

But building Shenzhen in a desert—if you think about the pipeline, so far we have relied first on the entire semiconductor supply chain. That industry depends on tons of inputs and materials, and it gets them from probably tons of random places in the world. Creating that infrastructure—doubling or tripling, whatever, the entire infrastructure—is very hard work. So probably you couldn't even do it. Even if you just had Shenzhen in a desert, that would be even more expensive than that.

On top of that, so far we have been drawing heavily on the fact that we have built up this huge stock of data on the internet over the past 30 years or something. Imagine you were trying to train a state-of-the-art model, but you only had 100 billion tokens to train on. That would be very difficult. In a certain sense, our entire economy has produced this huge amount of data on the internet that we are now using to train the models.

It's plausible that in the future, when you need to get new competencies added to these systems, the most efficient way to do that will be to leverage similar kinds of modalities of data. That will also require this—you would want to deploy the systems broadly because that's going to give you more data. Maybe you can get where you want to be without that, but it would just be less efficient if you're starting from scratch compared to if you're collecting a lot of data.

I think this is actually a motivation for why labs want their LLMs to be deployed widely. Sometimes when you talk to ChatGPT, it's going to give you 2 responses and say, “Well, which one was good?” Or it's going to give you 1 response and ask you, “Was this good or not?” Why are they doing that? That's a way in which they are getting user data through this extremely broad deployment.

I think you should just imagine that thing continuing to be efficient and continuing to increase in the future, because it just makes sense. Then there's a separate question: suppose you didn't do any of that. Suppose you just tried to imagine the most rudimentary, narrowest possible kind of infrastructure build-out and deployment that would be sufficient to get this positive feedback loop that leads to much more efficient AIs.

I agree that loop could, in principle, be much smaller than the entire world. I think it probably couldn't be as small as Shenzhen in the desert, but it could be much smaller than the entire world. But then there's a separate question of whether you would actually do that and whether that would be efficient.

I think some people have the intuition that there are just these extremely strong constraints—maybe regulatory constraints, maybe sociopolitical constraints—to doing this broad deployment. They just think it's going to be very hard. But I think that's overstated.

I think people's intuitions for how hard this kind of deployment is come from cases where the deployment of the technology wouldn't be that valuable. It might come from housing—we have a lot of regulations on housing. Maybe it comes from nuclear power. Maybe it comes from supersonic flights. Those are all technologies that would be useful if they were less regulated, but they wouldn't double.

I think the core point here is that the value of AI automation and deployment is just extremely large, even just for workers. There might be some kind of displacement, and there might be some transition that you need to go through in order to find a job that works for you, but otherwise the wages could still be very high, for a while at least.

On top of that, the gains from owning capital might be enormous. In fact, a large share of the US population would benefit. They own housing, they have 401(k)s. Those would do enormously better when you have this process of broad automation and AI deployment.

I think there could just be very deep support for some of this, even when it's totally changing the nature of labor markets and the skills and occupations that are in demand. I would just say it's complicated.

I think what the political reaction to it will be when this starts actually happening is hard to forecast. The easy thing to say is, yeah, this will become a big issue and then it will maybe be controversial or something. But what is the actual nature of the reaction in different countries? I think that's kind of hard to forecast.

The default view is, well, people are going to become unemployed, so it will just be very unpopular. I think that's very far from obvious. I expect heterogeneity in how different countries respond. Some of them are going to be more liberal about this and are going to have a much broader deployment. Those countries probably end up doing better.

Just like during the Industrial Revolution, some countries were ahead of others. Eventually, almost the entire world adopted the norms, culture, and values of the Industrial Revolution in various ways.

Actually, you say they might be more liberal about it, but they might actually be less liberal in many ways. In fact, that might be more functional in this world in which you have broad AI deployment.

We might adopt the kind of values and norms that get developed in, say, the UAE or something, which is maybe focused a lot more on making an environment that is very conducive to AI deployment. We might start emulating and adopting various norms like that. They might not be classical liberal norms, but norms that are just more conducive to AI being functional and producing a lot of value.

Robin Hanson

This is not meant to be a strong prediction; this is just illustrative. It might just be the freedom to deploy AI in the economy and build out lots of physical things at scale. Maybe that ends up being more important in the future.

Maybe that is still missing something. Maybe there are some other things that are also important. The generic prediction that you should expect variance and that some countries do better than others, I think that's much easier to predict than the specific countries that end up doing better.

Yeah. Or the norms that that country wants.

Robin Hanson

That's right.

One thing I'm confused about is, if you look at the world of today versus the world of 1750, the big difference is just that we've got crazy technology that they didn't have back then. We've got these cameras, we've got these screens, and we've got rockets and so forth. That just seems like the result of technological growth and R&D and so forth. It's capital accumulation.

Explain that to me, because you're just talking about this infrastructure build-out. I'm like, but why won't they just fucking invent the kinds of shit that humans would have invented by 2050?

Robin Hanson

Producing this stuff takes a lot of infrastructure build-out.

But that infrastructure is built out once you make the technology, right?

Robin Hanson

I don't think that's right. There isn't this temporal difference where first you do the invention. Often there's an interplay between the actual capital buildup and the innovation.

Learning curves are about this, right, fundamentally? What has driven the increase in the efficiency of solar panels over the past 20 or 30 years? It isn't as if people had the idea of 2025 solar panels. Nobody 20 years ago had the sketch for the 2025 solar panel. It's this kind of interplay between having ideas, building, learning, and producing.

Other complementary inputs also become more efficient at the same time, like you might get better materials.

Robin Hanson

For example, the fact that smelting processes got a lot better toward the end of the 19th century, so it became a lot easier to work with metal, maybe that was a crucial reason why aircraft technology later became more popular.

It's not like someone came up with the idea, “Oh, you can just use something that has wings and a lot of thrust, and then that might be able to fly.” That basic idea is not that difficult, but how do you make it an actually viable thing? That's much more difficult.

Have you seen the meme where 2 beavers are talking to each other and they're looking at the Hoover Dam? One of them is like, “Well, I didn't build that, but it's based on an idea of mine.”

The point you're making is that this invention-focused look at technology history underplays the work that goes into making specific innovations practicable and deploying them widely. It's just hard, I think. Suppose you want to write a history of this—you want to write the history of how the light bulb was developed or something. It's really hard, because to understand why specific things happen at specific times, you probably need to understand so much about the economic conditions of the time.

For example, Edison spent a ton of time experimenting with different filaments to use in the light bulb. The basic idea is very simple: you make something hot and it glows. But what filament actually works well for that in a product? What is durable? What has the highest ratio of light output versus heat, so that you have less waste and it's more efficient? Even after you have the product, you're facing the problem that it's 1880 or something, and U.S. homes don't have electricity, so nobody can use it. Now you have to build power plants and power lines to the houses so that people have electricity in their homes and can actually use this new light bulb that you created.

So he did that, but people present it as if it's, “Okay, he just came up with the idea. It's a light bulb.” I guess the thing people would say is, you're right about how technology would progress if we were humans deploying it for the human world. But what you're not counting is that there's going to be this AI economy, where maybe they need to do this kind of innovation and learning by doing when they're figuring out, “I want to make more robots because they're helpful, so we're going to build more robot factories, we'll learn, and then we'll make better robots,” or whatever.

But geographically, that's a small part of the world where this is happening. It's not like they walk into your building and then you do a business transaction with Lunar Society Podcast LLC. You understand what I'm saying?

For what it's worth, if you look at the total surface area of the world, it might well be the case that the place that initially experiences this very fast growth is a small percentage of the surface area of the world. I think that was the same for the Industrial Revolution; it was not different.

What concretely does this explosive growth look like? If I look at this heat map of growth rates on the globe, is there just going to be 1 area that's blindingly hot, with desert factories and all these experiments?

[Speaker?]

I would say our idea is that it's going to be broader than that. Eventually, it would probably be most of the world. But as I said, because of this heterogeneity, because I think some countries are going to be faster in adoption than others, and maybe some cities will have faster adoption than others, there will be differentials, and some countries might have much faster growth than other countries.

But I would expect that, at a jurisdiction level, it will be more homogeneous. For example, I expect the primary obstacles to come from things like regulation, so I would imagine it being more delineated by regulatory jurisdiction boundaries than anything else.

Got it. So you may be right that this infrastructure build-out, capital deepening, and whatever else is necessary for a technology to become practical—or even to be discovered—is necessary. There's an aspect of it where you discover certain things by scaling up and learning by doing; that's the learning curve. There's this separate aspect where, suppose that you become wealthier, you can invest that increased wealth in accumulating more capital, but you can also invest it in R&D and other ways. You get Einstein out of the patent office. You need some amount of resources for that to make sense, and you need the economy to be of a certain scale.

You also need demand for the product you're building. You could have the idea, but if the economy is just too small and there isn't enough demand for you to specialize in producing the semiconductor or whatever, then it doesn't make sense. A much larger-scale economy is useful in many ways: in delivering complementary innovations and discoveries happening through serendipity, and in producing consumers who would actually pay enough for you to recover the fixed costs of doing all the experimentation and invention. You need the supply chains to exist to deliver the germanium crystals that you need to grow in order to come up with the semiconductor. You need a large labor force to be able to help you do all the experiments, and so on.

I think the point you're illustrating is, “Look, could you have just figured out that there was a Big Bang by first-principles reasoning?” Maybe. But what actually happened is that we had World War II, and we discovered radio communications in order to fight and effectively communicate during the war. Then that technology helped us build radio telescopes. Then we discovered the cosmic microwave background. Then we had to come up with an explanation for the cosmic microwave background. And then we discovered the Big Bang as a result of World War II. People underemphasize the giant effort that goes into this buildup of all the relevant capital, all the relevant supply chains, and the technology.

Earlier, you were making a similar comment when you were saying, “Reasoning models, in hindsight, look pretty simple,” but then you're ignoring this giant upgrading of the technology stack that happened and took 5 to 10 years prior to that. I think people just underemphasize the support that it has from the overall upgrading of your technology, the supply chains, and various sectors that are important for that. People focus on specific individuals, like Einstein having this genius insight and being the pivotal thing in the causal chain that resulted in these discoveries, or Newton being extremely important for discovering calculus, without thinking about all the other factors that produced lenses, produced telescopes, got the right data, and made people ask questions about dynamics and so on that motivated some of these questions. Those are also extremely important for scientific and technological innovation.

As you were saying, one of Conquest's laws is that the more you understand about a topic, the more conservative you become about that topic. There may be a similar law here: the more you understand about an industry, the more conservative you become. Obviously, I'm just a commentator or a podcaster, but I understand AI better than any other industry I understand. I have the sense, from talking to people like you, that so much went into getting AI to the point where it is today. Whereas when I talk to journalists about AI, they're asking, “Okay, who is a crucial person we need to cover? Should we get in touch with Geoffrey Hinton? Should we get in touch with Ilya?” I just have this sense that they're missing the picture.

Maybe it's a similar phenomenon to Gell-Mann amnesia. We should have a similar attitude toward other industries. Robin Hanson has this abstraction of seeing things in near mode versus far mode. If you don't know a lot about a topic, then you see it in far mode and simplify things; you see a lot less detail.

In general, I think the thing I would say—and the reason I also believe that abstract reasoning and deductive reasoning, or even Bayesian reasoning, by itself is not sufficient or is not as powerful as many other people think—is because there's just an enormous amount of richness and detail in the real world that you can't reason about. You need to see it. Obviously, that is not an obstacle to AI being incredibly transformative, because you can scale your data collection and scale the experiments you do, both in the AI industry itself and more broadly in the economy, so you just discover more things. More economic activity means we have more exposed surface area for more discoveries. All of these are things that have happened in our past, so there's no reason that they couldn't speed up.

The fundamental thing is that there's no reason why economic growth can't be much faster than it is today. It's probably as advanced as it is right now just because humans are such an important bottleneck. They both supply the labor and play crucial roles in the process of discovery and various kinds of productivity growth. There's strong complementarity, to some extent, with capital, and you can't substitute machines and so on for humans very well. So the growth of the economy and the growth of productivity end up being bottlenecked by the growth of the human population.

So let me ask you a tangential question. What's been happening in China over the last 50 years? Would you describe that as, in principle, the same kind of explosive growth that you expect from AI? There's a lot of labor that makes the marginal product of capital really high, which allows you to have 10%-plus economic growth rates. Is that basically, in principle, what you expect from AI?

I would say in some ways it's similar, and in some ways it's not. Probably the most important way in which it's not similar is that in China, you see a massive amount of capital accumulation, a substantial amount of adoption of new technologies, and probably also human capital accumulation to some extent. But you're not seeing a huge scale-up in the labor force.

For AI, you should expect to see a scale-up in the labor force as well—not in the human workforce, but in the AI workforce. And I think you did, maybe not consecutive increases in the labor force…

[Speaker?]

The key thing here is just the simultaneous scaling of both these things. You might ask the question, “Isn't it basically half of what's going to happen with AI that you scale up capital accumulation in China?” But actually, if you get both of these things to scale, that gives you much faster growth and a very different picture.

At the same time, if you're just asking what 30% growth per year would look like, if you just want to have an intuition for how transformative that would be in concrete terms, then I think looking at China is not such a bad case. Especially in the 2000s, or maybe the late '90s, that seems slower than what we're forecasting.

Right. I think also looking at the Industrial Revolution is pretty good. Well, the Industrial Revolution is very slow, but just in terms of the margins along which we made progress in terms of products, the thing that didn't happen during the Industrial Revolution is that we just produced a lot more of the things that people were producing prior to the Industrial Revolution, like a lot more crops and maybe a lot more pre–Industrial Revolution-style houses or whatever on farms.

Instead, along pretty much every main sector of the economy, we got many different products that were totally different from what was being consumed prior to that. So, in transportation and in food—I mean, health care is a very big deal, and antibiotics.

Another question: I'm not sure I understand how you're defining learning by doing versus explicit R&D, because there's the way companies report what they call R&D for tax purposes. But then there's the intuitive understanding of R&D. If you think about how AI is boosting TFP, you could say that right now, if you just replaced the TSMC process engineers with AIs and they're finding different ways to improve that process, improve efficiencies, and improve yield, I would call that R&D.

On the other hand, you emphasize this other part of TFP, which is better management and that kind of stuff. But how much oomph are you going to get? You're going to get to the fucking Dyson sphere by better management?

[Speaker?]

But that's not the argument, right? The point is that there are all these different things, some of them maybe more complementary than others. The point is not that you can get to a Dyson sphere by just scaling labor and capital. That's not the point. You need to scale everything at once. So, just as you can't get to a Dyson sphere by just scaling labor and capital, you also can't get to it by just scaling TFP. That doesn't work.

I think there's a very important distinction between what is necessary to scale to get to this Dyson sphere world and what is important. In some sense, producing food is necessary, but of course producing food doesn't get you to a Dyson sphere, right? So I think R&D is necessary, but on its own isn't sufficient. Scaling up the economy is also necessary. On its own, it's not sufficient.

Then you can ask the question, what is the relative importance of each? I think our view here is very much the same. It is very connected to our view about the software R&D thing, where we're just saying there are these bottlenecks, so you need to scale everything at once. This is just a general view.

But I think people misunderstand us sometimes as saying that R&D is not important. No, that's not what we're saying. We're saying it is important. It is less important in relative terms than some other things, none of which are by themselves sufficient to enable this growth.

So the question is, how do you do the credit attribution? One of my missions in economics is to look at the elasticities of output to the different factors. Capital is less important than labor, because the labor elasticity of output is like 0.6, while for capital it's like 0.3. But neither are by themselves sufficient. If you just scaled one of them and the other remained fixed, then neither would be sufficient to indefinitely scale output.

4. Will there be a separate AI economy?

One question that Daniel posed to me—because I made this point about everything being interconnected when you were talking about it—is that another example people often bring up is: What would it take to build the iPhone in the year 1000? It's unclear how you could actually do that without just replicating every intermediate technology, or most intermediate technologies.

Then he made the point: “Okay, fine, whatever. Nanobots are not a crux here. The crux, at least for the thing he cares about, which is human control, is just: by when can the robot economy, or the AI economy—whether it's a result of capital deepening or whether it's a result of R&D—have the robots and more cumulative physical power?”

[Speaker?]

But he's imagining a separate thing called the AI economy. Why would you imagine that? I think it's probably downstream of his views about the software-only singularity. But again, those are views that we don't share.

It's much more efficient for AI to operate in our economy and benefit from the existing supply chains and existing markets, rather than set up shop on some island somewhere and do its own thing.

And then it's not clear. For example, people might have the intuition—I brought this up before—the distinction between what is the minimum possible amount of build-out that would be necessary to get this feedback loop up and running and what would be the most efficient way to do it. Those are not the same question.

But then people have this view that the most efficient thing in principle can't be done because—I think the example he might give is when the conquistadors arrived in the New World, or when the East India Company arrived in India, they did integrate into the existing economy.

In many cases, it depends on how you define “integrate,” but the Spanish relied heavily on New World labor in order to do silver mining and whatever. The East India Company was just a ratio of British people to Indian people, which is not that high, so they just had to rely on the existing labor force. But they were still able to take over because of—I don't know what the analogous thing here is, but you see what I'm saying.

He's concerned about by when they will, even if they're ordering components off Alibaba or whatever—and sorry, I'm being trite, but you see what I'm saying—even if they're going to get into the supply chains, be in a position where, because this part of the economy has been growing much faster, they could take over the government, or whatever, if they wanted to.

[Speaker?]

That's right. So I think that eventually you expect the AI systems to be driving most of the economy. Unless there are some very strange coincidences where humans are able to somehow uplift themselves and become competitive with the AIs by stopping being biological humans or whatever—which seems very unlikely early on—then AI is just going to be much more powerful.

I agree that in that world, if the AIs somehow coordinated and decided, “Okay, we should just take over,” or somehow coordinated to have that goal, then they could probably do it. But that's also probably true in our world. If the US wanted to invade North Sentinel Island, then probably it could do it. I don't think anyone could stop it.

But what does it actually mean? There's this dramatic power imbalance, but that doesn't mean—that doesn't tell you what's going to happen, right? Why doesn't the US just invade Guatemala or something? Why don't they do that? It seems like they could easily do it, because the value to the US of—

Not that high, right?

Yeah. So I agree that might be true for AIs, because most of the shit is in space, and you want to do the capital deepening on Mars and the surface of the Sun instead of New York City.

I think it's deeper than that.

So it's deeper than that. There's also the fact that if the AIs are going to be integrated into our economy, they start out as a small part of our economy or our workforce, and over time they grow. Over time, they become the vast majority of the actual work power in the economy.

But they are growing in this existing framework, where we have norms and rules for better coordination, and undermining those things has a cost. If getting the things that are making humans wealthier than they used to be and more comfortable means that you would probably be better off if you could just take that from them, the benefit to you, if you're already getting almost all of the income in the economy, will be fairly small.

I feel like the North Sentinel Island thing has one reference class that includes that. But historically, there's a huge reference class that includes the East India Company. It could have just kept trading with the Mughals; it just took over, right? It could have kept trading with the 50 different nation-states in precolonial India. But yeah.

That's right. I mean, that's what they were initially doing. And then whatever. I'm not going to go into that subject, but that is the reference class—

I agree.

So if the question is, if they have some totally different values and then they represent most of the economy, would they take over? I still don't know, because I'm not sure to what extent the class of all AI is a natural class. It's sort of like: Why don't the young people in the economy coordinate?

I agree that sometimes these kinds of class arguments are misused. For example, when Marxists are like, “Why doesn't this class rise up against the others?”

Daniel made the interesting argument that, if you look at the history of the conquistadors, when Hernán Cortés was making his way through the New World, he actually had to go back and fight off a Spanish fleet that had been sent to arrest him, and then go back.

So you can have this fight within the conquering AIs, and that still nets out to the Native Americans getting disempowered.

But with AIs in particular, they’re just copies of each other. In many other ways, they have lower transaction costs when they trade with each other or interact with each other. There are other reasons to expect them to be more compatible coordinating with each other than coordinating with the human world.

Sure. If the question is just, “Is it possible for that to happen?”—which is a weaker claim—then, yeah, it seems possible. But there are, I think, a lot of arguments pushing back against it. Probably, actually, the biggest one is: just look at the AIs we have today. Can you imagine them doing that?

I think people just don’t put a lot of weight on that because they think that once we have enough optimization pressure and once they become superintelligent, they’re just going to become misaligned. But I just don’t see the evidence for that.

I agree there’s some evidence that they’re good boys.

No, there’s more than some evidence.

No, but there’s also some evidence. There’s a new OpenAI paper where, in chain-of-thought, reward hacking is such a strong basin that if you were like, “Hey, let’s go solve this coding problem,” in the chain of thought they’ll just be like, “Okay, let’s hack this and then figure out how to hack it.”

So imagine that you gave students at a school a test and the answer key was on the back.

Right, but the reference class of humans does include Cortés and the East India Company.

Sure. So I think one issue here is that people are doing this very kind of partial-equilibrium analysis or something, where they’re thinking about the raw abilities of AI systems in a world where AI systems are dominant, and human civilization has done very little in terms of integrating itself, and AI is integrating itself into the human world.

Insofar as it’s poor at communicating and coordinating with AI, it should have an incentive to address those deficiencies and improve that. Insofar as that’s posing a risk or creating inefficiencies because it’s unable to benefit from coordinating and trading, then it should have this enormous incentive to address that.

Insofar as there is a lot of value to be gained from dominating and taking over humans, what you might get is a more negotiated settlement. If that’s indeed the case, then a war would just be inefficient, and so you would want to negotiate some settlement that results in some outcomes that are mutually beneficial.

There was a mutually beneficial trade that was made between the Qing dynasty and the British in the Opium Wars, right? But it was maybe better than pre-industrial China going to war with the British Empire, but it wasn’t better than never having interacted with the British Empire in the first place.

So I think one mistake that I feel people make is that they have this very naive analysis of what creates conflict. I think Matthew has written a bit about this—a colleague of ours—where they say there’s misalignment, and so that then creates conflict. But that’s actually not what the literature on what causes conflict says creates conflict.

It’s not just misalignment; it’s also other issues, like having a bad understanding of the relative strengths of your armies versus theirs, or maybe having these very strong commitments where you think some grounds are sacred, and so you’re not willing to do any trade in order to give up some of that to gain something else. And so then you have to posit some additional things other than just the base-value-misalignment part.

I think you’re making a good argument against humans taking up the spears and machetes and going to war against the AI data centers, because maybe there’s not this asymmetric information that often leads to conflicts in history. But this argument does not address at all the risk of takeover, which can be the result of a peaceful negotiation or human society being like, “Look, we’re totally outmatched, and we’ll just take these meager concessions rather than go to war.”

But insofar as it’s more peaceful, then I think it’s much less of a thing to worry about. I think there could be this trend where we indeed have this gradual process where AI is much more important in the world economy and actually deciding and determining what happens in the world. But this could be beneficial for humans, where we’re getting access to this vast, much, much larger economy and much more advanced technological stock.

Yeah. So I think it’s important to be clear about what is the thing that you’re actually worried about. I think some people just say, “Oh, humans are going to lose control of the future; we’re not going to be the ones that are making the important decisions,” which, however, I concede, is also kind of nebulous.

But is that something to worry about? If you just think biological humans should remain in charge of all important decisions forever, then I agree, the development of AI seems like a problem for that. But in fact, other things also seem like a problem for that. I just don’t expect it to generically be true.

Like, in 1,000,000 years from now, even if you don’t develop AI, if biological humans, the way we recognize them today, are still making all the important decisions and have something like the culture that we would recognize from ourselves today, I would be pretty surprised by that.

I think Robin Hanson has again talked about this, where he said that a bunch of the things that people fear about AI are just things they fear about change and fast change. So the thing that’s different is that AI has the prospect of accelerating much of this change so that it happens in a narrower period.

I think it’s not just the kind of change that would have happened from, let’s say, genetically modifying humans; it’s instead happening in a compressed amount of time.

I think the worry comes more from—it’s not just that change is compressed. It’s a very different vector of change.

Yeah, but what is the argument for that? I have never seen a good argument for this. You should expect a bunch of change if you accelerate just human change as well.

You might expect different values to become much more dominant. You might expect people who don’t discount the future as much to be much more influential because they save more and make good investments that give them more control—people with higher risk tolerance, because they are more willing to make bets that maximize expected value and so get much more influence.

So, just generically, accelerating human change would also result in a lot of things being lost that you might care about.

I think the argument is that maybe the speed of the change determines what fraction of the existing population or stakeholders or whatever have some causal influence on the future. And maybe the thing you care about is, look, there’s going to be change, but it’s not just going to be like one guy presses a button. That’s the software-only singularity extreme. It’s more like, over time, norms change and so forth.

5. Can we predictably influence the future?

So if you’re looking at the software-singularity picture, I agree that picture looks different. And again, I’m coming back to this because obviously Daniel, and maybe Scott to some extent, probably have this view that the software-only singularity is more plausible. With one person, we could end up in a situation where their idiosyncratic preferences or something end up being more influential.

I agree that makes the situation look different from if you just have this broader process of automation. But even in that world, I think a lot of people have this view about things like value lock-in, where they think this moment is a pivotal moment in history. Then someone is going to get this AI, which is very powerful because of the software-only singularity, and they’re just going to lock in some values. Then those values are going to be stable for millions of years.

I think that just looks very unlike anything that has happened in the past. So I’m kind of confused why people think it’s very plausible.

I think people have the argument that they see the future, again, in my view, in sort of “far mode.” They think there’s going to be 1 AI. It’s going to have some kind of utility function. That utility function is going to be very stable over time, so it’s not going to change.

There won’t be this messiness of a lack of coordination between different AIs, or values drifting over time for various reasons—maybe because they become less functional in an environment, maybe because of other reasons. And so they just don’t imagine that. They say, “Well, utility functions—we can preserve them forever. We have the technology to do that, so it’s just going to happen.” And I’m like, “Well, that seems like such a weak argument to me.”

Often the idea is that, because this is digital, you can preserve the information better and copy it with higher fidelity and so on. But actually, even if you look just at information on the internet, you have this thing called link rot, which happens very quickly. Information that’s digital isn’t preserved for very long at all.

And the point that Matthew was making is that the fact that the information is digital has led to—not maybe led to, but at least been associated with—faster cultural change.

Cultural change, exactly.

I mean, basically, technological changes can create incentives for cultural change just as they make preserving information more difficult.

I think there are 2 key arguments that I’ve heard. One is that we will soon reach something called technological maturity. One of the key ways in which society has been changing recently is—maybe, actually, its culture would have changed even more.

Actually, no, I think this argument that you’re making is wrong, because we do know that language actually changed a lot more. We can read everything that was written after the 1800s, when literacy became more common. But just go back a couple hundred years after that, and you’re reading Old English, and it’s hard to understand.

And that is a result of literacy and the codification of language.

[Speaker?]

Well, that information was better preserved. What about other kinds of cultural practices?

But I think the argument would be that change was a result of technological change in general, not the result of information being digitized. And maybe culture would have actually changed more if information wasn't as well preserved or technology had continued to proceed.

The argument is that, in the future, we're going to reach some point at which you've done all the tech, ideas have just gotten way too hard to find, and you need to make a CERN that's the size of a galaxy to progress physics an inch forward. At that point, this growth in technology, just churning over civilization, goes away. Then you just have the digital thing, which does mean that lock-in is more plausible.

So, the technological maturity thing, I agree that results in this slowdown in change and growth, and certain things might get more locked in relative to what preceded it. But then what do we do today about that?

[Speaker?]

Well, what could you do to have a positive impact, by our lights? Robin Hanson had this question of what someone could do in the 1500s to have a positive impact on the world today from their point of view, knowing all they knew back then.

I think this question is even worse than that, because I think the amount of change that happens between today and technological maturity is just orders of magnitude greater than whatever change happened between the 1500s and today. So it's an even worse position than someone in the 1500s thinking about what they could do to have a positive impact in expectation—something predictably positive today. I think it's just pretty hopeless. I don't know if we could do anything or find any candidate set of actions that would make things better post-lock-in.

That's assuming lock-in is going to happen. In the 1700s, a bunch of British abolitionists were making the case against slavery, and I don't think there's any in-principle reason why we couldn't have been a slave society to this day, or why more of the world couldn't have slavery. I think what happened is just the convincing of British people that slavery is wrong. The British Empire put all its might into abolishing slavery and making that a norm.

I think another example is Christianity and the fact that Jesus has these ideals. You could talk about these ideals.

[Speaker?]

I think the world is a more Christian place.

It is a more Christian place, sure, and also more of the kind of place—I'm not saying Jesus Christ would endorse every single thing that happens in the world today. I'm just saying he endorses this timeline more than one in which he doesn't exist and doesn't preach at all.

[Speaker?]

I don't know, actually. I'm not sure if that's true. It seems like a hard question.

But I think, in sum, from the Christian perspective, it was favorable cultural development for the West.

[Speaker?]

I mean, you don't know the counterfactual. I agree that is always true. I just think the world does have people who read the Bible and are like, “I'm inspired by these ideals to do certain things.” And it just seems like that's more likely to lead to—

So that is what I would call a “legacy effect” or something.

You can say the same thing about languages. Some cultures might just become more prominent and their languages might be spoken more, or some symbols might become more prominent. But then there are things like how cities look, how cars look, what people spend most of their time doing in their day, and what they spend their money on. Those questions seem much more determined by how your values change as circumstances change.

That might be true, but I'm in the position with regard to the future where I expect a lot of things to be different, and I'm okay with them being different. I care much more about the equivalent of slavery, which in this case is literally slavery.

Just to put a final point on it, the thing I really care about is that there are going to be trillions of digital beings. I want it to be the case that they're not tortured and put into conditions in which they don't want to work, or whatever. I don't want galaxies' worth of suffering. That seems closer to British abolitionists being like, “Let's put our empire's might against fighting slavery.”

[Speaker?]

I agree. But I would distinguish between the case of Christianity and the case of the end of slavery, because I think the end of slavery—

I agree you can imagine a society where, technologically, it's feasible to have slavery.

But I think that's not the relevant thing that brought it to an end. The relevant thing is that the change in values associated with the Industrial Revolution made it so that slavery just became an inefficient thing to sustain in a bunch of ways.

A lot of countries, at different times, phased out different things you could call slavery. For example, Russia abolished serfdom in the 1860s. They were not under British pressure to do so. Britain couldn't force Russia to do that; they just did that on their own. There were various ways in which people in Europe were tied to their land and couldn't move or go somewhere else. Those movement restrictions were lifted because they were inefficient.

There were ways in which the kind of labor that needed to be done in the colonies to grow sugar or various crops was very hard labor. It was not the kind of thing that probably you could have paid people to do, because they just wouldn't want to do it because the health hazards and so on were very great, which is why they needed to force people to do it. That kind of work over time became less prevalent in the economy. So, again, that reduces the economic incentives to do it.

I agree you could still do it. I would emphasize that the way you're painting the counterfactual is like, “Oh, but then in that world, they would have just phased out the remnants of slavery.” But there are a lot of historical examples where it's not necessarily only hard labor, like Roman slavery.

Yes, it was different. I interviewed a historian about it recently—the episode hasn't come out—but he wrote a book about the scope. I think it was something like 20% of people under Roman control were slaves. This was not just agricultural slavery. His point was that the maturity of the Roman economy is what led to this level of slavery, because the reason slavery collapsed in Europe after the fall of the Roman Empire was that the economy just lost a lot of complexity.

[Speaker?]

Well, I'm not sure if I would say that slavery collapsed. I think this depends on what you mean by slavery. In a lot of ways, people in feudal Europe were—

But his point is that serfdom was not the descendant institution from Roman slavery.

[Speaker?]

No, I agree. It was not descended from it. But, in fact, this point I'm trying to make is that the values that exist at a given time—what values we will have in 300 years, or, from the perspective of someone a thousand years ago, what values people are going to have in a thousand years—are much more determined by the technological, economic, and social environment that's going to be there in a thousand years.

Which values are going to be functional? Which sides, which values end up being more competitive and more influential, so that other people adopt their values? It depends much less on the individual actions taken by people a thousand years ago.

So I would say that the abolitionist thing is not the cause of why slavery came to an end. Slavery came to an end also because people just have natural preferences that I think are suppressed in various ways during the agricultural era, where it's more efficient to have settled societies in cities that are fairly authoritarian and don't allow for that much freedom. You're in this Malthusian world where people have very low wages, perhaps compared to what they enjoyed in the hunter-gatherer era.

It's just a different economic period, and I think people didn't evolve to have the values that would be functional in that era. So what happened is that there had to be a lot of cultural assimilation where people had to adopt different values. In the Industrial Revolution, people also become very wealthy compared to what they used to be, and that, I think, leads to different aspects of people's values being expressed.

People just put a huge amount of value on equality. It's always been the case, but I think when it is sufficiently functional for that to be suppressed, they are capable of suppressing it.

If that's the story, then this makes value alignment all the more important, because then you're like, “Oh, if the AIs become wealthy enough, they actually will make a concerted effort to make sure the future looks more like the utility function you put into them,” which I think you have been underemphasizing.

[Speaker?]

No, I'm not underemphasizing that. What I would say is that there are certain things that are path-dependent in history, such that if someone had done something different, or something had gone differently a thousand years ago, then today, in some respects, would look different.

For example, which languages are spoken across which boundaries, or which religions people have, or fashion, maybe to some extent, though not entirely. Those things are more path-dependent. But then there are things that are not as path-dependent.

So, for example, if some empire—if the Mongols had been more successful and somehow, I don't know how realistic it is, but they became very authoritarian and had slavery everywhere—would that actually have led to slavery being a much more enduring institution a thousand years later? That seems not true to me.

The forces that led to the end of slavery seemed like they were not contingent forces. They seemed like deeper forces than that. And if you're saying, “Well, if we align the AI today to some bad set of values, then that could affect the future in some ways which are more fragile,” that seems plausible. But I'm not sure how much of the things you care about in the future, and how much of the ways in which you expect the future to get worse, you actually have a lot of leverage over at the present moment.

Another example here might be factory farming, where you could say, “It’s not like us having better values over time led to suffering going down. In fact, suffering might have gone up because of the incentives that led to factory farming emerging.” And probably when factory farming comes to an end, it will be because the incentives start going away, right?

So suppose I care about making sure the digital equivalent of factory farming doesn’t happen. Maybe, all else being equal, it’s just more economically efficient to have suffering minds doing labor for you than non-suffering minds because of the instrumental benefits of suffering or something like that, right? What would you say to somebody like me who’s like, “I really want that not to happen. I don’t want the light cone filled with suffering workers,” or whatever?

Is it just like, “We’ll give up because this is the way economic history is”?

Robin Hanson

No, I don’t think you should give up. It’s hard to anticipate the consequences of your actions in the very distant future. So I would recommend that you discount the future—not for a moral reason, not because the future is worthless or something, but because it’s just very hard to anticipate the effects of your actions.

In the near term, I think there are things you can do that seem like they would be beneficial. For example, you could try to align your present AI systems to value the things that you’re talking about. They should value happiness and dislike suffering, or something. You might want to build up the capacity so that, in the future, if you notice something like this happening, we might have some ability to intervene.

Maybe you would think about the prospect that, eventually, we’re going to colonize other stars, and civilization might become very large, with communication delays becoming very long between different places. In that case, competitive pressures between different local cultures might become much stronger because it’s harder to centrally coordinate. So you might expect competition to take over in a stronger way, and if you think the result of that is going to be a lot of suffering, maybe you would try to stop that.

Again, I think at this point it’s very far from obvious that trying to limit competition is actually a good idea. I would probably think it’s a bad idea, but maybe in the future we will receive some information and we’ll be like, “Oh, we were wrong. Actually, we should stop this.” Then maybe you want to have the capacity so that you can make that decision.

But that’s a nebulous thing. How do you build that up? I don’t know. That’s the kind of thing I would be trying to do.

I think the overall takeaway I take from the way that I think about it, and I guess we think about it, is to be more humble in what you think you can achieve and just focus on the near term—not because it’s more morally important than the longer term, but just because it’s much easier to have a predictably positive impact on that.

One thing I’ve noticed over the last few weeks of thinking about these bigger future topics and interviewing Daniel and Scott and then you two is how often I’ve changed my mind about everything, from the smallest questions about when AI will arrive—it’s funny that that’s the small question in the grand scheme of things—to whether there will be an intelligence explosion, or whether it’ll be an R&D explosion, to whether there’ll be explosive growth, or how to think about that.

If you’re in a position where you are incredibly epistemically uncertain about what’s going to happen, I think it’s important to, instead of becoming super certain about your next conclusion, just take a step back and say, “Well, let me just take a step back. I’m not sure what’s going on here.” And I think a lot more people should be from that perspective, unless you’ve had the same opinion about AI for many years, in which case I have other questions for you about why that’s the case.

Generally, how we as a society deal with topics on which we are this uncertain is just to have freedom and decentralization—both decentralized knowledge and decentralized decision-making—take the reins, and not to do super-high-volatility centralized moves like, “Hey, let’s nationalize so we can make sure that the software-only singularity is aligned,” or to make moves that are incredibly contingent on 1 worldview and brittle under other considerations.

That’s become a much more salient part of my worldview. I think classical liberalism is the way we deal with being this epistemically uncertain, and I think we should be more uncertain than we’ve ever been in history, as opposed to many other people who seem to be more certain about these topics than they are about other, more mundane topics.

Robin Hanson

I think it’s very hard to predict what happens because this acceleration basically means that you find it much harder to predict what the world might be in 10 years’ time. I think these questions are also just very difficult, and we don’t have very strong empirical evidence. There’s a lot of this kind of disagreement that exists.

I would say that it’s much more important in a lot of cases and a lot of situations to maintain flexibility and the ability to adapt to new circumstances and new information than it is to get a specific plan that’s going to be correct, very detailed, and full of specific policy recommendations and things that you should do.

That’s actually also the thing that I would recommend if I want to make the transition to AI in this period of explosive growth go better. I would just prefer it if we, in general, had higher-quality institutions, but I am much less bullish on someone sitting down today and working out, “Okay, what will this intelligence explosion or explosive growth be like? What should we do?”

I think plans that you work out today are not going to be that useful when the events are actually occurring, because you’re going to learn so much stuff and update on so many questions that these plans are just going to become obsolete.

One thing you could do is look at, say, the history of war planning and how successful war planning has been at actually anticipating what happens when the war actually happens. For one example—I think I might have mentioned this off the record at some point—before the Second World War happened, obviously people saw that there were all these new technologies, like tanks and airplanes and so on, which existed in World War I, but in a much more primitive setting.

So they were wondering, what is going to be the impact of these technologies now that we have them on a much greater scale? The British government had estimates of how many casualties there would be from aerial bombardment in the first few weeks of the Second World War. They expected hundreds of thousands of casualties within 2 or 3 weeks after the war began.

The idea was that air bombing was basically this unstoppable force: all the major urban centers were going to get bombed, tons of people would die, and we couldn’t have a war because if there were a war, it would be a disaster because of this aerial bombardment. But later it turned out that that was totally wrong.

In fact, in all of Britain, there were fewer casualties from air bombing in the entire 6 years of the Second World War than the British government expected in the first few weeks of the war. They had fewer casualties in 6 years than they expected in 3 weeks.

So why did they get it wrong? Well, there are lots of boring practical reasons. For example, it turned out to be really infeasible, especially early on, to bomb cities in the daytime because your aircraft would just get shot down. But if you tried to bomb at nighttime, your bombing was really imprecise, and only a very small fraction of it actually hit.

People also underestimated the extent to which people on the ground, like firefighters and so on, could go around the city and put out fires from bombs that were falling on structures. They overestimated the amount of economic damage that it would do. They underestimated how economically costly it would be. Basically, you’re sending these aircraft, and then they’re getting shot down, while an aircraft is very expensive.

In the end, when the Allies started bombing Germany, for each dollar of capital they were destroying in Germany, they were spending 4 to 5 dollars on the aircraft, fuel, training the pilots, and so on that they were sending on missions. The casualty rate was very high, which later got covered up by the government because they didn’t want people to worry.

The planning that you would have done in advance was predicated on this assumption of air bombing going to be “nuclear-weapons-lite”—basically, that it would be extremely destructive.

I mean, it was, though, right? 84,000 people died in 1 night of firebombing in Tokyo. Germany had large fractions of its cities destroyed.

Robin Hanson

But that was over the period of 6 years of war.

Right, but there were single firebombing attacks.

Robin Hanson

That’s right, but if you look at the level of destruction that was expected within the space of a few weeks, and then this level of destruction took many years, there was a 2-order-of-magnitude mismatch or something like that, which is pretty huge.

I mean, there were single firebombing attacks. At the end of World War II, when they were looking for the place to launch the atomic bombs, they had to go through about a dozen cities because it just wouldn’t be worth nuking them; they were already destroyed by the firebombing.

Robin Hanson

That’s right, but if you look at the level of destruction that was expected within the space of a few weeks, and then this level of destruction took many years, there was a 2-order-of-magnitude mismatch or something like that, which is pretty huge. So that affected the way people thought about it.

An important underlying theme of much of what we have discussed is how powerful just reasoning about things is for making progress on what specific plans you want to make to prepare for and make this transition to advanced AI go well.

[Speaker?]

Our view is that it's actually quite hard, and you need to make contact with the actual world in order to inform most of your beliefs about what actually happens. It's somewhat futile to do a lot of wargaming and figure out how AI might go, and what we can do today to make that go a lot better, because a lot of the policies you might come up with might just look fairly silly.

In thinking about how AI actually has this impact, people think, “AI reasoning about doing science and doing R&D just has this drastic impact on the overall economy or technology.” Our view is that making contact with the real world and getting a lot of data from experiments, deployment, and so on is very important.

I think there is this underlying latent variable that explains some of this disagreement, both on the policy prescriptions and about the extent to which we should be humble versus ambitious about what we ought to do today, as well as for thinking about the mechanism through which AI has this impact. This underlying thing is: What is the power of reason? How much can we reason about what might happen? How much can reasoning in general figure things out about the world and about technology? That is a core underlying disagreement here.

I do want to ask: You say in your announcement, “We want to accelerate this broad automation of labor as fast as possible.” As you know, many people think it's a bad idea to accelerate this broad automation of labor and AGI and everything that's involved there. Why do you think this is good?

[Speaker?]

The argument for why it's good is that we're going to have this enormous increase in economic growth, which is going to mean enormous amounts of wealth and incredible new products that you can't even imagine, in health care or whatever. The quality of life of the typical person is probably going to go up a lot.

Early on, probably also their wages are going to go up, because the AI systems are going to be automating things that are complementary to their work. Or it's going to be automating part of their work, and then you'll be doing the rest and getting paid much more for that. In the long term, eventually we do expect wages to fall just because of arbitrage with the AIs.

But by that point, we think humans will own enormous amounts of capital. There will also be ways in which even the people who don't own capital are just going to be much better off than they are today. I think it's hard to express in words the amount of wealth and increased variety of products that we would get in this world.

It will probably be more than the difference between 1800 and today. If you imagine that difference, it's such a huge difference. Then we imagine 2 times, 3 times, whatever.

The standard argument against this is: Why does the speed to get there matter so much, especially if the trade-off against speed is the probability that this transition is achieved successfully in a way that benefits humans?

It's unclear that this trades off against the probability of it being achieved successfully. There might be an alignment tax. Maybe.

You can also just do the calculation of how much a year's worth of delay costs current people. This is an enormous amount of utility that people are able to enjoy, and that gets brought forward by a year or pushed back by a year if you delay things by a year. How much is this worth?

You can look at simple models of how concave people's utility functions are and do some calculations, and maybe that's worth on the order of tens of trillions of dollars per year in consumption. That is roughly the amount consumers might be willing to defer in order to bring forward the date of automation by 1 year.

In absolute terms, it's high. In relative terms, if you did think it was going to nudge the probability one way or another of building systems that are aligned and so forth, it's small compared to all of the future.

I agree. So there are a couple of things here. First of all, I think the way you think about this matters. We don't actually think that it's clear whether speeding things up or slowing things down actually makes a doomy outcome more or less likely. I think that's a question that doesn't seem obvious to us.

Partly because of our views on the software R&D side, we don't really believe that if you just pause and then do research for 20 years at a fixed level of compute scale, you're actually going to make that much progress on relevant questions on alignment or something. Imagine you were trying to make progress on alignment in 2016 with the compute budgets of 2016. You would have gotten nowhere, basically. You would have discovered none of the things that people have discovered today and that turned out to be useful.

I think if you pause today, then we will be in a very similar position in 10 years. We would not have made a bunch of discoveries. Scaling is just really important to make progress in alignment, in our view.

Then there's a separate question of how longtermist you should be in various different senses. There's a moral sense: how much you should actually care about people who are alive today as opposed to people who are not yet born. That's just a moral question.

There's also a practical question of, as we discussed, how certain can you be about the impact your present actions are actually going to have in the future?

Okay, maybe you think it really doesn't matter whether you slow things down right now or speed things up right now. But is there some story about why speeding them up from the alignment perspective actually helps? Is it good to have that extra progress right now rather than later on? Or is it just that if it doesn't make a difference either way, then it's better to get that extra year of people not dying and having cancer cures and so forth?

I think I would say the second. But it's just important to understand the value of that. Even in purely economic terms, imagine that each year of delay might cause maybe 100 million people—maybe more, maybe 150 or 200 million people—who are alive today to end up dying.

Even in purely economic terms, the value of a statistical life is pretty enormous, especially in Western countries. Sometimes people use numbers as high as $10 million for a single life. So imagine you do $10 million times 100 million people. That's a huge number.

That is so enormous that I think for you to think that speeding things up is a bad idea, you have to first have this longtermist view where you look at the long-run future. You think your actions today have high enough leverage that you can predictably affect the direction of the long-run future.

In this case, it's kind of different because you're not saying, “I'm going to affect what some emperor 1,000 years from now does,” like somebody in the year 0 would have to do to be a longtermist. In this case, you just think there's this incredibly important inflection point that's coming up, and you just need to have influence over that crucial period of explosive growth, or intelligence explosion or something.

I agree in relative terms. I agree that the present moment is a moment of higher leverage, and you can expect to have more influence. I just think in absolute terms, the amount of influence you can have is still quite low. It might be orders of magnitude greater than it would have been 2,000 years ago and still be quite low.

I think there's this difference in opinion about how broad and diffuse this transformation ends up being versus how concentrated it is within specific labs, where the very idiosyncratic decisions made by that lab will end up having a very large impact. If you think those developments will be very concentrated, then you think the leverage is especially great.

You might be especially excited about having the ability to influence how that transition goes. But our view is very much that this transition happens very diffusely, by way of many, many organizations and companies doing things, and for those actions to be determined a bunch by economic forces rather than idiosyncratic preferences on the part of labs or decisions that have founder effects that last for a very long time.

6. Arms race dynamic

Let's go through some of the objections to explosive growth. Most people are actually more conservative, not more aggressive, about the forecasts you have. One of the people who has articulated their disagreements with your view is Tyler Cowen. He made an interesting point when we did the podcast together. He said:

“Most of Sub-Saharan Africa still does not have reliable clean water. The intelligence required for that is not scarce. We cannot so readily do it. We are more in that position than we might like to think along other variables.”

We agree with this. Intelligence isn't the bottleneck that's holding back technological progress or economic growth. It's many other things.

This is very much consistent with our view that you need to scale up your overall economy, accumulate capital, accumulate human capital, and have all these factors scale. In fact, this is even consistent with what I was saying earlier. I was pointing out that good management and good policies just contribute to TFP, and they can be bottlenecks.

Right now, we could just plug and play our better management into Sub-Saharan Africa. No, we can't. It's hard. I don't think we can.

Okay, so maybe I should have said one could theoretically imagine plugging and playing—

I agree. I can imagine many things. But we cannot so readily do it because of—it's hard to articulate why—and it wouldn't be so easy to do in just capital or labor.

Why not think that the rest of the world will be in this position with regard to the advances that AI will make possible?

If the AI advances are like the kind of geniuses in a data center, then I agree that that might be bottlenecked by the rest of the economy not scaling up and being able to accumulate the relevant capital to make those changes feasible.

So I kind of agree with this picture, and I think this is an objection to the “geniuses in a data center” type view. I buy basically this. It’s also plausible that you’re going to have the technology, but then some people are not going to want to deploy it, or some people are going to have norms, laws, and cultural things that are going to make it so that AI is not able to be widely deployed in their economy—or not as widely deployed as it otherwise might be. And that is going to make those countries or societies just slower.

Some countries will be growing faster, just like Britain and the Netherlands were sort of the leaders in the Industrial Revolution. They were the first countries to start experiencing rapid growth, and then other countries, even in Europe, had to come from behind. Again, I just think we expect the same thing to be true for AI. The reason that happened was exactly because of these kinds of reasons: those countries that had a culture or governance systems or whatever that were just worse and bottlenecked the deployment and scaling of the new technologies and ideas. It seems very plausible.

But you’re saying as long as there’s 1 jurisdiction?

Yeah.

But then again, you also previously emphasized the need to integrate with the rest of the global economy and the human economy. So doesn’t that contradict…?

That doesn’t often require cultural homogeneity. We trade with countries—the U.S. trades with China quite a lot, actually. And there’s a bunch of disagreement.

But what if the U.S. is like, “I don’t like that the UAE is doing explosive growth with AI; we’re just going to embargo them”? That seems plausible. And then, would that not prevent explosive growth?

I think that would be plausible at the point at which it’s revealing a lot about the capabilities and the power of AI. You should also think that this creates both an incentive to embargo, but also an incentive to adopt the very similar styles of governing that enable AI to be able to produce a lot of value.

What do you make of this? I think people interpret explosive growth from an arms race perspective. That’s often why I think in terms of public-private partnerships for the labs themselves.

But this idea that you have the geniuses in the data center: you can have them come up with the mosquito drone swarms. Then those drone swarms will—if China gets to the swarms earlier…

Even within your perspective, is this a result of your whole economy being advanced enough that you can produce mosquito drone swarms? You being 6 months ahead means that you could decisively win—does it? I don’t know. Maybe you being 1 year ahead and explosive growth means you could decisively win a war against China, or China could win a war against you. So would that lead to an arms race-like dynamic?

[Speaker?]

I think it would to some extent, but I’m not sure if I would expect a 1-year lead to be enough to take a risk, because if you go to war with China… For example, if you replace China today with China from 1990, or if you replace Russia today with Russia from 1970 or 1980, it’s possible that their ICBMs and whatever technology they have are already enough to create a very strong deterrence. So maybe even that technological lead is not sufficient for you to feel comfortable going to war. So that seems possible.

Yeah. And actually this relates to a point that Gwern was making, which is that this is going to be a much more unstable period than the Industrial Revolution, even though the Industrial Revolution saw many countries gain rapid increases in their capabilities. Because within this span, if you’ve got a century’s worth of progress compressed within a decade, one country gets to ballistic missiles first, then the other country gets to railroads first, and so forth.

But if you have this more integrated perspective about what it takes to get to ballistic missiles and to railroads, then you might think, “No, basically this isn’t some orthogonal vector. You’re just churning on the tech tree further and further.”

[Speaker?]

For what it’s worth, I do think it’s possible if you have it just happen in a few countries which are relatively large and have enough land or something. Those countries would be starting from a lower base compared to the rest of the world, so you would need to catch up to some extent if they’re just going to grow internally and not depend on the external supply chains. But that doesn’t seem like something that’s impossible to me. Some countries could do it; it would just be more difficult.

But in this setting, if some countries have a significant policy advantage over the rest of the world and they start growing first, then they won’t necessarily have a way to get other countries to adopt their norms and culture. In that case, it might be more efficient for them to do the growth locally. So that’s why I was saying the growth differentials will probably be determined by regulatory jurisdiction boundaries more than anything else.

I’m not saying, say, the U.S. by itself, if it had AI but couldn’t get the rest of the world to adopt AI, wouldn’t still be sufficient for explosive growth. I think that would still be sufficient for explosive growth.

How worried should we be about the fact that China today, because it industrialized relatively recently, has more industrial capacity and know-how and all the other things of learning by doing and so forth? If we buy your model of how technology progresses, with or without AI, how are we just underestimating China because we have this perspective that what fraction of your GDP you’re spending on research is what matters, when in fact it’s the kind of thing where I’ve got all the factories in my backyard and I know how they work and I can go buy a component whenever I want?

[Speaker?]

I don’t think people are necessarily underestimating China. It depends on who you’re looking at, but it seems like the discussion of China is just this very big discussion in these AI circles, right? People are very much appreciating the power and the potential threat that China poses. But I think the key thing is not just the scale in terms of the pure number of people or number of firms or something, but the scale of the overall economy, which is just measured in how much is being produced in terms of dollars. There, the U.S. is ahead.

But we’re not expecting all this explosive growth to come from financial services. We’re expecting it to start from a base of industrial technology and industrial capacity.

[Speaker?]

No, financial services can be important if you want to scale very big projects. Financial services are very important for raising funding and getting investments in data centers.

If I understood you correctly, it just seems like, man, you know how to build the robot factories and so forth. That know-how which, in your view, is so crucial to technology growth and general economic growth, is lacking. You might have more advanced financial services, but it seems like the more you take your view seriously, the more it seems like having Shenzhen locally matters a lot.

[Speaker?]

I mean, relative to what starting point? I think people already appreciate that China is very important. And then I agree that there are some domains where China is leading, but there are very many domains in which the U.S. is leading, or the U.S. and its allies, where countries are producing relevant inputs for AI that the U.S. has access to, but China doesn’t.

So I think the U.S. is just ahead on many dimensions, and there are some that China is ahead in, or at least very close. I don’t think this should cause you to update very strongly in favor of China being a much bigger deal, at least depending on where you start. I think people already think China is a big deal. This is the big underlying thing here. If we were just very dismissive of China, then maybe this would be a reason to update.

7. Is superintelligence a real thing?

I get your argument that thinking about the economy-wide acceleration is more important than focusing on the IQ of the smartest AI. But at the same time, do you believe in the idea of superhuman intelligence? Is that a coherent concept in the way that you don’t necessarily stop at human-level Go play and you just go way beyond it in Elo score? Will we get to systems that are like that with respect to the broader range of human abilities?

Maybe that doesn’t mean they become God, because there are other ASIs in the world. But you know what I mean: will there be systems with such superhuman capabilities?

[Speaker?]

Yeah, I do expect that. I think there’s a question of how useful this concept is for thinking about this transition to a world with much more advanced AI. I don’t find this a particularly meaningful or helpful concept. People introduce some of these notions that, on the surface, seem useful, but then when you delve into them, they’re actually very vague and kind of unclear what you’re supposed to make of them.

You have this notion of AGI, which is distinguished from narrow AI in the sense that it’s much more general and maybe can do everything that a human can do on average. AI systems have these very jagged profiles of capability. So you have to somehow take some notion of average capabilities, and what exactly does that mean? It just feels really unclear what you’re supposed to make of this.

And then you have this notion of ASI, which is AGI in the sense that it’s very general, but then it’s also better than humans on every task. Is this a meaningful concept? I guess it’s coherent. I think this is not a super useful concept, because I prefer just thinking about what actually happens in the world.

You could have a drastic acceleration without having an AI system that can do everything better than humans can do. I guess you could have no acceleration when you have an ASI that is better than humans at everything, but it’s just very expensive or very slow or something. So I don’t find that particularly meaningful or useful. I just prefer thinking about the overall effects on the world and what AI systems are capable of producing those types of effects.

One intuition pump here is: compare John von Neumann versus a human plucked from the standard distribution. If you added a million John von Neumanns to the world, what would the impact on growth be as compared to just adding a million people from the normal distribution? I agree it would be much greater. But then, because of the Moravec paradox-type arguments that you made earlier—that evolution has not necessarily optimized us for that long along the kind of spectrum on which John von Neumann is distinguished from the average human—and given the fact that already within this deviation you have this much greater economic impact, why not focus on optimizing this thing that evolution has not optimized that hard on, further?

Robin Hanson

I don't think we shouldn't focus on that. But what I would say is, for example, if you're thinking about the capabilities of Go-playing AIs, then the concept of a superhuman Go AI—yeah, you can say that is a meaningful concept. But if you're developing the AI, it's not a very useful concept. If you just look at the scaling curve, it just goes up, and there is some human level somewhere. But the human level is not privileged in any sense.

The question is, is it a useful thing to be thinking about? The answer is probably not. It depends on what you care about. So I'm not saying we shouldn't focus on trying to make the system smarter than humans are. I think that's a good thing to focus on.

Yeah, I guess I try to understand whether we will stand in relation to the AIs of 2100 as humans stand in relation to other primates. Is that the right mental model we should have, or is it going to be a much greater familiarity with their cognitive horizons?

I mean, will we be able to cognitively access the kinds of considerations they can take on board? Humans are diverse, but no chimp is going to be able to understand this argument in the way that another human might be able to, right? So if I'm trying to think about my place, or a human's place, in the world of the future, is the relevant concept just that the economy has grown a lot and there's much more labor, or are there beings who are, in this crucial way, superintelligent?

Robin Hanson

I think AI systems will be very diverse, and so it's not super meaningful to ask something about this very diverse range of systems and where we stand in relation to them.

And then how important is it for us to have access, or in principle be able to access, those considerations?

Robin Hanson

I think it's not clear to me that it's particularly important that any individual human should be able to access all the relevant considerations that produce some outcome. That just seems like overkill. Why do you need that to happen? I think it would be nice in some sense. But I think if you want to have a very sophisticated world where you have very advanced technology, those things will just not be accessible to you. So you have this trade-off between accessibility and maybe how advanced the world is.

From my point of view, I'd much rather live in a world which has very advanced technology, has a lot of products that I'm able to enjoy, and a lot of inventions that I can improve my life with, if that means that I just don't understand them. I think this is a very simple trade that I'm very willing to make.

8. Reasons not to expect explosive growth

Okay, so let's get back to objections to explosive growth. We discussed a couple already. Here's another, which is more a question than an objection: Where is all this extra output going? Who is consuming it? If the economy is 100× bigger in a matter of a decade or something, to what end?

Robin Hanson

So first of all, I think even if you view that along what you might call the intensive margin, in the sense that you just have more of the products you have today, I think there will be a lot of appetite for that. Maybe not quite 100×; that might start hitting some diminishing returns. Current GDP per capita, on average, in the world, is $10,000 a year or something, right? And there are people who enjoy millions of dollars. And so there's a gap between what people enjoy and don't seem to be super diminished in terms of marginal utility, and so there's a lot of room on just purely the intensive margin of just consuming the things we consume today but more.

There is this maybe much more important dimension along which we will expand, which is—

Product variety.

Robin Hanson

Yeah, extensive margin: what is the scope of things that you're consuming? And if you look at something like the Industrial Revolution, that seemed to have been the main dimension along which we expanded to consume more. In any kind of sector that you care about—transportation, medicine, entertainment, and food—there's just this massive expansion in terms of variety of things that we're able to consume that's enabled by new technology or new trade routes or new methods of producing things. So that, I think, is really the key thing that we will see come along with this kind of expansion in consumption.

Another point that Tyler makes is that there will be some mixture of Baumol's cost disease, where you're bottlenecked by the slowest-growing thing. The fastest-productivity things basically diminish their own—

Robin Hanson

Share in output. That's right, yeah. We totally agree with that. I would say that that's just a kind of qualitative consideration. It isn't itself sufficient to make a prediction about what growth rates are permitted given these effects versus not; it's just a qualitative consideration, and then you might need to make additional assumptions to be able to make a quantitative prediction.

The convincing version of this argument would be if you did the same thing that we were doing earlier with the software-only singularity argument, where we were pointing to essentially the same rejection: there are multiple things that can bottleneck progress. So I would be much more convinced if someone pointed to an explicit thing, like: Here, health care is this very important thing. Why should we expect AI to make that better? That doesn't seem like it would get better because of AI. So maybe health care just becomes a big part of the economy and then that's the bottleneck.

If there's one, though, and it's a small part of the economy, then you could still get a lot of growth. You just automate everything else, and that is going to produce a lot of growth. So it has to quantitatively work out. And so you actually have to be quantitatively specific about what this objection is supposed to be.

Robin Hanson

Right. So first of all, you have to be specific about what these tasks are. What is the current share in economic output? The second thing is, you have to be specific about how bad you think the complementarities are. So in numerical terms, economists use the concept of elasticity of substitution to quantify this. That gives you a numerical estimate of, if you just have much more output on some dimensions but not that much on other dimensions, how much does that increase economic output overall?

And then there's a third question. You can also imagine you automate a bunch of the economy. A lot of humans were working on those jobs, so now they don't need to do that anymore because those got automated. So they could work on the jobs that haven't yet been automated. As I gave the example earlier, you might imagine a world in which remote-work tasks get automated first, and then sensory-motor skills lag behind. You might have a world in which software engineers become physical workers instead.

Of course, in that world, the wages of physical workers will be much higher than their wages are today. So that reallocation also produces a lot of extra growth, even if bottlenecks are maximally powerful. Even if you just look at all the tasks in the economy and literally take the worst one for productivity growth, you would still get a lot of increase in output because of this reallocation.

So I think one point that I think is useful to make: our experience talking to economists about this is that they will bring up these more qualitative considerations, whereas the arguments that we make make specific quantitative predictions about growth rates. For example, you might ask, “How fast will the economy double?”

And then we can think about an H100. There are some estimates of how much computation the human brain does per second, and it's about 10^15 FLOPs or so. It's a bit unclear, but then it turns out that an H100 roughly does on that order of computation. And so you can ask the question, “How long does it take for an H100 to pay itself back if you run the software of the human brain?”

If you run the software of the human brain, you can then deploy that in the economy and earn, say, human wages on the order of $50,000 to $100,000 a year or whatever in the US. And so then it pays itself back because it costs on the order of $30,000 per H100. And so you get a doubling time of maybe on the order of a year.

And then there's the response: “Well, you have Baumol effects.” Well, what does this mean? Does it double? Does this predict it doubles every 2 years or every 5 years? You need just more assumptions in order to make this a coherent objection.

The other objection that I've heard often—and it might have a similar response from you—is this idea that a lot of the economy is comprised of O-ring-type activities.

And this refers to, I think, the Challenger space shuttle explosion. There is just one component—I forgot what the exact problem with the O-ring was—but because that was faulty, the whole thing collapsed. I think it's quite funny, actually, because the O-ring model takes the product of many, many inputs, and then the overall output is the product of very many things. But actually, this is pretty optimistic from the point of view of having fewer bottlenecks.

I think we pointed this out before. Again, talking about a software-only singularity, I said, if it's the product of computer experiments with research—but if one of those products is zero, because of human—

[Speaker?]

But you have constant marginal product there, right?

Yeah, but if one of those products doesn't scale, that doesn't limit scaling. It means you're less efficient at scaling than you otherwise would be, but you can still get a lot of scaling. You can just have unbounded scaling in the O-ring world.

[Speaker?]

So actually, I disagree with Tyler: he's not conservative enough. He should take his bottlenecks view more seriously than he actually is, and yet I disagree with him about the conclusion. I think that we're going to get explosive growth once we have AI that can flexibly substitute.

I'm not sure I understand. Will there be entirely new organizations that AIs come up with? We've written a blog post about one such: AI firms. You might be a productive worker or a productive contributor in an existing organization as it exists today.

[Speaker?]

In the AI world, many humans might just be zero or even negative.

I agree. Why won't that just put that in the multiplication? But why would humans be in the loop there? You're both saying that humans would be negatively contributing to output, but then you're also saying that we should put them into the—

[Speaker?]

Okay, fair. The main objection often is regulation. We've addressed it implicitly at different points, but we might as well explicitly address why regulation won't stop this.

For what it's worth, we do have a paper where we go over all the arguments for and against explosive growth. Regulation, I think, is the argument that seems strongest against it. The reason it seems strong is that, even though we've made arguments before about international competition, variation in policies among jurisdictions, and these strong incentives to adopt this technology for both economic and national security reasons, I think those are pretty compelling when taken together. Even still, the world does have a surprising ability to coordinate on just not pursuing certain technologies.

Right. Human cloning—

[Speaker?]

That's right. I think it's hard to be extremely confident that this is not going to happen. I think it's less likely that we're going to do this for AI than it is for human cloning, because human cloning touches on some other taboos and so on. It's also less valuable.

Also less valuable.

[Speaker?]

And probably less important for national security in an immediate sense. But at the same time, as I said, it's just hard to rule this out.

So if someone said, “Well, I think there's a 10%, 15%, whatever, 20% chance that there will be some kind of global coordination of regulation, and that's going to be very effective. Maybe it will be enforced through sanctions on countries that defect. Then maybe it doesn't prevent AI from being deployed, but maybe just slows things down enough that you never quite get explosive growth,” I don't think that's an unreasonable view. It's like a 10% chance that it could be.

I don't know. Do you encounter any other objections? What should I be hassling you about?

[Speaker?]

Some things that we've heard from economists: people sometimes respond to our argument about explosive growth, which is an argument about growth levels. We're saying, “We're going to see 30% growth per year instead of 3%.” They respond to that with an objection about levels. They say, “How much more efficient, how much more valuable can you make hairdressing, taking flights, whatever, or going to a restaurant?”

That is fundamentally the wrong kind of objection. We're talking about the rate of change, and you're objecting to it by making an argument about the absolute level of productivity. As I said before, it is not an argument that economists themselves would endorse if it was made about a slower rate of growth continuing for a longer time. So it seems more like special pleading.

Why not just the deployment thing, where the same argument you made about AI applies? You do learn a lot just by deploying to the world and seeing what people find useful. ChatGPT was an example of this. Why won't a similar thing happen with AI products and services, where one of the components is that you put it out to the marketplace, people play with it, you find out what they need, and it plugs into the existing supply chain and so forth? Doesn't that take time?

[Speaker?]

It takes time, but it is often quite fast. In fact, ChatGPT grew extremely fast.

Right, but that was just a purely digital service.

[Speaker?]

One reason to be optimistic is if you think the AIs will literally be drop-in remote workers—or drop-in workers in some cases, if you have robotics—then companies are already experienced at onboarding humans. Onboarding humans doesn't take a very long time. Maybe it takes 6 months, even in a particularly difficult job, for a new worker to start being productive. Well, that's not that long.

So I don't think that would rule out companies being able to onboard AI workers, assuming that they don't need to make a ton of new complementary innovations and discoveries to take advantage of them. I think one way in which current AI systems are being inhibited, and the reason we're seeing growth be slower than you might otherwise expect, is that companies in the economy are not used to working with this new technology. They have to rearrange the way they work in order to take advantage of it. But if AI systems were literally able to substitute for human workers, the complementary innovations might not be as necessary.

9. Fully automated firms

Actually, this is a good excuse to go to the final topic, which is AI firms. This blog post we wrote together was about what it would be like to have a firm that is fully automated, and the crucial point we were making was that people tend to overemphasize AI from the perspective of how smart individual copies will be. If you actually want to understand the ways in which they are superhuman, you want to focus on their collective advantages, which, because of biology, we are precluded from: the fact that they can be copied with all their tacit knowledge.

You can copy a Jeff Dean or Ilya Sutskever, or whatever the relevant person is in a different domain. You can even copy Elon Musk, and he can be the guy who's every single engineer in the SpaceX organization. The AI equivalent of that is that, if it's not best to have Elon Musk or anything, you just copy the relevant team or whatever.

[Speaker?]

We have this problem with human firms, where there can be very effective teams or groups, but over time their culture dilutes, the people leave or die, or they get old. This is one of the many problems that can be solved with these digital firms.

Firms right now have 2 of the 3 relevant criteria for evolution: they have selection, and they have variation, but they don't have high-fidelity replication. You could imagine a much faster-paced and more intense sequence of evolution for firms once you have this final piece click into place. That relates to the onboarding thing, where right now they just aren't smart enough to be onboarded as full workers. But once they are, I just imagine the kinds of things I try to hire for. It would be such an unlock.

The salaries are totally secondary. The fact that I can say, “This is the skill I need,” or the set of skills I need, and I can have 1,000 workers in parallel if there's something that has a high elasticity of demand—that would just be such an unlock. I think it's probably, along with transformative AI, the most underrated tangible thing that you need to understand about what the future AI society will look like.

I think there's a first point about this very macroeconomic picture, where you just expect a ton of scaling of all the relevant inputs. I think that is the first-order thing. But then you might have more micro-questions about, “Okay, how does this world actually look? How is it different from a world in which we just have a lot more people and a lot more capital?” Because it should be different. Then I think these considerations become important.

I think another important thing is just that AIs can be aligned. You get to control the preferences of your AI systems in a way that you don't really get to control the preferences of your workers. Your workers, you can just select; you don't really have any other option. But for your AIs, you can fine-tune them. You can build AI systems which have the kind of preferences that you want.

You can imagine that's dramatically changing basic problems that determine the structure of human firms. For example, the principal-agent problem might go away. This is a problem where, as a worker, you have incentives that are either different from those of your manager, those of the entire firm, or those of the shareholders of the firm. I actually think the incentives are a smaller piece of the puzzle. It's more about bandwidth and information sharing.

With a large organization, it's very hard to have a single coherent vision, and the most successful firms we see today are where, for an unusual amount of time, a founder is able to keep their vision instilled in the organization. SpaceX and Tesla are examples of this. People talk about NVIDIA this way.

But just imagine a future version where there's this hyper-inference-scale mega-Jensen, where you're spending $100 billion a year on inference, and copies of him are constantly writing every single press release, reviewing every pull request, answering every customer service request, and so forth, monitoring the whole organization, making sure it's proceeding along a coherent vision, and getting merged back into the hyper-Jensen, mega-Jensen, whatever.

Yeah, I agree that's a bigger deal. At the same time, I would point out that part of the reason why it's important to have a coherent vision and culture and so on in human companies might be that incentive problems exist otherwise. I wouldn't rule that out, but I agree that, aside from the overall macroeconomic thing, I think the fact that they can be replicated is probably the biggest deal.

That also enables additional sources of economies of scale. If you have twice the number of GPUs, you can run not only twice the number of copies of your old model, but then you can train a model that's even better. So you double your training compute and your inference compute, and that means you don't get just twice the number of workers you would have had otherwise; you get more than that, because they are also smarter, because you spend more training compute. So that is an additional source of economies of scale.

And then there's this benefit that, for humans, every human has to learn things from scratch. They are born, and then they have a certain amount of lifetime learning that they have to do. So in human learning, there's a ton of duplication, while for an AI system, it could just learn once. It could just have one huge training run with tons of data, and then that run could be deployed everywhere. So that's another massive advantage that AIs have over humans.

10. Will central planning work?

Maybe we'll close up with this one debate we've often had offline, which is: will central planning work with these economies of scale? Again, the question is, “Will it work?” Will it be optimal? My guess is probably not optimal. But I don't think anyone has thought this question through in a lot of detail. So it is worth thinking about why one might expect central planning to be slightly better in this world.

One consideration is just communication bandwidth being potentially much, much greater than it is today. In the current world, information gathering and information processing are co-located: humans observe and also process what they observe. In an AI world, you can disaggregate that. So you can have sensors that don't do much processing but just collect information, and then process it centrally. Central processing might make sense for a bunch of reasons, and you might get economies of scale from having more GPUs that produce better models, and also be able to think more deeply about what they're seeing.

It's worth noting that certain things already work like this, for example, Tesla FSD. It will benefit from the data collected at the periphery from millions of miles of driving. And then the improvements which are made as a result of this are centrally directed; it's coming from HQ being like, “We're going to push an update.” So you do get some of this more centralized approach, and it can be a much more intelligent form than just whatever gradient averaging they do. I mean, I'm sure it's more sophisticated than that at Tesla, but it can be a much more deliberate, intelligent update. So that's one reason to expect it to work.

And the other reason, I guess, is current leaders or CEOs don't have bigger brains than the workers do.

Maybe a little bit. I don't know if you want to open that.

But not by orders of magnitude. And so you could have orders of magnitude more scaling of the size of the models that are doing the planning than the people or the agents or workers doing the actions.

And I think a third reason is the incentive thing, where part of the reason you have a market is that it gives people the right kinds of incentives. But you might not need that as much if you're using AI. So I think there's an argument that if you just list the traditional arguments people have made against “Why does central planning not work?” then you might expect them to become weaker.

Now, I think there's a danger when you're doing that kind of analysis to fall into the same kind of partial-equilibrium analysis where you're only considering some factors and then you're not considering other things. For example, things get more complex; you just have a much bigger economy. So on the one hand, your ability to collect information and process it improves, but the need for doing that also increases as things become more complex.

And one way to illustrate that is: imagine if Apple, the organization today, with all its compute and whatever, was tasked with managing the economy of Uruk. I think it actually could centrally plan the economy. The economy of Uruk might work even better as a result. But Apple as it exists today cannot manage the world economy as it exists today.

That's right. Yeah. All right, actually, this will be the final question. One of the things that makes AI so fascinating is that there is no domain of human knowledge that is irrelevant to studying it.

I don't know about that. There's no serious domain of human knowledge—

That's better—that is not relevant to studying it, because you're just fundamentally trying to figure out what a future society will look like. Obviously, computer science is relevant, but also economics, as we've been discussing, history and how to understand history, and many other things we've been discussing.

Especially if you have longer timelines and there is enough time for somebody to pursue a meaningful career here, what would you recommend to somebody? Because both of you are quite young. I mean, you especially, Ege, but both of you. You would think this is the kind of thing which requires crystallized intelligence or whatever, especially given what we said earlier about—look, as we get more knowledge, we're going to have to factor what we're learning into building a better model of what's going to happen to the world. And if somebody is interested in this kind of career that you both have, what advice do you have for them?

Yeah, that's a hard question. I'm not sure. I think there is an extent to which it's difficult to deliberately pursue the implicit strategy that we would have pursued. It probably works better if it's spontaneous and more driven by curiosity and interest than if you make a deliberate choice: “Okay, I'm just going to learn about a bunch of things so that I can contribute to the discourse on AI.” I would think that strategy is probably less effective. At least I haven't seen anyone who deliberately used that strategy and then was successful, it seems like.

Yeah, I guess not that I've contributed to discourse directly, but maybe facilitated other people contributing. I guess it wasn't a deliberate strategy on my end, but it was a deliberate strategy to do the podcast, which inadvertently gave me the opportunity to learn about multiple fields.

Yeah, so given that you're already interested and curious and reading a bunch of things, and studying a bunch of things, and thinking about these topics, on the margin there are a bunch of things you can do to make you more productive at making some contributions to this. And I think just speaking to people and writing your thoughts down and finding especially useful people to chat with and collaborate with, I think that's very useful. So just seek out people that have similar views and that you're able to have very high-bandwidth conversations with and make progress on these topics. And I think that's just pretty useful.

But how exactly? Should they DM you? How do they get in?

Yeah, sure. I don't know, set up Signal chats with your friends or whatever. Actually, it's crazy how much alpha I've gotten out of that.

But I think one piece of advice I would give to people in general, even if they are not thinking about AI specifically, but I think it's also helpful for that, is that people should be much more aggressive about reaching out. People have an impression that if you reach out to someone who looks really important, they're not going to respond to you. But if what you send to them is interesting and high quality, then it's very, very likely that they will respond. There's a lot more edge there that you can get, which is just being more aggressive and less ashamed of looking dumb. That's the main advice I would give.

Because if you want to be productive, then again, there are these complementarities, and so you need to be part of some community or some organization. And it goes back to the thing about reasoning alone not being that helpful.

Yeah, yeah, yeah. It's just like other people have thought a long time and have randomly stumbled upon useful ideas that you can take advantage of.

That's right. So you should just try to place yourself in a situation where you can become part of something larger. Which isn't working on the frontier; that's just a more effective way of contributing. And to do that, you have to let people know.

That's right.

And I think just coming to the Bay Area is especially useful if you have an interest in AI in particular.

Yeah, going to the Bay Area is nice. Just writing things and posting them where people can see them. Just aggressively reaching out to people with interesting comments, provided your thoughts are interesting and so on. I mean, they probably aren't. In many cases, my thoughts still might not be interesting, but people will tolerate my cold emails and will still collaborate with me and so forth.

The other thing I've noticed—tell me if this is actually the wrong pattern—is that with people like you or with Carl Shulman or something, as compared to a general person who's intellectually curious or reading widely, you tend to focus much more on key pieces of literature than, say, “I'm going to go read the classics or just generally read.” It's like, “I'm going to just put a ton more credence in something like the Romer paper.” And a normal person who's intellectually curious would not be reading key pieces of literature.

Yeah. I think you have to be very mindful of the fact that you have a very limited amount of time; you're not an AI model. So you have to aggressively prioritize what you're going to spend your time reading.

Even AI models don't prioritize that heavily. They read Reddit mostly, or a large part of their corpora, rather than key pieces of empirical literature—at least among you guys.

I think that's useful. I also think it's useful to read Twitter. I think we were having this conversation about how people often say that they're spending too much time reading Twitter and wish they spent more time reading arXiv. But actually, the amount of information per unit of time you get from reading Twitter is often much higher, and it's much more productive for them to read Twitter.

I think there are key pieces of literature that are important, and I think it's useful to figure out what people who have spent a lot of time thinking about this find important in their worldview. So in AI, this might be key papers. The Andy Jones paper about scaling laws for inference is a big thing. And in economics, this is the Romer paper, or the paper on explaining long-run population from Kremer or from David Roodman, and so on.

I think if people who you think think very well about this suggest a certain paper and highly recommend it, then you should take that seriously and actually read those papers. And for me, instead of just skimming a bunch of things, it's been especially helpful, if there's a key piece of literature—for example, in order to understand the transformer. There's always Andrej Karpathy's lectures, but one piece of research that was really useful was Anthropic's original transformer-circuits paper, A Mathematical Framework for Transformer Circuits.

Just spending a day on that paper instead of skimming it, making a bunch of spaced-repetition cards, and so forth, was much more useful than just generally reading widely about AI. I think it's just much more important here, if you want to prioritize things correctly, to be part of a community, to get input from a community, or to get input from people who have thought a lot and have a lot of experience with what is important and what is not. This is true even in academic fields.

So if you want to do math research but you're not part of a graduate program, and you're not at a university where there are tons of people who do math research all day for many years, then you're not even going to know: What are the open problems that I should be working on? What is reasonable to attack? What is not reasonable to attack? What papers in this field are important and contain important techniques? You're just going to have no idea. So it's very important to be plugged into that feed of information somehow.

But how did you know all this shit before being plugged in? Because you weren't talking to anybody in Ankara.

You don't need to talk. The internet is a pretty useful thing in this respect. You don't necessarily need to talk to people; you can get a lot of benefit from reading. But you just need to identify the people who seem constantly most interesting.

Maybe you find 1 person. Often, that person will know some other people who are interesting. Then you can start tracing the social network. One example I can give, which I think is actually accurate, is Daniel Ellsberg. So you look for a podcast he appears on. You notice that he's appeared on the 80,000 Hours podcast, which he has.

Then you notice there are some other guests on the 80,000 Hours podcast. So maybe there's Bryan Caplan, who has also appeared on the podcast. And then maybe Robin Hanson has also appeared on the podcast. And then maybe there are some people those other people know. Just tracing that kind of social network and figuring out who to listen to like that—I think that can be useful.

And I think you're doing a very big service in making that possible. I think your selection is often very good. I'm actually curious to hear offline what I got wrong.

Well, actually, I think I know the answer to that. And I think that makes it a bunch easier to track who the people doing the most interesting thinking on various topics are.

That's right.

Cool. I think that's a good place to end, with you praising me. Thanks for lending your studio.

Yeah, that's very generous.

Anyways, cool. Thanks, guys.