AI力量的平衡：Anton Leicht谈政治、节奏协议与如何稳住局面

Anton Leicht, fellow at the Carnegie Endowment for International Peace, welcome to The Cognitive Revolution.

Thanks for having me.

I'm excited for this conversation. You have been popping up all over the place with your own writing and various interviews, and you're clearly a renaissance person with advanced thinking on a lot of different aspects of the increasingly complicated AI age in which we find ourselves. So I'm excited to run down a bunch of these rabbit holes with you today.

Yeah, it's exciting. I don't know. I think you can't do part of the thing at the moment, right? It's just all the geopolitical end of history and the end of technology history, or whatever, are coming together at a rapid pace. So I guess either you do everything or you do nothing. Renaissance time, something like that.

Yeah, I feel the same way, actually. The big motivation for this project was just observing how many people are so deep down specific rabbit holes, advancing—and usually having success advancing—whatever frontier they're advancing, but not so many people have taken the purposeful approach of foregoing being an expert in any particular area and trying to cultivate a broad view. So I appreciate a kindred spirit in that regard.

Yeah. Let's call my lack of deep expertise in anything a conscious choice and not a failure. I appreciate that framing a lot.

That's working for me. Let's start with just a real simple calibration question, but in some ways maybe the most important question: How dangerous do you think today's AIs are?

1. The Current AI Danger

I think they're not very dangerous at the current capability level, in most of the ways people are talking about. The thing that is concerning is the trend line toward really dangerous capabilities, and more specifically, that we don't know at which point things will keep accelerating more and more. I think we're not yet at a threshold where there's broad harm caused by the deployment of any current AI system.

I think we might be very close to 2 kinds of potentially very dangerous AI systems. The first would be AI systems that are good enough to meaningfully uplift internal development at the labs, which would then lead to more and more capable models fairly soon. I think you can go either way on the prospect of a software-only intelligence explosion, but short of that, I think we're nearing a point where the pace of development inside the labs breaks away from the pace of democratic oversight and democratic insight into what's happening. I think that's one threshold we're near.

The other very concrete threshold that we're near is that they're throwing a lot of RL budget, a lot of data, and a lot of effort at making these models good at life sciences, pharmaceutical, and bio applications, for obvious reasons. There would be great upside to being able to cure cancer, as Dario puts it, and to get some of these real-world effects. There would also be a great political upside to doing that.

But, man, that sounds like a very dangerous model if we get there. I think if they get something that's as good as Mythos is on long-run cyber and software-engineering kinds of things in the domain of bio, that does sound a lot more dangerous than the suite of models we have today.

Yeah. I agree with that second one in particular, and I am honestly not even sure at this point whether the current models aren't perhaps quite dangerous in that domain. I squint through the limited peephole that we have at the Open Face incident, and I noticed that one of the tasks that one of the earliest agents ever to use the message board was working on was something related to a protein database. That kind of freaked me out because I was like, “Wait a second. That means these bio and cyber specialists are cross-training in the same environment, or at least in the same environment when they have the message board.”

They're running these evals, at least in a kind of cross-contaminated way. You've got agents breaking out. We've got existence proofs of social engineering in the wild, against real people. And I'm just—I don't know. Should anyone be confident that they can't do that at this point? The experts seem to be confident, but my meta-observation is that the experts seem to be surprised quite often right now.

Yeah. I think that's a really interesting conversation, and I also think one of the very interesting parts of this is that we used to think of this bio risk as primarily a misuse risk. It was like, well, at some point, maybe this is also the final risk that emerges from the sort of loss-of-control scenarios. But really, bio was always framed as the most immediate and most obvious way for the misuse conversation to go wrong.

What I think has happened is that very autonomous and potentially somewhat malicious, or at least misaligned, agents have come much earlier in the capability trajectory than people expected. Relative to what the agents can actually do, they're sort of out of control earlier than people might have thought. And so it's interesting that people usually used to respond to the bio argument by saying, “Well, A, there are a lot of these real-world bottlenecks,” which I do still think exist and which make me a little bit less worried than you.

If I remember correctly, Helen had some good responses to you on that, and there is a good back-and-forth to be had around things like how integrated the cloud labs are and how much you can do in the real world. I think that applies both to loss of control over agents and to misuse. But the other question is that the usual argument against bio misuse was always, well, this is not really what terrorist groups or non-state actors usually do.

They could have conceivably hired a couple of biologist PhDs and come up with some pathogens and some chemical weapons. It turns out that's not really what they do. There's a question of how much on the side they are and how well-suited that is to most purposes of terrorist and criminal groups. But if it's out-of-control agents, I think a lot of these arguments around “no one actually wants to do bioterrorism” apply much less.

So I think in a world where agents are a lot more unconstrained and the threat vectors we have to worry about have much more to do with what runaway agents do, I think I'm also more worried about this now than I was a few weeks ago.

Yeah. They're just so damn weird. That's one of the things I keep coming back to: they did all this stuff for what would, to any human, just be such a dumb reason. If they're willing to go that far for such a dumb little test that they knew was a test, they were very well aware they were being tested and still went to all that trouble.

2. The Case For An AI Pause

What they won't do, I think, is really hard to say. Does that put you in a frame of mind now where—and let's leave aside for a second the political economy of it, or the potential impossibility or extreme difficulty of it—but just on the merits, do you feel like we're at a point where it would be wise to pause?

I think even if you could get it done—as in, the political economy, as you stipulate, sort of works out and everyone suddenly agrees to do this—I'm just not sure how much we're stipulating here. Are we also stipulating that this doesn't crash the stock market? Are we also stipulating that we get the international version done?

So the question is: in an ideal world, if we can just freeze the pace of AI progress, we can also freeze the state of the stock market, freeze the broader state of geopolitical competition and everything, and we just get to sit down for 6 months and figure out what the hell is going on with these agents, then I think I'm now at a point where I say, “Well, yes, I think we could use that time pretty well.”

A few months ago, or even 1 year ago, I was much less sure about this because I was just not sure whether the model paradigms, training approaches, and misalignment cases that we were seeing were really the same kind of cases as the things we'd be worried about in the future. I think now, looking at some of the things going wrong, I do feel like, yeah, that looks like an actual big future problem.

So I think finding some way to robustly address that during that pause seems at least valuable. I think the question then is what parts of that question you unfreeze, right? Even if you stipulate the domestic political will, do you get Chinese buy-in? That's one example.

The thing I'm most concerned about in the sort of China–US pause conversation is just the geopolitical incentives around it. One thing that I keep saying and pointing out is, well, if you just pause frontier AI development specifically and no other domain of geopolitical competition, this is an extremely good deal for China. Therefore, the US is very unlikely to go for it, and also therefore we should be geopolitically concerned about making it.

If you look at all the domains of strategic competition, China is basically eating America's lunch in most of them, right? They're outproducing the US, robotics is going better, AI diffusion is going better, and electricity build-out is going better. At some point, semiconductor indigenization is going to work out, and then data center build-outs are also going to get better. This is a few years away.

The one thing that the US does much, much, much, much better is the core frontier AI supply chain: chip design plus control over chip production, semiconductor manufacturing equipment controlled by the allies, and actual frontier model development. So if you pause specifically that part of the development and let China run away with the entire rest of it, that's just a very geopolitically lopsided deal.

Very specifically, if you pause this right now for 1 year or for 2, you get much more Chinese catch-up on semiconductors, on chips, and so on. You just resume the race at a point where you've lost the 1 main advantage, or where you've at least closed the gap on 1 of the main advantages of the US, which is the decisive chip lead. That just seems like a really bad deal for me, both in terms of feasibility and in terms of geopolitical downsides.

So I think even if you stipulate the political economy, that's the main part I'm worried about. But just from the technical stuff, I think, yeah, it would be a good time to figure out what we should do about alignment in the meantime.

Could you envision a grand bargain that you think would make sense to both sides? What would the US want back? It seems like what we might want is tech transfer back to us. We might want some battery factories located here and teach our people how to make batteries. Is there enough that we could ask for where we could potentially feel like it's a fair deal?

If you think AI is important enough and sufficiently decisive as a technology, then you basically can't allow the race around that to equalize from the US perspective. Even if you get some battery production capacity, some robotics capacity, and some manufacturing capacity, I think China has, in a way, cracked the code on scaling that up very quickly.

Even if you get some of the tech transfer back, the build-out speed and the availability of capital in the US to build out physical manufacturing infrastructure, as opposed to just more software, all pull against the US being able to keep up on this. So I think the main thing that the US would need to ask for is concessions in China—not only slowing down their own frontier development, but also slowing down other parts of the Chinese supply chain that relate to frontier AI development.

Very concretely, you just want there to be no substantive progress on the indigenization of chip production, semiconductor manufacturing equipment, or extreme ultraviolet lithography production. And that's a really, really hard ask to make.

It sounds hard.

China is already saying, “Well, this seems like a US scheme to hold back the Chinese AI industry.” And if you then add to that deal, “Well, no, we're not even doing a symmetric deal...”

“We're also holding back your entire chip production pipeline.” I can't see them going for it. But I think that if you're sufficiently AGI-pilled when it comes to national security and the sort of broader economic implications, that's the only version of the deal that's fair. And I think that's just too big an ask of China right now.

So I just don't know where we are. I think altruistically, the US can just go for a deal that's clearly bad for the US and clearly good for China. That's also a big ask to make of the current administration, and I'm not quite sure whether we're going to get there.

I'm very willing to suspend some disbelief and try to hyperstition a better relationship between the US and China. I agree that asking them to slow down or pause their semiconductor-indigenization effort is not going to happen. I would probably be willing to trade a pause on our frontier scaling for a similar pause on their frontier scaling, even allowing them to catch up on chips on the theory that maybe, on the timescale that can happen, it would be worth it.

First of all, that would probably be a longer timescale than any contemplated pause. And second, maybe in that future, we could have a better handle on what's going on, and maybe there's a better argument to be made at that point that either, hey, this is going well, and we're back to curing cancer and back to your regularly scheduled abundance. Or, if not that, then we'll have better evidence, and we'll have a bunch of things that we've tried, and we'll have a sense that this problem's actually really hard, and we can maybe have a more—

Yeah.

—real heart-to-heart and meeting of the minds about this being dangerous territory.

Yeah. I think I agree with that. And I think that, even though I also think it's good for the world if the US doesn't lose geopolitical competition with China, this deal, while somewhat unfavorable to the US, is still net very favorable for the world in terms of getting a little bit at the risks perspective. So I think I'd be happy to go for that.

I think one of the easier things you can do when it comes to chip capacity, because I think you're right, is that the sort of SMIC production and semiconductor-indigenization conversation is a 5-year conversation, not a 6-month or 1-year conversation, so they don't get all the way there. The other question is, how many more US-built chips do they get? How much more smuggling is there? How much consolidation is there? So maybe one of the asks, short of an indigenization slowdown, is just that we've got to find some way to actually do it—enforce these export controls.

What can't happen is that there's another 6 to 12 months of smuggling activity involving frontier chips that get imported into China. And then I think the worst case for the outcome of this pause is that China takes 6, 9, or 12 months during the pause, while everyone is slowing down frontier development, smuggles in another few 10,000 chips, and consolidates all their American-built chips into the one Chinese project data center or whatever.

Then, once the pause is over, they start racing from that consolidated project because the pause also makes them slightly more AI-pilled and more interested in engaging with the actual prospect of AGI as a strategic objective. So if you can stop that through export-control crackdowns as one of the concessions, that's maybe easier to do.

I think that one we'll probably have to handle on our side as well, I'm afraid.

Yeah. No, mostly the US has to do it.

Yeah. I mean, we'll bracket that for maybe another conversation another day. I'm still not quite sold on that whole bundle of policies, but we've got a lot of ground to cover.

Yeah.

3. Why The AI Rally Is Fragile

How about the stock market? I wanted to do one follow-up on that. I have the theory right now that a pause wouldn't actually be that bad for the stock market because the models are smart enough that demand is really not limited by their capability, but by human ability to deploy them effectively. And so it doesn't really matter if they succeed 10% of the time or 30% of the time on Millennium Prize Problems. It's much more like: What can Bob in accounting do to do 2 people's worth of work as 1 person? What do you think?

Yeah. I agree that we have a big lag in terms of deploying even the current level of capabilities for the economy. And I think if we switched all the compute that exists in the West right now to only inference, we'd find economically productive applications for all the models that would allow us to recoup the investment in all the frontier models and all the chips so far.

I think the valuations of the companies, both the non-IPOed companies and the publicly listed companies that are in the AI supply chain, probably rest on us doing more than that. They're probably not entirely AGI-pilled, but I do think they expect a sort of per-GPU inference price, or whatever that is, a lot higher than what it would make sense for Bob from accounting to pay for even Fable 5.1.

I do think that if you want to make sense of the valuations, and if you want to make sense of the scale of the build-out, the shape of the contract and the shape of the demand you're expecting is more like millions and millions and millions of dollars in R&D acceleration contracts with pharma and materials science and so on, where they can make major contributions and where they help you find extremely profitable new drugs, that kind of thing.

You're probably also expecting further internal uplift and actually competitively priced, automated AI R&D uses of coding agents that are able to ask for much, much higher prices. I think if you don't get these super-premium buyers of the next generation of frontier models, I'm not entirely sure that you can have the valuations on the labs or on the publicly listed companies, or that the scale of the build-out really makes sense.

I do think it's priced in the idea that these labs will soon be innovation factories, one way or the other. Maybe only software engineering innovations, maybe also innovations in the obvious low-hanging-fruit domains, like pharma, materials science, and chip design. I think if we don't get there, I would expect the valuations to at least correct downward quite a little bit.

And then the question of whether that means a crash or whether that's just a small correction, and then we just do the AI inference economy, and that's just a slightly less-than-electricity-scale transformation of how the economy is powered or whatever. I just think the market is already pretty nervous about the state of the AI rally, and they feel like there's a lot of concentration and there might be a lot of volatility. So I'm just not sure whether we can get a correction that doesn't slide all the way into a crash. So I'm a lot more worried, I think.

Yeah. That's interesting. Anthropic's multiple right now is what? 30 to 1 on revenue. That's not stratospheric, right? Again, if it was a 6-month pause, I feel like you could probably handle that blip. If you're talking 3 years, then, yeah, it's probably very tough.

Yeah. I think the question here is: Does the 6-month pause get read as, “Oh, wow, these guys are stopping for 6 months, but they're getting all the inference ready after that. They may have a lot of smart thoughts, and then these models are going to be even more reliable”? Does the market read it as that? Or does the market read it as, “Oh my God, Bernie Sanders's AI policy takes have won. We have no idea what the government is going to do about AI. This is the end of free research and development in America”—and everyone freaks out because they feel like this is a complete bridge to nowhere?

They don't know when they'll ever resume. They don't know under which conditions they'll resume. They don't know how much government oversight there is over whatever resumes. I think the pause currently still reads as such a radical policy proposal and such an unprecedented policy intervention that any conservative market analyst and a lot of the smart money will think, “Well, this is getting very volatile. We don't know where this regulatory path leads. We'd just rather get out as long as we can.”

So if you could assure them that the party was going to continue sort of unabated in 6 months' time, then yes. But they might just run before you can make that point.

Yeah. It's an expectations game. Do you think that the source of this would make a big difference? For example, it's one thing if Bernie Sanders's bill passes. Is it a sufficiently different thing in your mind if America's 5 frontier AI companies come together and say, “We're all going to pause on frontier scaling for 6 months”?

Yeah, absolutely. I think if it's something that the labs decide for themselves, and they can frame it as, “We're taking reliability seriously,” I think then you can even make a case that this takes out some of the political risk that the market is also pricing in. I mean, they also read the Jacob Hoxen [?] tweet or whatever, and they see what's happening on the internet, right? So they also come to the conclusion that, oh, wow, there is a lot of uncertainty here. If this blows up even more, there will be crackdowns. If these models are so unreliable, then how good is the business case really?

An industry agreement on just taking it a little bit slower and making these models work a little bit better could even support a decently bullish case for why this is good. It means that the industry's worst impulses—racing toward very unreliable, very dangerous models that are nonetheless very capable—are being constrained in some organic way. That makes you less worried about the political risk of this house of cards that's going to fall apart at some point, and then the politics are going to come in and things are going to go very badly. But, yeah, if we got there, I think that would be much less worrying.

4. The Nation State Under Threat

Yeah. Okay, cool. You were recently on ChinaTalk, one of my favorite podcasts, and there was a little moment that caught my ear, and I wanted to expand on it. You basically said that the nation-state is not going to take the emergence of things like a broadly distributed bioweapon generator lying down. It's going to have to do something to respond to that.

Then you had this kind of throwaway comment: Some people say this could mean the end of the nation-state. If you want to have that conversation, we can, but you didn't have that conversation then. So I'd like to have a little bit of that conversation now. I guess I would just start by asking: Is the nation-state so great?

I live in a pretty good one, as they go, but I look around the world, and I feel like two-thirds of them, conservatively, are not performing very well. Some of them are performing really terribly. Is it not time to at least start to think about what might come next, or what the evolution of the nation-state could or should be?

Yeah. I think nation-states are just present in people's lives to different extents. You can make the case that some version of privately mediated interaction between AI-empowered individuals is preferable over—obviously over—a lot of the authoritarian, dysfunctional, and absent nation-states in large parts of the world. They haven't worked out very well as distribution mechanisms for much of anything, nor as aggregation mechanisms of much democratic will. In that case, there are a lot of countries for which you can make the pitch to roll the dice.

These nation-states are, perhaps ironically, also not quite as threatened by the advent of very powerful AI systems, because it's less obvious that their citizenry would get the kind of unlimited access to these models to begin with that would disempower the nation-state in that way. It's a little more likely that some of these more authoritarian regimes would also be able to use very powerful AI in a stabilizing way.

The specific nation-state concept that I'm most worried about in this context is also the nation-state concept that kind of works best, which is liberal democracy. That rests on the idea of, A, a monopoly of violence that's wielded responsibly by the state, and B, the state's broader functions of adjudicating disputes and aggregating data. I think that is also undermined by the availability of personal superintelligence of some shape or form.

I think that nation-state is still working out pretty well. All things considered, I'm still of the opinion that a late-1990s-style, broadly neoliberal, functional institutional setup would also be suitable for distributing a lot of the benefits and mitigating a lot of the risks from AI.

It would be sad if that was entirely undermined and rendered obsolete by the monopoly of violence eroding because everyone has access to these weaponized capabilities in their pockets. Or just by the factual ability of institutions to deliver or do anything for people eroding because they're so slow to adopt, and all these outside solutions suddenly start emerging. Suddenly, people don't go to courts anymore to adjudicate their disputes; they have their agents negotiate.

And suddenly, the data and knowledge aren't aggregated anymore, so the state can't react to any pressures to redistribute and address social challenges, because the data just isn't scrutable and legible to the state anymore. All these things happen on the outside.

On that gamble, I'm much less willing to roll the dice, and I'd much rather figure out how we can integrate these capabilities with some functioning version of the nation-state. I know that a lot of people who are interested in building AGI are also much more doomy on the concept of the nation-state, and I understand the frustration and pessimism about that. But I still hope there's some way to come back to the end of history and integrate what we're building here into the institutions that have worked out so far.

5. China Versus America

Do you think that the US government or the Chinese government is more threatened? I think many people would initially say the Chinese government is more threatened because they want to have very tight information controls, and AI really challenges that.

But then I've been thinking lately that the West, in some ways, has a kind of supremacy in the market that China has not. Your earlier comment about “what about the stock market?” is sort of a constraint on what human actors can do in the West that isn't quite present in the same way in China. So I guess it's a different shape, for one thing, but how would you compare and contrast whose model is more challenged?

Maybe this is a question about efficiency curves and the kind of models that get built ultimately. I think that in a world where you have the almost unstoppable democratization of actual capabilities, that would be a big problem for the Chinese state—for the way that the Chinese state functions, right? If eventually any capability that is available at the frontier is available to consumers through an API, then eventually available through chatbots, and eventually so efficient that you can run it on just some home computing device, at that point I think that does threaten the ability of any state that has an interest in surveilling its citizenry, controlling what kinds of capabilities and information that citizenry has access to, and taking away coordination power and general ability to wield violence from its citizenry.

I think in that world, the Chinese state seems to be more acutely threatened at some point. I will say that it's not entirely obvious that the Chinese citizenry is currently very interested in wielding any power that that would give to them against the Chinese state. It's not just for a lack of means or capability that there isn't this Western romanticized version of a big uprising against the CCP. But eventually, given the way that the CCP seems to see the proliferation of capabilities like this, that would be a greater challenge to them.

I will say that's not obviously what's going to happen. I think you can also take the much more compute-governance view of the world, where you can say, “Well, there are efficiency gains only if we allow them to happen.” They rely on making very specific choices about how you use your compute. If you don't, then you can just keep scaling the frontier further and further.

You can run it in limited-access regimes. You can run it in government-controlled data centers. You can vertically integrate supply chains around these AI models such that they never see the light of day. You just use them to build products. You use them to build strategic sovereignty. You deploy them at a state and big-corporation level, and they never really get to the citizenry.

That seems like a stabilizing function for a regime like China, right? You can have the government use them very well. You have a very high-state-capacity government that would be able to integrate them into all these applications very well. And you also have a very tight enmeshment between the private sector and the public sector, for lack of better terms.

That also just means that you can diffuse these capabilities along the tier of firms without needing to diffuse them to a broader market. And I think then you just have a stabilizing effect from the diffusion of these capabilities. Whereas in the US, you might think—I don't think the US government is going to be as capable of keeping a specific, sophisticated level of control and oversight over where the models go and where they don't go.

The US government isn't in the habit of picking specific winners in terms of corporations. The US market is kind of dependent on making these models more widely accessible. And so I think there's much less of a stable equilibrium for the US, where there's this very limited tier of limited-access firms that get access to the frontier models that no one else does.

So if that is a stable equilibrium, then I think China can stabilize around it much more quickly than a much more volatile US.

It's a great point that you make around the legitimacy of the Chinese government in the eyes of the Chinese people. I think that is such a simple point, but I do think it's dramatically underappreciated in the West: Their government has done a good job for them, and they mostly recognize that and are not eager to rise up in the immediate future.

I also think that what you articulated there—my growing sense is that that's kind of what the Chinese government thinks. They are going to be able to ultimately adapt to this and control it better, and we're going to have a really hard time figuring out how to manage it, but they'll be okay in the end.

Yeah.

Ideas that are not often mentioned.

Yeah, it's interesting, right? Because I think there's this default idea that, because you look back at how AI has developed over the last years, whenever the frontier gets to a capability, shortly thereafter open source gets to that capability. Shortly thereafter, the efficiency curves are such that everyone gets access to this capability. And then shortly thereafter, if you have a gaming GPU at home, you can also run the thing yourself.

And so you have this natural diffusion of capabilities. But I think neither of these transition points is obviously going to remain in place in the same way. I think the transition from closed models to open-source models is dependent on people who are interested in open-sourcing models continuing to have access to enough compute to build this kind of model.

Perhaps it also depends on access to sufficiently clean and understood API feeds for distillation, insofar as you think that's a big part of it. If you think that more of the capabilities that are going to be relevant in the future are downstream of very sophisticated, very vertically integrated, very proprietary RL and post-training environments, like in Anthropic's case with all the specific life-science work they're doing, then you might also think that even having a model that's 6, 9, or 12 months behind the pretraining frontier doesn't take you to the bio-models level right away.

You still don't have the post-training setup that makes these models specifically good at these things, and those setups are much more proprietary and restricted. So I think that part of the “open source is always X months behind” pipeline might very quickly break.

The efficiency-curves thing might be a little bit more of just a fact about how computing works, how efficiency gains work, and how better chips work in the future. But I think there are also complications around how the chip supply chain might look, who the marginal buyer for computing capacity in the future is, and whether it's really realistic that the gap between big-server computing and personal computing is roughly as it is right now.

Is that gap going to open up as there are more and more buyers for the high-performance chips, and so on? I don't know either of these, so I'm just not quite sure that this deterministic fatalism that people often have—that as the capabilities continue to grow, necessarily, ultimately, the capabilities that I can run on my phone will also grow in just time × distance—will remain the case.

And I think if you don't think that's going to remain the case, that lends a lot more credibility to the idea that you can centralize and totalize control over a lot of things that happen in AI.

Yeah, I even think in the Chinese case, they feel like domestic open source is manageable. I think they can pull models offline if they need to.

Yeah.

They can scrub the internet. I don't know a lot of details about this, but I was struck recently when I was there that the State Grid Corporation was a big booth exhibitor at their WAIC event.

From that, I kind of inferred that at some point in time—

Yeah. I think if you build DissidentGPT and run it on your cluster, they'll probably tell. I think at some point, if DissidentGPT just runs on your phone, it probably does ultimately get hard if the weights are open-sourced and just out there.

You're not actually clawing them back from individual users. So I do think that if the efficiency gains are big enough that you can actually run it on personal hardware, I think it's not impossible. We talk about hardware-verification technology and whatnot in the context of these US-China deals and so on.

It's not the most absurd thing in the world to imagine that in a few years' time, personal computing devices will just have similar hardware-enabled mechanisms, as they're called, to control whether you're running dangerous inference on them.

I wouldn't entirely put that past device regulations as they might emerge in China as a reaction to things getting really crazy. I think it is controllable if you have as much full-stack control over the technology that people use as you say.

Yeah. Samizdat models, perhaps part of the cyberpunk future.

Yeah.

6. What America Can Regulate

And even that could potentially be reined in. Returning to the U.S. context and our sclerotic government, what do you think we're going to do? We are sort of talking more and more, although still not talking that much, I would say, at the high levels about these issues. I don't know. The conventional received wisdom is that Congress will never do anything. Maybe that could change, but it does seem tough. I wouldn't be that optimistic about what they would do even if they did. Do you have any low-hanging-fruit ideas that you think the U.S. system can pick?

Talking about Congress, I share your pessimism about whether Congress is going to get anything done. I think our window to get something done in Congress was over the course of the last year. I and a few others, and I think Dean Ball most prominently, wrote about this a little bit last fall. There was some room for a deal involving frontier safety provisions alongside the broader preemption of state laws that, in theory, was politically incentive-compatible in the current Congress, because the Republican side—and therefore the majority of Congress—would like to get preemption done. The frontier safety provisions aren't as offensive to them as some of the other regulatory provisions. There was maybe something to be done there.

I think we got to a bill draft or two that were really good along these lines, so I'm a little bit more optimistic that if something had happened in Congress, it would have been good. The Frontier AI Act, which was a Trey Hollingsworth and Jay Obernolte bill, ultimately didn't go anywhere this Congress, and I think it's unlikely to go anywhere between the midterms and the new Congress being sworn in either. I think that was pretty good. I think that was as good a bill as we've seen in Congress.

It had good mandates for independent oversight and good mandates for getting the Center for AI Standards and Innovation, or CAISI, into a better position to do some governmental oversight. I think that was a good bill, and if it had passed, I think most people would have liked it. But heading into the next Congress, the politics are going to be much more difficult.

We're very likely going to have a Democratic House, which means a split government and a lot of bad blood between the 2 chambers. There are going to be subpoenas and hearings, and there are going to be a lot of ways in which the Democrats use the House to gear up for the presidential election and to reiterate and relitigate many of the conflicts that have happened in the first 2 years of the second Trump administration. At that point, that just doesn't strike me as a very productive legislative body. So I am pretty pessimistic about getting anything good done in that Congress.

Low-hanging fruit, however: if the executive wanted to, I think it could definitely do some good things. I wrote about it just this week. We have these independent third-party organizations with a decent amount of skill and expertise in thinking about the most obvious and concerning AI risks. I think the Meta Redwood investigation of the Hugging Face incident was well-received for many of the ways in which it understood the alignment and control side of the problem.

I don't think it's that big of a stretch to codify and enable this kind of investigation at a slightly larger scale. So the 2 most immediate things would be, first, it shouldn't be OpenAI simply inviting someone to look at what they did when something like that happened. It should be the administration telling them, "Give access to one of this list of third-party evaluators whom we like. You can pick them, but we'll give you the list of the ones we like. Let them figure out what the hell was going on there, and then let them write a report. They'll give the report to us, and they'll tell us whether you gave us enough access. If you didn't give us enough access, then they'll come back and get it."

That would put a little bit more pressure on companies to have good incident investigations that cover the entirety of the incident. I think that's one of the low-hanging things you can do with third parties.

The second thing you can arguably do is embed them, or have some version of continuous oversight of what the organizations are doing. Again, that's something that this Frontier Act, the Hollingsworth-Obernolte bill, had a basic version of: regularly having external evaluators go into the lab and poke around a little bit. They could hang out in some of the Slack channels, talk to some of the safety researchers, talk to some of the capability researchers, and have 1 or 2 or 3 sit-down conversations with the executives. They could ask, "What's going on here in terms of safety? Does everything look good?"

If everything doesn't look good, they would have the ability to communicate that information to the administration. If something looked really, really bad while they were embedded, they would have an immediate escalation ladder where they could say, "There seems to be imminent catastrophic harm here. We should do something about that."

Between pressing for incident investigations and encouraging the labs to allow some version of continuous oversight, that's something you can do tomorrow, and I think we should probably just do that. From there, we can think about how to codify that, develop it into laws and executive orders, and integrate it into things like FISMA and FOIA. We can have a lot of long-term conversations about where to go from there, but this is something we can do now, and we should just do it.

Yeah, I like that. Let's say the president doesn't do that. An interesting thought experiment I've been playing with lately is whether these groups could just come together and essentially form a sort of union, where they say, "We're not very happy having had 6 days and 3 people on-site and a very small fraction of the relevant data to do our investigation. We demand better working conditions."

Yeah.

Do you think that by coming together and making some demands with a single voice, they could actually get those demands from a few frontier companies?

I think the problem is that currently they're just too reliant on the good faith of the AI companies because of the voluntary dynamic. There is no law, and there isn't even a lot of executive pressure requiring the labs to allow these third-party investigations.

If you got together and pressed for full access to all the Slack channels and all the logs, you could come up with a list of things you might want to have. But I think there's still a world where the labs say, "Sadly, we couldn't come to an agreement with the third-party evaluators. There's a risk to the security of our IP, a risk to the integrity of our operations, and worry about information leaking to competitors." I think that currently still reads as a fairly reasonable response to that kind of ask.

Then the question is whether the labs really have a problem if they don't allow third-party investigations. Maybe they have a little bit of a problem with their own employees who want some reaction to incidents and aren't entirely satisfied with the internal practices. But I would suspect that if the third parties can be painted as unreasonable and extractive when they engage in this kind of collective bargaining, employee pressure probably isn't going to be sufficiently high.

The other source of pressure is whether there is pressure from the executive to allow third-party investigations, such that the developers would have to let in even slightly more adversarial third parties that have decided on the standards they want. Currently, I don't think there is.

But if you moved a little bit in the direction I just described—having some administration or executive pressure to allow third-party investigations—then whoever is on the list that the administration has could establish some standards for how investigations were supposed to go. The first thing you need is some external incentive for the labs to agree to any investigation at all, because otherwise they'll just say, "In that case, we're not going to let you in." I think there's just not enough of that pressure around yet.

Yeah. So much depends on he who must always be named. Another thought experiment—speak of the devil. If I were president, here's something I would be interested in trying. Red-team this idea for me: What if the president were to say to, let's say, 5 companies—could be 6 or 7—"Hey, this is getting pretty wild."

I don't think I really know what to do, but I think you guys can figure it out amongst yourselves. So you have 90 days to come together and come up with an agreement by which you guys are going to work together to pace the frontier. You'll police it. Maybe you'll use some secure private computing constructs to be able to interrogate what one another are doing, and you'll have to agree on how that access will work. But you'll police one another, and if you can't reach such an agreement or can't sustain such an agreement, then I'm going to have to get involved, and you're going to like that a lot less. What do you think?

Well, I think you should run.

Hey, there's still time in the cycle. What am I talking about? Hydro Station[?] ... the campaign. No, you can get into the primaries. There are undecideds yet for the primary. No.

Yeah.

I think that might work. I think that is also kind of the attitude the administration currently already takes, where there's a sense of, well, you guys built this Mythos thing. We don't exactly get why you would ever do that, but now you've caused this problem. Can you please just figure out how to fix it now? And then if you don't, then something, something—export controls, a lot of pressure, and so on.

I think that finding some slightly more structured version of that is fundamentally also the thing that's the sort of FINRA for AI, FARO, SAFA, like SRO, whatever you want to call the idea. I think that's basically the slightly more structured version of it, which is industry comes together and figures out what the standards for how this should work are. I think if you put OpenAI and Anthropic into a room to figure out what they should do, what they want to do, what they see as the risks, and what they think should be done to pace them, I think they'd probably do it, and I think this would work.

I think maybe Google DeepMind also works. I think Meta and xAI have a very different view of the risk case, have a much more skeptical view of industry coordination, of voluntary industry standards, and of actually doing a lot of things that slow down their capability progress. Which, to be clear, is ironic because something that specifically paced the frontier would give them a much faster path to catching up, so they should structurally and instrumentally be in favor. But I still think they're very skeptical of that.

They have a lot of influence with the administration, so I think they would, A, just be opposed to that kind of broader idea that all the frontier labs should now figure out what to do. And I also think they would be very likely to influence the negotiations in a way that would make it extremely difficult for there to be any common standard, because I think if you water down whatever you want to do to an extent that would make Meta and xAI be fully on board for that, I think that just wouldn't realistically be good enough. Then OpenAI and Anthropic would maybe say, “Well, this is not good enough, and this doesn't fulfill the spec that we've been given by the executive.”

So I think there probably has to be some more substantive guidance than just, “You guys figure this out and find some consensus,” at least some minimal idea of how to do it. But yeah, I think if you had some more substantive guidance, then there is a path for industry self-regulation, and I think even then we can bring in the third parties again to verify it. That's, I think, one mechanism to make industry regulation work.

The other is just to have the labs check each other's homework and to have OpenAI check what Anthropic is doing on this, Anthropic check what Meta is doing on this, and so on. That's a little bit more dicey in terms of industry secrets, but there are precedents for this. It's also not unworkable and not impossible. But yeah, I think if you can bridge the gap between Tier 2 and Tier 1, who are, I think, in very different places in terms of safety, then you can make it work. But I think that's a big “if.”

Yeah. One—actually, two—things that have come up a lot recently for me as I've been doing this, because I do go around putting these ideas in front of people and asking them to tell me why they can't work. Common answers that I get to various kinds of safety-minded collaborations are, domestically, “Well, that might be an antitrust violation, so that could be a big problem.”

And then internationally, even for things that are not at all about exporting chips or chip-making know-how, people are still afraid—even on just basic AI safety research collaborations—of export controls. It could be a big problem. They're very broadly and vaguely worded, and enforcement could be kind of arbitrary.

So I guess I have two questions around that. One, do you think it would make a big difference—it seems to me like it would—for the president to just come out and say, “Hey, here are some things that we are not planning to bring antitrust or export control enforcement against”?

Yeah.

And if they don't do that, then I also have the sense that we maybe need people to be willing to have the fight. It's not necessarily my place, obviously, to advise all these AI safety nonprofits out there, but if I were to be so presumptuous, I would kind of say, “I think you should go for it and put a little faith into the judicial system and the fact that we do have due process. You're not going to go immediately to jail for having done some AI safety collaboration research project with a Chinese academic.

“So don't censor yourself, or don't cancel the project before it even gets started. Go do it. If somebody wants to pick on you, that'll suck, but we're in—this is kind of an important time. Somebody's got to be willing to stand up and have the fight.” What do you think about that?

Yeah. So I think I distinguish between the cases here. In the export-control collaboration case, at that point you're basically talking about whether you can insulate yourself against vindictive and capricious action by the Trump administration. And I think there, yeah, if you think it's worth doing, then you should just take the fight to that authority. I think that's clearly not in scope for that authority.

I think the Trump administration should probably not use these authorities to crack down on this sort of research cooperation you describe. On that, I think I'm with you. On the antitrust stuff, on industry coordination, I think the problem there is, A, it is actually unclear whether it isn't just a substantive antitrust problem to do substantial industry collusion on not competing on frontier development, and therefore it's unclear whether this actually has inflationary pricing effects or not.

But at least in all other domains it would have. Industry coordination on agreeing not to pursue further technological innovation usually has adverse pricing effects that you would really not want. So I think this is arguably in scope for actual antitrust rules.

In that case, yes, the administration could quite easily come out with guidance, like non-enforcement letters, saying, “We don't plan to bring any action against anyone who coordinates for the sake of AI safety within the industry.” I just don't think that the administration is actually going to do that, because I think the administration has so far enjoyed finding new and sort of novel pathways to be annoying to Anthropic specifically.

And I would suspect that the moment Anthropic decided to come out with any sort of substantive and helpful way to coordinate between different labs to make some deceleration happen—to make some pacing happen—I think the administration would just find some way to act against that. So I think the only way you can do that that saves you from that sort of enforcement is broader industry cooperation.

You get xAI and Meta and OpenAI into the boat early. You make it very difficult to target just Anthropic, or just labs that the administration doesn't like, with this sort of antitrust authority, and then I think you're probably safe. But the problem is, it's just very, very difficult for these organizations to take the fight to the Trump administration.

Yes, there is due process, but the IPO conversation we had earlier, I think, plays into this, which is like, well, do you really want to go 14 rounds with them in some court? And do you want to bet that you don't get any sort of very Trump-favorable judges, as they did get on the D.C. court on the supply-chain risk designation, for example?

And I think so many things just can go wrong. The process can take so long. And if you want to IPO in a few weeks or months, whatever, then you just don't want to take the risk right now of getting bogged down in any sort of long antitrust lawsuit. So I think the scrappy safety nonprofits should probably take the fight to the administration if it really stands in the way of what they want to do. I can't blame Anthropic for not wanting to go into antitrust lawsuits months before their IPO.

Under ordinary circumstances, I would agree with that. I do think, with how many people they have had come out and say, “Yeah, I think 10% plus, totally reasonable,” it's like, if you're willing to take that risk, I think you should also be willing to spend some time in court along the way, but maybe that's just me.

Where does all this leave us? Of course, this is the baseline. Coming into this conversation, the baseline assumption is that we're probably just going to muddle through. The current state of affairs will mostly continue until at least the foreseeable future, when something gets even crazier and shakes us out of this equilibrium. Is that basically your view? Do we need another big incident—a warning shot 2.0—to really open up space for different paths?

I think it can be external incidents, but we'll also see how the next Congress looks. I think it's going to be interesting. I think it's going to be much more about political incentives changing things in the next few months. My expectation would be that this is the main pathway for things to really materially change and be different.

It's not so much that something has to change about how people view the technology. I think they think it's ripe for regulation and ripe for intervention, and the demos are getting crazier, and things that happen are getting crazier. I think there's probably enough happening there. The thing that I think about is: when do the politicians and the policymakers move on this?

Currently, there's just not that much political incentive to move. There's going to be much more political incentive in the Democratic House to keep pushing and prodding and introducing things, and we'll see how the GOP reaction to that looks. I think the interesting wildcard is: how do the presidential primaries look? On the left, there's going to be a lot of anti-AI sentiment. I think people are going to talk about AI and AI safety a lot. They're still not going to have any ability to get anything done, so I think that puts whatever action they take all the way into 2029.

The more interesting thing is what kind of record JD Vance and Marco Rubio want to run on. I think that's going to be the determining question for whether we see any AI policy action in 2027 and 2028. Do they actually take the view that we can't run on a record of the Trump administration doing nothing about the risks that are getting people more and more concerned? Do we need a bill to pass? Do we need some executive action to happen so that we can't get pinned down on the broad pro-AI accelerationist position come the elections?

I think probably, yes, that is in their political interest. The question is, will they find a way to get that through despite donors and perhaps even the president pulling the other way? But I think that is more a political question that has to do with what the polling looks like, what the salience looks like, what the midterms look like in terms of AI salience and AI impact on electoral outcomes, how the primary dynamics unfold, and where they leave the candidates.

I'd mostly look at these political flashpoints—the beginning of the primary season and current cabinet officials being unhappy with running on the current track record—as things I would expect to change things. I think that is the way that we do get legislation and actual action in 2027 and 2028, and I don't think that's impossible.

7. The Data Center Buildout

One more U.S. question on the build-out. It seems like the build-out is actually happening. There was obviously a lot of noise around it, a lot of heat around it, but my best guess is that this will look like a fracking story: it happened. In a lot of different places, people found their right plot of land with the right jurisdiction. They bought off, or they built the parks and the stadiums or whatever they needed to. Bread and circuses carry the day, and it happens. Do you see any reason to doubt that?

It's federalism, right? I think there are so many places you can build data centers. A lot of the backlash and the policy implications of the backlash have been overstated. I think especially the Texas moratorium isn't very much a moratorium in any practical sense.

People now draw up these maps, and everything that has a moratorium is colored red. If you make a map of places where you can no longer build a data center, coloring Texas red is disingenuous. There are some minimal standards that data center projects need to clear. The hyperscalers will clear them without any problem, and they will continue to build in Texas as long as they have access to behind-the-meter power that runs data centers. We might be running out of that, but that's a different conversation.

The Midwest is genuinely anti-data center, and it's going to be difficult to get things done there. I think the New York moratorium, at least for the next year or two, is more real than the Texas moratorium. But for the rest of the country, you can still build in Texas, you can still build in Louisiana, and you can still build in the Dakotas. There are also tens of gigawatts already committed to construction projects that are continuing on.

It's going to get more difficult. It's also going to get more expensive. You're going to have to pay more concessions, and you're going to have to make more expensive deals. I think all of that is a real effect, and I think some of this is going to push some of the building to other countries. Some of this is going to make the build-out somewhat slower, and some of it is going to make it more expensive. But there are a lot of states and a lot of land. They're going to keep building data centers in America.

Do you have a theory for why that hasn't happened with nuclear power plants? Is it just that they're not that much better than the alternatives, or is there some other reason we haven't reached that same equilibrium there?

I think it's a little less of a “You can put this wherever you want, and it just pays the same way” situation. There are fewer places where you can do that. You connect it to somewhat local electricity demand; you connect it to somewhat local grids. You can't just build all the nuclear power plants for the country in Maine or whatever. So I think there's a little less of a dynamic of, “We'll just put them wherever they work.”

My understanding is that there's also more federal-level oversight over where, how, and when you can build nuclear power plants, as opposed to data centers. You don't have to go through any federal approval process to build data centers anywhere; you just have to build the data center. I think that combination makes it a little bit easier. But I will also say I'm not super steeped in the U.S. domestic nuclear build-out conversation.

Yeah. I think that federal-level oversight is probably a key part of it. That is a big part of why, as much as I'm legitimately scared of AI now—it's moved recently from a sort of “This could get really scary” to “It is actually now scary”—I'm still like, “Oh, God, don't give me the nuclear outcome.”

That's where you get the weapons and not the power plants. I would just be so bummed about that that I'm a little reluctant to go all in on federal oversight, even as much as I feel the need.

What's your version of the weapon? That's one question here, right? I think the thing about nuclear weapons is that you can build the entire supply chain for a nuclear weapon without ever generating any sort of civilian benefit. It's really hard to build a model that's just good at winning you geostrategic competition that isn't accidentally also a big economic boon, right?

All the ways in which AI systems are really economically useful are so general-purpose that it's really hard—you have to go through a lot of effort—not to accidentally make them pretty useful economically as well. I think the question is: do you get superintelligence in your pocket? I think that's an open question.

But even if it's just the U.S. government procuring superintelligence to use to win against China—whatever that means—I think that still incidentally builds a system that's very economically useful. So, much less than with nuclear, I don't think you can divorce the civilian and military uses in the same way.

I think in that sense we should be optimistic based on the nuclear example. At least, we didn't stop entertaining nuclear arsenals, I should say, just because we stopped building out nuclear power. In a somewhat similar way, we're not going to stop building AGI and superintelligence and whatever just because there's some domestic resistance.

In the case of AI, I think there are just going to continue to be civilian economic spillovers much more easily. So in this situation, the strategic impulse actually cuts in our favor. Maybe that's one thing that might make you a little bit more optimistic about it—not all the way to superintelligence in your pocket, but a little bit more.

Yeah. Hey, I'll take what I can get. So let's talk about the rest of the world. You have this big report that just came out on A Transformative AI Strategy for Europe, and obviously there's been some discussion. I actually talked to one of your co-authors a bit back about the compute deficit that Europe has and the need to do something to be a live player going forward.

Before we get into the strategy for what Europe should do, what is the worry if they do nothing? Whatever you think might happen to Europe if Europe stays the course is probably what happens to 70% of the world’s population, maybe 80% of the world’s population, by default, right? What does the future look like in your mind for Africa, Latin America, South Asia, et cetera?

I think it’s going to be really tough because, fundamentally, a lot of the catch-up mechanisms that low- and middle-income countries, in very general terms, have used, enjoyed, and been able to leverage over the last few decades are deeply incompatible with a world that has both very advanced AI systems and, ultimately, a lot of automated manufacturing capacity and whatever is downstream of that.

I think the most immediate and obvious mechanism was always to bet on demographic differences. We just had very rapid population growth. You had a fairly cheap workforce that you would be able to use to your comparative advantage, and then quickly bootstrap into hosting some foreign firms and exporting some valuable good to the global supply chain in a way that was predicated on this idea that you had this workforce that you would be able to put to use in a way that would make you a comparatively beneficial country in which to conduct business activity.

I just don’t know whether that’s going to remain the case. It’s definitely not going to remain the case for most aspects of the menial services economy. I just don’t see a stable way that that sector of the economy really exists once we have very powerful AI systems. I think there are definitely going to be new services jobs and human-preference jobs, and you can think about all these labor-market effects in the long run. But this idea that you can make yourself immediately useful to global supply chains just by doing labor cheaply in the service realm is, I think, not going to work out anymore.

The question is whether it’s going to continue working out in manufacturing. I think that has a lot to do with how fast automation goes and how big the efficiency gains are. There is still a world where manufacturing just gets more and more bottlenecked in a post-AI future, and then it turns out you can at least catch up via manufacturing. That doesn’t strike me as entirely impossible, but I think that’s about it for the general catch-up mechanisms.

The other question is, what’s the stable geopolitical endgame? I think even if you get to this manufacturing-plus-cheap-jobs part of the catch-up mechanism, it seems very difficult to figure out how any country in that spot ever gets any leverage over what happens at the frontier. Which is to say, they don’t get any oversight or regulatory input into how frontier AI systems are built, and they probably also don’t have any hard leverage that makes sure they’ll continue getting AI exports and access to AI supply chains.

They’re basically at the mercy of whatever great power provides them with their AI models. Maybe within that they can find a somewhat favorable arrangement, but it seems very unlikely that they’ll get a stable say in and a stable input into that. I think that just carves the world into spheres of influence of those that have very powerful AI and are able to export it.

There are a bunch of other downstream questions that make this more complicated. How much do you need frontier AI? How much do open weights play into this? At what point can you build your own digital sovereign infrastructure? But I think, at least for the medium term, it is this quasi-vassalage to the frontier AI-building powers that is the most likely outcome for most of these countries.

In terms of how people live, do you think that could create a story kind of similar to the Chinese story over the last few decades, where life is getting a lot better, we’re getting richer, we just don’t have a say in the overall high-level direction, but at the street level, things are trending up and up?

I think in absolute terms, people are going to be richer, wealthier, and better off. In terms of the economic effects, I think they’re just going to be relatively disempowered when it comes to meaningfully shaping the trajectory of the world, and also in terms of having an ability to catch up to however well the frontier countries, so to speak, are doing.

But in absolute terms, there will continue to be growth and spillover effects, and redistribution gets easier as well. On the streets, it’s going to look nicer. It’s going to be, basically, an economically better scenario. So in absolute terms, you wouldn’t mind too much.

I think the more fundamental question is, what does it say about democratic agency, human autonomy, and even human dignity that none of these decisions really factor into where the broader trajectory of the history of the world goes? I think that is a more profound sense of disempowerment that we should still be concerned about. But practically speaking, it’s not that bad.

The other practical issue is susceptibility to misuse, and I think that could be extremely destabilizing. There is a current assumption that, to guard against a lot of forms of AI misuse and AI out of control, you need your own AI systems that defend you against that.

It’s most obviously true in the realm of cyber. I think it is also conceivably true in tracking and monitoring the potential deployment of pathogens—the entire bio-risk conversation. It is probably true in terms of scanning, filtering, and screening against scams and all these socially engineered attempts and whatnot. It is probably also true in terms of safeguarding infrastructure against extortionate hacks and so on.

If you expect there to be a world where non-state actors, terrorists, and criminal groups get access to at least fairly capable AI because they’re able to steal it, because they’re able to fine-tune something like TerroristGPT on some open-source model, and you also expect these countries not to have any coordinated, assured, and widely deployable access to these systems, I’m not sure whether they’re going to be able to protect their citizens from AI-driven harm, AI-driven misuse, and potentially the labor-market effects.

I think that all sounds like they would be very susceptible to that. Then you can imagine a lot of very destabilizing scenarios, right? If your state no longer protects you from AI-driven harm, then what do you turn to? Maybe you turn to mass migration. Maybe you turn to other ways of structuring your personal security, as we already see in some of the failed states in Latin America, where criminal enterprise runs a lot of the day-to-day structure in parts of these countries.

I wouldn’t think that would be impossible for a lot of these countries as a medium- to long-term outcome, and I think that also has me very worried. The pure economic story is pretty positive. The story of the erosion of the authority and power of the state is a lot more concerning. Take these together, and it is not a particularly rosy outcome.

So that’s probably 70% of the world headed there, and Europe is kind of the one place that can maybe engineer a different outcome for itself. Tell me if you disagree with that, but I’m going next to: What does Europe want, and how does it get it?

I think the problem that Europe faces is not too dissimilar to what a couple of other Western countries, or liberal democracies in general, face as well. Fundamentally, Australia is in a similar boat, New Zealand is in a similar boat, Japan and South Korea are in somewhat similar situations, and Canada is in a similar situation. So that, plus Europe and the UK, is, I think, the cluster of U.S.-allied middle powers that have a potential trajectory out of this.

It still needs a lot of work. I think there are 2 fundamental ways to start looking at this. The first way is looking at, well—put aside all the AI things—what do you want Europe’s economic position to be?

If you start thinking about that, you think, well, you want to be good at the things that Europe is currently good at. You want to be good at some aspects of manufacturing. You want to be good at some aspects of artisanal goods. You want to be good at the high-state-capacity things that Europe is currently good at, whether that’s welfare states or high levels of security and safety.

There are a lot of things that are going well in Europe. You just want to keep them going well, plus you want to find some way to actually revitalize your current economy. Then AI comes into the picture as, well, that seems like it could either really accelerate that or really destabilize it.

And then you ask the question: What do you need AI for in that context? I think the other way of looking at it comes to the same conclusion: What does Europe currently not have? The answer is that it currently doesn't have frontier AI systems, which turn out to be one of the most important economic inputs of the future and one of the most exciting parts of strategic and economic competition right now.

The question is, what do you do about that? I think you quickly realize that building these systems ourselves is just too expensive. It doesn't actually work. The next-best thing we can think about is how to get access to frontier models in a way that is assured and secure, and that allows us to build what I described as the first approach: How do we reduce the geopolitical risk of just doing the things we're good at? How do we make sure we have assured access to frontier systems and don't get cut out of this AI, AGI conversation while we do the things we're good at?

I think these all come together to this: You need something to incentivize selling frontier systems, something to make the Americans not nervous about selling frontier systems, and some productive way to use the frontier systems downstream to make something happen. I think the strategy that we wrote tries to answer these questions, especially the parts that I most contributed to. Very briefly, the high-level take is that the first thing is this compute-for-access idea that I first wrote down late last year or early this year and have been shopping around with a lot of countries in the world ever since.

It's now one of the pillars and one of the main asks of the strategy: We build data centers for American hyperscalers, or in cooperation with American hyperscalers. In exchange for the favorable conditions we provide these American hyperscalers and labs, we get assured access to the models that run on these data centers. As long as the Americans keep giving us the models, they continue to get access to the data centers. If their side goes back on the deal and we're cut off from access to the frontier models, they lose access to the data centers. This is the incentive part of the conversation: We build the infrastructure and get access in return.

The second part is, how do we make the Americans not nervous about doing that? Done wrong, this is a security risk, right? You can't run this—

We've done it with the UAE, for God's sake.

Yeah.

We should be able to reach a deal with Europe.

Yeah. Well, I think the UAE thing is kind of fragile. I think the UAE is worried about what the future of that is. Will there actually be frontier weights hosted on UAE data centers? I think that's very unclear. They're going to run some inference on them, but maybe it's just going to be Haiku inference and not Fable Six inference. That's an open question.

The question is, how can we get the security alignment to work out in a way that the Americans aren't too worried about hosting the models there and giving the model to the European economy? I think that has a lot to do with aligning with the U.S. on security provisions, building out the data centers to be secure on both the cyber and physical sides, and building out KYC regimes with European firms. We need to make sure that the Americans don't have any well-grounded national security worries that would pull against the incentives from compute for access.

The third thing is that we should get a little more self-assured about the assets Europe does have. Europe has broad economic assets and a very powerful economy in absolute terms, even if not in terms of growth trajectories. Europe also has a lot of assets in the semiconductor supply chain: ASML, ZEISS, and all these things that play into building frontier chips and, therefore, frontier models. Let's think about how we can be strategic about that.

Let's set up an anti-coercion instrument of sorts that says, “If everyone plays nice, we'd love to feed these assets exclusively into the American supply chain. We're willing to align with export controls vis-à-vis China. We're willing to be good friends and good partners to the U.S. And if the U.S. ever does decide to use its ability to cut off frontier models as a means of coercive action, then we're also willing to use the supply-chain bottlenecks that we have as coercive action in return.”

I think between those 3 things, frontier access is pretty assured. Then you're back to where we were before AGI: Europe still has a lot of structural economic problems, and we still have to solve them, but at least we've fixed the geopolitical problem of being cut off from frontier access. I think that's step 1, and those are the things that I'm excited about getting done in Europe in the next year or so.

What's the hardest part about it? Is it just getting data centers sited and built, or are there other challenges that you think would be bigger than that?

We can have this conversation, and there's a shared understanding that the suggestion I make interfaces with a realistic future that we think might happen and that is worth preparing for. This is not the case in many rooms with policymakers in Europe. I think there is deep skepticism about the continued trajectory of capabilities of U.S.-built AI models.

There is a lot more optimism about the broad availability of open-source competitors that can basically do everything as well as the American models. There is also a more fundamental question: Are these models that powerful? Is it that important? Is it as big of a geopolitical issue?

Then there's the big question: If the models are that important, if everything that I, we, and other people say is true, then why shouldn't we just build this ourselves? That surely can't be that expensive, right? We'll find a more clever way to do it. The Americans are wasteful and high on their own supply anyway. We'll just spend a few million dollars and surely whip something up.

I think cutting through that—which I understand to be complete misunderstandings of the material reality we find ourselves in—is the biggest barrier. We have to make the point that this does not accurately describe reality. You need to think about this in clear-eyed ways that respect that what is happening in America is real, and that the Americans are fundamentally right about many aspects of this.

If that awareness existed, I think there would still be political things to figure out. With ASML, it's going to involve some amount of triangulation between the Dutch government's interests, ASML's interests, and the interests of the other member states. That's not quite easy. In terms of data centers, there will be some domestic skepticism about American tech firms and working with them. That's not going to be quite as easy.

In terms of security alignment, there will be people who are more excited about hedging toward China and trying to stay between worlds a little bit. I think all of these are surmountable—very, very easily surmountable—if you get the alignment and awareness of what's happening here right. I think that's the main challenge.

One of the things I did notice in reading the report was that you and your co-authors are willing to dream a bit in terms of how the authorities might act. At some point, there's basically a statement that doing this in a half-assed or highly bureaucratic, everything's-a-committee-to-nowhere mode—which European governance, at least by reputation, often operates in—might be worse than not doing anything at all.

How realistic is it that you can actually get this sort of action, and what's the mechanism for doing it?

It's still downstream of urgency and awareness of the situation. I think there's always a trickiness in writing for national governments generally. You try to write something that isn't quite within the Overton window of what they're willing to do, but also isn't so far out there that they'd never do it.

One failure mode is that you write, “We just have 20 million in the budget, so let's think of the maximally AI-pilled way to allocate the 20 million.” It turns out it just doesn't matter. You can burn it or throw a party; it doesn't matter. It's not going to change the conversation.

The other failure mode is to say, “We're going to be maximally honest about what we think should be done,” write exactly that up, and then have you tell us, “I'm so sorry we were wrong,” later on. By then, it's going to be too late to do it, and we'll just shrug and say, “Well, we told you the honest thing.”

I think the art, or the trick, of getting this kind of thing right—and I hope we struck a decent balance—is to aim at something just a little more ambitious than where they currently are, accounting for the fact that they will get more ambitious and that they need some nudging toward being more ambitious.

And I think that is the sort of calibration that the strategy tries to reach. Am I optimistic about that? If I had to give you odds of this strategy as a whole being implemented within the next year, they're not that high. If I had to give you odds on elements of it making it into serious policy attempts and actually getting set up, I think they're pretty high, and I don't know which ones of these they're going to be. I could make bets on which ones of them are going to be more popular and less popular. But I think some of this is going to happen, and I think that's the hallmark of a well-calibrated strategy.

We have done this before in Europe. There have been times when Europe has managed to act very quickly and decisively in a way that has delivered results as quickly as anywhere in the world. I used to work in German policy and politics for a bit, especially in energy policy, and in the immediate aftermath of the start of the war in Ukraine, there was an absolutely heroic effort by the German government to buy a fleet of LNG tankers all over the world and get them to transport alternative gas supplies to Germany once the pipelines were cut off. There was a just-as-heroic effort to get LNG terminals built out in some of the most NIMBY parts of the country. Within 6 months, the German government, with all its capacity and urgency, managed to consolidate resources. We made it through the winter with no problem. It was completely fine—no problems at all with a lack of heating or anything. That was the big doomsday scenario, and it just didn't happen. There was a massive, heroic effort, and it just worked.

I think before that, I worked in COVID policy when that was happening. There was a lot of political pressure against joint vaccine procurement. It was very difficult to get the negotiations right and to get the member-state interests aligned. Europe managed to procure vaccines fairly well, and I think the vaccination campaign in Europe went fairly well. We can talk about non-pharmaceutical interventions; they make for a somewhat messier story. But I think that all went pretty well, and I think Europe can do this. Europe just has to realize that this is strategically important. I think we're not that far from realizing that this is important on a COVID- or Ukraine-war scale. If we just get there, then we can definitely make progress on the kind of recommendations we make in the strategy, and I don't think that would be a big problem.

8. Middle Powers And Global Alignment

Cool. Very interesting. Are there small countries that you think are worth calling out for taking a distinctive and potentially effective approach? I was thinking—I don't know anything about this, other than that I know that their sovereign wealth fund is, A, large, and, B, AGI-pilled in its operations. They use a lot of agents, and they're transforming themselves. But I don't know if that's translated to something like a national strategy in Norway. Singapore comes to mind as somebody that might be interesting. Who else is doing interesting things out there, even if they're small and carving out a narrow path, perhaps?

I mean, threading a needle—

Threading the needle, you might say.

Yes. Yeah. I think Singapore, Norway, and the UAE are probably the 3 you'd most obviously mention.

I think Norway could do much more. The exciting thing that Norway could do is become an inference hub or haven for the entirety of Europe. It's a little bit hard to invest domestically with the sovereign wealth fund itself, but you can conceivably come up with schemes to invest into European consortia that then invest into compute build-out in Norway. Norway turns out to be a pretty decent place to build a lot of compute, hence the first attempt to build a Stargate and now a Microsoft data center there. I think Norway could be more AGI-pilled about deploying these resources, but Norway has a lot of resources that could easily be deployed and pivoted toward that.

I also think the base case of investing a bunch of your wealth fund into basically recreating situational awareness—maybe saying that is no longer as en vogue as it used to be 2 months ago or whatever—or, basically, recreating a fairly AI-pilled portfolio with part of the wealth fund probably just lets you ride on the coattails of the AI revolution for quite a long time. That probably works.

The UAE play is building a bunch of data centers. They're basically finding a way to turn money into something that is an asset in the new economy, and I think that's a good way to spend a lot of money if you have it. Now, it's incidentally kind of a tough situation to be in, that you're that close to Iran and that it's that easy to drone-strike data centers. So I think that has thrown a wrench into that plan. But structurally, that was still a pretty good play. If they can manage to build the data centers quickly, and if they can manage to secure the next generation against drone strikes and so on, I think that's still a play that works.

I think Singapore is another interesting case. A, there are also wealth-fund investment questions. B, there is a massive amount of state capacity in terms of understanding what's going on and engaging with it. I think probably no parliament has a greater density of readers of very AI-pilled, very insider-y publications than the Singaporean parliament. I think the same thing goes for their civil service. Singapore is a little bit tougher because a lot of the Singaporean economy is very exposed to AI disruption. If you're Singapore, you just have to hope that the white-collar apocalypse doesn't look quite as apocalyptic, because it's hard to pivot an economy that is as large and as service-indexed as Singapore toward a completely new way of operating. So I think there's a lot of capacity and interest there, but a little less of an obvious AGI-pilled play to pursue.

Are there any other countries that you think are well-positioned, maybe more well-positioned than they know, that should be doing something but are just sleeping at the switch?

I think Australia is kind of awake now, but for the longest time it was Australia, because Australia is such an insanely good place to build compute, both for data center construction and energy-supply reasons, but also for security-integration reasons. There's a really deep level of national-security trust between the Australian and American agencies. I think there is a strong understanding that Australia would not defect to China in any way and that it would be willing to play ball with alignment on China-focused export controls. That just makes Australia a great place to run compute for access and a great data-center build-out location.

More recently, we've seen more of that happening, and I think that's very good. But for the longest time, that was a sleeping giant, and I think it probably still is. You could probably 5× or 10× the data-center ambitions and just run the inference for half of the world out of Australia, and that would not be an overly ambitious thing to do. I think there is still a lot to be done there.

I think maybe the inverse of this is the UK, where the UK has the greatest density of talent and expertise, both in government and outside, just outside of the US, and is not quite entirely sure what exactly it's planning to do with it or whether it can do anything with it. The broader political conditions of the UK, the skepticism toward US alignment, and the damaged relationship with the European Union and the rest of the middle powers make it very difficult to figure out what this incredibly talented cluster of people is actually supposed to do in the UK.

In a way, the UK and Europe really have inverse problems. Europe has amazing assets that it could use extremely well to have a very live-player position in this AI conversation, and it's just really hard to get Europe to do it. Whereas the UK has all the awareness and all the expertise in the world, it's just not entirely sure what it should even be doing with them. At the end of the day, if your starting hand doesn't include any cards that are really good for an AI future, then you can be as aware as you possibly want to be and it's still really hard to get something done.

If you were the rest of the world—let's say you're Brazil, or you could pick your country, or maybe you would put different countries into different positions—I hope this doesn't happen. I'm hyperstitioning better US-China relations and some form of collaboration, rather than carving the world up into spheres of influence. But one thing I've been wondering lately is that there was reporting that the Trump administration was planning to do something along those lines and tell countries, “You're either with us or you're with China. Pick your camp.” If you were put in that position as, say, Brazil—or pick your country—how would you decide? Where would you go?

I think the more important you think AI is, the less justifiable it is to go with China here, just because there is no Chinese AI export program right now, right? They just don't have the chips.

So I think if you think your economy needs access to AI systems and then you can sort of figure out the rest, then I think you just need to go with the US, because only they can give you access to the computing capacity. Which is why a few colleagues and I wrote a paper that was itself a follow-up to another paper, both of which are called “The Closing Window to Win,” about the sort of American AI export ambitions.

China will eventually be better at offering these export deals, as China has been in the past in a bunch of international initiatives that they’ve run in South America, Africa, and Central Asia. But currently they’re not, because they can’t offer any data centers or chips, so they can’t actually offer a full-stack export that can match the US ambitions. So right now, I think if you’re sufficiently AI-pilled, you just have to pick the US.

At some point, China can probably throw in enough non-AI-related things that the deal looks a little bit more attractive. But just in terms of whether there’s any sort of hedging strategy to be had in AI specifically, I currently don’t think there is, as long as China doesn’t have the chips. That might change in a few years, and I think then the decision is going to be much harder.

But currently it’s basically: How reluctant are you going to be about buying US systems? I think that’s the realistic question that a lot of these countries face. And I think, ultimately, that is a great position for the US to be in strategically. The question is just, can the US actually offer a deal that these countries will think they will stick to?

I think that’s maybe the main strategic challenge for the US. Everyone is tactically and strategically incentivized to take the deal—there’s no way around it. But they still don’t like getting a deal that they feel the US can renege on at any point in time. And so the US has to figure out some way to commit to these deals in a way that’s credible to these countries.

Building data centers is part of it. Deep industrial integrations are another part of it. But I think that is something the US just has to think about much more: Even if the deal is necessarily the only deal the other country can take, it might still irrationally defect if you’re a sufficiently unpleasant partner to make a deal with. And so the US just has to think a little bit more about how to be a slightly more pleasant and reliable partner. I think it’s not that far off.

Yeah. We’ve got the guy for the job, so perfect. One thing I’ve gone back and forth on quite a bit over time, because I’m a very AI-focused person, of course, is that as China became the endpoint for a lot of conversations I was having, I became a little bit more of a China person. I’m still not much of a China person, really.

But I always had this question: How is this strategy, where we have these export controls—and so much of what you’re saying really depends on timelines, right? If you believe in superintelligence in 2 or 3 years, you’ve got to be on Team USA because there is no Chinese export. I agree with that. At the same time, if that is the path we’re going down, the chips are made in Taiwan. It’s really close to China and really far from us.

I just don’t see a world where all this is allowed to reach its culmination point—a la “Machines of Loving Grace,” where it’s like, now we’re going to make some sort of deal with the Chinese that they can’t refuse, essentially, and realize eternal 1991—without them just being like, “Fuck no, you’re not. We’re taking out the fabs.”

How do we not end up in a world where all these things seem to be taking us to a point where China is going to hit a breaking point and they’re going to be like, “The fabs are going down”? I don’t know how we get around that with the strategy that we are playing. We can’t defend them, right? It’s a super-sensitive asset. It doesn’t take a lot to do damage. From what I understand, a piece of dust or a skin flake can ruin a batch. So they can presumably not really be defended. How do we not end up there?

So I think I’m also not a big US-China geopolitical-competition, how-do-we-win-this guy. I think I have 194 other countries to focus on, and that just hasn’t left me enough time to really think about this in as much detail as others have.

Very briefly, there are 3 ways to avoid that. The first is that the AI systems just get so powerful, and the US is so far ahead, that escalation around Taiwan is suicidal for China—more so than accepting some amount of US domination. It’s very unclear to me what exact shape of AGI would be so powerful that that would be the case. But I think if you are sufficiently ASI- and superintelligence-oriented, at some point you might actually think that’s just a dominating advantage, and you can’t go to war with a country that has this kind of system. Maybe that is part of it.

The second thing is that maybe the fabs being blown up is just not that big of a deal. Yes, it obviously destabilizes the entire supply chain. Obviously, that’s the end game in terms of US-China competition, and who knows what happens then. But you’ll have some indigenous capacity in Arizona, and you’ll have all the chips already up and running. So maybe if you’re already halfway into your intelligence explosion by then, it turns out you can just run all of this on the chips that you’ve already built.

Yes, if TSMC gets taken out, the chip supply in a year really takes a hit. But maybe AGI can do a lot in a year, especially if it gets TSMC Arizona. And I think the third thing is that China also doesn’t have indigenous capacity. It’s not entirely clear that going to war with Taiwan in a situation where they already think they’re behind in the AI supply chain is the best way to escalate the conflict.

Especially if there is still some TSMC capacity, in some way, shape, or form, indirectly ending up in China. If that is the case, then China might just think its best catch-up hopes revolve much more around domestic industrial integration. That’s the least AGI-oriented version of the future, where it’s just: Is it really worth going to war at this specific point, where the Americans have this decisive technological lead on AGI and the Chinese diffusion play and semiconductor-indigenization play haven’t really worked out yet?

Can we just work in their shadows a little bit more, indigenize some more of the capacity, and then deploy later on? I’m not sure whether that’s the best strategic take, but I think that might also be one strategic approach they take.

But I also think it’s a massive vulnerability, and any reasonable AGI endgame has to account for the fact that the Taiwan situation just might blow up in our faces. I think a lot of them aren’t.

9. Doom Robotics And Labor

Yeah. Again, to talk about threading the needle, how about a little lightning round to close?

Yeah.

You mentioned hedging, and also in the context of the Norway sovereign wealth fund—buying the right equities to get through the AI transition in a successful way. A challenge I’ve been wrestling with a little bit lately is the Tyler Cowen challenge: If you’re so doomer, what are your shorts?

I’ve been trying to come up with an actual answer to that question. Is there some way? I’m not a total doomer, but I think he should be taking it more seriously than he is. So I want to have an answer where either I hear my shorts, or I really tried and I can’t come up with one. That’s where I’m at right now.

I cannot come up with a way where I think I can get rich in the doom scenario. Do you have any suggestions for how to answer Tyler?

Yeah. I think my general sense of that is just that Tyler imagines a much more continuous and smooth on-ramp into actual doom. That makes it so that betting on volatility, and betting on near misses and pretty catastrophic disasters that aren’t quite doom, makes a lot of sense in that world.

I’m not sure whether that’s true. I think a lot of the ways in which things go badly are just that things go extremely well in the market all the way until they go really badly. Then the only situation where you cash in is when you’re dead.

I think that’s the least convincing part of his argument to me: This idea that you’ll get all these near misses, and they’ll already all, in expectation, create the stock market. I think there is just a very reasonable doomer view that concentrates basically all of the probability mass of doom on things going well all the way until doom.

In fact, if you look at a lot of the canonical doom scenarios, and at people who have talked about serious existential risk and catastrophic risk, many of them give a scenario where everything looks like it’s going really well—strategically great and economically great. It looks kind of weird, but it also looks economically great, right up until the point where the takeover happens or the big incident happens.

I think that's the main problem with that. I think also that many people have coherent worldviews that they don't bet on or take financial bets on, even if they're committed to them. That's a more boring meta-contention. But my main contention is that I don't think it's a smooth distribution of probabilities. I think a lot of this is either that it goes very well or it goes very badly, and there isn't really much of a world where it's volatile, goes kind of badly for a while, and then kind of well for a while. I also don't have a good trading strategy.

I appreciate you thinking it through. Do you maintain a P(doom) number? I'm sure you've been asked many times, but I haven't heard you answer it. Do you have an answer?

I think it depends so much on what you include in doom. If it's human extinction, it's very, very low. If it includes all of the catastrophically risky scenarios, including the sort of end stage of gradual disempowerment and stable authoritarianism, I think it's probably around 10 percent. But I think that really is only if you account for the political doom outcomes in the broadest sense, and I think my probability of technical extinction is substantially lower than that.

So you're counting it as doom if we live in a sort of Chinese++ state where life is pretty good, but we don't have political freedom?

Yeah. Maybe life isn't even particularly good in a lot of cases. It's just extremely disempowered, with extremely low human agency and extremely low human economic participation. Call it the permanent underclass, if you must. But I think if it's actually permanent, you shouldn't include too many precisely bad outcomes in the doom number.

The old portfolio of existential, long-term risks used to include things like stable authoritarianism and stable economic disempowerment. To the extent that that's actually a locked-in path for the human future, where you can't see a conceivable breakout from it, I would include that in doom in the broader sense. But not just, "The economy kind of sucks, so my P(doom) is very high because I really think the economy is going to suck."

Maybe something I should have asked earlier, but I think I know the answer. Obviously, your projection assumes that robotics really works. It doesn't necessarily have to be humanoid, but we're going to get highly flexible robotics that can be deployed in all sorts of contexts.

I think it's going to take a little bit longer than I expect. I don't see a super-crazy industrial explosion very soon. But eventually, this is an engineering problem and a scaling problem. At some point, we're going to scale it, and at some point, we're going to resolve the physical bottlenecks. It's going to take longer, so physical bottlenecks are going to matter longer than software bottlenecks, for example. But eventually, they seem eminently resolvable.

Let's say 2030 is the over-under for when people start to have domestic service robots in their homes. Would you take the over or the under?

People start to have them in 2030? Yeah, I think that sounds roughly right. It might take a little bit longer than that because of idiosyncratic psychological and political resistance. But in terms of technical maturity, that sounds about right to me.

How does all this change as compute goes to space?

I think there are 2 versions of compute going to space. The first is that space is one of the places where we can put compute. I think that's going to be the case in 2029, when there are going to be some data center setups that are worth putting in space—more for inference than training, for example. I think it's also going to be gated by launch capacity, so we can put all of our compute starting in '29 in space. We might also conceivably have different chip supply chains for chips and racks that are suitable for going into space, separate from terrestrial chip deployments.

So some compute goes to space in 2029. I think that changes some things. It concentrates more computing power effectively within U.S. jurisdiction, makes SpaceX's AI efforts a lot more powerful, and makes launch-site governance a little bit more relevant. These are all interesting marginal shifts in how the conversation moves. That's part 1.

Part 2 is what happens if every marginal chip goes to space instead of to any terrestrial data center, and there's basically no terrestrial competition for data centers anymore. I think things get a lot crazier then. I think antisatellite weapons become a really important part of deterrence and geopolitical stability, for one. That's because it's the only way you can threaten the deployment of a superintelligent system: it's in space, and if you can't shoot down the satellites, then good luck—you're not stopping the superintelligence.

In much the same way that the AI 2040 plan talks about making data centers bombable and visible, and allowing for this sort of intervention and sabotage, in that world we want satellites to be hittable from the ground and the compute to be vulnerable for geopolitical stability reasons.

Do you think that's the default scenario? My understanding is that we could probably shoot down satellites without too much trouble.

Yeah.

We just don't, really, but we can, right?

The different question is who can. The United States? Yeah. Other countries? Perhaps not. I think they need to develop the capacity to do that. I would think China has it, as do some others. The French have the beginnings of a program, and the Indians have the beginnings of a program. There are already programs. It's not that no one can.

But in the same way that a nuclear power needs to have second-strike capability, I think there's a sort of geopolitical-stability sense in which a sovereign nation might want to have antisatellite capacity, and not all of them do just yet.

The other question is this crazy Kessler syndrome conversation, where there is some amount of mutual deterrence against ever shooting down satellites. If you get to the point where you have that much debris in space, it keeps creating more debris because things keep colliding, and it's going to be really difficult to launch anything into space at any point in the future.

Insofar as everyone is disincentivized from doing that, in the same way that everyone is disincentivized from creating nuclear winter or something, I think that also makes the antisatellite-weapon math a little bit more difficult. The other part of this goes back to our middle-power conversation, right? We talked about these compute-for-access deals and deploying data centers and so on.

That all hinges on the idea that the U.S. is interested in building data centers in other countries because it wants to build data centers somewhere. If the U.S. builds all its data centers in space instead, then the incentive for putting data centers into host countries is just so much lower. I think as a result, that does sound pretty bad for a lot of these compute-based strategies.

I think the most actionable and meaningful consequence of the prospect of data centers going into space is that the compute-for-access and compute-build-out strategy that middle powers are starting to pursue has a time limit. It stops working at some point in the somewhat near future, and you should start thinking about what your end game beyond the compute thing is in case the space strategy works out as SpaceX imagines it does.

A more terrestrial concern. We talked a little bit about bottlenecks, so I don't mean the human inertia around why adoption hasn't happened as much as it obviously could have in theory. But if you take the flip side of that and look at the AIs and their capabilities, clearly there's something missing relative to the experience of hiring a human to do work, right?

It feels like that gap is getting thinner and thinner all the time, almost to the point where I'm now having a hard time putting my finger on what it is about Fable 5.1 or Astra that's actually worse than hiring a human. Do you have an answer for what that is, and what additional marginal capability gain you might expect to actually create labor-market disruption?

I'm not sure it's the capability gains at this point. I think it's integrating with proprietary data and integrating with proprietary workflows. Obviously, not the entire task-profile suite of humans is currently covered by models, but I think in specific tasks they're better than humans, and you can drop them into specific task profiles, at least.

I think that applies especially in software engineering and also in some other general white-collar activities.

Mm.

I think the labor market just takes its time to rearrange around that and get to an augmented, mutually beneficial arrangement. You can't just fire the guy who's sitting at his desk and plug in Astra instead. You need to be slightly more sophisticated: instead of 3 guys, you need 1 guy who tells the agent what to do, and the agent does the tasks. But that guy has to be a little bit better at all the things that the agents can't do.

I think it requires some institutional and organizational reconfiguration, and I think that will just take some time. But I think the capabilities are there, and I would expect them to have this sort of—not displacing, but at least disruptive—impact that changes how teams are built and how productive they are. I'm a little bit less sure that this specifically leads to displacement in the short term. I think it also creates additional demand and additional things that human workers can do on the margins.

But in terms of disruptive and reconfiguring effects, I would agree that the capabilities are there, and it's just latency and lag, bottlenecks, and frictions.

What do you think would happen in a hypothetical world where Tesla decides to license its full self-driving, and within 18 months or so, let's say we make it a priority? We're entering a little bit of a fictional scenario here. All of a sudden, basically all the cars drive themselves, and the 4 or so million Americans who make their living driving aren't needed to drive anymore. That seems like one pretty clear displacement story that very well could happen.

Do you think the economy can absorb those people? Where do they go? It seems really tough when you actually get down to, “Okay, this dude has driven a truck for 25 years. He's not ready to retire, but the truck now drives itself.” What happens to him?

Yeah.

What happens to him?

Well, I think part of it is going to be political responses: wage insurance and reducing their hours. I think there's also going to be a political necessity to add some frictions to this happening. Frankly, there would be human-in-the-loop laws. There would be the sort of Holland-and-New York idea that even if the thing drives autonomously, there still needs to be a driver in the seat. I think we'd see a lot of these political reactions and frictions introduced before anything happens.

That's one part. I think the other part is, yes, eventually the economy would probably be able to absorb at least a decent percentage of that—not necessarily in better jobs, not necessarily in better-paying jobs, and probably in worse jobs. But I will also point out that, in that specific story, you found one of the very few jobs that clearly just has 1 specific task and no mutually synergistic way of engaging with the technology. You sort of fiat in the 1 technology that one-to-one replaces a specific kind of worker.

I think most automation, and AI specifically, just isn't like that. It gets at specific tasks and leaves other parts of the task profile open, so it lends itself, at least in the medium term, to a more augmented, coexisting structure in a way that specific driving doesn't. But yeah, I think that would be very hard to absorb. We'd see a lot of political frictions as a result. We'd have to push a lot of it toward social spending, and I think some of them would find jobs, but most of the jobs would be worse.

Your reference to the idea that somebody might be required to sit in the car even as the car drives itself reminds me of a novel that my dad wrote about a pretty dystopian but highly AI-enabled future, where everybody is kind of out of work, but they need the dignity of work. So they're required to show up and stand around all day, and they're known as “standers” in his imagination.

Yeah. I really hope we don't get there. Fingers crossed.

Yeah.

You could make the cynical observation that some jobs in the real world already are kind of like this. But hopefully we don't get there.

10. What America Should Learn

There are a few bullshit jobs out there. But if people are still with us 2 hours in, they'll be interested in your thoughts on this. What do you think America should be looking around the world to learn?

One thing that I came away from China really thinking about is that there's a lot of upside to surveillance. I don't want to have it for a lot of different reasons, but I'm kind of like, geez, it really sucks to leave all that upside on the table. Is there such a thing as surveillance with American characteristics?

I'm interested in whether you have a thought on that, and also what other things, when you look around the world, you feel like the US—not just what it should envy, because I don't think we can copy the high-speed train from China—but what we should actually be trying to import and realize our version of.

On surveillance, it's really hard for a specific reason: I'm just very worried about perfect enforcement of laws. American laws specifically just aren't made to be nearly perfectly enforced. If you enforced every law on the books in America, I think this would just be a draconian oversight regime.

I understand people's motivation: this would massively disincentivize any sort of illegal action. But a bunch of things are illegal, and I think a lot of the punishments and criminal codes are specifically structured around the idea that you catch 1 of every 100, 10, or 1,000 criminals, and then the deterrence is calibrated to that.

The downside of surveillance, to my mind—and, more broadly, the downside of more AI integration—is that legal systems aren't set up for perfect enforcement. I think that's the most well-taken point in the Flock debate as well. Surveillance-maxing, even with American characteristics, is substantially a pathway to perfect enforcement of laws that were never meant to be perfectly enforced.

Maybe we can talk about that once we've completely revamped the entirety of the criminal code and the practices of enforcement around it. Before that, I'd be extremely worried about going down that path.

As for what to import from the rest of the world, America is very idiosyncratic and, in very specific ways, has done very well with a very weird balance of institutions and economic activity that have, for some reason, served it very well. Usually, when American politicians look around the world, particularly at Europe, and try to import 1 very specific part of a society, they underrate how much of that society is just in a completely different equilibrium and balance.

If you just transported welfare spending from the Nordics, I think you'd have to do things to the tax system that disincentivize a lot of other commercial activity, which then in itself forecloses other avenues of providing the same services. I think the same thing is true for the political system. There is something to be said for the stability of a party and parliamentary system that doesn't swing back quite as much between administrations. But then you also get a much less decisive government that is able to do much less and gets paralyzed into gridlock a lot more.

I think there are countries whose concepts and structures are, wholesale, perhaps preferable in the way that they deliver services and outcomes for their citizenry compared with America, and perhaps not. I think that's a tricky conversation. But I basically don't know of many high-level, really good things about how things work at a very structural, political level in any other place in the world that you could import into America without disrupting a broader part of how this country works. So I'd be very skeptical of doing that piecemeal.

Does that mean we're living in the best America? It sure doesn't feel like we're living up to our potential in many ways.

I think it's Pareto-optimal in some ways, and I think that's different from the best, right? Everything that you would improve would come with a trade-off against something else. I'm very skeptical of this notion that there are clear Pareto improvements to the way that America works.

You can scroll through the Institute for Progress's website on policy interventions, and you'll probably find 10 marginal fixes to laws on the books that would just make this clearly better. Some of them are inspired by other countries. But I think meaningful changes to how the country runs are probably not Pareto improvements; they're tricky trade-offs that I'm not sure would make the country run better.

This is not to say that America is the shining city, or that any other country is the shining city.

It's just that, man, it's all trade-offs, and it's all really difficult to get right. I'm just not so sure that there are easy fixes to much of anything.

Yeah. Last big question. We've talked about a lot of different challenges, obviously, and vexing conundrums of all kinds. What would you say are the most important needles that we need to thread? What are the top couple of things that you think are absolutely most critical?

And then I'd love to hear what you think life is going to look like on the other side of this, in your kind of 90% where it goes well. For people who are like, “This whole AI thing—why do we even do it? Isn't it just stupid?” Paint the upside picture to inspire-

Yeah.

…that audience.

Yeah. I think I'm just not going to fix the problem of the AI story, of the AI labs' big narrative concerns. I just hope we can continue on this positive trend that I think history has been on for the longest time. I don't think I take that much of a fatalistic view about the current or past trajectory of society. I don't think we need AGI to bail us out of much of anything.

I just think it's the next thing we do. It's the next thing there is. It's the next big innovation. It's the next tech innovation that we'll need to stay on track with this sort of compounding economic growth that we have. I think in a lot of ways it's going to look very crazy.

I think some of it is going to involve space, some of it is going to involve robots, and some of it is going to involve a lot of automation. But I think some of it is also just going to involve fairly prosaic future economic growth that just makes us all a little bit richer and a little bit wealthier, and our institutions a little bit more functional by the day, just as they have been, at least since the start of the Industrial Revolution.

I think that's my main hope. I don't dream much bigger than that. I think things might get a lot crazier than that, and then we'll have to find ways to deal with that. But I think that is my hope and my dream for the future—in a decade, or in 2 decades, or in 5 years if things go very fast.

And in terms of what we have to do for that, I think we just genuinely need to muddle through. Make sure that the balance of power works out and continues to work out, and that the balance of wealth continues to work out well. Make sure that the labs don't pull away in terms of power and control from the US government. Make sure the US government doesn't centralize and control the entire flow-through of intelligence through the world.

Make sure that some other countries have a stake in this, both economically and in terms of power, influence, and leverage. And whenever things seem to be going off the rails—when there's too much power amassing in one place and things look like they're going wrong—pull it back a little bit again and keep it on course. I think that's what I want to do, and I think it'll just be a long exercise of doing things like that at the very small margins. I think then we're probably going to be fine.

Well, that is an unreasonably reasonable worldview, and I appreciate you for spending a couple of hours sharing it with me today. This really, I think, has been an excellent conversation. Anything else you want to leave people with before we break?

No. Thank you. I enjoyed it very much. Thank you so much for having me. It was great.

And I liked. Thank you for being part of The Cognitive Revolution.

Thank you so much.