行动自由：美国的 ASI 战略——对谈 RAND 的 Joel Predd、Ben Boudreaux 与 Matt Chessen

Today, my guests are Joel Predd, senior engineer and director; Ben Boudreaux, policy researcher and professor of public policy; and Matt Chesson, resident technical expert at the Center for the Geopolitics of AGI at the RAND Corporation. Together with their coauthors, they’ve recently written a new RAND Perspective paper titled “A U.S. Strategy to Secure Geopolitical Advantage on an Uncertain Path to Superintelligence: Maintaining Freedom of Action.”

This is one of the clearest pieces of AI strategy writing I’ve encountered. It’s both a great introduction and orientation tool for people who are just now entering the AI strategy debates, and a useful organizing framework for anyone who’s been frustrated by how many AI strategy conversations consist of smart people talking past one another because of differing, often unstated assumptions about critical questions.

Their first contribution is a taxonomy of 7 archetypal strategies that aim to achieve distinct, coherent win conditions, and the 5 pivotal uncertainties that will need to be resolved before the best strategy can be identified. The 7 strategies, which I think will be familiar to Cognitive Revolution listeners, include acceleration, aka the a16z plan to let technology take its natural course; dominance, aka the Dario plan, where the U.S. builds such an advantage that it can dictate terms to other countries; co-development, which you might think of as the Demis plan for international cooperation; preparedness for coexistence, which seems to be OpenAI’s plan, as far as I can tell; a moratorium, aka the pause plan; deterrence, which is something like Dan Hendrycks, Alex Wang, and Eric Schmidt’s mutually assured AI malfunction plan; and, finally, investments in what they call continuity of society, which might be best described as prepping for an inevitable AI doomsday.

Meanwhile, the 5 uncertainties are: 1) How close are we to extremely powerful and dangerous AI systems? 2) Is it feasible for humans to coexist with such systems at all? 3) Is it possible for one actor, which could be a country or perhaps even a private company, to use these systems to achieve a decisive strategic advantage over its competitors? 4) Can humanity come together to cooperatively restrain itself from developing such dangerous systems? And 5) if not, can the United States or a coalition of governments prevent the development of such systems, including by force if necessary?

The core argument, as you’ll hear, is that because there is no consensus on these critical questions, even as AI capabilities continue to advance at a blistering pace, the U.S. government should invest now in all of the foundational technologies, institutions, and other capabilities that the various strategies would require. That way, we can keep our options open long enough to get the answers needed to make a wise high-level strategic choice and ultimate commitment.

Personally, I would argue for co-development by default, with a moratorium as a fallback in case it does become clear that we simply can’t continue to develop AIs safely. But I do feel this recommendation is a very good one for a national government that’s just now beginning to understand the nature of AI and remains, for a mix of good and bad reasons, very doubtful that it can trust its rivals to cooperate on such a high-stakes issue.

Along the way, we also discuss how the politics of AGI have shifted in D.C. over the last couple of years; how AGI-pilled the government is today; how realistic a decisive strategic advantage really is; the similarities and differences between the strategic dynamics of the current AI situation and the Cold War nuclear arms race; the magnitude of the investment they’re recommending, which they’re working to quantify in upcoming work on the cost and lead times for 106 distinct capabilities they’ve listed in a security architecture catalog; and what we can do, both in our organizations and even just as individuals today, to maintain as much agency and control as possible even as we seek maximum leverage from AI in our work.

This conversation and the Perspective paper itself stand out for abstracting away the many messy and often confusing details of week-to-week AI developments and focusing our attention on the most critical decisions the United States will need to make, as well as the core reasons it remains premature for now to commit to any single strategy. So, without further ado, I hope you enjoy this clarifying synthesis of AI discourse and simple, high-level argument that the U.S. government should do everything it can now to maintain its freedom of action on the path to superintelligence, with Joel Predd, Ben Boudreaux, and Matt Chesson of RAND’s Center for the Geopolitics of AGI. Joel Predd, senior engineer and director; Ben Boudreaux, policy researcher and professor of public policy; and Matt Chesson, resident technical expert at the Center for the Geopolitics of AGI at the RAND Corporation. Welcome, all of you, to the Cognitive Revolution.

Great to be here. Thank you for having us.

Yeah, I’m excited about this. I’ve been looking through what I think is a really clarifying document that you guys have put together. It is called “A U.S. Strategy to Secure Geopolitical Advantage on an Uncertain Path to Superintelligence: Maintaining Freedom of Action.” I think we’ll benefit a lot, and the audience will benefit a lot, by just walking through this. I find it to be unusually well-crafted and easy to absorb.

I think going through it in a pretty linear way will be well worth our time. Then we can potentially expand to some other topics, and maybe I’ll push you guys on some of the recommendations you make as well. Maybe, just for starters, a little bit of context before we get into the strategy document: one question that’s always kind of in the air is where we are right now on the path toward governments around the world taking AI as seriously as I think probably all 4 of us think they should. I’ve seen a lot of signs that I didn’t expect to see this soon, but I also see a lot of indicators where I feel like maybe they’re not so serious after all. It’s a little bit hard for me to read from the outside.

I’ll take a first swing at this. Matt, Ben, you may have sharper answers than this, but here we go. It is quite remarkable, the moment we’re in. When the Center for the Geopolitics of AI was launched 2 and a half or 3 years ago, it was such that we kind of had to whisper “AGI” for fear of being thought of as too out there, too consumed by science fiction scenarios. It didn’t deter us or scare us from doing the kind of work that we’ve been doing, which is to think about the implications of frontier AI that could advance to that threshold. But nonetheless, that was the climate. It is quite remarkable that now you can’t get into a meeting without calling it superintelligence. So times have changed.

One of the things that we always say in this space, mostly with reference to the technology, is that we should be focused on the trends, not the timestamps. The trends are quite remarkable, as we’ve all observed over the last 2 and a half years in the pace of progress in the technology. But I think one of the things you’re observing is that the trends are also somewhat favorable, even if they’re way too slow when we think about the governance of this technology.

We have governments waking up and paying attention to it. We have the firms building the technology talking about governance solutions. We have civil society forming our public debate now. The trends are very rapid, or even accelerating, in the governance space. Are they fast enough? It’s very difficult to keep pace with this rapidly advancing technology. We’ve spent a lot of time over the last 2 and a half or 3 years looking for the folks who were taking it as seriously as necessary. I regret to say that there aren’t nearly enough as there should be.

I’d just add that I think we are seeing lots of signs that the U.S. government has become more aware of the type of strategic importance that this technology may bring.

But it’s not clear that the U.S. has settled on a strategy, nor have the frontier companies. That might be challenging because many capabilities we might need, like verification technology, technical expertise in government, evaluation capacity, and things like crisis response and communication, may take quite a long time to build. By not proactively building many of those capabilities while capabilities advance and various deployments continue, we may be making a choice that forecloses options we might need down the line. I think we’re seeing signs of greater awareness, but not yet signs of truly adapting and trying to proactively get ahead of the transformations we’re likely to see.

Yeah, I think that’s a great bit of foreshadowing for the strategy document as a whole because, spoiler, it kind of leads up to this idea of building the capacity to commit to a strategy once we see a couple more cards turned over and have a better sense of the nature of the technology and the challenges we’re dealing with as a result of its development. Fundamentally, we still have a lot of uncertainty about the nature of the technology and what challenges will be most important. It strikes me that what you’re recommending ultimately is kind of like launching a lot of sub-agents in parallel in the U.S. government, so they can go out and prepare for a lot of different eventualities.

Before we get there, though, let’s maybe start at the beginning. The first big thing I took away from the document is a really clear taxonomy of 7 archetypal strategies that people are throwing around. I think listeners to this feed will have heard variations of them before, but I hadn’t ever quite seen them enumerated in such a succinct way. So maybe let’s start with a quick review of the archetypal strategies that should at least be on the table for consideration. Then we’ll get into the uncertainties that will drive the decisions about which one will ultimately be the right strategy.

1. Seven Strategies and Five Pivots

Our work started with trying to answer one question: Which national strategy is the best one for managing the transition to superintelligence? The bottom line up front is that we believe there’s just too much uncertainty right now, and none of the archetypal strategies for superintelligence can be justified. If you pre-commit prematurely to any one strategy, it forecloses other strategies and doesn’t build the capabilities that might be needed later. We think the best way for the U.S. to secure geopolitical advantage and ensure humanity’s survival with agency is by executing a freedom-of-action strategy.

This started with a paper we put out where we looked at the fundamentals of what superintelligence actually means for humanity. It’s called “Finding Common Ground in AGI Strategy Debates.” In that paper, we came to the conclusion that there’s a diagnosis and prescription we need to focus on for superintelligence. The diagnosis is that superintelligence is going to be transformative, and the stakes include decisive geopolitical advantage and humanity’s survival with agency. What do we need to do about that diagnosis? We need to build an ecosystem in which AI and humans can coexist and humans can maintain agency, and we need to implement a security architecture that keeps the world stable enough to build that ecosystem.

In analyzing all these different strategies, security architectures, and proposals for international organizations and the like, we found that focusing on any one of them right now is unwise. We found that each strategy can be represented by 1 of 7 archetypes. The first is dominance: you lead at the frontier and suppress rivals, but this could lead to extreme power concentration and value lock-in. Then there’s co-development, the idea of having a U.S.-led consortium that probably needs to include China. You pool compute, have some sort of layered verification, and jointly develop superintelligence together.

This strategy also has a pathology—the dark side of what happens when it’s functioning well—where the founders’ values really become the default and the consortium can become an arena for international competition. There’s a strategy called preparedness, which is much more focused on international coordination and resilience. The problem here is that when you start muddling through, you can normalize accumulating a lot of damage. Then there’s a moratorium strategy, which people are talking about a lot now: some sort of verifiable global halt or pause. The pathology here is that enforcement can become global surveillance; the type of regime you need to maintain these compute caps could be quite intrusive.

There’s a deterrence strategy, where we halt our own frontier and suppress rivals’ programs to make sure that nobody achieves superintelligence because it’s too dangerous. This has a pathology as well, where you get into an eternal standoff that never quite resolves itself. Then there’s a strategy we call continuity of society. You get to a point where you realize that superintelligence is dangerous and inevitable, and it’s going to be diffuse, so we’re just going to have to learn to live with it. This is really focused on resilience, but it could lead to a situation where humanity survives as a relic and doesn’t have a lot of agency.

Then there’s the acceleration strategy. If you believe danger is not close, you’re going to tend to accelerate to maximize the benefits and your competitive advantage. But this also has a potential pathology: markets basically don’t price in the harms, and we become very dependent on systems we don’t control. We revealed these 7 archetypal strategies by developing a classifier, which reveals the sometimes unstated assumptions that underlie superintelligence strategies. The classifier, which you can take a look at in the perspective we put out, asks 5 questions.

The first question is: “Is danger close?” Are we close to superintelligence that could be dangerous? The second question is: “Is coexistence feasible?” Can humans actually live with advanced forms of artificial intelligence? The third question is: “Is restraint feasible?” Can the United States or a coalition of countries cooperate to restrain their progress on AI? The fourth question is: “Is a decisive strategic advantage reachable?” Can one country proceed, achieve recursive self-improvement, and launch itself ahead of all other countries?

The last question is: “Is suppression feasible?” Can you actually use advanced AI to suppress other people from proceeding to dangerous levels of superintelligence? The answers to all these questions reveal strategic preferences and result in these archetypal strategies. As an illustration, if you believe danger is far off, you’ll push for acceleration because danger is far away, so you can maximize the benefits and your competitive advantage.

If you think danger is close, but we can’t coexist with superintelligence—it’s just too dangerous, and we don’t have alignment or control techniques figured out—but we can restrain ourselves and cooperate with others to do so, then you might naturally favor some sort of moratorium. That would buy time for coexistence research to catch up to capabilities. On the other side of the pivot tree, if you think coexistence is possible and you can get a decisive strategic advantage, you might favor a dominance strategy where you push ahead and try to get some sort of monopoly.

The problem is that there’s really no consensus on the answers to these 5 pivot-factor questions, and there’s no evidence that could produce consensus on the timeline these decisions demand. You may not be able to agree even if the evidence is there. If you pick wrong, you’re going to foreclose strategies you might need later. If you pursue more confrontational strategies, like dominance or deterrence, you can destroy the trust you might need if you have to pivot to a cooperative strategy later on.

Likewise, if you pursue acceleration, it can underdevelop all the capabilities you need if superintelligence turns out to be extremely dangerous. Right now, the U.S. doesn’t really have a declared superintelligence strategy. The evidence doesn’t conclusively support any of the archetypal strategies, and we may be running an acceleration strategy by default. Meanwhile, the clock is ticking, and capabilities we might need for any superintelligence strategy are not getting built.

I know one of the big purposes of a document like this is to brief policymakers in various venues and try to help them adopt this framework for thinking through these issues. I’d love to hear what intuitions you’re hearing from people on the five questions, and how those are changing. I know you can’t say everything, and it’ll be Chatham House Rules at a minimum, but I’m interested in what you’re hearing.

For example, I’d think “Is danger close?” has probably had a significant vibe shift over just the last few months. I’d be interested in all five: what was the status quo not that long ago, and how is it shifting as we see these colorful alignment incidents and other headline-making developments, Millennium Prizes, and so on? From my position well outside the halls of power, I’m very unclear whether these things are changing where people started, and how much they’re changing people’s minds.

First, I want to acknowledge the importance of your question. In an environment subject to this much uncertainty and disagreement, monitoring people’s beliefs is very important, particularly in a complex and changing geopolitical environment. Simply observing how individuals in power are changing their views on these questions will be indicative of the kinds of instabilities we could confront as actors pursue different strategies.

The second point I want to emphasize is that, oftentimes, when we talk about this, we emphasize the uncertainty. But one of the points that Matt has made, which I find very important, is—

Irrespective of the uncertainty, which we may disagree on even within the uncertainty, we simply disagree on these points. You or I may have a different take on how close we are to danger, whether we could conceivably enter any kind of negotiated restraint, whether we could fundamentally coexist, and so on. But as Matt said, it seems unlikely that the evidence will persuasively convince everyone, which is what would be needed. So this reinforces the importance of your question.

To the extent that there’s any shift, I think it’s primarily on that first pivot factor. Matt and Ben, you may disagree, but I think this is the factor where we might see the most movement. I’m less certain whether anyone in a position of power is changing on that variable, but it’s the one getting the most attention. I don’t know that there’s been as much movement on the more technical question of whether we could coexist with superintelligence. It’s interesting that we’ve chosen the word “coexistence” because we’re explicitly avoiding the word “alignment,” which boxes us into a specific set of solution techniques. We think the more fundamental concept is coexistence. I don’t know that there’s been as much movement on that one; it’s such a difficult question to answer. And the question of negotiated restraint is so caught up in a complex domestic and international political environment, with trust levels seeming so low, that if there’s been any movement, it has probably been toward greater skepticism. Matt and Ben, you should weigh in with your own takes.

What I’d emphasize is that our focus throughout this work is on preserving strategic optionality, or the capacity of the U.S. to make a smart decision about which of these strategies to pursue. I like that emphasis because it makes clear that we still have some agency here to make a choice. Some of these pivots, the way they’re phrased, make it sound like they’re simply exogenous things happening to us. But in reality, and what this is hopefully trying to demonstrate, is that there are choices we could make to push danger further off, make restraint more feasible, or perhaps develop certain types of diffusion capabilities that enable a more decisive lead.

These factors are more endogenous than a glance at the pivot table might suggest. What we’re seeing decision-makers do is start to grapple with what they might be able to do to ensure that danger is farther off and coexistence might be more feasible. The overarching approach is to think across the strategy space: If you want to do all those things, what can decision-makers build now to ensure they’re using their agency now to preserve it for later as well? That’s the shift I’m seeing: a greater recognition that AI development and diffusion aren’t just things happening to us, but things we can shape. Discussions about pacing, and even the morally binding commitments recently announced by the White House, show that there are things that can be done to help us navigate these pivot factors.

We should repeat that phrase, “morally binding commitments,” as often as possible.

Yeah, I love it. I’ll keep tying the frontier companies to their moral responsibilities.

Matt?

I would add that the question “Is danger close?” is one of the most important questions you can ask in AI policy right now. A lot of governments are very awake to the opportunities of AI, and I think they’re waking up to some of the dangers. I don’t know that anybody quite has a handle on what the strategy should be for the path to superintelligence. That’s where I think our strategy provides people a lot of options and a framework for thinking about strategy.

But this question of whether danger is close: when you talk to people at the frontier companies and some of the organizations closest to AI, I think they’re very clear that danger is close, and that this is something we need to pay attention to. When I talk to them about this, I ask, “If you think danger is close and there’s a lot of evidence for it, why isn’t that getting translated well to policymakers? Why isn’t leadership awake to the dangers quite yet?” I think this is a gap the AI policy community needs to address. If evidence exists that danger is really close, the AI policy community has to do a better job of packaging it up in a way that policymakers can take on board. Policymakers have lots of different things in their portfolios, and they might not be experts in AI. They need the information packaged in a way they can absorb and use to update as new information develops.

We can’t just wait for crises. I think the Hugging Face incident was a mini-crisis. If we’re going to rely on crises to raise awareness, I think the AI policy community can do better.

So, the “danger is close” one seems to have fallen out of favor. Matt, you said a minute ago that if we don’t think danger is close, we might be playing an accelerationist strategy by default. I think that seems to be falling at least somewhat out of favor, although there’s no consensus yet.

We’re talking on the day when Muse Wonder was just released—not to forget Muse Wonder Full—and it’s coming from a company that doesn’t seem too concerned about any near-term danger. It certainly doesn’t seem inclined to do much pacing beyond whatever baseline consumer safety testing they’re known for in all their products, which I think we should have full confidence in. That was somewhat of a joke, in case that wasn’t obvious.

Yeah.

I would’ve said it seems like we’re playing a dominance strategy by default, because we’ve been in this chip-denial mode for a couple of years. As you may know, I’m definitely a skeptic of that policy. One reason I’m skeptical, and one thing I’d love to get a steelman from you guys on, is decisive strategic advantage, or a decisive lead. Is that really possible? Who makes the best case for it?

I always look back at Machines of Loving Grace and think, “Boy, you skipped a lot of points along the way, Dario.” I love the first half: all the diseases we’re going to cure, the better mental health, and all these new genius outputs. But then it’s like, “And something, something, Eternal 1991. We’ll put China in a box, make them an offer they can’t refuse.” That’s all we need to say about that.

What’s the best thinking on what a decisive strategic advantage could plausibly look like? Who articulates that best? Sorry if that’s an impossible question.

Yeah.

One reason this is a challenging question is that you have to answer the question of the scope of this decisive strategic advantage. When you talk to people who want to advocate for that strategy, they may have a different source of advantage in mind.

Oftentimes, a decisive strategic advantage brings to mind an enduring lead in the technology itself. In that case, you’re prompted to ask whether you merely mean maintaining a consistent lead, or whether you mean preventing your rivals from ever crossing some specific, hard-to-identify threshold. Given that this technology is driven by economic forces as much as technological forces, it does seem a little hard to believe that you can prevent an adversary from crossing a given threshold, although it’s not impossible. That does seem to be a hard bar, and it may be the one you’re thinking about.

Given the pace of things, it feels less difficult to imagine that you could maintain a steady lead over time. But there are other domains in which you might maintain a decisive strategic advantage economically. The question there is whether you can translate your leadership at the frontier into consistent economic gains that would underwrite growth in your economy and scientific innovation.

It’s interesting that the things you would need as a society, as a nation, to gain and maintain a decisive economic advantage, as our colleague Toby Sitzmann has written about in a separate paper we could refer you to, are quite different. They aren’t just the technological underpinnings of the technology. In another domain, as you would imagine, there would be a decisive strategic advantage in the military domain. As with the economic source of advantage, you would need to translate this into weapons systems, concepts of operation, and a strategic doctrine, none of which come for free.

To the extent that a decisive strategic advantage in an era of very advanced forms of AI means all these things at once, that’s where the rub is.

It just seems super hard to me. I actually had a chance to play a tabletop exercise, essentially an almost-fast-takeoff scenario, and I got assigned China. At the end, like most players in the game, we were thinking, “Well, that all happened really fast.” We kept being a step behind events.

As usual in these tabletop exercises, we were trying to simulate what we thought the real actor would do. I do feel like there’s at least some chance that everybody will be paralyzed by indecision to the point where we get to some outcome before anybody really plays their cards. But I also felt like, as China, I had a lot of cards to play. It was really my restraint, and my desire to seek a constructive deal first before I started playing super hardball, that was limiting me.

I guess I just have a hard time seeing what the win condition is supposed to be. In Machines of Loving Grace, it really does sound a lot like regime change at the end. I think the sentence is almost verbatim: “China, we’ll convince China that it’s better to give up competing and instead join our coalition to share in the benefits.” That doesn’t sound like the sort of thing where we’re making nice. It sounds more like we’re giving an offer they can’t refuse.

It feels like you can only get there if timelines are super short. To what degree does the plausibility of a decisive advantage depend on a super-fast, FOOM-like scenario, where the gap widens long enough to take some pivotal action before the other side can act? The things I’m considering as China: all these TSMC fabs are awfully nice. It’d be a shame if something happened to them, right? They’re right in my corner of the world. How do you think about what determines this? Is it about timelines, or something else? I still really struggle to see it.

I’d say a few things here. Joel already distinguished ways in which decisive advantage might appear differently in different domains. The U.S. might have a decisive advantage in one domain while China has it in another. This picture is trying to roll them all up, but that might not be possible, or at least they might not all be commensurate in some clear way.

Your question also suggests that the dominant strategy might differ based on the temporal windows we’re talking about. It might be accelerated by a FOOM-type fast takeoff that would extend a leader’s lead over the rest. But of course, in a FOOM, you need to worry about whether you can maintain control. Decisive advantage is only good insofar as the leader can continue to lead the AIs.

That might also be in jeopardy in a fast-takeoff situation if the governing leader isn’t prepared to control and steer the technology in ways that serve its interests. Having more time might be necessary to ensure that the decisive advantage can be implemented. The FOOM case might not give you enough time to build in all the control and steerability mechanisms you need to really leverage it.

This also makes me think that dominance, like some of the other strategies, might not be something you can ensure will be sustainable over the long run, even if you commit to it and execute it. The U.S. might be pursuing a dominant strategy and even achieve dominance for a period, but at a certain point, through model theft, distillation, diffusion, or many other processes, you might need to pivot, rebalance, or move to a different strategic paradigm.

The choice among strategies might never be fully complete for all eternity. It’s perhaps going to be a continual process of adaptation as the technologies and the geopolitical landscape change.

So, just to lay my cards on the table: I feel like if humanity were wise and really up to this challenge, the right strategy would be some sort of co-development, with a moratorium as basically the fallback. In other words, we don’t know if danger is close or far. If it’s close, and if there are scientific problems that prove very difficult to solve, then we’re going to need to agree on some sort of moratorium. So let’s work together until such time as we have clarity on that, and then we can move into that mode if we need to.

It is sad to me that this seems to be, as you described it, Joel, something people are growing more skeptical of over time. I don’t know to what degree you guys want to put a thumb on the scale in terms of what you think humanity would do if we were wise, but I’m interested in whether you would generally agree with my take there. And then I’m interested to unpack: Given that it seems like, if anything, danger looks more likely all the time, why are we not trying to figure that out, and why are we instead growing more skeptical of it?

Yep. I’ll jump in on this one. I would agree that if humanity were united and coordinated, some sort of co-development strategy with a moratorium as a fallback might make a lot of sense. But the fact is, humanity is not a unitary actor. There are very real differences in our political systems between authoritarian governments and democratic governments.

If you’re betting on co-development or a moratorium, you’re betting on restraint feasibility, and you’re betting on the other side actually agreeing to and abiding by the rules that are set up for these restraint systems. One of the big concerns for me, because we do have these differences in values and perspectives on the right political systems, is that if you go with some sort of co-development strategy with China, you’re really locking in nondemocratic founding values. You’re locking in those authoritarian values as persistent for humanity over time in the AI systems that are being built. I think we really need to think about whether that’s something we’re interested in. Do we want to lock in those values, or do we think that enshrining democratic values in AI systems is actually a better outcome for humanity?

Can I come back to your dominant strategy question? We left some work on the table there. There’s a paper by our colleague Phil Root on our website that has tried to do the game theory under some of this. It begins with the premise that the race to these advanced forms of AI is a kind of prisoner’s dilemma. Under some reasonable assumptions about what that game looks like, one of the initial insights is that in a competitive environment, the default strategy you’re led to is one of sabotage over cooperation. That is to say, even though some form of cooperation produces higher collective gains, over time, both players get into an equilibrium in which mutual sabotage of each other’s AI progress is the equilibrium strategy.

One of the things that disrupts that, however, is a dominant strategy in which AI is used to create something we have called a wonder weapon in some of our other work. That is, AI as an engine of scientific and military innovation creates a new capability that allows us to hold others’ AI programs at risk. If AI, or superintelligence, were to yield such a capability, that’s the circumstance in which a dominant strategy would work. One reason I think your intuition about how unbelievable that is makes sense is that you’re stipulating a capability that would be given birth by the AI itself. But that does end up being the equilibrium strategy under those assumptions. It’s worth checking out.

Can you put a little more meat on the bones of a wonder weapon? What is that? What does that look like?

It is exactly the right question, and I don’t think I’m going to give you an answer that will be very satisfying. Wonder weapons in the history of military innovation are very rare. What we mean by that is a military capability that very rapidly and discontinuously disrupts the military balance. Most of the time, military innovation is one in which the disruption comes through adoption. You have to develop force structure, concepts of operation, and doctrine.

In the vast history of military innovation, very rarely is one machine gun, one tank, or any other military innovation going to change the game. But there are instances, and the most canonical instance is nuclear weapons. They were an example of a military capability that, simply by their existence, offset the military balance in a way that was disruptive for stability and deterrence and all the rest of it.

In this context, we’re stipulating something similar. What exactly would it look like? Some have argued, I think somewhat compellingly, that Mythos or Mythos-class models, and their ability to very rapidly discover and exploit cyber vulnerabilities, were the first wonder weapon we’ve seen of the AI era. We can speculate about fully autonomous cyber weapons, which would be a kind of wonder weapon simply by their mere existence. If they were in the possession of a state actor, they could essentially overnight—perhaps not literally overnight, but very rapidly and discontinuously—shift the military balance.

In this game theory work I’m referring to, I’m speculating on the idea of an AI-enabled wonder weapon that would be used to ensure a stable advantage in frontier AI, which you could then translate into a decisive strategic advantage, probably in the cyber realm.

One of the obvious fundamental challenges of all of this is how much we have to imagine, and the almost canonical impossibility of knowing what the AI that’s smarter than us is going to invent. So that definitely makes it tough.

There are 2 things about that that I think are really important. One is—and I don’t mean to interrupt you—coming back to this, what you believe is important. That is a remarkable finding we have. We’ve run, quite literally, hundreds of tabletop games on these kinds of dynamics: crisis simulations by our colleagues like Rich Girven, Greg Smith, and Sena Bagui, and U.S.-versus-China strategy games by Abby Dow and Carl Mueller. There are a lot of reports on the RAND website about this.

One of the recurring findings we get over and over is the degree to which players’ ingoing priors about these kinds of pivot factors will determine the outcomes of their games. If you go into one of these games and someone is a believer, so to speak, in a decisive strategic advantage, they can connect the dots in their own minds, or they believe that danger is close, or whatever. You get a very different outcome in these games than you would otherwise. I think that’s a remarkable point. Your perceptions about the facts of the matter matter as much, if not more, than the actual facts of the matter.

And I will say this: That was probably true of the nuclear era in a way that I don’t appreciate because I didn’t live through it. Despite the massive uncertainties that existed in that era, there were some basic certainties that we lack in the current environment. The scientists at Los Alamos had the physics and the mathematics to run a calculation that gave them confidence they could proceed with the Trinity test without igniting the atmosphere. There was a lot of anxiety and uncertainty in that, but they could at least run a calculation that gave them confidence to proceed. We don’t have a similar science to run those kinds of calculations right now on whether danger is close, whether we can coexist with superintelligence, or whether decisive strategic advantage is a thing.

We are very much anchored on the perceptions of these load-bearing factors, for better or worse—probably worse.

When you say “for better or worse,” could you give me a little sketch of what the policymaking and military leadership community tends to come in with? Given that only relatively recently so many people have started to take this stuff seriously at all, I would guess that most people have had the background assumption that you’re probably not going to have a superintelligence spit something out that makes a nuclear arsenal irrelevant. I take it that would have to be the level of power it would need to achieve, right?

The cyber weapon thing always confuses me, too. This is a bit of a digression, but we have a war going on in the world right now where there are 2 countries with a lot of great programmers. They’re fighting with drones a lot, and there’s definitely a technology and AI story to that to some degree, although I don’t have a great sense of how AI-enabled the drones are versus just being flown by kids in basements. But the cyber story there hasn’t been that big, right? I have a very hard time seeing the cyber story getting to the point where I can somehow take the adversary’s nuclear arsenal offline and then do whatever I want. If I can’t do that, they still have a pretty strong hammer that I have to be very threatened and constrained by, right? Am I making sense?

You’re exactly right. Now, of course, Nathan, as a good host, you’re going to try to put us in a corner and make us take a position on this, and it’s unlikely that you’re going to get us to do that because we’re slightly too—

You’re trained professionals.

Yes, exactly. So we’ll continue to talk about assumptions and uncertainties. However, I’ll agree with you in some sense. If we were to build a general intelligence, a superintelligence would be born in a world with nuclear weapons. It would be very different if superintelligence were born in 1930 or whatever. I think this is related to one of the failure modes of a dominant strategy: you have to have a theory for how you will avoid escalating to a catastrophic war in which you’re trying to permanently suppress your rivals. That’s a substantial risk of that strategy.

Another point here is the pace of things, and Matt and Ben should weigh in on this. I’ll be curious to hear their take. We also have to be careful about what we mean by stability in this context. We enjoyed a decades-long period of stability in the nuclear era, and we’re looking for a similar, multi-decade source of stability in the AI era. But it’s unclear that we will ever enjoy that kind of stability. We may go through these different phases in our AI strategy more rapidly.

I think this may provide an interesting lens for looking back on nuclear strategy. Our colleague Anne Stickles is going to write a Substack post on this, which may come out this week. Our strategy for the nuclear era has shifted remarkably over time. There was a period when we were sprinting, and that was called the Manhattan Project. Then there was a temporary monopoly, about a 4-year period, during which we dominated and used that advantage geopolitically. During that period, some of us were quite anxious about the dangers of nuclear weapons and a nuclear monopoly in particular, and some proposed something resembling a moratorium. We didn’t proceed with it, but eventually entered into verifiable agreements with our rivals.

Later, we used something that looked like a suppression or deterrence strategy to prevent the technology from proliferating into Iran and North Korea, with limited success. The nuclear era continues, and the technology continues to proliferate. Who knows what’s coming next? One reading of the history of nuclear strategy is that it was a kind of freedom-of-action strategy. No policymaker ever stood up and built optionality as we’re proposing, but that’s because they had the luxury of time to build this kind of strategic optionality. We won’t have that here. The diagnosis of this freedom-of-action strategy is that, unlike in the nuclear era, we may still transition through different strategic epochs. To navigate them, we’ll need to pre-build the elements of a security architecture that will make the transition possible.

We’ve beaten up on the dominance strategy a bit. I definitely want to get to the recommendations you’re making about what we should be investing in now to maintain freedom of action next. Is there any other nuance around these other strategies that you think we should spend more time on?

Throughout our discussion and our paper, we use the term “coexistence” to represent humans living with advanced AI while maintaining agency. Alignment is often treated as a catch-all for coexistence, but it’s just one path among several, alongside control mechanisms, architectural schemes, and methods not yet known.

We use “coexistence” throughout to represent the concept of humans living with advanced AI while maintaining agency. Let’s talk about the archetypal strategies and why we think focusing on one particular strategy is unwise. We looked at all the strategies, architectures, and proposals for international organizations that were out there, and found that they could be represented by 7 strategy archetypes. Each archetype has real advocates and a coherent win state, but it also has a pathology—the dark side of what happens even when the strategy is working well.

For example, with a moratorium strategy, you have Eliezer Yudkowsky and Mary advocating for some sort of pause. The win state is that the United States and its rivals would halt capability development and buy time for the coexistence work to catch up. But it has a pathology: the enforcement mechanism needed to enforce a global pause could harden into some sort of global surveillance regime.

Schmidt, Hendrycks, and Wang proposed a type of deterrence strategy with MAIM. The win state there is that the United States would prevent dangerous AI by creating some sort of moratorium at home, while also suppressing it internationally, including through force if necessary. The pathology is that this might harden into a standoff that never really resolves into any sort of peace.

The audience will certainly be familiar with Eliezer Yudkowsky’s calls for kinetic strikes on data centers if and whenever needed. Dan Hendrycks is a past guest on the podcast, although I talked to him before they came out with the MAIM document. These authors are definitely, many of them, familiar names.

Continuity of society: I planted a victory garden this year for the first time in my life. The squirrels got most of it, but it did reinforce for me the pathology of just trying to hunker down in some subsistence mindset and scrape by, or eke out a survival in the case that we can’t avoid creating them and coexistence isn’t very feasible.

The other nuance I want to give you, Nathan, is that we really see this as a strategy for America, not just for the U.S. government. Sometimes when we talk about that, we elevate the importance of the U.S. government because there are certain non-delegable responsibilities that are only available through the authorities the U.S. government has. Maintaining the viability and integrity of the nuclear deterrent is inherently a U.S. government responsibility. Defending U.S. national security in general is also a non-delegable U.S. government responsibility.

But there are many things called for in the security architectures we propose that we might too quickly assume are government responsibilities, when they’re really opportunities for businesses and civil society across America to step up. For example, one practical thing we think would be needed to support a freedom-of-action strategy is investment in highly secured data centers. In principle, the U.S. government could support that, but in practice, we could create a market for it. Firms across the country should have a demand for it, and we could rely on the market to create it.

Another investment called for is incident reporting and response. You can easily imagine a government role there, but you can also imagine civil society filling that function. One virtue of this freedom-of-action strategy is that it exposes opportunities not just for the government to marshal its forces, but for all of American business and civil society.

That reminds me of my grandfather, who had tuberculosis and didn’t go to World War II as a result of his illness, but nevertheless told stories of how we won the war. It was victory gardens and carpooling to work to save on rations.

And the point was, everybody has a role to play in such a national effort as that. I like the framing of reminding everyone that this is not a highly top-down civilization that we’ve built for ourselves, and that there are a lot of things people can and should be doing without necessarily being told to do so, without waiting for anybody else to give them any orders. That is a big part of the strength—

Yeah.

—of the American system.

I think that is especially true for the elements of our strategy that call for investments in resilience, and we see that already. Ben, this is something I know you’ve thought a lot about. Do you want to weigh in on that?

Yeah. In that vein, managing our dependencies on AIs means being able to operate critical infrastructure and develop and implement strategy at non-personal levels, right? We need to be able to think things through ourselves and maintain our individual-level agency. I think all of that is part of implementing basically any of these seven strategies. Even things like moratoriums will really only be feasible if we’re able to live without advanced forms of AI.

Some of what we need to do is preserve those human capacities, both at the individual and collective levels. This relates to why we use the terminology of coexistence, and not just alignment, which makes it sound like we’re just training the models to conform to some set of established values. What we’re envisioning is a process of adaptation and perhaps even co-evolution, where, just as the AIs are becoming more capable, we’re becoming more capable of living with them—and perhaps also of living without them. That’s a key way to manage some of the vulnerabilities that come from those dependencies.

Safe to say the Overton window is wide open these days. Let’s zoom out and go through the seven objectives one by one. You can provide as much commentary on each one as you want, but I think it’s worth going through the high-level objectives you set out.

Great. We should talk through the different objectives and strategies. Thanks for the invitation to do that. The objectives really group into two categories. The first is that we need to make investments to build, and the second is the operational elements, or the things we need to actually do. These operate on slightly different timescales, so it’s worth unpacking them. The first thing we need to build is the human-AI ecosystem. Ben, you should talk about that.

Sure. This relates to what I was just describing about ways humans and advanced forms of AI can coexist, and how humans can continue to thrive and maintain our agency in that context. A lot of this is traditional safety research. Our analysis is that we haven’t invested nearly enough in some of the alignment and control techniques. We will continue to need new evaluations and ways of identifying when advanced models can tell they’re being evaluated and perhaps are sandbagging, so that we can understand their capabilities.

We need to build additional tools that assume we’re dealing with untrustworthy models and ensure that we have containers and security environments that, for instance, prevent them from breaking out of sandboxes, or allow us to contain their spread. Some of this is ensuring that our institutions remain at the center of high-stakes decisions and that humans are still calling the shots. We’re still the ones steering the future, and we’re still the actors capable of making decisions that affect the various pivot factors, like whether suppression might be feasible.

I think this requires a continual process of humans reflecting on the decisions we want to keep in human hands. Some decisions in the broader discourse, for instance about pacing or issues related to AI-enabled research and development, might be core decisions where we want to make sure we have human checkpoints. Even if an AI is seeking to persuade a human, we want to ensure those circumstances are caught and that humans are making reflective decisions and deliberating with others about how much decision space, autonomy, and how many authorizations we want to delegate to these models. I could talk a lot more about any of that, but I think that’s probably enough for now.

One of the things Ben and Alvin have written about that I think is really interesting is the idea of an agency eval.

Yeah.

A lot of the evaluations we hear about are evaluating model performance on discrete tasks, including tasks that should concern us if the models are capable enough to do them, like engineering a bioweapon. One of the insights Ben and Alvin had—and Ben, you should speak to it if you want—is that we should be evaluating models in the context of their deployments, for circumstances in which we may cross some threshold and have effectively, perhaps unknowingly, given up some agency.

Very briefly, it’s about mapping the human decision space and then seeing what roles AIs play when they enter into those decisions, either by disenfranchising various humans, persuading humans, or controlling the agenda over which humans are deliberating. I think these things aren’t typically evaluated now. They’re going to be hard to evaluate going forward, because individual rational decisions to delegate something to an AI may not be easy to see. What you need to do is track these decisions to delegate to AIs over time and in a more systemic way, looking at trends and trajectories rather than single, discrete decisions.

You really have to think longitudinally about the roles humans and AIs are playing in these high-stakes decisions, like whether to deploy a new model or whether to let an AI develop the next fad, or something of the sort. This is work that’s currently ongoing, and we hope to have a paper out on it soon.

Another big objective in the strategy is to adapt our legacy national security architecture. Unfortunately, the old problems aren’t going away, so we’re going to have to adapt our counterterrorism enterprise for terrorists who are in possession of very advanced forms of AI. We’re going to need to adapt our intelligence apparatus, both to embrace the opportunities of artificial intelligence and to anticipate how our adversaries will do the same. We will need to adapt our conventional military for the new demands that will be placed on it and the opportunities that will be given to it.

We may use our conventional military to defend U.S. interests in the way we’ve grown accustomed to, but we may also give it new missions to enforce an AI arms control agreement, say, or defend interests in new ways. One of our colleagues, Zach Burdett, is leading a big project on what a new conventional military in an AGI era would look like. We’ll need to update our nuclear enterprise. It’s unlikely that nuclear weapons are going away, as we may have referred to earlier, but the way we think about nuclear weapons in our strategy, and what it means to have a resilient nuclear architecture, could be quite different.

My colleague Edward Geist is thinking about those kinds of questions. Another area is biosecurity. A big team led by Steph Guerre at RAND is thinking through what that would look like. All of our legacy, pre-superintelligence security architecture elements are going to need to be adapted for the very new challenges and opportunities that AI will afford them. But we’re also going to need a new element: a new security architecture for AI itself. We’ll need an AI security architecture. Matt, do you want to speak to that?

Sure. We think that we need a security architecture specifically for AI. This is a set of technical and institutional capabilities that keep both the technology safe and the world stable. The really important thing to understand is that, traditionally, a security architecture is something governments establish. But AI is quite different, in that a lot of the elements of a security architecture are actually going to be built by the private sector.

Another thing to understand is that what we’re proposing with the Freedom of Action strategy is not one specific configuration of capabilities. What we’re proposing is building the baseline capabilities to run a variety of different strategies. We feel like emerging events, or even the evidence resolving in a particular way, could mean that you suddenly find out AI is very dangerous and the danger is very close, or that coexistence isn’t possible.

You may have to reconfigure your security architecture on the fly. The Freedom of Action strategy calls for looking at the capabilities needed by all the different strategies, and at the capabilities that have a very long lead time. They might not be needed by every single strategy. For example, verification technologies aren’t needed by every strategy, but they take a long time to set up.

The Freedom of Action strategy would call for investing in all of those, so that you have flexibility later on. If you decide to run a preparedness strategy and suddenly realize you need to switch to a moratorium, you’ll already have some of that infrastructure set up and built. We think the most important thing to build right now is navigation capabilities: the capabilities that let you understand where you are in the strategy space. They let you understand the current capabilities of different models, the capabilities of Chinese models, and what visibility we have into compute and where the compute clusters are. Without understanding the answers to these questions, you really can’t answer fundamental questions like, “Is danger close?”

One thing that occurs to me with this whole strategy—and I think this is somewhat trendy in the policy world right now, perhaps downstream of COVID—is that we realized going all in on efficiency creates resilience problems. Maybe we’ve partially learned a lesson there. There seems to be some appetite now, perhaps only at the think tank level; I don’t know if we feel this at Congress yet, to invest in portfolios where some of the individual investments will, by definition, prove wasteful in the end. But because we don’t know, we want to make all these investments now so we have the option available to us later. Hopefully that’s a fair echo of a core part of the strategy.

What among these things is going to be expensive? I think it all sounds good until we start looking at the bill. Which parts will be the big bites that have to be sold to people with the argument, “You’ll be really glad you did this if you need it, even if you don’t think you will right now”? What will be hardest to sell because of the price tag or otherwise?

I think this is one of the most important critiques we’ve received of the Freedom of Action strategy: the unwritten price tag. One of our colleagues told us, when this work was being peer reviewed, that a strategy without a price tag is just a vision.

I think they were using Claude to generate that critique, by the way. That sounds very Claude-like.

It does.

We’ll put that in Pangram.

That’s right. I think they—and Claude, if that was the case—are right about that. Our response to that, though, is twofold. We’re on the cusp of a transformation, and I’ve got to be honest with you: any of the archetypical strategies we propose will be expensive, with the exception of acceleration, although the expense may be borne out in the downsides. In all cases, these strategies are expensive. We’re talking about a society-level reorientation, perhaps on the scale of the reorientation we had with the dawn of the nuclear weapons enterprise and confronting the Soviet Union, or the reorientation of our society after 9/11, when we pivoted to counterterrorism and counterinsurgency.

To a first-order estimate, all of these strategies are expensive in the resources, attention, and focus they’ll require. The difference between any one of them and the Freedom of Action strategy is dwarfed by the difference between any of them and the status quo. I’ll say that most of the things our Freedom of Action strategy calls for are no-regret investments. In most of these strategies, you’re going to need to maintain some visibility into compute and govern the flow of compute. All of them will require some situational awareness, as Matt referred to. They’ll require investments in the resilience of our critical infrastructure, our national security systems, and our biosecurity. Unfortunately, it’s expensive, but all these strategies are quite expensive. Matt, would you want to add to that?

Yeah. I’d just note that not all of these costs fall on the federal budget. You have companies and other actors funding much of the alignment, evaluation, and security work. It’s not just something where you go to Congress and ask them to write a check. That’s why this is a national strategy: lots of actors play a role in executing the Freedom of Action strategy.

Do I understand correctly that the idea is that if it costs 1x to prepare for any one of these strategies, preparing for all of them simultaneously might cost 1.5x? I think you said the difference between doing one and maintaining freedom of action is less than the cost of doing any one thing. Is that a fair way to think about it?

I think it’s a reasonable way to think about it. We haven’t done those cost estimates. Our research is going next toward thinking about how we operationalize the Freedom of Action strategy: What is it actually going to take? I don’t know if we’ll come up with budget-quality estimates that we can take to Congress, but we’ll look at what it will actually take and who needs to do what under different assumptions about the division of responsibility between governments, businesses, and civil society. What is our option space there?

Intuitively, what you said makes sense. The difference between the status quo and any one of these strategies is dwarfed by the difference between any one of these strategies and the Freedom of Action strategy. I don’t have an exact number for you, but they’ll all require a reorientation of resources, our attention, our politics, and our focus across the board.

How are you guys using language models in your work? It strikes me that if you have a fairly detailed sketch, you could probably get agents to run that budget question to ground pretty effectively these days, right?

Matt, you should speak to this. Matt has been very forward-leaning on this class of problem. Go ahead.

Yeah. We use large language models quite a bit, mostly for organizing and remixing information rather than doing a lot of primary research. We’ve found them incredibly useful, especially project knowledge bases for keeping all of our various memos and everything we’re compiling together.

On that particular question, we have a catalog we call the Security Architecture Catalog, which has 106 different capabilities needed to build out this ecosystem and security architecture. We used LLMs quite a bit to gather this information and compile and organize it in a way that humans can digest. For some of these things, there is a primary source—a paper out there that can tell you exactly how long it will take to build, say, verification technologies. Often, though, nobody has ever done that study or assessment.

That’s one of the things we’ll be working on over the next couple of months. We have a sprint going on to look at the Security Architecture Catalog and figure out the lead times for each of these capabilities, a rough magnitude of their costs, who’s working on them, who should be working on them, and whether the U.S. government should own these capabilities. We’ll publish that information after the sprint is complete.

What I might add is that the Freedom of Action strategy provides a set of decision criteria you might use to look at a policy proposal or some of the capabilities from the Security Architecture Catalog Matt was just talking about. It lays out a set of questions you would ask about any given policy proposal. One main question is: Which of the archetypes does a given proposal foreclose, and how does it do that? That might be a real cost that’s not just the economic cost of the capability. If it prevents you from having a choice you might need later, that’s a significant cost that needs to be factored in.

The issue of foreclosure is something I’ve found engaging to explore with LLMs: seeing ways in which given policy proposals might foreclose different strategic choices. I think that’s important, as is the extent to which those options are permanently foreclosed—whether your choice to pursue a certain policy was irreversible, or whether your policy could be reversed in time.

You talked about export controls as an example. One might argue that they foreclose certain forms of co-development, but also that this foreclosure isn’t necessarily permanent. These policies could be reversed or shifted in certain ways.

That’s one criterion you might look at: whether a given policy forecloses other strategies, and the cost associated with it. But also related is what strategies a given capability opens up. That’s what Matt was just discussing, and some of the things Joel was talking about—the most important things we ought to be prioritizing because they help us advance the entire strategy space across all 7 strategies.

One way that a given policy or capability might do that is by providing information. If it helps us better track the pivots or provides information about capabilities or what China is up to, that provides real value. That might offset some of the costs associated with the capability. If it just allows you to make better decisions down the line, those upfront costs might be very much worth it.

The last criterion that grows out of this way of thinking is: What does it signal to the rest of the world, to other geopolitical actors? That might be where your point about export controls is relevant. Maybe the real cost was that it signaled we weren’t interested in some forms of partnership. But are there other things we could do to signal that we’re interested in various forms of partnership if the time is right, we have the right verification mechanisms, and we’re not worried about someone defecting from their agreements?

Those criteria—looking at the foreclosure of strategies, what strategies are opened up, what information we might get from a given policy, and what that policy signals to other geopolitical actors—are how I think through what we should prioritize, what policies make sense, and what the costs are.

Do you have any other thoughts? A couple of quotes I pulled out of the report were on international relations and signaling: that the U.S. should “signal to hold deterrence and cooperation credibly open at once.” Another quote I particularly appreciated, given my relative dovishness, is that we should “weigh carefully any action that would harden rivals’ resolve.”

My sense, having spent a couple of weeks there this summer—and I’m by no means a China expert—is that people were saying, “Yeah, these chip restrictions are hardening our resolve.” That seems to be happening at least somewhat. I agree that you can reverse that policy and start selling chips again, or come to some new agreement if circumstances or attitudes change. At a high level, how would you think about balancing deterrence and cooperation at the signaling level? How do we even know if we’re getting that right?

That’s a great question. The idea here is that you want to demonstrate capabilities selectively and share the existence of capabilities selectively. Often, you’ll want to withhold your intent behind capabilities. Sometimes, you’ll want to signal your intent behind capabilities.

For example, if you have a credible, independent evaluation and verification regime, it serves both sides of that posture and doesn’t foreclose anything. Building this capability, announcing it, and sharing it can be really useful and preserve your optionality. In areas that might be more confrontational, like offensive counter-AI tools, you might not want to disclose that you have them. Or you might want to disclose that you have them, but not explicitly state their purpose, because that type of signaling can foreclose some of your options.

Can we stay coherent enough to do some of these things today? When I look at China, I don’t know what’s going on behind closed doors, but they definitely present a pretty unified front. Whereas we’re counter-signaling. We’re signaling to ourselves so much that I think it’s very hard for us to have any sort of coherent signaling strategy on the international stage. Does that feel right to you, or do you think we can do that more than I give us credit for?

I think we can do that. It requires some policy discipline, but sometimes that ambiguity is actually beneficial. If you look at the ambiguity around Taiwan and what U.S. intentions would be there, I think some of that ambiguity can serve our purposes.

Very briefly on this, we have the example of the Cold War, when our open society, the freewheeling public discussion, and the critiques of leaders were sometimes seen as a disadvantage. But in retrospect, I think that actually made our system much more resilient and much more able to take on different possibilities. There was more legitimacy and support for the decisions that were ultimately made. I personally see that as a huge advantage. I know it can look a little messy, but ultimately I think that’s to the good.

Going back to the other question about deterrence and cooperation, I think there’s a way in which the freedom-of-action strategy is kind of asking us to do it all. It’s like having your cake and eating it too, or walking and chewing gum at the same time. I think in some sense that’s true, but that doesn’t mean this is feckless or not particularly demanding.

We think there’s a lot that can be done to advance both deterrence and cooperation. Matt was just laying out parts of the evaluation ecosystem or verification tools. Those are very clear examples that support both. Another example is incident response, both domestically and through crisis communication channels with adversaries like China or with other countries around the world. That, again, supports multiple strategies at once.

We don’t have these up and running yet. Even if we only advance the things that serve the entire strategy space, that gives us quite a lot to do. Then we don’t have to wrongfully commit to something that would be premature, deprive us of options later, or perhaps signal things to China that would be overly provocative. We can do the things that enable both deterrent and cooperative approaches at the same time.

Nathan, on this—

Amen.

On this point of weighing carefully any action that would harden our rivals’ resolve, I think this is really important. Moving towards some sort of suppression capability can basically send a signal to your rivals that the window is closing for them to take action. It can push them to accelerate.

There’s been an argument that chip controls basically fed China’s push to create its own domestic AI GPU accelerator market and start producing those, because now it’s existential. They can’t procure GPUs on the international market as much as they could, so they have to produce them themselves. I think there’s some logic to that.

I think one of the most dangerous things the United States could do, if there’s a dominant strategy being run, is signal that the monopoly created by that dominant strategy would be used to threaten the Chinese Communist Party. Anything that turns existential for a particular rival puts real pressure on them, hardens their resolve, and focuses their efforts. That’s where you could get dangerous consequences. You want to be very careful when you’re signaling, to make sure the message is received in a particular way and you’re not boxing your adversaries into a corner.

I was definitely hardened in my view that they’re going to have a chip industry at this point, and let’s hope it hasn’t hardened too much beyond that.

Are there other hidden-gem policy ideas that you think deserve more attention right now? My friend Zvi always hammers on the Jones Act. Repealing the Jones Act is his sort of quixotic quest in life. Are there things that aren’t the headline but are favorite ideas among you guys in the know—things that are neglected when there’s not enough time to get into the weeds, like we have today?

I’ve written on some of this already, but I do think a lot about both how we’re going to respond to incidents and prevent incidents from happening. Some of this gets a little conceptual, relating to what it means for us to hold AIs at risk—to ensure, just as we are vulnerable to them, that they remain vulnerable to us, going back to the conversation we were having earlier.

I think ensuring we have various capacities for shutting down models, and being able to credibly threaten them that we will do such a thing, is important. Some of that comes down to our resilience and ability to live without them, to make those types of threats credible. It also means ensuring we have the right technical approaches, the right authorities in place, and the right crisis processes so that we can act quickly.

I think all of that is pretty important, especially if we’re worried about the very high-end risks from advanced AI. Our ability to hold them at risk, to be able to shut them off, is pretty crucial, and maybe doesn’t come out as explicitly in this particular piece.

Yeah.

2. Keeping Human Agency

I’d be interested in any double-click you have on that, or even just any sort of advice for the rest of the country outside the government.

Yeah.

I just had this experience yesterday. I’ve put a lot into my own personal AI infrastructure over the last nine or ten months, and it’s gotten pretty good. I just happened to be giving a couple of talks this week, and one of the things I’m going to leave people with is, “Hey, here are a couple of public versions of my setup that are sort of depersonalized. Not that you want to copy them, but go talk to your agent about it. Maybe you’ll copy some. Maybe you already have some. Whatever—you can maybe learn something.”

I’m also going to offer a sort of agent-to-agent exchange, which I think could be interesting. But as I was depersonalizing my setup, Claude decided to rotate the passwords on the agent console software I use to talk to the agents, which is increasingly how I’m doing all the work. There was a moment when I was locked out of my own system.

Mm.

I was like, “Oh, shit. I don’t even know where that password is. It just changed it. Claude can change my passwords.”

Yeah.

Claude isn’t fortunately scheming against me at this point, so I just went to a terminal and asked it, and it gave me the password. Then I logged back into my system. But it was a little wake-up call: I’ve already intermediated between myself and the actual levers of action in the world so much, just in October 2026, that if Claude turned bad on me, it could go really badly. It could lock me out of all my stuff. This is getting real for me already. Do you have any tips or best practices you want to—

Yeah.

—propagate, or habits of mind you want to cultivate in people?

I think about this less in terms of which passwords I want Claude to manage, and more in terms of what kinds of analytical work, writing, and other types of critical thinking I’d like Claude or other LLMs to do. I’m also a teacher, a professor, and I have four kids. I think about which capacities it’s important that they maintain, and the best practices for how they interact with AIs.

What I advocate for is always starting from your own considered judgment and point of view. Before I ask a model for its opinion on something, I want to already have real clarity about what my question is and what good or bad answers would look like. I want to go into it with a clear vision of what I’m hoping to get out of the interaction, then let the model cook and expand my thinking, build the tool, or do all the things I wouldn’t have been able to do without it.

At the other end, that critical judgment and faculty need to come in again. I want to be thoughtful about the response. I’m not just putting my name on it and passing it forward; I’m really thinking thoroughly about it. The human comes in at the beginning, and then it comes in at the end. I want to inculcate that cycle in myself, my students, my kids, and the like.

In terms of more institutional decisions about when we should rely on AI—for example, critical infrastructure or managing your desktop—it’s hard for me to suggest one best practice for everyone. I try as much as possible to ensure I can continue to operate without these things. Some of it is just the cutesy stuff, like trying to navigate to my friend’s house without looking at GPS, to flex that type of muscle.

More institutionally, it’s about thinking through human backstops: fail-safes that allow manual operation of different systems, and still having paper copies of health records, for example. All of that helps ensure that if there’s an incident, we can perhaps withstand it. That’s where resilience comes in. It also makes our threats to shut these systems down much more credible, because it makes doing so a lot less costly to us societally and on a personal level.

I think of it as being very clear about which dependencies our institutions have, thinking about what we can do to lessen those dependencies or at least have fallback options, and then exercising and practicing them to make sure they’re really available when we need them. I haven’t perfected this. It’s very much a practice—something I strive to do and often fail at—but that’s how I think of it.

It almost seems like we need an AI Sabbath day, even if we’re not observing a religion. We need a day a week when we just don’t do AI stuff.

A hundred percent. You can extend that to other examples. Already, schools in LAUSD and even New York are AI-free. That was an explicit choice made by the school districts, and I think it’s along the lines of what you’re suggesting: AI-free spaces.

Do we need to establish more of these? Community centers where AIs aren’t welcome, where human activities are happening—people exercising, convening, and doing other very human things—free of AI mediation? That way, we can practice those human abilities and be prepared for a situation where we might not have AI or might need to stop an AI from intervening.

I think a lot about those types of things. An AI Sabbath, or a human reserve labor force for critical infrastructure, hospitals, or the like, might be important. I try to take an AI Sabbath. I often fail, unfortunately, but I do my best.

That might be something I need to start.

3. Beyond the Singleton Model

When you’re talking about things where I think there needs to be a little more attention, one of the things we were looking at was the typology of superintelligence. When we were assessing all the strategies out there, we found that most of them focused on the idea that superintelligence would be very centralized—either a singleton, or maybe the United States has one and China has one.

Those strategies became very brittle when you started talking about some of these other models for superintelligence. There could be lots of different superintelligences out there at varying levels of capability, very widely distributed. There could be superintelligence that’s very narrow, where you use architectural controls to make these more like an oracle, without a lot of generalized intelligence and with lots of checks and balances.

But one of the more interesting ideas was that you could get superintelligent types of behavior from agent swarms that are sub-ASI or even sub-AGI. We did some initial research on this, and there’s really just not a lot of thinking about it. I think what we’ve seen with the agent swarm hacks this summer is maybe a preview of what’s to come. This is less like dealing with a singleton or even a constellation of superintelligences, and more like looking at these things as an ecology.

Some of our colleagues at RAND—Ed Geist and a number of other authors—put out a paper called “The Ecology of AI Risk.” I think this is an emerging area of research in artificial intelligence, and it is enormously important as we start getting more and more agents out there. They may be interacting with each other, forming different coalitions, and having these types of emergent behaviors that we don’t understand.

We’ve spent thousands of years figuring out all the checks and balances for how human agents interact with each other in the world. We have very little time and very little experience with how these types of agents are going to interact with each other in the world. I think this is very, very important for how things are going to turn out.

Yeah. Ecology of AIs—it’s wild times. I’ve had a very similar impulse. Of course, China is always at the end of a lot of AI conversations, but I also sometimes think about Prime Intellect. What’s going to happen at the distributed layer, in a community, in a participatory fashion, in some stories or timelines, may be just as important as what the Chinese government decides to do.

4. Where Not to Deploy AI

Another question, kind of the opposite side of the coin from all the investments you recommend that we make to maintain option value: Are there any things where we have enough information right now that we should all be able to come together and agree that we just shouldn’t do them? One that I like to float, which you may or may not accept, is that I don’t think we should train AI generals. If we train AIs that are capable of winning military campaigns, the foreshadowing is so obvious. We’re training the thing that is potentially going to be fighting us at some point, and I don’t want AIs to be that good at fighting military campaigns. Yes, they might hypothetically beat our adversary, but I worry about our ability to fundamentally control that technology. Is there anything you’d put in the category of, “Yes, this is obviously such a bad idea; we definitely just shouldn’t do it”?

If I may, I’d like to understand your example, Nathan. Is the idea that we shouldn’t even train such a model or agent, versus training it and perhaps leveraging it, while humans still fundamentally determine whether to deploy it or act on the recommendations from this general? Is your objection to the training itself, or to how we deploy this agent?

I would probably stop a little bit upstream of the final decision to deploy. If we suddenly hand that over to military decision-makers, it might be pretty hard to resist.

I’d probably say the same thing about training AIs to go out and make money on the internet, and just rewarding them based on how much money they make. That seems like a pretty clear way to get all kinds of lying, cheating, and conniving AIs. They’ll be economically valuable, but with the techniques we have right now, they may also be dicey. As you alluded to earlier, methods not yet known could come online and slot in with this to make it okay. But what I’ve learned from Andon Labs-type stuff is that even the models we have now, which haven’t been trained in that way, are sometimes pretty dicey to deal with.

Yeah.

So I just think if we go down that path, we’re going to have a very hard time not deploying them.

Got it.

We’re going to have a very dangerous situation where it’s like, “Shit, the military’s now run by the AI. Now we have to fight that?” Whether it’s a—

Right.

—really tough place to be.

Yeah. I’d be interested to think more about this one. To my mind, the line is more around giving this agent access to various tools or capabilities that allow real-world impact.

In the making-money case, I’d feel like that ship has sailed. I don’t see how we can pull back. These companies are designing systems that are supposed to have some economic value and run over long horizons, and so that one might be harder. But in the military context, we’re accustomed to complying with various restraints and obligations for the good of humanity. These are codified in the law of armed conflict and international humanitarian law.

I think those same types of legal constraints should apply to our use of AI in military applications. They require, for instance, reviewing operations to ensure they don’t have a disproportionate collateral effect, distinguishing civilians from combatants, and ensuring that someone remains accountable for these decisions.

To me, that suggests that even if you’re training these agents and getting some value out of them in the planning process, we still need to create institutions where humans remain fundamentally accountable. We have a practice of that in the military, with clear hierarchies, and we just need to double down on those. That might mean bringing in more military lawyers to review operations and more people thinking about what’s appropriate. I wouldn’t necessarily say we’re completely unprepared for such an eventuality. We just have to go back to what’s worked historically and ensure that those existing institutions adapt and are relevant for today’s challenges.

But sorry, that’s not answering your question. I’m just interested in that as a side issue.

Yeah, that’s good. I offered it for you to beat it up a little bit. Anything else come to mind in the “thou shalt not” category?

Yeah. One thing we talk about a lot is that rushing to put AI into your nuclear command and control is probably a very, very bad idea. This needs to be done in a very deliberate fashion, if at all, with a lot of demonstrated safeguards. Anything touching on the nuclear domain is an area where you need to proceed very, very cautiously.

There’s also the use of agents in military affairs. Some research has shown—and I think this is all very anecdotal—that some models, when put into simulations, tend to escalate a lot. If you have agents that are reward hacking, with escalation on the table, that seems like something you want to be very, very careful with, too.

Yeah. Chilling, honestly. All right, last one. What are you tracking most closely?

Just to revisit the structure of this whole discussion: We want to maintain freedom of action. To do that, we’re going to make parallel investments in a lot of different things. The reason is that we have seven archetypal strategies, and it’s too early to decide. We have these five pivotal questions that we’ll have to work through and get answers on in order to decide what strategy we ultimately commit to.

With that in mind, what are you guys tracking most closely right now? What do you consider the likely next data points that will start to shift your sense of how these pivotal questions are going to shake out, and, downstream of that, what strategy we want to adopt?

The things I’m tracking are probably not a surprise; they’re probably what a lot of the community is tracking. One is continuing to see developments along the pacing proposals: what actually gets enacted, what kind of access independent evaluators actually get internally at the frontier companies, what shared standards these companies decide on, and whether various security controls are sufficient to manage internally deployed models.

So far, we’ve seen that the companies have not created sufficient security internally, and I’m very much watching to see whether they’ll get a handle on that. If they don’t, what kind of accountability or consequences can follow? I’m also watching for signs of recursive self-improvement, or what the trajectory of the AI R&D space looks like, since that can really change timelines across the broad range of decisions we’re tracking.

The last thing I might mention, of course, is U.S.-China dynamics, both from the broader diplomatic standpoint and more narrowly on AI issues: whether some sort of incident-management channel can be created, or whether there’s more shared understanding of the risks. That’s the stuff I’m watching.

Matt?

Yeah, I would just add to that. I think the important things to watch are—as I said—“danger close” is probably the most important question we can answer right now.

And for that, we want to really be looking at the pace of recursive self-improvement. How are the labs managing that? They’re doing their own pacing. Are they actually able to release new capabilities safely? Are we continuing to have more and more escalating incidents around agents, or are we effectively developing standards around how agents are used?

I think one of the interesting things to look at right now is cyber capabilities. Mythos was released about nine months ago, and typically open-source, open-weight models lag the frontier by six to 18 months. Those types of capabilities are now starting to be available in more of these open-weight models. We’re going to need to see how malicious actors and state actors are using these capabilities as they become more and more available.

When you talk to people in the cyber world, I think there’s a lot of concern about the next couple of years. Over the long term, there’s a view that AI might be positive for cyber defense, but a lot of that depends on whether defenders can actually take up the frontier capabilities quickly enough. If there are open-weight capabilities out there that enable malicious actors, terrorist groups, and criminal groups to go and hack, it may be that Big Tech is fine, but what about the small and medium enterprises that aren’t able to take up the latest frontier models as quickly?

I think we may be in for a rough patch in the cyber world over the next 12 to 24 months, and hopefully it’s just a narrow rough patch. That’s something I’ll be keeping a close eye on.

Yeah. Offense can stay dominant longer than you can stay in security through obscurity, I suppose.

Yeah.

Well, this has been great. I really appreciate all the time and the hard work that goes into making this. This is really one of those documents I would encourage people to share and go through the hierarchy, because you actually took the time to write the short version, and I think the clarity really comes through. I appreciated that a lot. Anything else you guys want to touch on that I didn’t think to ask, or that we just didn’t get to, before I let you get back to work?

I would just add one thing. I was just at The Curve last weekend, which is one of these big AI confabs in Berkeley, and I presented on the Freedom of Action strategy. A lot of the feedback I got was that this really enabled people who were newer to the AI debate to understand the different dynamics at play in looking at and choosing strategies for advanced AI.

I think folks out there in the audience shouldn’t look at this as just an academic RAND report. People kept citing this table we put in there that outlines the five pivot factors and how they lead you potentially to the different archetypal strategies. They said they were going to take it back to their stakeholders and constituents to try to explain how AI strategy works.

For folks out there who wouldn’t normally read a RAND report, I would encourage you to take a look at that and at the pivot factor table. It might help you understand more intuitively what these strategic choices are all about and what the underlying assumptions for most of the AI strategies out there are based on.

Yeah, I agree with that. For people who’ve been with us on this beat for a long time, all these things are kind of in the air and familiar. I was at The Curve too, actually. I’m surprised we didn’t have a chance to meet there.

For folks who are catching up now, I think the structure and the way you guys present this information is an excellent primer. I do think it’s definitely something that should be forwarded on to people who are just entering the discourse at this possibly late hour. I guess we don’t know yet exactly how late it is. Download it, and you’ll be glad to have something you can send to friends when they need a primer. I think it’s excellent for that.

Thank you.

Joel Predd, Ben Boudreaux, Matt Chessen, thank you guys all for being part of The Cognitive Revolution.

All right. Thanks for having us, Nathan.