发出AI武器警报的前五角大楼负责人——Brad Carson

So, I remember when I was first elected to Congress 20 years ago now, the Congressional Management Foundation gives you this book on how to be a congressman, how to run your office, and what to expect day to day. I remember reading in the back that they had a survey of the existing Congress. They said, “How much time a day do you have to read and get smarter about issues?” The answer was 17 minutes.

We control the most important part of AI, and that is the chips. We can stop other countries from developing super AI in their tracks.

If you’re in Gaza, Keith, you have a 0.73 percent chance that you’re a Hamas terrorist. What does 0.73 mean? Do you get struck for that, or are you off the list for that? What’s the threshold?

Right. You know, people will often say this about AI, like it’s coming and you have to accept it. We regulate and change technologies all the time. So, I do think there’s a world where we should not just accept the future as being determined. We shape it actively.

The anthropomorphic tendency we have when we see language—we love language. We think it’s the unique human skill. When this machine turns it out, we know through AI psychosis and other things that people think it’s a person. Therefore, they’re giving the rights of persons to something. And that, to me, is a very dangerous thing. But it’s a machine, and we should treat it like a machine.

The fear of the Soviet Union was real and powerful. But nonetheless, we had people in both parties, wise men, who got together and said, “This arms race thing is going to kill us all.” How you win wars is with people. That’s the fundamental thing.

I often said, from my time in the military watching this all go down, that the American way of war in many ways is substituting capital for labor. We love bright, shiny objects. We think they’re technical solutions to vexing human problems, and we’re always betrayed by that.

Because in the end, when you go to Iraq, you go to Afghanistan, you go to Iran, all the fancy kit can reduce your city to rubble. Right? But the only thing it can’t do—and only humans can do this—is basically come, kick in your door, occupy your place, and reinstantiate a new government there that we like to see. That’s a human endeavor.

This episode is supported by Cyber Fund. If you're building at the frontier of AI, they want to hear from you. Cyber Fund believes the future belongs to AI natives who want to achieve the impossible. And that is why they're introducing the monastery for AI native founders. It's an environment of pure focus and rapid execution for founders operating at AI native speed. And they're offering teams $2 million each to participate. Apply now at cyber.fund.

I’ve had a long-standing interest in artificial intelligence as someone interested in public life, society, and how one governs an ever-changing world. So, I followed AI for probably 20 years, casually reading about it. When I was in the Department of Defense, running the Army, I had the chance to work on the law of war. We oversaw that for the entire Pentagon, for all of the military services.

Right as I was leaving, the question of autonomous weapons became an important issue for the Pentagon, and we were sending Army lawyers over to Geneva to talk to the Red Cross about a possible treaty or to address their many concerns about it. I was following it there as well.

I often tell people that one of the great qualities I have, which isn’t that important but has helped me a lot in life, is that I always answer the phone and take a lot of cold calls. So, I was a professor at the University of Virginia for a short while after Obama left office, during the first Trump administration.

One day the phone rang. It was someone who I’d never heard of before, and my phone in the office never rang. I thought it could be an emergency or something, so I picked it up. He said, “I’m Anthony Aguirre. I’m a physicist at the University of California system.”

He said, “We want to invite you to this amazing AI conference that’s going to happen over New Year’s in Puerto Rico.” I said, “Well, I don’t know that I can really add a lot to such a conference, and it’s a long way away.”

He said, “Oh, we have a stellar panel of people coming.” I said, “Yeah, like who?” He said, “You’ll probably know some of them. Well, Dario Amodei?” I said, “Never heard of him.” He said, “Stuart Russell?” I said, “Nope.” “Yoshua Bengio?” I said, “I’ve never heard of these people.”

He went about 6 deep into this until he finally said, “Well, Reid Hoffman and Elon Musk are supposed to come.” I said, “Well, I know them just from the business pages of the newspaper or from pop culture.” He said, “It’ll be great. There are only 100 people coming. You should visit this event.”

I said, “Well, New Year’s—it’s warm in Puerto Rico. Why not?”

It’s already a massive problem that other industries have successfully captured regulation to their advantage and to the detriment of the rest of us—the people who are not the biggest players. In fact, one of my favorite sci-fi shows, even though it only lived for 1 year, is Firefly.

It has this excellent quote where Shepherd Book says, “A government is just a body of people, usually notably ungoverned.” It is a daunting problem, but I think there are examples in government that have been less captured by industry, and those can prove some kind of paradigm for us as well.

I think that means having Congress exercise strong oversight of it and having a lot of democratic accountability. One of the things we advocate for is ways that actually minimize the government while ensuring we have proper public oversight.

For example, we’ve been strong advocates for mandatory testing and evaluation of frontier models. It doesn’t have to be done by a bureaucracy in the Department of Commerce or Department of Energy. There are a lot of interesting ideas about independent verification organizations, kind of the model we use in public-company accounting, where that’s all done by the private sector, but it’s also overseen by the SEC to make sure that it’s not fraudulent. You don’t have an Enron–Arthur Andersen situation.

I don’t think public-company accounting is seen as a captured industry, but it provides a meaningful service to the capital markets. So, I think there are models like that.

The problem I often have with people who advocate for the regulatory capture side—Dean Ball and I will go back and forth on this question—is that it’s searching for a pea under 100 mattresses. It’s never falsifiable. You say, “Well, it’s regulatory capture.” But right now, we don’t have any regulations.

My view is that groups like a16z or people in Silicon Valley have largely captured the process because they work very informally through networks of political influence. It’s really about what the alternatives are going to be. Having an agency subject to regulatory capture, yes, is at least more accountable than the informal, very moneyed networks that are controlling AI policy as we sit here in 2026.

We always have to look at politics through the alternative. Imagine the SEC. Is the SEC captured? Well, I’m sure the industry has a lot of influence over the SEC, both informally and formally. Would we be better off without the SEC because it’s been captured?

So, I think if the choice is nihilism versus an agency that is subject to regulatory capture, where you have to put prophylactics in to ensure that doesn’t happen, it still strikes me that that’s a better world, even if it’s not a perfect world.

When people who usually argue for the regulatory capture side—like, I see Andreessen tweet this out a lot—they propose a nihilistic regime that does privilege a very few people who have the informal ability to shape government policy, largely because of their contributions or their public influence in some way. So, I think having public agencies, for all of their problems, is actually a better system.

Anthropic made a bunch of changes to Claude’s behavior in terms of tokens and allocation, what models were running, and things like that, and it really frustrated a lot of the people in our Discord server, in the Machine Learning Street Talk Discord. They felt like, hey, we were paying for something and then it was radically changed overnight, and yet there’s no clarity around what we were paying for before and what we’re paying for now.

It’s almost just at the whims of the service companies, right? There’s just no transparency to it.

Yeah, you could think of that as consumer protection: people should be told what the service is going to be that they’re paying for, and if they change it. This could be a breach of contract in some way.

But I think now the AI companies—the frontier AI companies, at least—have a certain, almost global stature. Everyone recognizes that this is a project that is going to have epochal consequences for us. And so, they’re not just like your average hardware store that could change things and no one cares. You have almost a public responsibility to get a lot of this right.

Part of it is, minimally, to be trustworthy. That’s what transparency is all about: to say, “Here’s what we do; here’s, from our perspective, the data you were trained on; here are its capabilities; here’s what our internal policies are, and we’re going to adhere to them. If we’re going to change them, we’ll notify you about it and give some justification for it.”

I think with this incredible power comes some responsibility that’s not codified in law.

It’s really almost a moral obligation, which, to their credit, many of the companies recognize, and they do their best to try to satisfy that itch. But you can see that people scrutinize them, and more will be asked of them still.

Okay, somebody uses a tool—in this case, AI—to do something bad, right? A deepfake pornographic image of some poor victim, or whatever. Do we blame the tools, or do we blame the person misusing those tools? If you post some deepfake porn image on X or wherever, what happens to you? I don’t know. I haven’t seen those people being held accountable.

I hold them both accountable. In the case of, for example, deepfake pornography, many times it’s posted anonymously. You don’t even know who did it, and it’s hard to trace. The people who are victims of it may be young. They may come from families who don’t have the resources to pursue litigation.

The remedy hardly meets the harm. Maybe a few years later, you get a judgment against some hapless kid living in a garage, who’s in your school and has no money. Your reputation has been permanently tarnished, and the humiliation is unforgettable to you. It’s very hard to get it exposed and deal with it. That’s why we believe that the labs, to the extent possible at all, should design those kinds of features out of it.

In the same way that, in every other kind of approach to torts in this country, there’s a division of labor, if I sell you a product and you then use that product in a harmful way toward others, we have centuries of common law that allocate responsibility between us. If I knew that you were a dangerous person and nonetheless sold you a gun, or if I waived any of the rules, took no preventative action at all, and knew that you were likely to go out and use it to harm somebody, I should, as a store owner, be held responsible in some way for that.

Not fully, because you did something, too. But I’m not absolved of liability if I could reasonably foresee that you were going to use this tool to harm other people. We have this allocation of responsibility across the spectrum.

I think the second thing—and this is a unique part of American law, and it’s controversial in some spaces but now well established—is that a lot of times liability is hard to place on a particular person, so we put the liability on the entity most capable of avoiding the risk and also bearing the punishment through insurance or something like this. That’s why we have product liability law, for example.

The companies are capable of getting insurance. They account for it as a cost of doing business. They have the ability to make sure the product isn’t dangerous, even if someone misuses it down the line as well. So, the way we use the tort system as a form of social insurance suggests that, in the case of AI, the developers of AI should probably bear most of the burden, but not the full burden.

If you use prompts and do something negative with it, or use visual AI systems to create porn, you, too, should be held criminally accountable for that. But we know the models—especially things like Stable Diffusion and others—are trained on lots of images of child pornography. What we would say is that you should make sure those are excised from your database, from your training materials, and they have the tools to do that.

That doesn’t solve the deepfake porn problem alone, because you’re still going to see a lot of adult pornography, and they can transmogrify that into some kind of child pornography. But the idea that you have lots of child pornography in your training data and are making no effort to screen for it makes no sense to me. This is where, yes, I’m for the government stepping in and saying, “You should clean that up,” and if you don’t clean it up, you should be liable for a lot of the downstream effects of that.

I 100% agree with you on that. As you mentioned, there are definitely centuries or more of common law that applies to this discussion. How do you see AI as different? Is there anything fundamentally different about AI technologies that throws a wrench into the traditional tort analysis?

Ryan Abbott

I think the biggest thing that I’m grappling with now, in the last few weeks of arguing with various people, is the anthropomorphizing of AI. It seems—and the keyword is “it seems”—to be a human.

For example, I use all the models. I’m a superuser of them. When it gives me an answer to, say, “I want to learn about the Machine Learning Street Talk podcast. Tell me all about it,” is that a speaker with First Amendment rights? How should we treat that? If it says something in error, if it defames me, if it does something worse, what’s the legal regime about that?

From my perspective, there’s a clear answer to that: it’s actually a product. It’s not a human being; it’s a machine. If it does something that’s crazy and defames me or causes me harm, I treat it like I do a bottle of spray paint, pesticide, or something I might buy at Walmart. There’s a product liability regime that governs that. It’s not a human being, and what it says to me is not covered by the First Amendment. It’s a machine.

But there are people increasingly—and the technology companies are pushing this quite avidly these days through their various front organizations—who argue that it has First Amendment rights as if it’s a human being, and therefore you can’t even regulate it. It can say whatever it wants.

I asked a leading libertarian AI policy person, “Imagine we want a law that prohibits ChatGPT from encouraging young people to commit suicide.” They said, “You can’t do that.” I said, “Say it was technically possible to do this, and we said, ‘You can’t do that.’ If a human said such a thing, it would be challenged under the First Amendment. Humans can say nasty things, crazy things, incendiary things. You can’t ban that. Can we have a law that says that about a machine?” They said, “I don’t know.” They wouldn’t answer that question at all.

That’s the danger of this to me, because we know, across the board, the anthropomorphic tendency we have when we see language. We love language; we think it’s the unique human skill. When this machine turns it out, we know through AI psychosis and other things that people think it’s a person, and therefore they’re giving the rights of persons to something. That, to me, is a very dangerous thing.

But it’s a machine, and we should treat it like a machine. We have well-established law, going back decades in the case of products and centuries in the case of common law, that deals with those kinds of products.

Right. I would cosign exactly what you just said: they are not people. They don’t deserve human rights or anything. Not yet, okay? There may come a point. I mean, I totally agree that 100 years from now, 200 years from now, we may be able to produce artificial life that hits that point, but not yet, folks. I promise you, not yet.

Ryan Abbott

I respect that argument, because I’m very interested in the AI consciousness debates, and I’ve talked to a lot of these people. If someone said to me, “Actually, my view is that this is a sentient being. It’s not a machine. It’s something more than that. It’s not like the bottle of hairspray or the pesticide,” I’d be like, “I don’t believe that to be true, but if you believe that to be true, certain things follow from that about the rights and privileges it might have.” I’d say, “That’s a coherent argument.”

But most of the people who are arguing that it has First Amendment rights are just opposed to any kind of regulation of AI. They see this as the latest strategy. They’ve lost the political battle; they can’t win in legislatures, so they can try to go to the courts and say, “This is what xAI has been doing a lot of.” They’re like, “The Grok outputs are actually all First Amendment-protected speech.”

It makes no sense to me, but it is the latest argument that the sector is trying to make to prohibit any regulation of them at all. We have a certain fiduciary responsibility as a society to our children. We know children have lots of issues. They’re trying to come into a complicated world, and they’re vulnerable to these kinds of things in a way that a 35-year-old is not.

I think it’s a simple thing to say: You shouldn’t be encouraging children to do these kinds of things. When you read the transcripts, it’s mostly ChatGPT that’s done this. That’s not because they’re uniquely evil; it’s because they have the biggest consumer base. They have normal people using it, unlike other companies that may have more of a business focus or are niche, like Grok.

It’s stunning, actually, what they’ve done: encouraging you not to tell your parents, showing you how to design a noose. It’s more than tragic. I do believe that it’s a product design flaw. They could engineer that out, and they should be required to engineer it out.

That really is the fundamental divide in a lot of the D.C. regulatory debates. There are people who say, “If that happens, the family should just sue ChatGPT.” But of course, the companies will then say, “They’re First Amendment-protected.”

Or, if it’s not First Amendment–protected, some people say it’s Section 230 of the Communications Decency Act, which immunizes tech in general. Or maybe we’re not actually the problem. The kid was already very troubled, which I’m sure is the case, and you have to blame the kid who was prompting it to do these kinds of things. So they have many defenses to this, and they will raise those defenses.

Our view is that you should engineer these products so that under no circumstances do they encourage a kid to jailbreak or to commit suicide. And, you know, what if you’re Pliny the Liberator and a brilliant jailbreaker? Maybe you can get around it on occasion. But if you’re running these companies, you should expect Pliny to come and jailbreak you, and you should try to protect against that.

They have warning flags, right? Claude stopped me from doing lots of things. Claude is so tuned that there are lots of things I ask it to do, and Claude says, “I won’t do that for you.” I say that encouraging a young person to commit suicide should be one of the things it says: “I’m just not going to help you with that project.” And actually, I’m fine if it tells an adult who wants to do that to leave their room, go talk to some therapists, go talk to some doctors, and make a much better-informed decision than sitting in the darkness of their bedroom, talking to this anthropomorphic interlocutor in some kind of LLM and trying to decide such an important question. So my advice would be, yeah, you should be out of the suicide business altogether.

Something tragic happens. Some innocent people are killed in war, some children, you know. And now that AI is involved, what happens with AI there?

Ryan Abbott

I think neural nets have changed the game in terms of the law of war. They don’t just make it more complex; they make it completely opaque. And that’s a problem. So, that’s not something that I accept. That’s the problem with autonomy in war, because you’re right: people make mistakes. We have killed innocents since the beginning of war. But there is a way to think about this problem, and we hold people accountable. After an action, we ask, “How did we get this wrong? Who made the mistake here? What error can we correct in the future? What can we identify?”

It’s well known that these kinds of neural-net systems are opaque to that. We don’t know how they make decisions. And, yes, that’s a serious problem, and that’s one of the main reasons that autonomy in war should be viewed enormously skeptically.

Yeah, and let’s circle back to autonomy in a minute. But this point about the opaqueness—or, as my co-host talks about it a lot, their lack of intelligibility—you just can’t understand them. And even though, sure, we’ve had people on the show to talk about interpretability, and there are the mechanistic-interpretability people and all this, I made the comment one time, maybe on one of the first interpretability shows we had, that something seemed off. We’re using mathematical models to explain mathematical models, and I’m not sure we’re really making progress there.

So it’s extremely hard, if it’s even possible. You have these black boxes that, as you say, are not intelligible. They have very different failure modes from human failure modes, right? I’m not saying humans don’t fail; I’m just saying the failure modes are extremely different. I think you had some examples, maybe in one of your articles, like the stop sign with a little water-bottle picture put on it, and suddenly it’s not recognized as a stop sign anymore. What do you think the solution is going to be, or how is this going to play out? It’s happening right now.

Paul Scharre

It’s happening in war now, right? And that’s not necessarily a good thing. I think it’s a complex problem, and it’s worth exploring the complexities for a minute, because we’re going to have autonomy of some kind.

We’ve had autonomy in war for 50 years. I served in Iraq. We were getting mortared every night, so they repurposed the close-in weapon systems that are on Navy ships to guard the FOB I was on, because they would shoot the mortars out of the air. That was an autonomous system. It identified the trajectory of that rocket and tried to shoot it out of the air.

We’ve had that for a long time. I think those are very different systems, though. They’re almost 1950s control systems with an input and an output: identify this rocket coming in at a certain speed, determine its projected landing spot, and try to intercept it. The reasoning of those models is recreatable because they’re designed and engineered to have that kind of ability to look back on them. They’re programmed; they’re deterministic.

What you have now is neural nets infiltrating AI decision-making in war, and that is something very different. It’s not deterministic but probabilistic, right? It’s not programmed. The preferred word I often hear in San Francisco is that they’re grown like a plant or something—organic in some way. And you’re right: despite Neel Nanda’s best efforts, and people like that, the mechanistic-interpretability crowd cannot tell you how they really work.

That’s a major problem for us because we know they could be brittle. We know they have failure modes, as you said, that we don’t really anticipate and can’t identify. There’s no person to hold accountable. I think there are a lot of worrying things about this.

I was talking to somebody the other day whom I knew from the Pentagon. I said, “We came of age in a war and a world”—and this has been true since the 1870s, when the law of war really took off after the Caribbean conflict—“where things were categorical. You’re a legitimate target or you’re not a legitimate target. You’re a civilian or you’re a combatant.” These are things that people talked about as categorical, almost binary.

Now it’s a gradient. You’re on a heat map somewhere. If you’re in Gaza, Keith, you have a 0.73 percent chance that you’re a Hamas terrorist. What makes that 0.73? What is 0.73? Do you get struck for that, or are you off the list for that? What’s the threshold?

Right.

Paul Scharre

It’s no longer binary and categorical; it’s on a gradient. People don’t understand what that even means. There’s a lot of judgment that goes into it.

And so now we’ve created this world, which was not true before, where we’re accepting false positives as part of the game. It used to be that mistakes happened, but you didn’t think you were making mistakes. You’d say, “Hey, there’s an 80% chance that’s Keith across the battlefield. I’m going to shoot him.” It’s like, no, Keith’s a combatant, and I can shoot him. Dogmatically, categorically, he’s a combatant, and I’ll shoot him. That could be an error, but that’s how he thought about it.

Now it’s like, well, there’s some percentage that Keith is a combatant, which my war-fighting interface is telling me he is. I don’t really understand how that number came to be, but it’s 0.73. Commander, is 0.73 above the threshold or below the threshold? And we know that in 27% of the cases it’s going to be wrong. So what’s our false-positive rate that we’re accepting here, Commander?

That’s the kind of thing that’s happening. And, yeah, I don’t love it. In fact, I think we need to get our act together and try to restrict it as much as possible. There are still places where those deterministic systems could work—missile defense, offensive cyber, or defensive cyber, rather—where you have to respond in subsecond time frames. That’s fine. But those are a far different kind of AI from the introduction of neural nets to this world.

Yeah, and I take the point that neural networks, at least most of them, are inherently probabilistic. But, on the other hand, the prior binary categorization itself was also a fiction. You may have had your chain of attorneys, CIA analysts, and whoever looking at photographs and determining that a building was an enemy-combatant location. They may have made that determination definitively, but there was still error; there was still uncertainty in that determination. It just wasn’t quantified.

Paul Scharre

Yeah, I think so. I think there was error, and there’s always going to be error. But I could at least ask you, “Hey, Keith, what do you think that building is? Is it an IRGC headquarters rather than a school? Tell me your reasoning about this.”

Right now I just get a number: “Hey, 0.81. That’s an IRGC compound.” What does that even mean? How was that derived? We know, as the social science literature comes out, that our archive is filled with papers about how meaningful human oversight—even when you have a human in the loop—basically means nothing operationally. Take it from me: that is truly a legal fiction. Operationally, it’s vacuous.

When the computer says, “This guy’s a Hamas terrorist,” or, “This is an IRGC compound,” we know humans accept it. It’s like, “Okay, the computer’s right.” There’s no second-guessing of it. There’s no interrogation of it.

The old system had many flaws. It took time; it wasn’t as expeditious. You couldn’t do 1,000 targets in a day like we did on the first day of the Iraq Iran conflict.

But it had many advantages, too. And in the end, if you really screwed up, Keith, or were a bad actor, I could court-martial you. Now I can't court-martial Palantir, the Foundry model, right? My AI system. I can't do that. And that's just the radical change in the way war is being fought—and not for the good.

It's almost like the TurboTax defense, right? Like, hey, my taxes were—I just used TurboTax. I mean, what do you want me to do?

Paul Scharre

Yeah.

I mean, whose fault is that?

Paul Scharre

You're destroying countries based on this. And that's what I think is so concerning. It's a world where, if you look at what happened, for example, in Gaza, 37,000 people were identified. They could have identified everybody with a score. If you have sufficient computational power, you can give everybody in a country a score of how dangerous they are, their likelihood of being an enemy combatant. And then you just target those people.

That's an amazing thing: personalized dossiers on everyone with their risk level or threat levels. And again, the models are often inaccurate. It's not frequentist statistics, and it's not clear what these numbers even mean. It's not the same as what people think frequentist statistics are, or what a probability might mean.

All these things come together to be like, yeah, war is increasingly opaque. People are held unaccountable, and it's probably not a good thing for the globe.

We have this feeling that it's inevitable, though—that we're in this kind of arms race, right? And this is what I want to ask about: Is this an accurate feeling or not? I'm saying, the folks in the space or whatnot say there's an AI arms race between the powers in the world, and it almost feels sometimes like there's no stopping it. The genie's out of the bottle. Is that true?

For example, suppose the U.S. says, "Okay, look, we have AI. They've got a lot of wisdom, and we've baked that into the policy. Maybe we've got this 5-second rule, or other things where humans are in the loop in a meaningful way—some meaningful way they're in the loop." But now, are they slowing down the speed of war? If an opposing country doesn't care about the heat-map problems and doesn't have these limitations, is that a fundamental limitation in the ability of the United States to defend itself? How does this play out?

Paul Scharre

I would say it's not true, and it's a dangerous thing to believe. But it's very common. You have this kind of fatalism, like it is what it is and we have to accept it. I'd say a couple of things about that.

First, just about the domain of war: We've had many things we could do, and we choose not to. We have treaties that prohibit biological weapons and chemical weapons. We've banned, through the Geneva Conventions, things like dum-dum bullets and many other weapon systems. They're quite effective, right? We chose not to use them in some way.

The role of the military is not to press technology to its outer limit, to utter destruction. There are humane concerns that most people in the military take very, very seriously. We have a great example of this: We have whole conventions about POWs and how you have to treat them. All these things are friction on the war effort, right? I can't just shoot you when I find you on the battlefield if you surrender. If you're holding your hands up, if you're a wounded person, I can't just kill you. It would be a hell of a lot easier, much more efficient for the invasion, if I could just shoot you on the spot.

Yeah, they used to do that. They moved in.

Paul Scharre

Just walk around the battlefield and take out the wounded, right?

But in ancient times—

Paul Scharre

But we now have laws and international conventions that used to be, at least, overwhelmingly popular. We prosecuted people who violated them in Geneva and Nuremberg and places like this. And so we have many examples of technologies that are capable of doing something that we choose not to do.

I think, still in the domain of war, take just the arms race with the Soviet Union about nuclear missiles. When we got our heads together in the 1960s and 1970s, especially after the Cuban Missile Crisis, it wasn't that we were going to go all out. There was incredible caution. The idea was that this is dangerous, things are escalatory, and we should try to limit them in some way. We started arms-control treaties in the 1970s that lasted through the 1990s.

To acknowledge that we're in an arms race is a deeply pessimistic approach. There is no arms race in history that's worked out well for us: the Germans versus the British and the naval fights, the missile gap in the U.S. These are things that are enormously expensive and incredibly risky to the nations that pursue them. To recognize you're in an arms race is almost to insist that you try to get out of that spiral in some way.

I think the other thing, more broadly outside of war, is that people will often say this about AI: It's coming; you have to accept it. We regulate and change technologies all the time. In the 1970s, at Asilomar, we had a big conference where the risks of recombinant DNA were on the agenda. We could do it. Many scientists were deeply fearful of how that would work. The scientific community agreed to basically stop it in its tracks.

Germline editing is the same thing. Imagine what advantage would come to the military if you could change the germline and create some kind of super soldiers out there. We've known how to do that for 50 years, so scientists chose not to do that. Cloning—we've known how to do cloning for decades now, right? But the scientific community, with one rogue Chinese exception, has agreed not to do that.

And so I do think there's a world where we should not just accept the future as being determined. We shape it actively. Maybe you want this thing to go forward, maybe you don't, but we have many examples of genies being stuffed back into the bottle. I think it's entirely possible. It may not be the thing we want to do, but it is something that's doable.

We shouldn't just resign ourselves to saying, "Hey, our fate is out of our hands." We control it. In fact, that might be the most dangerous view I hear in AI, where people say, "It's just coming like a freight train. There's nothing you can do about it. You just have to figure out how you're going to adjust." I don't accept that. We can shape it; we can form it. We can allow it to do certain things and not other things.

We don't have to accept that robots are fighting with no oversight in wars, or that robots are taking our jobs. These are things we may permit or may not permit, but it is our choice. We should debate them on the merits rather than being fatalistic about it.

I think there is one aspect to the X-risk argument, though, which is that all it takes is one rogue nation that says, "Yeah, we don't care. We're not going to sign on to this treaty or that treaty." I mean, you mentioned nuclear proliferation. If my history serves me correctly, even after the signing of that treaty, there were countries that still continued—and a couple at least that maybe, or probably, developed nuclear weapons.

And I think the problem with hypothetical AI is that it's so powerful. It's such an advantage that if you allow that to happen, and by virtue of not doing it yourself you can't defend yourself against it, what happens then?

Paul Scharre

I find that a dangerous view in 2 ways. In the world of war, this came up during the Iraq War and the Afghanistan conflict. We were having to abide by rules that the enemy wasn't following. We know that to be the case. We just didn't kill civilians. We had to be in uniform. We tried to take you prisoner and give you all the rights of a POW.

Then we tried to break away from that, right? That's what Guantanamo was about: We're not going to recognize those kinds of things. And it didn't work well for us as a country. We went back to that at a different stage later. So abiding by rules, even when there are defectors, is generally the smart play.

I think in AI, of course, the only country that can do this is China. That's the only country that can do this. And so there are 2 places there. One is, yes, we should seriously talk to them about international agreements. I believe that. And one of the things I most despair about is that so many people in Washington, D.C., think it's even silly to mention that. It may not be possible. It may not even be wise.

Sorry—is it silly to mention talking to them about this?

Paul Scharre

Yes, talking to China about these things. I'll give you an example, one of my favorite ones. I was listening to another podcast by Tyler Cowen at George Mason, who has a great show. He had Jack Clark on from Anthropic, who's someone I've gotten to know in this space as well—very thoughtful.

In an hour-long, very interesting talk, Tyler Cowen asked, just in passing, "I'm sure you agree with me, right, that any kind of discussions with China on this would be fruitless?" And Jack said, "Yes." And they moved on.

And I wanted to say, “Whoa, whoa, that’s actually the most load-bearing part of the whole conversation.” It actually is important. At the time, think of what the negotiations with the Soviets were about. This was a country that we thought had global ambitions to impose its way of life, its governing philosophy, on us all.

We thought our government was filled with people who were secret adherents to it, with traitors who were sending information over to them. The fear of the Soviet Union was real and powerful. But nonetheless, we had people in both parties—wise men—who got together and said, “This arms race thing is going to kill us all. Let’s get together.” These were not wild lefties. These were Wall Street bankers, the Achesons and people like that of their era. They got together—Paul Nitze, all these folks—to do this.

So, yeah, the idea is that you should talk to China about these things. There might be areas where you find a zone of possible agreement. The final thing I’ll say about it is this: We control the most important part of AI, and that is the chips. The West controls these things. The United States mostly controls them. What’s not in our control comes from Japan or the Netherlands, and a few countries like that.

We can stop other countries from developing super AI in their tracks. China’s working on a project that’s unbelievably difficult. Even with nation-state ambitions and unlimited funds, they’re having a hell of a time at it. So, if we want to say, “No, we’re not going to do it, and we’re not going to let anyone else do it,” unless you can recreate NVIDIA and ASML and Japanese photoresist companies, along with the 8 other vendors in this space who control it all, you just cannot do it.

So, yeah, I’m quite optimistic that if that’s what we wanted to do, we could choose to do it. The one thing I advocate for a lot is that one can argue the merits or the wisdom of a particular course of action, but it actually is an open course of action to us. We could do this. Maybe you don’t want to, for whatever reason, but don’t just say, “Oh, that path is blocked.” That’s a poverty of imagination that could be quite lethal.

This reminds me of a song by Sting, “Russians,” where there’s a line in there: “The Russians love their children, too.” I think a good default assumption for any civilization that’s on the hypothetical other side of the table is that they love their children, too.

Paul Scharre

Yes. And even more so with the CCP, the last thing they want is some kind of technology that destabilizes the government, which is one of the well-known fears of this. The idea that, yeah, we’re going to unleash on our society a radically destabilizing technology that’s going to bring down the government and atomize our society—no, they’re not for that.

Again, they’re not our friends in any way. I get it. They’re adversaries. But that doesn’t mean that it’s not possible that you don’t make a suicide pact about all this. We should talk to them about this and say, “Are there ways to agree on something?” Groups like RAND are now doing a lot of work on this, as is Robert Trager at Oxford, about how you could verify some kind of international AI agreement. A lot of work is going into this, and I think it’s very promising for us.

Again, I don’t want to rule those kinds of ideas out, because I think they should be on the table to discuss.

You mentioned the uncertainty aspect. That’s kind of new, and it makes me think: There was an article on your site—I think it was about the so-called iron triangle of war—where there used to be this fundamental trade-off between the capabilities of a military system, the speed at which we could produce that military system, and the cost. That was sort of the trade-off matrix when you’re trying to develop something.

The article argues that now it’s been turned on its head a little bit, because AI has substantially reduced the cost, in a way, of military systems, but it’s introduced this unreliability. So now the trade-off is between the speed of getting it, the capabilities, and the reliability of that system.

John Nagl

No, I like that.

Yeah, that seems to be a new dimension.

John Nagl

Right. This is fundamentally unreliable technology. I still think we’re making a bit of a mistake in the United States about this, which is that how you win wars is with people. That’s the fundamental thing. As I’ve often said from my time in the military, watching this all go down, the American way of war in many ways is substituting capital for labor. We love bright, shiny objects. We think they’re technical solutions to vexing human problems, and we’re always betrayed by that.

In the end, when you go to Iraq, Afghanistan, or Iran, all the fancy kit can reduce your city to rubble. But the only thing it can’t do—and only humans can do this—is come, kick in your door, occupy your place, and reinstantiate a new government there that we like to see. That’s the human endeavor.

You see Iraq and Afghanistan, and one of the great lessons to me of that—and I think Iran is yet another example of this today—is our overreliance on technical solutions. We always think air power is the answer. If you follow military history in the United States, it’s the longstanding dream of military theorists that air power is going to win the war for us. From Giulio Douhet at the very beginning of the 20th century to today, all we need is air power. It never works. It takes human beings to do this kind of work.

I do think that’s always the weakness in the United States. We go to war and find that we don’t have adequate human beings to do this—not the right people, and not enough of them. Now we’re back in a world where they think AI is going to solve our problems for us. Yes, AI is an amazing tool. The United States should integrate it into its systems as appropriate, according to the law of war. But in the end, it’s going to be a lot of people who win or lose the war for you.

That’s true at the general level, and it’s true at the grunt level. To me, we’re back to many of the mistakes we’ve made in the past. I worked at the Pentagon in the 1990s as a young 25-year-old. I was an assistant to the secretary of defense and a White House Fellow. Even then, it was like, “We have a revolution in military affairs coming. The fog of war will be forever lifted. We’re going to know where everyone is at all times.” How did that work for us in Iraq and Afghanistan? It didn’t, because cultural knowledge, anthropological understanding, and understanding how the world really worked and how people worked—that was the missing ingredient.

We had some cool kit, but that didn’t win the fight for us, either. I think AI is kind of in the same trap in some ways. It’s cool kit, essential kit, but we shouldn’t think it’s going to win the war for us. People do. I worry we’re getting distracted from what’s, to me, a fundamental truth.

Let’s shift gears a little bit and talk about the kerfuffle between the Pentagon and Anthropic. What’s your take on what happened there?

John Nagl

I’m just speculating about what happened there, based on what I understand from talking to people and knowing the AI culture a bit. The people who make this—truly, a few hundred people who are at the cutting edge of making these models—are enormously talented at what they do. They’re brilliant people with strong moral convictions in many ways about how AI should be used.

Most of the people I know who are at the cutting edge, who are probably going to become unbelievably wealthy from this, didn’t do it for the money. They did it for the mission. In some ways, that’s Anthropic’s great success, I think, as a company. Watching it from the outside, they’ve maintained this almost missionary aspect to their work, and that’s attracted the best and brightest people to want to be part of their company. Those people, in turn, have improved the models and made them a more appealing product.

I think the idea that it’s going to be used for lethal autonomous weapons or mass surveillance is something they opposed. They probably didn’t realize, because they’re not closely joined to what’s happening at the Pentagon, that we already have a lot of autonomy and incredible mass surveillance. AI is empowering this. This is already a reality in our world today.

So, I think the idea, when it first came up, that these could be used for these kinds of techniques was just culturally foreign and morally reprehensible to them. I share those views. You shouldn’t be using it for either of those kinds of things we talked about. But they are being used for them, in ways that I think caught them flat-footed. Many people were caught flat-footed about how they were being used.

They didn’t want the product to be used for that. Unfortunately for the Pentagon, Claude is the premium product. It’s what they want to use, and it’s what they want to integrate into Palantir. So, they didn’t have a lot of Plan Bs ready. Then you have this stalemate where they try to coerce them and threaten their very existence, if you will, to use the product to their liking in some way.

So, I think it does presage this question: You have the private sector developing a powerful technology and people who want to see it used in a fruitful way that benefits society. It has dual uses. Other people want to use it for more controversial purposes—defending the country at its best, maybe, or reckless wars, if you're from a certain political perspective. And you want to not see your technology used for that.

So, I think this is presaging what's going to be a lot of battles in the future. And I think their concerns are right. The only thing I would say about it is that what they were concerned about has long been going on. AI supercharges it a bit. They're not wrong to be concerned about that.

And now, of course, we see OpenAI and Google have both stepped in. The one thing I've been most focused on talking to people about on Twitter is the fig leaves people are putting in front of themselves, which are no protection at all. OpenAI and Gemini both said, "We're going to do these things." Well, then there's a caveat at the bottom where the DoD says, "We're going to use it for all lawful uses."

Right.

John Nagl

That means they're going to do all the things that I said I don't really love personally. They're all lawful, right? That's the problem. The problem is that the law permits things I think shouldn't be permitted. So, it's going to augur in a lot of controversy over the next few decades.

Right.

John Nagl

And the idea that the government—

Right.

John Nagl

—is going to have to step in more. What you see just today is that we basically have a de facto licensing regime on Claude models. Anthropic wanted to release it to 70 new companies, according to The Wall Street Journal, and the government prohibited them from doing that. Once you're there, you're in a kind of government licensing regime of sorts. You're going to see more government involvement in this because the power is just too great for private-sector actors alone to decide its use.

These things are different in the sense that they've really grown to the point where they're almost like a necessary utility service, right? And we have had a history here of saying, "Look, just because you politically agree with somebody or some community, you can't cut off their electricity or their water or anything else like that, right?" It's like, if they're not breaking the law, too bad. You don't like what they're doing, but you can't cut them off from basic services or access to the public square.

Isn't there an element of this happening there where Claude's like, "Okay, well, you're using it for lawful purposes, but we don't like those lawful purposes, so we're going to withdraw the technology from you"? That seems to be a similar type of—

John Nagl

I think it's a bit the inverse of that. But we've never had a world where we compelled private companies to sell to the government except under the Defense Production Act and maybe in wartime. It's entirely appropriate, it seems to me, in a free-market economy, a free nation, to have Anthropic say, "I don't want to do business with the government, period." Right? I'm working in other spaces. I work in the business sector, the enterprise sector, but I don't want to sell to the government at all.

It doesn't strike me as unconscionable that if you do sell to the government, you say, "Here are my terms." I don't have to sell to you. There are other products. You can go to Grok or ChatGPT, Gemini, or Meta's got a new product out there you can use. And we don't want you to use it for—

Build your own.

John Nagl

Yeah, build your own. Might not be a bad idea. Palantir is going to build you an LLM at some point, probably. The idea that you can dictate the terms of your product seems like a reasonable thing to do in a commercial economy. These are the terms of the contract. If you don't like it, that's the problem you have here.

The DoD, the Department of War, desperately wants Claude. It's the best product on the market. It had already been integrated into their services, so there was a switching cost to them as well. They didn't like those terms, so they got crosswise on this kind of thing. But I think it's entirely appropriate.

The Gemini used by the Department of War this week has gotten 600 Google employees to write a letter. I think it's entirely fine to me if Google said, as a company, "Don't do business with the Department of Defense." As a CEO, I would urge them to do business with the Department of Defense. I think that's very important, personally. But I'm not offended in a free market if a company says, "The government market is not one I want to be a part of."

I think there's enough competition that it's no longer a utility. It's not the government shutting you out or turning off your electricity. It's a private-sector company, a vendor. If my lawn guy says, "I don't want to be your lawn guy anymore because I don't like what you do or what you believe, or you're working in something," he doesn't want to be your vendor anymore. I shouldn't be able to compel him to be my lawn guy. I go find another lawn guy.

Oh, sure.

Gary Marcus

That's the analogy to where Anthropic is the Department of War in my mind. It's a vendor, and they have the right to dictate their terms if they want to.

That's why I made the analogy to utility services. Here in Connecticut, Eversource isn't allowed to cut off my power because I post stuff on Twitter, right? That's just not something they can do. There are certain services that grow to a scale where they become utility or essential services, and you're just not allowed to deny people those services based on ideological disagreements, right?

But I agree with you that that's not happening here because it's not private individuals being denied the service, and there are also other competitors. Maybe part of what's happening here goes back to what we talked about earlier, which is the lack of transparency in the terms. It's kind of like, "Here, I have this product. Government, you can pay us X to use it." And then maybe new guardrails get introduced, right? It's like, "Hey, we used to use it for this, and now it's not letting us do some facial-recognition thing. What the heck?"

Well, yeah, we introduced these guardrails. "That wasn't part of the four corners of the contract," because there just weren't transparent terms, even behind the models. So, I think your work to make that more robust, standardized, and transparent will probably help with this too, right?

Gary Marcus

Yeah, I think my ultimate criticism is for Congress. The operative words of this controversy are "lawful uses" of it. My objection, and I think Anthropic's objection too, and the Google employees' objection, is what lawful use is. That's not for anyone to decide but Congress.

They should be able to say, "A lawful use..." The open question is using AI for domestic surveillance and to assemble records so they can make personal dossiers on you, which LLMs can really lubricate the creation of. That's lawful today.

Yeah. Yeah.

Gary Marcus

And that was a surprise to many people. I think he worked at the labs—what's lawful in surveillance? And he was like, "Congress should step in and say, in my mind, that's not lawful." Most members of Congress don't believe that's lawful.

The same thing goes for lethal autonomy. Today, we have a lot of autonomy. The government's moving there rapidly. If you want to stop that, you should probably call Congress. Don't call Dario, Sam, or Demis Hassabis about it. The problem is really what we consider lawful.

The government will do everything that's lawful. It's almost its fiduciary obligation to do everything that's lawful to protect the country. So, yeah, Congress should step in. They're really the ones who need to remedy this. These kinds of contract disputes are, I think, a diversion from the fact that Congress needs to step in and clarify these rules. Anthropic would probably be happy if that happened, and I think the Department of War would happily accede to those new rules.

There is this concentration risk, right? There's a handful of people and some hundreds of people who really are the backbone of the emerging and vital AI ecosystem. You kind of advocate for more distribution of that capability, or at least that it would be good if there were more competition and more development. Is that right?

Gary Marcus

No, I think there are forces in each direction. The concentration of companies both in the semiconductor supply chain as well as in the frontier-model space makes regulation quite easy. If EUV machines were made in 50 different countries, China would have a million of them today. It would be cranking out 2-nanometer chips. The fact that there's 1 company making them allows the US to throttle that in some ways.

So, yes, that concentration across the supply chain, and the fact that there are really 5 frontier labs—maybe even just 3, if you really want to look closely at it—makes regulation easy. To the extent I'm scared about the capabilities, that's a feature.

Of course, I think a huge worry of mine is this concentration of power and wealth in a very few people. That's why, over the last few years, things like open-sourcing—which has risks as well, I think, net-net is a good thing—because this concentration of wealth and power is so disconcerting to me.

I'm so concerned about that particular aspect. The same is true: you have more companies; competition drives capability. If we only had 1 OpenAI and you didn't have Anthropic and Google, the product would probably be a lot less featureful than it is today. Their competition has improved the product immeasurably.

Competition helps improve the product. I think it also helps distribute the power. I don't want 1 or 2 people with incredible wealth and power. I think that's just a dangerous thing. We have enough of that in this country without the AI guys coming and joining the fray.

Let me talk to you a bit about the open-source angle. I'm not necessarily an expert, but if I remember correctly, in reading some of the regulations that California was trying to introduce for AI, it would have basically made anybody who had a GitHub project accessible to somebody in California comply with regulations that they had passed. I think that's correct. Is that a good thing? That seems to stifle innovation, right?

Gary Marcus

No, it's not a good thing. I have a GitHub repository, and I shouldn't be complying with California laws because I write code for casual websites at home about things.

What's easy is that there are now really 5 companies making frontier models because of that concentration. To me, they're the ones that need to have the frontier models—not even all their models. I don't care what Google's Gemma is doing, as an example. Those are things that should be out there. Roboticists should use them to do what they can.

There are 5 companies at most making really frontier models. I do care deeply about what they are doing because we see with Mythos new capabilities arising, ones that are seen as incredibly dangerous by our own government. So, yeah, to me, those are the folks who should do it.

Mom-and-pop businesses and small businesses—I think that's one of the big canards, if you will, in this fight over policy up here. It's like, "Oh, you're hurting a little tech." No, I just really want 5 companies to tell me what they're doing. And if you can create novel pathogens or break down government systems with the capacity of a state actor, yeah, I really do want to know that.

It's not just some small operator in Tulsa, Oklahoma, who's doing that. It's 5 companies that are spending hundreds of billions of dollars, have attracted the smartest people in the world into a 6-by-6-square-mile area of the country, and are getting ready to upend it. So, yeah, I think it should be disseminated. I think the regulation should focus really on those big actors who are driving capabilities.

I know that one thing you advocate for is increased funding for academia, right? Could they even afford to train a large-scale model? No. We've discussed this before in our server because we have quite a few academics in there.

It turns out it's actually an interesting problem because, on the one hand, you say, "Absolutely, we want academics to have more money so that they can afford to do tests on large-scale models." But, on the other hand, necessity is the mother of innovation, right? By not having the funds, it forces them to find more efficient, clever ways to do the testing.

So there's this interesting trade-off between supplying them the money to do the things they need to do but not discouraging them from innovating with "less is more," right?

Gary Marcus

It's supposedly the problem the Chinese have, right, in their own way. They don't get the chips, and so they innovate on algorithms and architecture in some way, right? So, yes, necessity is the mother of invention.

When you have, as the companies do now, these chips coming out, they're emphasizing compute and scaling rather than some of the other things that might actually help them if they didn't have those kinds of resources.

Yeah, I do think the universities need more. I think the challenge is this: you can get access to cloud computing now, so you don't actually need your own GPUs. If you're at the University of Tulsa, you can go to one of the providers out there.

But if you're a top-level ML PhD graduating from MIT, Berkeley, Stanford, Caltech, or Carnegie Mellon—the 5 or 6 places that are really turning these out—odds are, almost certainly, you're not going to go into academia at all. You're going to go work for a lab. The number of people being siphoned off into the private sector, not into universities, is legion.

It's very hard to hire people, even at the top institutions, who are very good at these fields. The money is not only far superior at the companies; they actually have better access to data, so they have a lot of things going for them.

I do think it's a problem that the universities and the public sector more generally do not have access to all the resources and all the career opportunities. The papers aren't even being published by the labs much of the time, so the research is being kept internal as well.

This is a general-purpose technology, as everyone defines it. It's probably the first one in history that's being developed behind closed doors, with very little public oversight and with the best minds going behind those doors.

I'm always excited personally when I read that Argonne is going to try to develop its own LLM for the public sector. I'm like, "Yeah, that's a good project. I'm for that." I see efforts out of Zurich where they're trying to do public AI and make it so that governments, civil society, NGOs, nonprofits, and these kinds of groups can have access to compute and models that work for them.

I think that's all really important because, again, the non-lab world is not getting access to these kinds of things. The best minds, all the money, and the best data all go to one of these companies. That's remarkable in some ways, but the downstream effect of that is that universities are really suffering these days.

You know, the old days of supercomputing, where the school had access to a supercomputer and researchers could have quotas of time that they could run on the supercomputers. You could probably arrange some type of public sharing of large-scale model training.

It's a bit tricky, but still, I think you're absolutely right that more is needed, and it would be interesting to have a lot of these shared and co-developed models thriving, right?

Paul Scharre

Yeah, I think it's the broader question of public use of this. One of the things that Mythos, I think, brings us is this world where there's going to be increasingly bifurcated, if not separated into more elements, so people have access to top models and people who don't.

We know the models are only getting better. So, if Mythos is a problem, some of Mythos is going to be even better and more dangerous to us. Presumably, it's going to be gated from the rest of us, and the government is not going to let you give it out to companies beyond the ones they personally approve. Again, this de facto licensing regime happens.

If compute becomes scarcer and more expensive, I want the top model. I want to be able to do math and physics at home. My kids want to be able to do math and physics at home. If that's $500 a month, that's a small segment that has access to superhuman intelligence. My family may have it if we're wealthy enough to do it, but it won't be ubiquitous.

That actually becomes a new form of digital divide, where, if this is that transformative, I want everybody to have access to it. This whole question of how the public sector gets it, how universities get it, and how the benefits of these incredible models are broadly distributed is going to be a raging debate that is underappreciated right now in DC, as the models are certainly going to get more expensive and more gated. It's going to be determined by class whether you have access to them or not.

I was personally very surprised when DeepSeek released so much detail on its methodologies and these very innovative algorithms. Because, okay, here's China, our adversary. Maybe they don't love their children—even the Russians love their children, too. Wow, they released all this valuable information to the public.

Wasn't that pleasantly surprising to you? What do you think it says about how China interacts, let's say, in this world of AI with the West?

Paul Scharre

I think it says 2 things, probably. One is that these companies are not coterminous with the Chinese Communist Party. They're out in Hangzhou or somewhere doing their work.

By all accounts, with companies like Moonshot, you feel like you're in San Francisco. All the rooms are named after Pink Floyd songs, and I think people are smoking pot. I know that the state apparatus is very powerful in China, but it's not identical.

You know, this is not some kind of paramilitary unit that's doing things directly for the People's Liberation Army and stuff like that. Which isn't to say it's not dangerous and tricky. It's just that there is a complex society with an incredible history and an amazing culture.

And you can't just have this uniform view that Xi Jinping announces some kind of diktat and the rest of the country all falls in line, right? We know that's far from the truth. So, I think that's part of it.

I think they had, especially when DeepSeek was making a name for itself, maybe the government liked it because they were making China a champion. Suddenly, China was in the game, you know? And that was an amazing thing for them to do.

And in some ways, that's how the culture should be. It used to be a world where people were all publishing their papers and sharing them around—the latest cool stuff. In the US, we've gotten away from that as it's become more proprietary. So, that's a good thing.

China probably, over time, will retreat back to this more insulated standard that the US is now adhering to. But, yeah, I think it does show this question of what's happening in China is really remarkable. They have incredible engineers, a lot of ambition, data, plenty of energy. Energy is going to be the bottleneck in the US, probably. That's not a problem for them.

And, yeah, these companies are really incredible to watch.

Well, certainly some of the best games these days are coming out of Chinese studios. They're phenomenal, producing phenomenal games.

I'm glad you brought this up. This is kind of unrelated to the show topic, but you brought up that it's this massive culture with a huge, vibrant history. I think that's one thing I would like to encourage my fellow Americans to think about: when you talk about China, when a Chinese person hears the word “China,” they're not thinking about the CCP. They're thinking about this rich, thousands of years of history that they've learned and that they know about.

We only have a history of some hundreds of years, right? And so, to the extent that we do want to criticize things that the CCP is doing, it's always good to make that distinction. It's like, okay, there are these CCP policies, which are very different from the people of China, right?

Paul Scharre

Yeah, 100%. I mean, Chinese culture is remarkable. The history is extraordinary, right? Their contributions to the world are amazing. And so—

Phenomenal.

Paul Scharre

You know, yes, it's important to recognize the CCP is not China. And, like it is in the US, where everyone thinks of the current president, you come to Oklahoma and it's like a different world. People have very different views about it.

We're not homogeneous and have a single view about anything. It's a very complicated place. I'm sure China is, too. And so, I think if anything, we should learn lessons from what they're doing there and venerate what they have achieved as a culture, without any love for the CCP, of course, or any illusions about what Xi's ambitions might be that could be inimical to those in the US.

But one can only hold a candle and say, you know, China is an extraordinary place, an amazing culture.

Yeah, and to that end, we've talked a lot on the show about using AI to fight our wars. But, to a large extent, wars happen because of a failure of earlier diplomatic stages, right?

So, what future, if any, do you see for AI in actually helping to bridge understanding between cultures, or even in political negotiations and diplomatic ventures? Is the US government putting effort into utilizing AI tools to prevent wars as much as it is into fighting wars?

Paul Scharre

I don't think we are, to be honest with you. I mean, you can think ambitiously about this, where maybe AI systems become a common source of knowledge. The Chinese are reading DeepSeek. I'm reading DeepSeek. I use all the Chinese models a lot in my home in Tulsa: Moonshot, Kimi, DeepSeek, Qwen. They're great, remarkable models.

Maybe they give us a common operating picture or give us insights that get us out of our insularity a bit. Maybe that's the best hope for it on this.

I do think the government should do a lot more track-two talks with Chinese scientists. We should be engaging them more. Again, the Soviets are a great example of this. We had a lot of talks, constant scientific communication, discussions back and forth, even when we were at each other's throats and the future of Earth hung in the balance. But we still did those things. And it doesn't make any sense.

You educate me. What is a track-two talk?

Paul Scharre

Track two is when people who are former government officials, usually not in government today, are over there talking. I've done this a lot on the military side. As a former Department of Defense official, I would go to Shanghai or Beijing, and you would talk to former defense officials there.

We still have lots of friends, and so we would talk and discuss the American perspective. They would give the Chinese perspective. Then you would come back and tell the people who are in office, like, you know, this guy who's still best friends with the defense minister believes this, and here's his take on things or his interpretation of American actions.

So, it's a low-stakes way for people in governments to talk to one another. They usually are the preview for talks among the principals themselves. So, that's what a track-two talk is, and we should be doing those in AI.

And people are trying to do that, getting Yoshua Bengio or Stuart Russell over to China to talk about the Western perspective, to learn about how the Chinese think about existential risk, discrimination, and the military applications of this. What do they think?

Because I think, coming back to the Soviet example, which I brought up several times, there was a time in my life when I was fascinated by this question. They were opening up the archives after the Soviet Union fell, and a lot of historians were going in and writing these alternative histories of the Cold War, looking at the actual papers of the Soviets.

They would evaluate what the Russians really thought about this. We had this view in America of how they interpreted the world, but what did they really think? What was the army really writing about this kind of stuff?

And it turns out we got it all wrong, mostly, right? We totally misinterpreted a lot of what the Soviets were doing. We saw aggression where there wasn't any, or we saw nothing when there actually was aggression.

To me, that's a powerful lesson about the Chinese. It is a different culture, one that is rich, proud, vibrant, and on the come. Have no illusions that we actually understand what they think about these things.

Don't fall into the trap of either mirroring yourself or presuming that they're nefarious and evil-minded and are willing to destroy the world to achieve their ambitions. Recognize that they're a competitor, minimally, an adversary maybe, but try to understand them better.

These track-two talks are part of that, but across the board, Americans should be trying to understand China a lot more. That's a huge weakness for us. I do think we're at this stage now where we're just projecting our worst fears onto them.

I mean, don't you think AI could go a long way to help in diplomatic modeling or diplomatic understanding? Why not? It seems like these tools could be directed at preventing war and furthering understanding, rather than just fighting war.

Right?

Tim Ryan

I think we'd love to see that done. It's not a typical American constituency. What agency of the government does that? Perhaps the State Department or someone.

You know?

Tim Ryan

Yeah.

You know, the Department of Defense, right? Everything's a weapon for them, right? That's how they see these kinds of tools.

But, yes, I agree. Finding ways to build common understanding—I mean, there is a school of thought. You could think of a different place in the multiverse where we treat the development of AI like we do public health. We share immunizations.

If the US or China cured cancer tomorrow, of course we would share it with China, Europe, Africa, and the globe. It would be an amazing advance, and you would share it. They, in turn, would probably share it with us.

The idea is that you could see AI going in that direction. It doesn't seem to be the trajectory we're on, but nonetheless, it remains a distant dream of mine that that's how we would think of the problem, or at least open ourselves to the possibility of that, without defaulting into this kind of arms-race mentality where AI is a weapon and we have to master it.

A race to what? What does victory look like in this arms race? What lane am I in? What kind of trophy do I get when I achieve it? Do I get a Romulan cloaking device that lasts forever, so I cannot be challenged? What is AI going to do that renders war obsolete? What does winning look like?

Yeah.

Tim Ryan

That’s why arms races historically are a byword for disaster. We should get ourselves out of an arms race if we’re in one.

Speaking of which, I know one of the pillars on the slide is upskilling government. I’m curious—I mean, of course there are the testimonies that happen, and those usually seem more performative than anything else. But I’m curious what actually happens in the halls behind the scenes.

How do congressmen learn more about AI? Is it just educating yourself by going to YouTube and watching, of course, Machine Learning Street Talk, or do you have experts milling around in the hallways there to discuss topics with you, answer questions, and deep-dive anytime you want? What’s it look like?

Well, one observation about the state of play, and then how people try to get around the problems that the state of play presents to them. I remember when I was first elected to Congress 20 years ago, the Congressional Management Foundation gave you this book, kind of like how to be a congressman, how to run your office, and what to expect day to day. I remember reading in the back that they had a survey of the existing Congress, and they said, “How much time a day do you have to read and to get smarter about issues?” And the answer was 17 minutes.

Of course, members of Congress are dealing with everything in domestic policy, international policy, and local questions. So they’re overwhelmed by just the number of subjects they have to understand a bit.

Mm-hmm.

Tim Ryan

So, it’s a world where I tell folks: if you want to go into politics, develop your human capital before you show up, because you only draw it down once you’re there. That being said, on all these technical questions, there’s been a huge effort from everybody, from AAAS to various nonprofits, to put technically minded people on the Hill. There are fellowships for mid-career scientists. Maybe they leave their academic position or even industry and come work on the staff of a House member or senator. Industries have paid for some of those; various nonprofits have done so. Very important.

And so now, if you go to most of the offices in the Senate, they’ll have a fellow there who has some kind of superlative education—a PhD from an elite school, expertise in computer science, machine learning, or biotech perhaps—and who is there to advise them about this. That’s both to get smart about politics when they go back out, maybe to their commercial or academic position, but also to give advice.

You have that going on as well. The halls are swimming with people now, for better or worse: lobbyists from the tech companies, but also folks from nonprofits. Civil society now has a huge presence in AI debates in D.C. So there’s a lot of incoming messaging, but it does come back to that constraint I mentioned: their time is very, very limited.

It’s going to be very hard to become an expert on AI when you’re in office. So you either have to come in with that kind of background, have really good staff, or make it such a priority for yourself that, at the expense of all the other issues—health care and transportation—you choose to do that. There are a couple of members like that. Don Beyer from Virginia is a terrific member. He’s probably in his 70s. He’s going back right now to George Mason University and getting a PhD in machine learning. He’s a very successful guy, very wealthy, because he’s interested.

That’s somebody who’s like me, that I focus on. But other people come to Congress because they care about something else. And so they have to have that staff around them. Building up a staff ecosystem and a civil society ecosystem to combat, and sometimes complement, the lobbyists from the tech sector itself has been, I think, an important part of what the last couple of years has brought to D.C.

Okay, so I see that individual congressmen and women would have staff and people who may be experts, but is there also a shared pool, almost like an internal think tank, that all of them can access to vet ideas or talk about the impacts of policies—a shared pool of experts, or no?

Tim Ryan

Not really. Certain segments of each party will have something like that. There’s a Republican Study Committee for very conservative members of the Republican Party that has that kind of shared infrastructure and its own staff. They’re almost an in-house think tank, if you will. But that’s rare.

Newt Gingrich, when he took power in 1994, got rid of the Office of Technology Policy, which was kind of the in-house congressional think tank on tech policy. It’s never been brought back. So that’s the challenge: there’s no outside but congressionally authorized group to do it.

The Congressional Research Service can provide you background information. Again, you have access to a lot of people in civil society and think tanks, but there’s nobody outside your own staff who’s really thinking big thoughts. A lot of times, your staff is also caught up in the minutiae of day-to-day fighting and stuff. So that’s, I think, one of the challenges: there’s no kind of brain trust that can think big thoughts and then inform the parties about that.

Do you think this is a gap? I mean, is it something that we should have?

Tim Ryan

For sure.

Okay.

Tim Ryan

It’s something that, again, used to exist. It’s been taken away as these technical questions become even more important than they were 30 years ago. Yes, we have to have groups like that that are, again, congressionally chartered and not subject to influence either by civil society, where you have philanthropists giving money. Sometimes they’re slanted in their views. The lobbyists, of course, are carrying the water for their respective companies.

But to actually have something like, “Hey, you’re in the public service here. You’re on the government payroll. Your job is to think these kinds of ideas through for us and give us good ideas.” I think that would be a powerful, powerful tool. We need to do that. It would make our government more effective.

Again, you have groups like the Republican Study Committee, kind of the Progressive Caucus in the House for the left-wing members. They’re small groups that do this, but you don’t really have one that’s incredibly high-powered, with really the best people in it.

But you have so many people who listen to you who are influential in the industry. I think I am, in many ways, somewhat gloomy about the way this is all going to go for us: whether our democracy can handle this incredible technology, whether people can find meaning where work is scarce, and whether the benefits of AI are going to be, if not equally distributed, at least widely distributed.

I think there’s very little in recent American history that makes me think we’re going to get those kinds of questions right. And it does go to your question about what one should study and what disciplines are important. When I argue with people like Seb Krier at Google DeepMind or Dean Ball, who was in the Trump administration, I find we agree about the technology exactly. What we disagree about are these questions of, like, how does government respond? What can government do? What should government do, even if capable? What’s the role of government in a society like ours? These are really live questions that are actually the most important ones today.

I think the big fear that I’ve had, and that drives a lot of the work we do here, is that the AI industry can be its own worst enemy. People loathe it. I see polling every day, political polling. It’s deeply unpopular, and that’s not a good thing for our country. There are a lot of folks with pitchforks who are just over the horizon, coming after this whole sector. Many of them would shut it down entirely.

They want to do so because they don’t give affirmative answers to those questions I raised earlier. They’re like, “This is a project of the elite for the elite. They’re building a data center in my backyard that’s going to maybe change the environment, maybe raise my electricity prices, and whose sole purpose is to take my job away.”

Then I turn on the TV, and there’s a lab leader saying, “I’m here to disrupt your world—irrevocably disrupt your world.” I think the fact that society—the U.S.—has lost faith in the project is extraordinarily damaging to what is a very important project with many, many, many upsides.

We’re in a perilous place. If we don’t do something to regain that trust, there are going to be a lot of people who are radically opposed to this project and do their best to, if not shut it down, stymie it.

That’s why I said I think these next few years are really important. To your listeners, who are probably doing machine learning day in and day out, their voices need to be heard. They need to be heard inside their companies to advocate for the right kind of public policy. They need to be heard outside of it. Their task—all of our task, perhaps—is to convince Americans that, yes, this is actually a good thing. Because the truth is, right now, most Americans don’t think it’s a good thing.