Superintelligence: To Ban or Not to Ban? Max Tegmark & Dean Ball join Liron Shapira on Doom Debates

Okay, let's do opening statements. Max, the starting point of our debate today was a dispute between you and Dean over your statement on superintelligence—the Future of Life Institute's statement on superintelligence—that was published on October 23rd. The statement says, “We call for a prohibition on the development of superintelligence, not lifted before there is: 1. broad scientific consensus that it can be developed safely and controllably, and 2. strong public buy-in.” So why should we ban superintelligence?

Well, if you negate that statement, then you're saying that we should be allowed to go ahead and build artificial superintelligence even if there's no meaningful consensus at all that it can be kept under control or that people even want it. And if we were to say that, then we would be basically doing the most spectacular corporate welfare, because we don't do that in any other industry.

Yet right now, there are more regulations on sandwiches than on superintelligence in the US. If you want to sell drugs, medicines, cars, or airplanes, you always have to demonstrate, to the satisfaction of independent scientists who don't have a conflict of interest, that this is safe enough and that the benefits outweigh the harms. I'm just saying we should treat superintelligence the same way.

Right now, 95% of all Americans, according to a new poll, don't actually want to race to superintelligence. And most scientists who work on this agree that we have no clue at the moment how to keep something that's so vastly smarter than us under control.

Okay. And Dean, you oppose the public statement, and you don't share Max's views on prohibiting superintelligence. Give us your opening statement. Why do you think we shouldn't ban superintelligence?

I think the concept of a ban on superintelligence in general is quite nebulous, and that is the fundamental issue that I have. AI systems that could pose substantial danger to humans are not disallowed by the laws of physics, at the very least. I think there are really serious questions about how close those things are and how likely we are to build them in the near future.

My guess is that 5 years ago, if you were to try to describe general superintelligence in a law that a lot of people could agree to—which would be the way that you would effect something like a ban—you would run into serious problems. All the things Max referenced are things that we impose those requirements on through laws, right? On airplanes and drugs and whatnot. So if you're going to have a law, you're going to have to define superintelligence in a statute.

I think the problem you will run into there is that you will define it in such a way that you actually end up banning many things that we would want. There are many ways that you could plausibly define superintelligence that would negate technologies that I think would be quite beneficial to humanity. Imagine an AI system that has largely solved mathematics, right? It has solved all the outstanding problems that we have in mathematics. It has advanced certain domains of science—maybe many domains of science—by the equivalent of a century compressed into a decade, or compressed into 5 years, let's say.

It's accelerating the development of AI research itself. It's doing that in meaningful ways because one of the areas of science that it knows how to do experiments in is computer science and AI research. It's a better legal reasoner than you or me or anybody else. It's better at coding than you or me or anybody else. I can imagine such a system existing.

In fact, my guess is that such a system will exist by roughly 2030 without posing the kinds of risks that Max is worried about, which, again, I don't think are impossible. I just place a lower probability on them. And so I worry that what you end up with in practice, if you tried to effect such a ban, would be, you know, “We're going to ban N plus 1, N plus 2,” right?

So there's GPT-5, and that's N. There's GPT-6, which would be allowed, and then GPT-7 would be the thing where we say, “No, we've decided that's too scary, and so we're going to basically ban that.” And then what happens after that?

In order to figure out anything about whether superintelligence is safe or not, you can't just do that research speculatively, right? You have to actually build the thing to some extent and put it in a constrained setting to figure out if it's safe. You have to build at least big parts of it. And once you've done that, it's like, well, okay, but there's a ban. So only the specially sanctioned group is allowed to conduct this research.

At that point, you have a monopoly, perhaps a global governmental cartel of some kind, that is developing this. And I also think this could potentially be dangerous. That is, of course, assuming that you were able to get the international cooperation you would need to effect such a ban, which I also doubt. So that would be my comprehensive statement.

Okay, Max, Dean raised a few points about the practical difficulties of doing this kind of superintelligence regulation, even going so far back as to define what superintelligence is for the purpose of this ban. How would you respond to that?

I'm afraid we might disappoint you by agreeing more than you want, because you want the fierce theorists to clobber each other. I think it's actually quite easy to write this law, and I don't think it requires defining superintelligence at all. Let me explain a little bit what I mean by that.

If we treat AI like we treat any other industry that makes powerful technology, we would have safety standards, right? There are safety standards for restaurants. Before they can open, they have to have someone check the kitchen. So if you had safety standards for AI, they wouldn't need to define superintelligence. They would just say that if there's a system that plausible experts think could cause harm, here are the things you have to demonstrate to the FDA for AI, or whatever, that this is not going to do.

You might want to demonstrate that it's not going to teach terrorists how to make bioweapons. If it's a very powerful system, you would probably make one of the safety standards that you have to demonstrate that you can keep this under control. If the company selling it can't convincingly make the case that this thing is not going to cause the overthrow of the US government, then reject it. Come back when you can, right?

I didn't mention superintelligence here at all. It's the company's obligation to demonstrate that they meet the standards. To take an analogy that might help clarify what I'm talking about here, let's talk about thalidomide for a little bit. This was a medicine that was given to women in the US to reduce morning sickness and nausea during pregnancy, and it caused over 100,000 American babies to be born without arms or legs.

The dumb way to prevent such harm would have been if the FDA had a special rule saying, “We have a ban on medicines that cause babies to be born without arms or legs.” What if someone comes out with a new medicine now and the arms and legs are fine, but the baby has no kidneys or no brain? That's not the way to go about it.

The way you instead go about it is you ask the companies to do a clinical trial and provide quantitative evidence of all the different side effects that people might not want. Quantify them—what percentage get each one—and then quantify the benefits. You give this to independent experts who don't have money on the line, so they can't work for the companies, for example, who look at the benefits and the harms and decide whether this is a net positive for the American people. Then they approve it.

This is how we do regulations in all other areas, and this is how I think it's quite easy to do it for AI, too. In summary, you don't define superintelligence. You just define the harms that society is not okay with. Very broadly, it boils down to demonstrating that the harms are small enough to be acceptable. Then it's the company's job to make all the definitions they want, quantify things, and persuade these independent experts. Does that make sense?

Yeah, I'm happy to let you guys cross-examine each other pretty freely, and I'll just step in once in a while.

Okay, cool. Basically, then, instead of saying we should ban superintelligence, what you're saying instead is that we should have a kind of licensing regime—a regulatory regime of some kind—with respect to frontier AI systems.

Yeah. Very much inspired by how we do it for other tech.

Yeah, yeah, yeah. So, I’d say a couple of things about that. First of all, most preemptive regulatory regimes that I’m aware of don’t generally require you to prove—you can’t prove a negative, right? The FAA, the Federal Aviation Administration, doesn’t require you to prove that your plane won’t crash. It requires you to make affirmative statements about really not the plane itself, but many subsystems of the plane, right? So, the turbines of this jet engine have XYZ chemistry, which conforms to XYZ technical standard, and so on.

In fact, the way that often ends up working is that there are layers and layers and layers of regulations. The plane maker has to buy jet engines only from people who conform to certain standards. Those standards often have to do not just with the object-level properties of the component in question, but also with things like how information flows through the business, through the company.

In other words, if you’re a turbine manufacturer, if you make turbine blades for jet engines, you’re probably subject to implicit and explicit regulations that have to do with risk management inside your company, who the designated risk officer is, and all these sorts of things.

But the point is—I’ll just jump in—I agree with everything you said here. What the companies need to demonstrate in the safety case is a high-level thing. The government wants to know: How many flight hours, on average, do you have until a failure, and so on?

If the companies can solve that whatever way they want, it’s in their interest not to use flaky manufacturers, to have good procedures, and to have people study crack formation, the physics of it, and so on. Then they’ll switch to another alloy if that works better.

It’s the same for medicines. The government doesn’t come in and micromanage, “Oh, this chemical is allowed; this ingredient is not allowed.” Rather, if a company has an effective medicine that seems to be pretty good—suppose there’s a new antibiotic that seems really good against bronchitis, and it contains lead, aluminum, and cyanide in small doses—people will look at the company and say, “You know, we’re having a hard time demonstrating the safety of lead. Maybe this works even without the lead. Maybe we can swap out this thing.” So, all the innovation is driven by capitalism, by market forces.

They get to come up with the quantitative risk bounds that they want to meet. Nuclear reactors are a great example, because what the law actually says there is that the company has to make a real quantitative calculation and demonstrate that the risk of a meltdown is less than 1 in 10,000 per year to even get permission to start building it. The company has free rein to come up with whatever reactor design it wants, and then it will innovate. Whoever first meets those standards is going to get the big bucks.

I think it’s considerably more complicated than that. In principle, that’s true, but in practice it’s considerably more complicated because there are all sorts of things—there’s what’s called soft law, which is guidelines and all these other things that push people in certain technological directions and away from others.

But that’s actually not even my point. My point is that, at the end of the day, in order to have a regulatory system like this, you have to be able to make affirmative statements about safety. The problem, I think, would be: What are the affirmative statements about safety when you consider that the systems we’re talking about are, by their very nature, extremely general?

Obviously, AI systems today are being used in areas that already have regulatory structures like the kind you’re describing. So, this regulator would either have to be so general and have such a broad projection of authority, or it would have to be really, really narrow. I kind of doubt that it would end up being really narrow in the context of democratic politics, because there’s going to be more than just existential-risk-type issues.

Even if you could formulate some statement about existential risk—“Okay, you have to prove that the model will not do XYZ, which demonstrates catastrophic misalignment”—fine. But I would say in practice you’re likely to end up with a situation where, for example, the model cannot result in job loss.

That would be a really good example of this. This gets back to an article that I wrote more than a year ago called “The Political Economy of AI Regulation,” which is to say that, because this is so general, and because the technology is going to challenge many entrenched economic actors and aspects of the status quo in its positive adoption—not existential risk, not anything like that—those people, if a regulatory regime of the kind you’re describing exists, are going to be able to use it as a cudgel to prevent technological change that I think we would all agree—well, not all, as in all people, but probably the 3 people in this discussion would agree—is good for the world.

So, I will push back a bit on this idea that it’s so hard to get started on this, but I’d love to just give you a chance first to answer a very simple question. Do you think it’s reasonable to have 0 safety standards on AI right now? Do you feel it’s reasonable that there should be less regulation on superintelligence than on sandwiches, now in 2025?

Well, I certainly think we overregulate sandwiches in the U.S. And, just for the listener who doesn’t have context, I think what Max is probably referring to is sandwiches served in restaurants—public-health regulations, local regulations, and all sorts of things like this. That’s true: There are probably ways in which we overregulate those things, and probably many other ways in which we don’t.

Generally speaking, in America, we succeed when we regulate at the level of the restaurant that serves the sandwich. That restaurant has many computers in it. It probably uses computers in many different ways, including to get the ham and the bread that brought the sandwich to us. We don’t regulate those computers with respect to their conveyance of ham to the restaurant. We treat them as general-purpose technologies that can do lots of different things.

Right. But if you go to that sandwich shop and notice that across the street from it is OpenAI, Anthropic, Google DeepMind, or xAI, and they had developed superintelligence this year—which I think is highly unlikely, but suppose they did—then they would be legally allowed to just release it into the world without breaking any law, because there are no safety standards they have to meet. Do you feel that’s at all reasonable?

Well, I wouldn’t quite say that. Fundamentally, I think you should be able to develop new technology and release it so long as you’re not behaving with reckless disregard or gross negligence—reckless conduct or gross negligence.

According to whom?

So, this is the thing that already exists. To say it’s illegal would imply that it’s a violation of criminal law, which may or may not be true, but certainly it’s a violation of civil law. If release of that system were to result in physical harm, loss of property, or the death of any human, that company would be subject to common-law liability.

Yeah. Well, human extinction, sure. But I’m sort of skeptical that that’s what we’re going to have on day 1. If we’re all extinct, then common-law liability doesn’t help you.

So, yes, in the tail-risk case that we all die, common-law liability does not help you. In general, it’s true that common-law liability is not a great solution for most tail risks, to the extent that the damages incurred vastly dwarf the balance sheet of even the largest companies.

You create—let’s not even say kill all the people—let’s say you create a pandemic. Someone makes a pandemic with your model, and we’ve decided that that was reckless misconduct for which OpenAI, the creator of the AI model, bears some form of liability. That’s a lawsuit you can bring against them. But if the damages are $100 trillion or something, it’s very unlikely that you’re going to be able to recoup that amount of money from OpenAI, even with all the money that they have. You’ll bankrupt OpenAI and still not be fully compensated for the harms that you suffered.

It is true that, as a general matter, tail risks are one of the classic examples of where government—where public policy outside of reactive liability—makes sense. I don’t dispute that. When it comes to the foreseeable tail risks that AI models might pose, the current ones that people talk about are things like catastrophic cyber and bio.

I think there are a lot of things that you can do downstream of that that avoid creating this large-scale regulatory regime.

A lot of people talked about extinction too, wouldn't you say? I mean, a lot of people do, but there still is not the kind of persuasive evidence for extinction in terms of—not just theoretically, but mechanically—how would that work? I just don't think we've seen that to nearly the same extent that we have.

We haven't seen any extinction yet, of course, by definition; otherwise, we wouldn't be talking here today. But I mean, it's very clear that we should put it on the risk list. So I'm just thinking it's interesting what you mentioned there about the pandemic example, because I think it's quite relevant. You know, as you know, it's very controversial right now whether COVID-19 was actually the result of gain-of-function research.

Yes.

Funded by the U.S. government and Peter Daszak, or not. But if you just consider that there's some probability P that Peter Daszak and his research group did create it with help from others, then if someone were to sue them for the millions of deaths that it cost, it would be pretty meaningless, because Peter Daszak doesn't have that kind of money, and the university where he worked doesn't have that kind of money.

For that reason, the U.S. government has now kind of clamped down on gain-of-function research again and said, “No more of this gain-of-function research until we better understand what you're doing.” We also have biosafety labs at Level 1, Level 2, Level 3, and Level 4. So if you're doing something that seems even less scary than what they do or did, you have to do it in a special facility, and you have to get some pre-approvals.

And then you contrast that now with digital gain-of-function research. We had Sam Altman at a press conference the other week being so excited about building automated AI researchers, right? And ultimately, a lot of people are excited about recursive self-improvement, which is very analogous to biological gain-of-function research. Why should we have regulations on biological gain-of-function research and still be content with having no binding regulations at all on digital gain-of-function research? That makes no sense.

I'm trying to answer the first question you asked me, which has to do with safety standards. But first of all, let me just say: Yeah, what you said is completely consistent with my assertion that tail risks are not typically contemplated very well by the common-law liability system.

But with that being said, I think that, for example—actually, you can do essentially automated gain-of-function research with a nucleic-acid language model today. You can basically simulate the evolutionary process that allows for more virulent viruses or whatever else, and we've seen the early stages of this from people like the Arc Institute in California. Those are not ChatGPT-style models, but it's the same architecture trained on nucleic-acid sequences, right? So we know that's a thing. I think, as a practical matter, though, the issue that you have is that—

Those things are bits, and it's very, very hard to just purely regulate bits. So what do we do? Well, instead of imposing regulations at the layer of the model, which is a really difficult layer of abstraction on which to do it, in the same way that we don't tend to place regulations at the layer of computers, software, or transistors, because these are really important general-purpose technologies—and undoubtedly all 3 of those things have killed lots of people at this point—we don't regulate at that layer of abstraction because it's not very practical.

It's not a good unit of abstraction and not a good conceptual unit of account for regulation. So what do we do? Well, there are all these choke points in the physical world. Some of them are labs of certain biosafety-level categories, BSL-3 and BSL-4, as you said. Some of it is at the layer of nucleic-acid synthesis screening—

Which, basically, you have to say, “Well, if you're going to order the creation of a certain kind of nucleic acid, we're going to, as a matter of policy, require that you screen that against some sort of methodology that allows us to test for whether or not you're trying to make a pathogen.” And again, I worked on some of those policies when I was in the Trump administration.

So these are all things that we do. Again, those are safety standards that exist, that are emerging, that are kind of downstream, in many ways, of advancements in AI. I think the urgency of policies like nucleic-acid synthesis screening goes up because of AI.

What do you mean by long run? Because Sam Altman talked about 1,000 days to superintelligence, and he might be wrong, but I'm curious if you're thinking less than 3 years or more than 3 years.

I'm thinking that it will happen gradually over the course of the next decade, or maybe—

After superintelligence, maybe?

After superintelligence, maybe. Yes. But I think the broader point here is that this is traditionally the way that we do things in the United States: You build a technology, you gain experience in practice with its utility, and you diffuse it throughout the economy in this very complicated way. Sometimes there are demonstrated harms, and when there are demonstrated harms, the first thing we do is deal with that through the liability system.

And again, I would point out that there are common-law liability cases—not copyright cases, but “You caused physical harm to me” type of liability cases—against OpenAI, Google, and other companies for chatbots, right?

Yeah.

And I think that at least some of those companies are likely to lose. I mean, they'll be determined by courts. And then gradually, over time, we codify around a set of standards that are shaped by experience, that are broadly agreed to by many different actors, and then eventually we codify those in the form of government standards. Eventually, that becomes part of an international standards body.

Nobody is disagreeing that this is a process in which we need to invest substantial time, money, and energy. And in fact, I would say the Trump administration should get points in your book because part of the reason that the administration renamed it—which was called the AI Safety Institute by the Biden administration—

They renamed it the Center for AI Standards and Innovation to reflect this reality: the ultimate goal of an organization like that is to produce technical standards. So you have to produce these standards. It takes time to do, but when you actually have them, they are coherent because they're formulated through experience.

I think the problem is when you try to change the sequencing of that and try to come up with standards without any experience, sitting in the ivory tower or the regulators' conference room. I think you have a tendency to create standards that are unrealistic and burdensome.

Mhm. I completely agree with you, Dean. It's great that the current administration is taking biosecurity more seriously, and I get a sense that they're also taking AI-assisted hacking more seriously. I completely agree with you also that this is how things have been done in the past: let technologies come up. People invent the car, a bunch of people die, and then gradually you mandate seat belts, traffic lights, speed limits, and other things to make the product safer.

But I think it's important to remember that science has been getting progressively more powerful from ancient times until now. As a result, technology also keeps growing exponentially in its power. At some point, the technology gets powerful enough that this old, traditional strategy of just learning from mistakes goes from being a good strategy to being a bad one.

I think it worked and served us well for cars. It served us well for things like fire. We invented fire first. We didn't regulate it to death. Later, we decided to put fire codes in place and have fire extinguishers, fire trucks, and things like that.

I would argue that nuclear weapons are already above this threshold. We don't want to just let everybody who wants to buy hydrogen bombs buy them in supermarkets and then say, "Oops, that didn't go so well. We had a nuclear winter, and now 99% of Americans have starved to death." Let's regulate those things. It was very obvious to people that one mistake was one too many, and we already have a bunch of proactive laws about how to deal with hydrogen bombs.

In fact, even despite all your work in the government, you are not allowed to buy your own hydrogen bomb, even though I would trust you with it. I know you're a nice guy. I'm not allowed to start doing new plutonium research in my lab at MIT, even if I pinky-promise that I'm going to be careful, just because one mistake there is viewed by society as one too many. They know I don't have enough cash to pay the liability if I get sued afterward.

I would argue that artificial superintelligence is vastly more powerful, in terms of the downside, than hydrogen bombs would ever be. There have been some pretty careful calculations recently showing that, in the worst-case scenario, about 99% of all Americans would die and starve to death if there were a global nuclear war with Russia. So there's still 3 million who survive. Whereas if we lose control of artificial superintelligence because somebody sloppily built a new robot species that just took over, it really is game over in a way that nuclear war wouldn't be.

So the way I see this is not that there's anything wrong with the traditional wisdom for how to regulate things. I think that's very appropriate for all technology below a certain risk threshold. We're very lucky with AI that so many of the great benefits we have are not particularly risky. AlphaFold is an absolutely superhuman tool for folding proteins, great for drug discovery. Autonomous vehicles can save, I believe, over 1 million lives every year from these pointless road deaths.

There's so much productivity that can be gained from building controllable AI tools, and I think it's very feasible to continue having the sort of liability system you're describing in the traditional way: learn from mistakes and then fix them. It's only fringe stuff, like artificial superintelligence in particular, that is on the wrong side of that threshold. Right now, there's not much upside, frankly, to sprinting to build superintelligence in 3 years. If we could do it safely in 20 years instead, we would be much better off just doing controllable tools until then.

That's why it irks me so much that I think people conflate these 2 things a lot. I'm not saying you do, but a lot of folks I've spoken to on the Hill do, I think, and think that the only choice we have is more AI or less AI, or go forward or stop. Whereas I see the development as branching into 2 paths.

Either we continue going very aggressively forward to build all these great tools, but insist that companies demonstrate to us that they are controllable tools, or we go all in on building superintelligence. I have to give you a compliment, Dean, also. I was so pleasantly surprised when I read the action plan that it didn't mention the word "super" a single time, and not even AGI. You must—and I think that was really wise, because it highlights that there's so much great stuff we can do with AI tools without having to even get into the whole question of superintelligence.

I don't know if there's anything you're allowed to share with us.

Well, a lot of people contributed to the action plan, but thank you very much. I appreciate it. Actually, I'd say the reason we didn't use terms like AGI and superintelligence in the action plan, at least from my perspective, is because it's really hard to know that we're talking about the same thing.

This is where I think maybe we ought to spend some time: the question of what exactly we're talking about when it comes to these things. I'll give you an example of an area where I've had an evolution in my thinking.

About 1.5 or 2 years ago, I was opposed to a bill called SB 1047, which was a California state bill that had to do with regulating models with respect to potential risks relating to extreme cyber events, bioterrorism, and other sorts of bioweapon events that could cause more than $500 million in damage.

At the time, the frontier models were things like GPT-4o, Gemini 1.5, and Claude 3. It wasn't obvious to me, sitting at that time—in, say, the spring of 2024—that the next time we cranked the pretraining wheel, the next time we went up another order of magnitude in terms of pretraining compute, we'd get to models that posed these very serious bio and cyber capabilities.

I was thinking, "Well, I don't know. You're talking about cross-entropy. You're talking about minimizing cross-entropy loss here on the broad internet corpus. Is that really going to create something that can cause a bioweapon?"

I said something, though. I said, "If you showed me a model that had demonstrable System 2 reasoning and that sort of led to the performance that I think it would—System 2 being deliberative, reflective reasoning—then I would change my mind about some of this stuff."

Then, right around the time SB 1047 was vetoed, OpenAI released a model called o1, which did exactly this. It had this System 2 reasoning. The performance on cyber, on mathematics, and on a lot of different areas of science, including biology, went way up.

At that point, shortly after that model, I said, "This changes my risk calculus with respect to catastrophic events like bio and cyber, because it's clear that this reinforcement-learning and inference-time-compute-based paradigm is going to rapidly lead to capability increases in some specific areas that we're worried about."

I can paint a really clear picture. I can go from, "This model can reason about biology, and it can also, by the way, use tools like AlphaFold. It can also itself use other biology machine-learning tools," to a novel virus being synthesized in a lab somewhere.

The virus self-replicates. It infects 1 human host. Now, there's a lot of complexity there. It doesn't mean that that kind of thing is guaranteed to happen, but it means that if you're doing the expected-value calculation and thinking about, "Okay, plausibly, what are the chances of this causing a pandemic?"—even if the chances are still low—the chances just went up a big fraction. So we're going to need some degree of targeted regulation to deal with this topic.

That is why I was supportive of SB 53, which in many ways was a somewhat more tailored version of SB 1047 that came a year later. That wasn't just because the law changed to become a little bit more favorable to things I care about. It was also because the facts on the ground changed.

So why does that matter? Because there is a clear link between emergent model capabilities and an actual harm that is cognizable to me. I think the issue with the human-extinction thing is that it's very hard to demonstrate in concrete ways what this looks like.

Oh, to this point, if you could formulate things that would make you feel better in affirmative, technical, empirical terms about models—for example, "We've stress-tested the model in this way, we've run this eval, or we've done this thing"—

And we have shown that the model passes what we view as an acceptable threshold, then I would be totally willing to say, "Yeah, you don't even need to pass a law."

Make it an eval. I kind of promise you that if you made that an eval and got enough credible people around to support it, labs would probably just run it without having to pass a law. So why not just do that?

Yeah. You’re raising a number of successful policy approaches from the past here. Let me summarize some good things I think you said there and add a little bit to it. More broadly, with regulation of things, let’s take drugs again, so as not to drown companies in big red tape and stifle innovation. One tends to look at rough plausibility and then divide all the products into classes. We have Class 1 drugs, Class 2, Class 3, and Class 4, right? There are much higher safety standards for fentanyl or other new opioid drugs than there are for new cough medicines for adults or for new vitamins.

If you take the same approach to AI, what you would say is that if there’s some new software that translates English into Chinese or Japanese, the most embarrassing thing that could probably happen is a sort of repeat of the Monty Python sketch with the fake Hungarian phrasebook. Some people get red-faced, whereas if you have an AI that is really state-of-the-art in protein synthesis or DNA synthesis, it’s pretty obvious that that should be subject to higher levels of scrutiny. What we do in industry now is let companies make the safety case rather than the government.

You said there that it’s hard to foresee exactly how superintelligence—if it’s just smarter than all humans combined—would kill us all. But being a scientist for so many years has made me really humble about these things. It would have been really hard for the makers of thalidomide to predict that it would cause babies to be born without arms or legs when all it did was reduce nausea in their mothers. We didn’t actually understand how that would happen, but it did happen.

Because of that, it would have been pretty reasonable to say, “Well, okay, we’ve noticed that there are a lot of things that can cause birth defects. We don’t understand exactly how it works, so before we try it on all American mothers without a prescription, let’s try it on a small number of mothers and then see what happens to their babies, and then kind of go from there.” You shift the burden of proof away from politicians having to articulate why this is going to be dangerous and onto the companies. We just have to do some basic research to make the safety case, right?

And I think, again, if we did this with AI today—if I had a magic wand and we created an FDA for AI—you would have Class 1, Class 2, Class 3, and Class 4, or AI Safety Level 1, 2, 3, and 4 systems, a little bit along the lines of what many AI companies have already done in their voluntary commitments, right? There would be very easy requirements for ASL 1 and so on. But for the higher-level systems, the companies would have to do a lot more to quantify the safety case.

I think what would happen then is that we would end up in a golden age of AI progress, where we would soon get flooded with all sorts of new medical treatments, amazing autonomous vehicles, and great increases in productivity. The one area that would get slowed down noticeably is precisely the race to build actual superintelligence, where I think nobody would be able to make a safety case yet. I think that will be just fine. If we have to wait 20 years to get that done properly, it’s way better than racing to it, bungling it, and squandering everything. Liron, it looked like you wanted to jump in.

Honestly, you guys are doing such a great job. I don’t know how much value I could add, but I’ll give it a shot to orient the viewers. You guys are talking about how to regulate these new AIs. Dean, in your case, as Max pointed out, when I read your AI policy document, America’s AI Action Plan, it doesn’t really mention superintelligence. Do you think that’s a wise way to go—to basically not look at the possibility of superintelligence currently when making policy—or do you think we should do anything to prepare for the possibility of superintelligence?

Well, I should say the action plan, in the AI policy world, is very heavily associated with me, but of course the action plan was written by many people within the government. I played a big role in it, for sure, but by no means was I the only one. It was not my unilateral product, for sure.

One thing I would say is that part of the reason the action plan doesn’t talk about superintelligence is because it would be very hard to build consensus, whereas my Substack does, from time to time, talk about superintelligence because it’s very hard to build consensus in a document that has so many authors as to what we really mean. This maybe gets into where my concerns are with laws and drafting, and exactly what you mean and what you don’t mean.

I think about a model like GPT-7, the sort of ostensible GPT-7, and I think to myself, man, if this is a model that advances the frontiers of science in many different domains and solves a lot of math problems that have flummoxed humans in some cases for centuries, is better at legal reasoning and all these other things than any human, and is better at coding than any human, it doesn’t seem inherently dangerous to me. It also seems like—how is that not superintelligent? It’s not Bostromian superintelligence, right? It’s not that specific definition.

But I guess my view is that the concept of superintelligence was created quite a long time ago, in the grand scheme of things, with respect to how fast AI advances. It’s not obvious to me that it remains the most useful abstraction. I think that concept of superintelligence was a really useful way of thinking about advanced AI systems. Nick Bostrom wrote the book Superintelligence in 2014, I want to say—about 11 years ago.

So, Dario talks about this sometimes with respect to AGI—Dario Amodei, CEO of Anthropic, for the listener—where he says, “AGI 10 years ago was like, ‘We’re driving to Chicago,’ but once you actually get closer to Chicago, it’s like, ‘Okay, what neighborhood are we going to? What street? What’s the house number?’”

I think that as we get closer, we actually need to develop new and more specific abstractions for what we are talking about because there are all sorts of things that we will probably, in the fullness of time, have really specific kinds of technical standards and maybe even statutory requirements for what you can and can’t build with AI. One thing I want to be very clear about is that I’m not saying this needs to be unregulated for all time. In fact, I would say you made the point earlier about how we regulate different medicines with different levels of—

Rigor.

Yeah, rigor, based on their potential risks. I think we already do that with the frontier language models, right? Because—

There are no binding regulations right now in America for anything. There are tons of binding regulations on frontier AI.

There’s no binding regulation preventing people from launching things afterward, right? As opposed to drugs, where I think that’s an interesting distinction for the listener.

You can’t release any drug in the US until you’ve talked to the FDA about it.

Not quite. Not quite. But this actually gets into technical definitions and things where these things matter. You can release, for example, CRISPR-engineered bacteria without consulting the FDA because those are probiotics according to the statute. A company called Lumina released a CRISPR-engineered bacterium that you’re supposed to brush your teeth with. Ostensibly, you’re infecting yourself with a bacterium that you’ll be infected with for the rest of your life, and every person you ever kiss will also be infected with it.

There’s a lab in Wisconsin that’s been taking this bird flu strain that kills 95% of humans but is pretty harmless because it’s not airborne. They’ve been working on trying to make it airborne.

Yeah, so there’s room for improvement there.

But I think we agree on the basic situation here: you can’t open your restaurant or release a new type of opioid before you’ve been FDA-approved. There may obviously be some differences in opinion about things, but there are some things that I think are more in the confusion category, which are really helpful to clear up. One of them is around definitions. Whenever you have any term that starts to catch on, historically, every hype is going to try to latch on to it and have it mean something else, right?

So Alan Turing, when he said in 1951 that if we build machines that are way smarter than us, the default outcome is that they take control, and when Irving J. Good talked about superintelligent machines in the 1960s and recursive self-improvement, the definition of superintelligence that was implicit in that was obviously that they could do everything way better than us—which meant that they could also do better AI research than we could.

They could build their own robot factories and make more robots. They didn't need us anymore. And therein lies the risk. After that, I agree with you.

Right now, there's just so much hype and BS about this. Mark Zuckerberg talked about superintelligence in a way that almost made it sound like it has something to do with Meta's glasses. We have so many different ways people have redefined AGI from the original definition that I actually—I don't know if you saw the paper I was involved in that we did with Dan Hendrycks, Yoshua Bengio, and many others, called “Defining Superintelligence.”

I welcome people to come up with other actually empirically useful definitions, but we found with this definition that we're absolutely not even at AGI. GPT-4 was 27% of the way to AGI. GPT-5 was 57% of the way there. There are still a lot of areas where today's best AI systems really suck—long-term memory, for example—but we're getting closer.

I think that if we're thinking about only putting the first FDA-style safety standards on AI in 3, 4, or 5 years, there's some reasonable chance that that'll happen only after AGI, and maybe even superintelligence, have been created. That would be a pretty big oopsie for humanity.

I think there are very useful, clear definitions of what we mean. As I said in the beginning, the way to write a law is not to define superintelligence and ban it. I think instead you ban the outcomes that you don't want: something overthrowing the U.S. government, or something making bioweapons for terrorists. Those are very easy to define.

As soon as that law is in place, it's going to spur massive innovation in the companies. I love comparing pharmaceutical companies' budgets with AI companies' budgets. The leading AI companies now spend maybe 1%, give or take, on safety, whereas if you go to Novartis, Pfizer, or Moderna, they spend way more than that on their clinical trials and on safety because that's the financial incentive.

They're in a race to the top. Whoever can be the first to come out with a new drug that meets the safety standards makes a ton of money. People really respect the AI safety researchers in those companies. They don't think of them as whiners who slow down progress. They think of them as people who help them win the race against the other companies to make the big bucks.

So I think as soon as we start treating AI companies like we treat companies in other industries, we will incentivize amazing innovation.

Can I also throw out a question? I want to clarify Max's nightmare scenario here because I think that's important to frame the discussion. Max, I think you're not even just concerned about something like thalidomide, where a bunch of people die—hundreds of thousands, or whatever it was. You're concerned about this runaway process where it just becomes too late to regulate forever. Is that fair to say?

Yeah. I can take a minute and clarify a little bit for listeners who haven't thought so much about this.

Many, many times humanity has been thinking a little too small. People thought nuclear weapons were science fiction until they suddenly existed. People thought going to the Moon was science fiction until we did it.

From my perspective as a scientist and as an AI researcher, if you think of the brain as a biological computer, then there's no law of physics saying you can't make computers that are better at all tasks than we are. A lot of people used to say, “Yeah, but that sounds so hard. It's probably decades away.” In fact, most professors I know thought, even 6 years ago, that we were probably decades away from making AI that could even master language and basic knowledge at a human level.

They were all wrong, it turned out, because we already have it now in systems like ChatGPT and Claude 4.5. If we consider what would happen if we actually built huge numbers of humanoid robots that were better than us at all jobs, including research, mining, building robot factories, and so on, we would have built something that is not just a new technology like the printing press, but really a new species.

These robots can build new robots in robot factories, and they don't need us anymore. It could be great. It could mean that we don't have to do the dishes anymore and we're going to live in abundance, with them taking care of us. But it's not guaranteed.

Alan Turing, as I mentioned—the godfather, really, of our field—said in 1951 that the default outcome he thought was them taking control. We have the 2 most cited AI professors on the planet, Geoffrey Hinton and Yoshua Bengio, saying similar things today.

If you let go of this idea that AI is like the new internet or whatever, and you just think of it as actually a new species which is in every way more capable than us, there's absolutely no guarantee that it's going to work out great for us. I'm not just talking about how we obviously couldn't get a job or get paid for doing work because they could do it all cheaper. I'm talking even about the fact that we don't really have any say after that, necessarily, in what happens on the planet.

A lot of people are working on this. It's called the control problem or the alignment problem. There's broad scientific consensus that they're not solved yet.

This is the scenario which I think we'll end up in if we just race as fast as possible to build these superintelligent machines, rather than focusing on the controllable tools that can cure cancer and do all the other great stuff, and then taking it nice and slow with the things that we don't yet know how to control.

So, okay, there's a lot there that I think I can respond to. First of all, I would start just by pointing out that you correctly observe that lots of people will take terms like superintelligence and redeploy them to mean completely different things. I would submit to you that maybe Sam Altman, when he talks about this existing in 3 years, is doing a bit of the same thing.

You can't talk out of both sides of your mouth. You can't say, “Well, this happens,” but also say that these people are going to build it tomorrow. You have to pick one.

But the other thing I would say that's more serious is that I wasn't talking about whether I'm concerned about this regardless of whether it happens in 3 years or 10 years. The key thing is whether I think right now we're closer to figuring out how to build superintelligence than we are—

Figuring out how to control it. Okay, so I think the only way we fix that is simply by making sure no one is allowed to build it before it can be controlled.

Okay, so let me just respond now. You have—basically, the fundamental difference here is that I am saying the technology will be regulated in a wide variety of different ways which are fundamentally, and mostly, reactive. That doesn't mean that we won't pass laws. There are already laws that I've supported which have to do with AI regulation, and I don't think they impose substantial burdens on development.

I would also say the development of this technology is a national security priority. It seems really hard—like a really big cost—to impose something that would self-consciously slow ourselves down when others are not doing that. But I'm not even going to—I think that's a valid point, but that's not even where I want to go.

I feel like our crux is that you kind of want this precautionary-principle-based, preemptive regulatory regime that would require some group of people to say affirmatively yes before you're allowed to do something—

Like we do it with pharma and every other industry.

Yeah, which is what those are. Which is what those are.

I think that there are huge costs associated with a regulatory regime of that kind. I don't think the government could do it very well. As someone who has observed lots and lots of these regimes play out, I think it's very possible that by doing that in practice, you would actually end up not just being worse for innovation, but being worse for safety.

Are you arguing we should close the FDA, or did I miss an important distinction?

I mean, I would say the FDA is an organization that is in need of deep and profound reform. One of the things that happens when you impose a top-down regulatory regime like this is that you lock in all sorts of assumptions that you have about the world. So let's take the FDA as an example.

Before you give this answer, which I'm super interested in, it's very different saying the FDA needs to be reformed, with less regulatory capture and so on, from saying it should be shut down. Are you saying that we're worse off having it than we would be if we didn't have it at all, or are you arguing simply for a better FDA?

Well, let me just explain where I'm coming from here. I think these analogies of the FDA to AI are not really very good. It's not to say that I don't think we need something like an FDA. It's not that I don't think we need to test drugs before they go into people.

But then I'm confused by why you don't think we should have the same for AI. So what's the difference between AI?

Let me just make an uninterrupted point for a few minutes, if you don't mind.

Okay.

Okay. So first of all, when it comes to the FDA, we have this huge problem right now with a lot of drugs. What we have realized after several decades of modern science, as opposed to when the FDA was created 100 years ago, is that diseases are way more complicated than we thought they were.

They’re not really discrete things, and there’s kind of no such thing as cancer and no such thing as Alzheimer’s. They’re much more complicated, broader failures of very complex biological systems and circuits. The issue that you run into with that is that you need highly personalized treatments in order to solve these problems, because your cancer is different from my cancer, which is different from someone else’s cancer.

The FDA’s regulatory regime turns out to be entirely unsuited to deal with that because it was based on this sort of industrial-era assumption that diseases manifest themselves the same way across large populations of people. What you want to do is test a treatment over a big population and get average statistical results, as opposed to safety results for one person. What that means is that we have locked into place an entire economic structure for the way that we treat disease that is wrong for modern science.

It’s really nonobvious how we change it because there are a lot of entrenched interests associated with the current system, including the people who run the clinical trials that we operate at great expense—hugely more expensive than they should be. That would be one of a huge number of examples of problems that can manifest themselves with top-down regulatory regimes of this kind. The idea that we need such a thing for AI carries an enormous cost with it.

I don’t think that we really have the evidence that the cost is worth paying with respect to AI, compared to the many benefits that we get from not doing that and regulating it more like we have regulated things such as computers, the internet, software, and many of the other general-purpose technologies that have actually worked, grown, and made our lives so much better. I mean, not to say that medicines haven’t, but the things that have been real growth areas have been regulated more like this. A lot of why medicine has done so well has to do with software, computation, the internet, and things like this, as opposed to pure object-level advances in biology. That’s why we’re going to cure cancer on chips that were originally designed to play video games.

I guess I’m saying that you need to demonstrate a very high burden of proof—not to say it will never be met, but to say it hasn’t been met. Every single top-down regulatory system we have has carried with it a similarly high burden of proof. If you made a statement, going back to the ban-superintelligence statement, that said, “We need to investigate such and such,” or, “We need to come up with the guarantees that we want AI labs to be able to make in terms of empirical evaluations about their models—what are the guarantees we want them to be able to make?” I would be open to it. I don’t know if I’d sign on to it; it would depend on the specifics. But certainly, I would not have had the kind of visceral negative reaction that I did to the ban-superintelligence statement.

Cool. There’s a lot of good stuff in there. Let me pick out 3 things I’d like to respond to: one about regulation, one about perceived vagueness, and one about national security that you brought up.

On the first one, on regulation, it sounds like we’re actually in agreement that even though you’d like to see reforms to the FDA, you would not want restrictions on biotech to be completely eliminated. You would not want people to be able to do biosafety level 4 research to make that 95%-lethal bird flu airborne, for example, just because it’s cool and people can sue them later.

You would like there to be something, whereas for AI you feel there should still be nothing to prevent companies from deploying things yet. Maybe you’re open to it, maybe in 3 or 4 years—you just want us to think more about it. Whereas I guess my position there is that if someone releases actual, true superintelligence that takes over the world, it’s going to be too late to regulate it then.

On the second point, on vagueness, this is really important. Many people have said to me, “This statement that we put out on superintelligence—why isn’t it written much more concretely so you can make a law out of it or something like that?” That was very deliberate because, if you look historically in the United States when we’ve had new laws passed—for example, a law against child pornography—you could have pushed back and said, “If someone says they’re against child pornography, how do you define that? That’s too vague. What do you mean by ‘child’? Is it under 16 or under 18? And how do you define pornography? That’s pretty complicated, right?”

In the law, you can’t just say, “You know it when you see it.” But I think that would have been totally fine, because you can do that. There started to be a broad consensus that we need some kind of ban on child pornography. That created the political will for experts to sit down and hash out all these details. This is something you are very, very good at: looking at how you would actually draft the laws.

The idea with our statement was very analogous. We see that 95% of Americans don’t want to race to superintelligence. A lot of people are super excited about AI tools and view the idea of losing control of Earth to a new robot species as kind of dystopian, including David Sacks, no less. If we can start getting public knowledge that most people actually don’t want an uncontrolled race to superintelligence, just as most people want some kind of ban on child pornography, then that can create political will for brilliant policymakers like you to sit down, talk to all the stakeholders, and come up with carefully crafted language for how this would actually work and whether there should be a new agency.

In summary, this was not something I view as a bug—the vagueness—but I think it was a feature. What we were going for here was just some moral leadership: we would like there to be some kind of restrictions on a race to superintelligence.

Let me jump in for a second because I think you guys may actually dovetail more on policy itself than it’s sounding like. Maybe the real crux of this disagreement is your mainline scenario of what things would look like if we just went on cruise control and didn’t do much more than we’ve already done in the way of policy.

Dean, let me ask you this question: What is your P(doom), defined as just letting AI play out, not layering on additional regulation, waiting 10 years, and then the probability that it goes wrong and we get this runaway superintelligence that’s now too late to control or regulate? What’s your P(doom)?

Doom being defined as human extinction.

Yeah, a catastrophe—something on an extinction scale. Maybe half the human population dies and then we go back to being cavemen, or just something extremely catastrophic, or even extinction.

Would a permanent 1984 also count?

I mean, if what we’re talking about is AI systems taking control over the world and killing large numbers of people, my P(doom) is very low. It’s sub-1%; it’s 0.01% or something like that. It’s very low.

That’s not to say that I think there aren’t all sorts of other outcomes from AI that seem very bad and seem way more plausible to me, and those are things that I work on a lot. But the specific doom scenario just doesn’t really seem all that likely when you think about many different things.

One area would be this: I think that if you passed a law saying that a group of people would have to take a straight-up-or-down vote, and the results of the vote would be public—a group of 7, the Supreme Court, let’s just say we send it to the Supreme Court of the United States—the Supreme Court would have to look at every frontier language-model release and take a vote on, “Do we think this model is likely to take over the world?”

I would be unconcerned about that. I wouldn’t really support that law for a lot of different reasons, but I would not be particularly concerned if that were literally the law. The problem is that it’s not. I think this is where you get back to why the superintelligence-ban statement was written in the way that it was: a lot of the people who signed that statement, I would predict, have a much more nebulous set of concerns about AI than the very specific ones, Max, that you have.

It’s not to say, by the way, that you don’t have other concerns. I’m not saying you’re not worried about misinformation, deepfakes, job loss, or whatever else. But when it comes to where you actually think things could go wrong, I think we would both agree that the job-loss thing is really complicated.

Matt Walsh, the conservative influencer, had a tweet a couple of days ago about AI costing 5 million jobs over the next 10 years or something like that. I forget what it was. I was like, “That’s an extremely optimistic scenario.” In the grand scheme of things, eliminating 5 million jobs—if you just focus on what gets eliminated, sure, it’s not that much. But you have to assume some jobs are created. The economy creates millions of jobs and destroys millions of jobs every year. That would be a very slow rate of change.

Actually, if it only destroyed 5 million jobs over the next 10 years, that would be low compared to literally just the normal churning of the economy. I think the job thing is certainly complicated, but can we agree that there’s a plausible version of GPT-7 that’s really good and prosocial but might also displace a lot of jobs?

If we had a regulatory regime that said, “We need to take a vote on whether this will take over the world,” it would be 0 to 9 in favor of “This is not going to take over the world.” But if we had a regulatory regime staffed with union representatives and various other stakeholders—representative stakeholders, let’s say—and their task was to answer questions like, “Do you think this will be good for the economy? Do you think it could create job loss? Do you think it could be dangerous?” more nebulously, that group of people might vote against the release of that thing, and that might actually end up being a bad outcome.

Do you see that failure mode? Do you believe that failure mode is a real one?

I totally see things like this getting very political. It was very interesting for me because I spent a lot of time talking to many of the people who signed the initial signatories’ statement. Of course, for the ones who did sign, I was very interested to hear why, and there were indeed, as you say, many different reasons.

The national security people, like former Chairman of the Joint Chiefs of Staff Mike Mullen, for example, I think control loss was very central for him because they’ve used that as a national security threat. Regardless of whether the US loses control, the US government gets overthrown by a foreign power or by superintelligence, it’s an existential threat.

On the other hand, there were people from Steve Bannon to Bernie Sanders who felt that if we end up in a system where we actually have superintelligence that, by definition, makes all humans economically obsolete, then American workers would be dependent on handouts—either from the government in the form of UBI, which the conservatives who signed often view as socialism and don’t like, or from the companies. Handouts from Sam Altman’s Worldcoin or whatever would be viewed by people like Bernie Sanders as incredibly dystopian: the most massive power concentration in human history to a tiny clique of people from San Francisco who don’t necessarily share the same moral values.

That, I think, is something that bothered a lot of people. Then we had a lot of faith leaders who signed this for fairly different reasons. They felt this was really going to be harmful to human dignity. A lot of people we both know in San Francisco joke about a superintelligent god and so on, and a lot of these people are like, “Wait a minute. I already believe in a god. Why should I support some atheists in San Francisco building a new one that’s somehow got to run the show?” That sounds very undignified.

People have many different reasons, but I think, in short, there are 2 separate questions. One is whether there should be any kind of safety standards, like we have for biotech or restaurants. The second question, which is much harder, is what exactly should be on the list.

I would be very happy if we could start with just 1 very light requirement: companies have to make a good quantitative case that it’s not going to overthrow the US government before we launch it. Then we can have a broader political discussion.

Before Dean responds, Max, speaking of reasons to sign the statement and these nightmare scenarios, what is your p(doom)?

We actually wrote a paper—I and 3 graduate students from MIT—where we took the most popular approach for how humans can control superintelligence, known as recursive scalable oversight. We got very nerdy on it and tried to calculate the probability that control fails.

In our most optimistic scenario, we found that it fails 92% of the time. I would love it if people who think they have a better idea for controlling superintelligence would just publish it openly so that it can get peer scrutiny from others.

Until that time, if we go ahead and continue having nothing like the FDA for AI, so people can legally just launch superintelligence and worry about getting sued, I would think it’s definitely over 90% that we lose control over this. One percent versus 90% is a pretty big crux.

Wow. I just have this sneaking suspicion that if the models seemed like they were going to pose the risk of overthrowing the US government, or anything in that vicinity, I don’t think OpenAI, Anthropic, Meta, xAI, or Google would release that model. I think they probably wouldn’t do that, would be quite concerned, and would probably call the US government and tell them. It just doesn’t seem like a realistic scenario to me.

I agree there. I think the makers of thalidomide would not have released it either if they had known it was going to cause 100,000 American babies to be born without arms or legs. But it was complicated, and they just didn’t realize it.

It’s similarly very complicated here for people at these companies to know. Dario Amodei has himself talked about 15% to 25%. Sam Altman has also talked about how it could be lights out for everybody. So they’re clearly comfortable with 5% or 10%.

You can’t deny the possibility that something like that could happen. You can’t prove a negative, so you can’t say it’s 0%, right? If you’re being intellectually honest, you can’t say it’s 0%.

Here’s the thing: I’d say I have 2 observations about this. The first is that some of the negative effects you’re describing—including, by the way, a lot of the labor market effects—are going to be an emergent outcome of a general-purpose technology interacting with society. That’s very hard to model in advance in the way you’re describing with drug testing. It’s not impossible; it’s just very hard.

I think these emergent phenomena are easy to overstate. When you’re thinking about things like labor—and this is why a preemptive regulatory regime scares me—if a group of people are sitting around thinking about the potential risks of something, you tend to overstate them.

You hear this all the time when people talk about the impact of AI on society. An economist would say that they don’t endogenize the impact, which is to say they model AI as an exogenous shock, like a meteor that’s coming to society and going to hit us, while we remain completely in place and say, “Oh my God, there’s a meteor,” without doing anything.

The reality is that if I had shown you, 5 years ago, all the generative AI tools that exist today and simply said, “This exists in 2025,” you would have said, “Wow, I bet you...” You would have predicted a bunch of things, and I would have too, that would probably have been wrong about the downsides of the technology as they manifested. You’d guess, “My God, their elections must be completely over. Their media environment must be destroyed. There must be huge labor market dislocation. There must be no software engineers.”

In reality, society is an adaptive complex system itself. Just like the human body, it has the ability to internalize many different things, and it’s incredibly adaptive. Humans are quite ingenious. I think that’s an easy thing to discount.

The other point I want to make is about this recursive self-improvement thing, because this is another thing that gets to me sometimes. Every general-purpose technology in human history exhibits what you would call recursive self-improvement in the context of a human in the loop.

What I mean by that is, you come into the Iron Age and then use iron to make more iron and better iron. You turn a mill with an iron hammer that you’re using to manufacture more iron. You use computers to make better computers. You use oil to get more oil and electricity to get more electricity.

Every general-purpose technology exhibits these kinds of recursive loops because it’s a general-purpose technology. One of the things a general-purpose technology does is make the general-purpose technology better. That’s very common throughout the history of technology.

So, I think you can't just cite the fact that AI is likely to have recursive loops of self-improvement and that AI will be useful for AI, because every general-purpose technology is like that.

So, we don't disagree on anything here. It's just—I wrote the book, but there's a reason that, in the case of every other general-purpose technology, the recursive feedback loop tends to be autocatalytic, often produces nonlinear improvement, but never results in these runaway processes, right? We've never seen that. We used energy to get more energy, and then all of a sudden blew up the entire universe. That didn't happen.

Okay, Max, maybe you can explain why you still think there's a doom scenario despite Dean's point. Then I've got one more question for Dean. I'd love to comment on the national security angle, which I think is super important, also because it's the main reason given in Washington for why we should not regulate.

Yeah, so I completely agree, of course, that technological progress itself has always involved these self-improvement loops. I have written about that extensively, and that is fundamentally why GDP has grown exponentially over time, because we use today's technology to build tomorrow's technology and so on. But there have always been humans in the loop. When there are not humans in the loop, things can go quite fast.

If you look at a slow-motion video of a nuclear bomb exploding, there is no human in the loop. You get 1 uranium atom decaying, and then you get 2, then 4, then 8, and so on. The reason we've never seen anything blow up fast and wholesale with our technology is that we've always had humans in the loop as a moderator. Right.

Mhm.

It's pretty obvious, unless you think that there's some secret sauce in human brains that you can't build into machines, that it is possible to build machines that really don't need us. If those machines think 100 times faster than us and can instantly copy all the knowledge that other robots have learned into themselves, then we could see more progress in a month than we had seen in 1,000 years when there were humans in the loop.

This isn't my idea at all. I. J. Good articulated this very nicely in the '60s. I think this is just a very basic, simple argument. We can't say, "Oh, it never happened before, so it won't happen again in the future," because we've never built superintelligence before. All the other technology, like the Industrial Revolution, just replaced some aspects of human work, like our muscles. We made machines stronger and faster. We've never had machines that could entirely replace our cognitive abilities.

Can I just comment on that national security angle a little bit here? When I talk to politicians on the Hill, especially when I listen to AI lobbyists—of whom there are now more in Washington, D.C., than there are farm lobbyists and fossil fuel lobbyists combined—the main talking point they use to explain why we should not have any binding regulations is, in one word, China. "But China," they say, right? I think that is just complete baloney.

There isn't 1 race against China. There are 2 separate races that people really need to stop conflating with each other. One is a race for dominance, which was very eloquently articulated, Dean, to your credit, in the AI Action Plan: a race for dominance economically, technologically, and militarily. The way to win that kind of race is by building controllable tools, which I'm all for. You need big data centers and all the other stuff to build these powerful tools that we can control.

Then there's a second race: who can be the first to release superintelligence that they don't yet know how to control. That's the one which I've been arguing is a suicide race, because we're closer to figuring out how to build that than we are to figuring out how to control it. The Chinese Communist Party—Xi Jinping as well—clearly really likes control. I think it's quite obvious that they would never permit a Chinese company to build technology if there were some superintelligence that could just overthrow them and take over China.

I even got a firsthand anecdote about that from Elon Musk. He told me in spring 2023 that he had a meeting with some quite high-up people in the CCP, where he said to them, "Look, if someone in China builds superintelligence, after that China is not going to be run by the CCP. It's going to be run by the superintelligence." The reaction in the room was hilarious. Elon said there were a lot of long faces, like they really hadn't thought that through, and within a month or so after that, China rolls out its first-ever AI regulations.

I'm also quite confident that the Chinese have much more surveillance on DeepSeek and their other AI companies than the U.S. government has on our companies, and have both the ability and the willingness to stop something that they think could cause them to lose control. So, I think this is actually something where, the way I see this going—and when I said P(doom) of over 90%, that was, Liron, to be clear, if we don't do any regulation, right?—I'm actually quite optimistic that things are going to go well instead.

I think there's no way China is going to allow a race to superintelligence, since we don't know how to control it right now. I think China is going to continue steaming ahead, trying to build all these powerful AI tools for the race that Dean was describing in the AI Action Plan, but absolutely not let anyone build superintelligence. And I think that's what's going to happen in the U.S. also.

I think I even know already that a growing number of people in U.S. national security are beginning to view this as a national security threat. Maybe they listened to Dario Amodei talk about a country of geniuses in a data center in 2027, and they're like, "Wait a minute. I have here a list of countries that I'm keeping track of as national security threats. Did Dario just say the word 'country'? No, maybe I should add that country to my watch list also." Mhm.

And then suddenly we end up in this really great situation where the U.S. will also prevent anyone from building stuff that they don't know how to control. We'll have this race to see who can build the best, most powerful, and most helpful tools. That's the future I'm really excited about living in.

Okay, Dean, let me just ask you the last question, and then you guys can make your closing statements.

At least 90%.

At least 90%. Okay. And then, Dean, you see that as a very low-probability scenario, but you do have your own nightmare scenarios, which are on the side of regulating AI too much, right? If I understand correctly, your 2 nightmare scenarios are losing the AI race, because your AI Action Plan focuses so much on winning the AI race. And you have another nightmare scenario, which is that some kind of overregulation could lead to a tyranny situation and just undermine democratic governance, right? So explain your nightmare scenarios.

Certainly, I think all manner of tyranny is possible with AI, and I worry quite a bit about that. Fundamentally, what I worry about is—I'll start by saying that AI is going to challenge the structure of the nation-state no matter what, in the good scenario and the bad scenario. It challenges that in various ways and requires institutional evolution, conceivably revolution in certain places in the world. So buckle up, because that's coming no matter what.

But there's a version of that institutional evolution that basically looks like what we get is a rentier state. We get a state that is run—think of the Middle Ages—by a small number of people who control something. That thing is certainly a tool of violence, but they're not quite legitimate in the way that we think of democratic legitimacy. There's some sort of middle class of rent-seeking humans who have legal protections from that upper class.

Then there is this large underclass of people who have very low practical agency, very low ability to meaningfully contribute, and they're kind of stuck in some sort of dystopia. That seems very likely to me. I think there are many regulatory regimes, including a licensing one, that make that outcome substantially likelier. So we face these kinds of trade-offs no matter what. Should we hit on losing the AI race, or should we hit on a tyranny scenario? Or do you think what you described is kind of the main nightmare scenario?

Okay. All right. We've covered a lot, so let's go to closing statements, starting with Max.

All right. We've talked a lot about doom here, and you've kept nudging us in that direction because your brand here is the Doom Debates. But I'd like to end on an optimistic note. The real reason I'm so engaged with this topic is because I'm fundamentally quite an optimistic person.

I've spent so much time playing with this guy, and I'm very excited about the potential for him to have an amazing future where he doesn't have to worry about dying of cancer, and we can prosper like never before on Earth—not just for an election cycle, but in principle for billions of years. Maybe that even spreads out into the cosmos. We've completely underestimated, as a species, how much opportunity we have.

That's why I think it's so important that we don't squander all this greatness by making some hasty chess moves here and blowing it all. I think there are 2 very clearly different paths that we're choosing between right now, and we have to make up our minds within the next year or 2, probably. One of them is the pro-human future.

America was founded to be a country run by the people, for the people. There was a real emphasis that America was supposed to be good for the people living in America. It was not founded to be good for the machines of America. That was not the idea here.

The way to get there is to stop the corporate welfare toward AI companies, which is hard because they have so many lobbyists. But hey, so did the other industries that we eventually ended up putting safety standards on. Then steer technology to really be pro-human: to make life better for humans, cure diseases, make us more productive, et cetera, but make sure it's always us in charge.

So the first one is a very pro-tech scenario. Notice, we go full steam ahead with ever-better AI tools. The other scenario is that we race to build superintelligence, which, by its very definition, is superhuman, right? By the very definition of this, none of you can earn any money after that's been built doing anything, right? So you're going to be dependent on handouts from the government or some tech CEO, or you're not going to have any money, and life will be very bad.

That, to me, is an incredibly unambitious vision for the future. Why should we, after hundreds of thousands of years as a species on this planet working so hard to build ever-better technology so we can finally become the captains of our own ship, so we don't have to worry about getting eaten by a tiger or starving to death, throw away all this empowerment we've gained through all this hard work by deliberately building something that's going to take over from us? It's ridiculously unambitious.

I want us to take charge of this. I think 95% of all Americans, in these polls, clearly agree with this: to deliberately say, “Okay, we're in charge now. Let's keep it that way.”

A journalist asked me, “What on Earth do these different people who signed the statement have in common? I don't even understand.” She said, “I thought about it for a while. What does Bannon have in common with faith leaders, Susan Rice, and Chinese researchers, et cetera?”

Well, then it hit me: they're all human. So, of course, they want the pro-human future where humans are in charge, right? If we found out that there was an alien invasion fleet heading toward Earth, obviously we would all work together to fight the aliens to make sure it's us in charge.

And now you have a quite small fringe group from Silicon Valley with very good lobbyists basically saying, “Yeah, we should build all these aliens, and they're probably going to take over.” Elon even said that openly the other week. That is the most unambitious ending to this beautiful journey of empowerment I can imagine.

To me, the inspiring future that I'm excited about, and that I think we will actually have once people understand more what this is all about, is one where we remain in charge, keep AI a tool, and create a future that's even cooler than the science-fiction authors could imagine.

All right, great. Let's go to Dean.

I think the fundamental thing to think about here is really assumptions. Most AI doom debates, to use the name of the podcast, revolve around at least 1 of the interlocutors—usually the one who believes in doom—assuming their conclusions.

We've had a lot of conversation in this discussion about how superintelligence has many different definitions. It could mean many different things, and we don't quite know what it means. There's 1 version of it that you can articulate that means all sorts of bad things. But that doesn't mean that thing is likely to be built, doesn't mean that thing is going to be built, and doesn't mean that thing is possible to be built quite in the way that we imagine.

It just means that it's a thing you can say. It is a valid linguistic sentence in the English language. But then it takes a big leap to assume that that's what we're actually going to build. I don't think we should assume that.

I think, as Max said at the end of his statement, the future is often profoundly stranger than we can possibly imagine. The future that we live in today would have been unbelievably alien to someone 50 or 100 years ago, and certainly 200 years ago. In many ways, it would have been incredibly alien.

Many of the jobs we do would have seemed quite odd, as would the relationships we have with one another and our institutions—all of it would be deeply alien. My guess is that that continues. My guess is that the things we assume today about the technology of the future are probably wrong.

We don't want to embed too many of those assumptions into the law or into regulation. We want to maintain adaptability and flexibility right now, more than at any other time, given the speed with which AI evolves.

I just wouldn't assume that superintelligence means the bad thing. I would instead at least consider that there are many worlds in which humans can thrive amid things that are better than them at various kinds of intellectual tasks, and that humans can still have a role because there are certain things that are inherently not replaceable by machines.

We can gain a tremendous amount of wealth, live much better lives, and find all sorts of new things to do that are economically and practically useful. That's been true so far throughout human history, and it wouldn't have seemed that way to people at the time.

Since we have a written record, we know what people's reaction has been to new technology. It's always been like this. You can say this time is different, and that's fine, but I think we should demand a higher standard of evidence.

Max talked about how America is of the people, by the people, and for the people. True, but we also have a system that makes it quite hard to pass new laws. There is a reason that we made it quite hard to pass new laws, which is that our Founding Fathers were deeply distrustful of raw democratic impulse.

The word “democracy” was pejorative to the people who wrote our Constitution. It was an insult to say that a proposal seemed too democratic because they believed that you had to balance raw democratic will—people's raw intuitions about things—with deliberative bodies that make it hard to pass laws.

Laws are ultimately rules passed by people who have the monopoly on legitimate violence, and that's a very sacred and important thing. We don't want to just give them, willy-nilly, all these new powers.

We've done that a lot, and I have very serious issues with the idea—logically, philosophically, and even morally—that we're just going to be able to pass a new regulatory regime and everything's going to go fine, with no side effects. I think there will be tons of side effects, and I think we will ban tons of technological progress and stave off a lot of great, wonderful possibilities for the future.

There are many ways to investigate and interrogate the concept of superintelligence and to advance the safety and controllability of that thing. There are many ways to do it that don't involve banning it, which was the original topic of this debate.

I note that Max did not spend that much time defending the thing that actually was the subject of the statement that FLI put out. But I think you can build something much better than just a regulatory regime. You can build a society that is capable of grappling with this technology, and institutions that are capable of evolving with it.

I think that's ultimately going to be a much healthier, better outcome for the world. That's the one that I work on every single day. It involves taking the risks seriously, and it involves taking the technology very seriously.

You shouldn't be a radical in either direction when it comes to this technology. You should be willing to update your beliefs frequently. But at the same time, details matter.

Getting this right is not going to be a matter of taking regulatory concepts that we've developed for other things off the shelf and applying them to this. It's going to be much more difficult than that. So I guess I'll close with that. Thank you. I'm thankful to both of you for stepping up to debate the difficult policy questions around superintelligent AI.

It’s such a complex issue, and there are so many different positions. It’s not black and white. It won’t work to do it in an echo chamber. It won’t work to reduce AI policy to left-versus-right politics. Respectful debates between smart people with different views are what we need right now as a country, as a species.

That’s how we can stress-test different ideas and bring out important nuance. I’d go so far as to say debate is a key piece of social infrastructure. So, thank you again, Max and Dean.

Thank you, Dean, for a really great conversation. Thanks to you, Max. Thanks to you, Laurel. This was great. Wow, what an illuminating debate from 2 people who are actually in the room for these kinds of policy discussions.

Regarding America’s AI Action Plan, the document that Dean Ball helped draft, both Max and Dean were happy that it doesn’t mention superintelligence, but for very different reasons. Max was saying we need a whole other statement about superintelligence, and he even proposed a statement saying it should be banned until there’s broad scientific consensus that it will be done safely and controllably, and until there’s strong public buy-in.

That’s what Max thinks we should do regarding a superintelligence statement. Dean is saying, “Yeah, it’s good that we didn’t mention superintelligence because it’s too vague right now.” Dean is saying that, from our current perspective today, we don’t know what superintelligence will look like. Maybe it’ll just work out really great and won’t need that much regulation.

So, it’s a very diametrically opposed perspective: pushing to ban it until there’s consensus versus, “Well, we’ll deal with it later. It’s fine for now.”

The crux of the disagreement between Max and Dean, as we uncovered during the debate, really does come down to their P(doom). If you remember, Max was saying his P(doom) is greater than 90%, assuming that we don’t have these tough regulations on AI, but Dean’s P(doom) is only about 0.1%. It’s a much, much lower P(doom).

Dean is basically not worried about plowing forward and dealing with issues as we get to them, whereas Max says we’d better be preemptive because it might be too late to regulate if we don’t start right now.

The reason I say P(doom) is the crux of their disagreement is because that’s the thing that I think would really change their mind about the other stuff. I think their policy recommendations are totally downstream of what they see as the probability of doom.

For example, if they were to meet halfway—if Dean were to go up from 0.1% to 25% and Max were to go down from 90% to 25%, or just anywhere in that middle range, 40%, whatever—they’d start coming up with very similar policy ideas.

I think in that case, Dean would naturally say, “Okay, well, we need very tight security on this development. You can’t just retroactively do it because there’s a high risk of total destruction, right, of runaway AI.” They would just naturally be thinking along the same lines. It’s all going to be downstream of how much doom they expect.

So, they went on to talk about the FDA analogy because they said, “What does good regulation look like?” Max was saying, “Isn’t the FDA a success story? Don’t you like the FDA?”

Dean’s response was that he didn’t go full libertarian. He didn’t say, “Oh, the FDA is evil. We shouldn’t regulate things like that.” He said, “Yeah, the FDA is helpful, but you can see it has this baggage. It has this legacy idea that 1 medicine has to treat 1 disease.”

Really, science is a lot more complex than that. So even the FDA is kind of putting this straitjacket on, where you have to jump through all these hoops, but you’re not necessarily getting a lot of productivity. You’re paying a high cost in terms of drag.

Dean’s analogy is that when we get to artificial general intelligence or artificial superintelligence, this kind of straitjacket regulation could be an extreme case of why Dean doesn’t like the FDA, even though he admits that some amount of regulation is good. So that’s kind of where they landed with the FDA analogy: the idea makes sense, but there’s this failure mode that’s going to bite.

But once again, from my perspective, the reason they’re not on the same page about the FDA analogy is just because Max is seeing this runaway risk, this doom risk, and Dean is like, “No, it’s really just about yet another technology,” right?

This comes up a lot in other debates: Is AI just another technology? My take is that Dean is very much leaning on the yes side of that and Max on the no side.

In conclusion, what a stark divide. We have a scientist saying there’s a high risk of a catastrophic outcome from superintelligent AI in potentially less than 10 years, and a policymaker saying, “You haven’t made a strong enough case for why the risk is high, so we shouldn’t ban a hugely valuable line of research, and regulation by default is burdensome, so we should be constantly worried about overregulation.”

It’s quite a difference of opinion to reconcile, and what’s crazy is that the stakes are so high and the timelines are so short.

My prediction is that we’re going to keep seeing policy that’s downstream of the policymaker’s P(doom). One of the benefits of having Dean Ball in this conversation is that we heard his perspective on P(doom) more explicitly because it wasn’t explicitly mentioned in America’s AI Action Plan.

More generally, I think debates about P(doom), or doom debates, if you will, are extremely productive for discourse. It’s not just about this one disagreement between America’s AI Action Plan and the Future of Life Institute statement to conditionally ban superintelligence. It’s a bigger picture. It’s about building the social infrastructure for high-quality debate.

The world is complex, and debate is one of the most powerful tools that we have as a society to navigate our way to appropriate policy decisions. But it has to be high-quality debate. The people have to be informed. It has to be respectful. It has to stay focused on the issue, not on finger-pointing, character assassination, or scoring political points.

It has to be nuanced, with the 2 sides actually trying to find common ground, if at all possible. And it has to be productive for policymaking.