The Compendium - Connor Leahy and Gabriel Alfour

If you can't even put together a Manhattan Project for this, right? If you can't even put together all of our greatest scientists working on this for a generation, you're not going to make it. You're not even trying.

This is a thing that has made discourse around AI so bad. If someone comes to you and tells you, “This is the definitive difference between humans and animals, between ChatGPT and pre-ChatGPT AI, or between ChatGPT and humans,” you should be like, “Well, they're a crank.” This is a scientific problem where we put a lot of effort in, and we failed. Why do you think your intuition, just because it feels like it, is the correct one?

We want to understand intelligence. We want to understand coordination. We want to understand alignment. These are obviously problems that are extremely relevant to our species' long-term happiness, but they're also very dangerous.

The goal is just to buy more time. Right now, there is a willy-nilly race. There is almost no thought given to safety.

If one does not address these problems, then you lose, and currently we're losing.

So Tufa Labs is a new AI research lab I'm starting in Zurich. It is funded from, uh, past ventures involving, uh, AI as well, and so we are Swiss version of DeepSeek, so a small group of people, very, very motivated, very hardworking, and we try to do some AI research starting with LLM and Owen Stein models. What we're looking for now is, uh, chief scientist and also research engineers. You can check out positions at tufalabs.ai.

Okay, guys, welcome to MLST. It's so good to have you both here.

You ready to be back? It's been a while.

Yes, indeed. In November, you guys published a piece called The Compendium. It's a monster. It's about 115 pages long, and I really enjoyed reading it. Maybe, Connor, first, can you give us the elevator pitch of that?

1. The Compendium's Core Case

Very simply, we just want to make the full case for why we think extinction from superintelligence is very likely, why we think this happens, why we think it's still happening, and what can be done about it.

Okay. In the first section, you're talking about the state of AI today, and you said that recent AI advances are due more to scale. You're saying that safety is being sidelined. OpenAI, DeepMind, and Anthropic are pouring resources into building more powerful models. They're openly acknowledging the risks while they continue to go full speed ahead.

I think it still holds true. The advances that we're seeing in AI are not from understanding different aspects of intelligence. It's not making intelligence into a more measurable, objective property. It's not factoring it into natural concepts, and so on and so forth. It's still mostly black-box approaches: iterating on algorithms until we find one that works, and then pouring in more compute.

I still think that the risks are openly acknowledged, and I still think they're racing for it. It seems pretty transparent from that point of view. The world in which we learned more about intelligence and started to understand more about intelligence and how it works is very different from what is currently happening, basically.

But, Connor, why is it the case that intelligence can just be a black box that we can continue to train? You made the comment in The Compendium that this type of technology is not designed; it's grown. Why is that the case? Why is it so easy, if you like, to mechanize intelligence?

2. Intelligence Remains Unsolved

The truth is, I think this is one of the great mysteries of our universe. I think this is genuinely an unsolved question. I think it's genuinely surprising that if you have a bunch of particles and you shake them around in a bit of a soup, eventually a human pops out. I think this is genuinely weird and not something that would be obvious.

Even if we had a chimp, right? If you tell me, “I take a chimp”—I don't know if you've seen a chimp. They're kind of terrible, you know—and you give it a bit of a bigger brain, now they start making art and nuclear bombs. This is not obvious, right? It's not obvious that you take a bunch of neurons and put a bunch more neuron soup together, and eventually you get something that can invent mathematics.

This is a fact about reality, though. For me, scaling and neural networks working isn't something that was predicted by some deep underlying theory of physics. It's just something we observe happens, and we don't really know why. Some people take this as, therefore, it won't keep happening, or therefore it won't matter, but I think of it kind of the other way around. The fact that we don't know why this happens doesn't allow us to make any predictions about when it will stop or how it will go forward.

One thing that I was taken aback by is that, in a way, this argument diminishes how clever we are as biomachines, right? We self-replicate when we have children. There are no power grids, no orchestration network. We just have sex, and 9 months later a child pops out. It just seems incredibly efficient, right?

You also made the comment in The Compendium that there are no physical laws that prevent this type of intelligence from emerging. But I think of humans as being in a different category.

You can think of it that way. I think it definitely makes sense that humans were optimized by a different process for a different thing. I think that's a fair way of seeing things. I just think it's kind of irrelevant.

What I take from this is that weird things are possible, and predicting what is possible is very hard. I think it's just very, very hard to predict what is or isn't possible. It's definitely true that humans are extremely clever in many of the ways they're designed. In some ways, extremely stupid, you know?

The fundamental thing is that we don't understand intelligence. We don't really know what it is, how to build it, or how to structure it. We don't have a science. We're pre-scientific.

It's kind of like how alchemists thought about chemistry. Alchemists could look at acids and metals and organic compounds and be like, “Well, some shit's going on here,” and obviously it's important. I think we're in a similar circumstance with intelligence. I don't think we can't figure it out. I just think we currently haven't.

I'm just trying to rationalize this, because a lot of people will hear you saying we need to basically make AGI illegal. That seems like—not hyperbolic, but it, for the average person, seems like a very strong thing to say. What's the justification for that?

What you just said is not true. The average person actually completely agrees with this and thinks this is completely commensurate. We've done polling on this.

But among ML researchers, it was 10%.

Yes.

I think the fact that you still get 10% on extinction risk, that you still get many people talking about extinction risks, is a good sign that you can go against incentives. Geoffrey Hinton left Google to talk about extinction risks. Incentives are not an iron law that confines you and prevents you from being an actual human with agency and free will.

Isn't there this notion, though, that even if AIs could self-replicate, they would need to have factories? They would need to have access to materials. It feels like there's this argument that intelligence is all you need, and it's almost an appeal to omnipotence: if only they are smart enough, they can do all of these things.

But surely, in the real world, you need to be moving molecules around. You need to be working with materials and so on. It seems like, in the physical world, it's a really difficult thing to do to build this kind of intelligence.

3. The Economy Becomes Extended Cognition

Yes, absolutely. Building things is hard, and the fact that humans can build microchips and chimps cannot is both downstream of us having thumbs and of us having higher intelligence. These things are very closely related.

But when we're thinking about building an AGI, we're not talking about building a thing on a rock in outer space, you know? We're talking about a thing that's deeply integrated into the economy. These systems are being integrated as hard as possible, as fast as possible, put on the internet, given all human knowledge, given access to tools, given human supporters, human helpers, and the wider economy.

The economy is much smarter than any individual human. The economy can build machines that I can't build, that no individual could build. I don't think any individual person can build a lithography machine, but the economy can. Humanity can build lithography machines.

I think it's useful to think about how, in practice, when we see AGI or ASI systems emerge, they will be part of these systems, and they will use these systems as part of their extended cognition. This powerful AGI system will be able to do anything a corporation can do, and probably will, at least for a little while.

It will probably purchase machines. It will probably do research. It will probably pay humans to do things. These are useful levers. Will it do that indefinitely? Probably not. If there was something that's much smarter than the entire economy and it gets a couple of years to develop new technology, I don't know what technology it would invent.

Maybe it would involve paying humans. Maybe it wouldn't.

Interesting. One of the comments that you made about intelligence, Gabe, was that you define it in quite simple terms. It's just the ability to solve problems. Why is that a good way to think about intelligence?

4. Intelligence Has No Clean Boundary

I'll be mean. I think a nice reason for why it's a good way to define intelligence is that it's quite underspecified. You can pick whatever classes of problems you want. If you want to pick formal problems, you can get into computability theory. If you talk about problems with different time and space resources, you get back into complexity theory. If you think about more logical descriptions or grammar-based descriptions, you get into Chomsky hierarchy theory.

It's vague enough that it lets you define the types of problems that are of interest to you. For instance, we can just take a couple of benchmarks, like ARC-AGI, and say that this is the definition of intelligence. So it's a very practical definition—not in the sense that it's very grounded, just in the sense that you can use it however you want. That's why I like this definition.

You made this very good argument in your paper that there are always people who say there's something missing, right? If only we had intentionality or something else. You were just touching on the Chomsky hierarchy and so on. Chomsky said that humans are different. There was this Promethean moment where we had some weird genetic mutation, and we could now do the merge operation in our brains. We became Turing machines, and that's the reason why we dominated the world, right? We have this ability that we didn't have before.

That seems to go against this notion of a continuum, right? Connor, you were just saying, look at chimp brains, look at our brains. The brains get a little bit bigger, and now all of a sudden we're dominating the world. But it's more interesting, from my perspective, to think that we are GIs. Chimps are not GIs, and if you think about it, ASIs will also be GIs. They'll be Turing machines. We'll be able to communicate with them. It seems to me that it would be different with us and ASIs than it would be with us and chimps.

Maybe. It depends on what you mean by different. I don't know if you've ever actually observed chimps for an extended period of time or read a book about chimps. We have a lot in common with chimps. Chimps have politics, you know. Not just, “Oh, this guy is higher rank than this guy.” They scheme. They make plans: “We're gonna ambush this guy so we can make this guy the leader, and then he will repay me.” This is the complexity that chimps have.

I really recommend reading actual books by actual primatologists about chimps, just telling stories about chimps. One of my favorite genres of philosophy, which Chomsky was a great example of, is philosophy of mind that could be completely disproven if its proponents had met 1 animal in their entire life: animals are way smarter than a lot of people realize. Dogs can plan. They can recognize patterns. They can remember things. They can understand identity. They have theory of mind.

It's not perfect, right? Sure, it's much more primitive than humans, but it's obviously on a spectrum. Chimps, orangutans, and so on are on a spectrum. Even rats have a lot of sophisticated behaviors. I'll look at a rat do something, and I'll be like, “He's just like me, for real. I understand why he did that. He had an emotion there. Look, he's angry.”

I think especially with mammals, this isn't just a noise. There is kind of a steady increase of a thing. A lot of people have tried very, very hard to use all of their big-brain IQ points to come up with something that will cleanly, beautifully separate humans from everything else. I think the story of science, ever since Darwin and beyond, is that there is no clean separation. There is no clean cut. There is no Turing machine, not-Turing-machine distinction.

All the evidence points to the opposite. You just do more things, you have more patterns, you have more general cognition, you have more compute, you have more things, and you have better algorithms, but it's all on a spectrum.

I agree with so much of what you just said. I think it's possible for both things to be correct. There was a great book by Max Bennett, A Brief History of Intelligence. He was saying that mammals have the same prediction apparatuses as we do.

But maybe there's something to what Chomsky is saying: that we do have a capability that is discontinuous, and there is a continuum of the other capabilities that you're talking about. It goes to your point that we can galaxy-brain ourselves, and we can point out specific instances of capabilities that we have that animals don't have, and that ASIs might have. But your premise is that this is just a pointless discussion.

I think there is a point to the discussion where you must take a step back. People make the mistake many times of being like, “Oh, the difference between animals and humans is tool use,” and then animals use tools. Okay, well, it was not tool use; it's language, and then animals use language. So you keep postponing what the actual thing is.

On one end, you could be like, “Well, there's no such thing.” On the other end, you can still be like, “Well, I predict that the next generation of humans might be able to build nukes, and I still predict that the next generation of chimps won't, and I predict that no other animal species will.” We have an internal model, an internal understanding that there is a difference. What we must recognize is that we don't have a good understanding of what that difference is. Is it the last difference? We don't know. We do not have a good understanding of this.

We can have hints, intuitions, and various other things. I will happily share mine right after. But I think the first thing to do is to actually take a step back and be like, “Well, when scientists tried to define this difference, they failed.” I think it's a very important step in science, which is to realize that we failed to solve a problem.

For instance, P versus NP is a very salient problem where we realize that it fails. Whenever someone comes and says, “I have a solution to P versus NP,” our first reaction is, “Well, you're a crank.” I think we should have a similar attitude with intelligence—not necessarily to that extreme. If someone comes to you and tells you, “This is the definitive difference between humans and animals, between ChatGPT and pre-ChatGPT AI, or between ChatGPT and humans,” you should be like, “Well, they're a crank.”

This is a scientific problem that we've put a lot of effort into, and we failed. Why do you think your intuition, just because it feels like it, is the correct one? Personally, I think this is what has made discourse around AI so bad: basically anyone feels like they have the correct intuition. I think having the humility to take a step back and be like, “I have my guess, but these are my guesses,” is quite hard.

There are discontinuities and continuums. Let's say at some point you must compound enough that you discover, I don't know, oral tradition or writing or something that is higher bandwidth. If you don't get there within a generation or at a tribal level of complexity, you never develop it. If you get there, then you can use this new bandwidth, this new type of system, to do more.

Once you have writing, you can have traditions that persist way past the degradation of oral transmission. When you have a clear language with a clear grammar—even before writing, with more systematized grammar—you can have oral transmission that lasts for longer. I'm not saying it's writing or oral transmission. I'm literally just using those as examples. But what I mean is that you can also have discontinuities that emerge from continuums.

You're better at language, and through this and some chance, you get to a place where you develop some oral tradition that lets you do much more—possibly proto-religions that were impossible for previous species. This lets you do more and more and more. I think there's also this type of stuff that's easy to forget: these continuums emerge from continuums. Qualitative differences can emerge from mere quantitative differences. Ultimately, it's all physics and quantum waves, this type of stuff.

I think for ASI, there are similar considerations. If you're a human and must learn something—even just learning 1 fact as a human—you must do spaced repetition, and it's still not that good, and so on and so forth. Teaching everyone 1 fact is basically impossible, whereas for AIs, doing a fine-tune of all the deployed instances is trivial.

With ASI, you expect that you will have these types of discontinuities from continuums. You can realize some dynamics, but even as you realize them, you should keep in mind and be very focused on what you do not understand. You should make clear what you're confused about, and so on and so forth.

I just don't know what they are, and we should not be pseudoscientific and use pseudo-technical terms to talk about those extra things. Right now, our theories only support saying vague things like this. We do not have a strong enough science of intelligence to be that precise.

I kind of agree with you. First of all, you were talking about the nuance of the dynamics of collective systems, systems thinking, and emergence, right? A lot of people, certainly in the olden days when we thought about intelligence, thought all intelligence happened inside the brain. Even if we needed to have some kind of emergent properties, they thought that it would only emerge from a baseline, a basis of capability inside the human brain, and that's not necessarily the case.

So we might say you need to have Turing-machine brains, but maybe you don't, because you can build systems of lower-level brains or systems of LLMs that are Turing-complete in some way, so that the whole thing becomes a wash. The problem, though, is that we have this legibility problem. We need to have precision in regulation, right? If we're saying all of these things are extra, we can't use the word agency, we can't use the word consciousness, then how do we have the precision and the legibility to actually talk about the things that we need to regulate?

5. Regulating An Unsolved Science

When I think about doing hard things in the world, I think it's very important to separate hardness—how much effort it takes to do something—from whether something's possible. For me, these are different things. Most of the arguments of the kind of shape that you're talking about are something like, “Well, we don't know exactly what the intelligence thing is, therefore it's impossible to regulate.” I disagree with this. The argument that we don't know exactly what the thing is, so therefore it's hard to regulate, I agree with.

I'm sure most people have heard about Big Tobacco, especially in the ’50s and the ’60s, and its massive campaign against the emerging scientific consensus that smoking caused cancer, which is, of course, bad for business. There's a fun thing where, to this day, we still don't actually know which chemical exactly in tobacco smoke causes cancer. This is still not 100% clear. We have many good guesses and so on, but it's not 100% settled.

So back in the ’60s, the tobacco companies said, “Well, of course, if smoking caused cancer, we'd want to regulate it. But let's first find out what the chemical is, and then we'll figure it out.” This is the same thing that's happening here: “Sure, of course we should stop AGI once we've figured out what it is. So you come back to me once you've figured that out.”

This is just an isolated demand for rigor. This is a completely disproportionate demand for rigor. It would be great if we figured out the proper metrology of intelligence, if we figured out the proper signs of intelligence, and boom—here's the thing: if you do this thing, all problems are solved. That would be great, but this is not the world we live in. You have to say, “What is the smallest circle I can draw where I'm guaranteed it's in there?” The more we understand intelligence, the more we can shrink the circle. But if we don't understand intelligence, that circle has to be quite large.

This reminds me of that expression—it’s like pornography, right? “I know it when I see it.” And you're saying it's like intelligence. It's dangerous intelligence: “I know it when I see it.” So it feels like an appeal to our intuitions.

“I know it when I see it” is talking about something grounded in reality. In regulation, you can talk about things that are grounded in reality. For instance, you can say, “Let's ban all projects that have the explicit aim of building AGI or ASI.” This is not a thing that you can measure, but it's a thing that a judge can decide on. This is why we have a judicial system and why not everything is done in laws through the legislative systems.

We can mention fraud, we can mention deception, and so on and so forth. They're not fully grounded definitions, but there's still a judge who can come and be like, “Yep, that's it.” So first, that's one thing that you can keep track of in reality, which is just the intent of people. We have ways to talk about people's intent in law. It's normal. We're people. Laws are for people. So if we could not talk about this, we'd be pretty screwed. We can leverage this.

For instance, we can talk about compute and be like, “Yes, let's ban training runs beyond a certain threshold of compute,” and so on and so forth. But compute by itself is not ASI. You can have ASIs under thresholds of compute. If you just limit compute, what you're doing is that you're funneling the resources that were going into compute into algorithms or into more theory, and so on and so forth. So you must actually go wide. You must look at the different precursors to superintelligence.

It can be like: I'll ban large amounts of compute. I'll ban this type of open-source models. We can ban automated AI R&D, so just AI systems improving themselves without any human in the loop. We can ban systems that require less than 10 days of effort to jailbreak. We can ban fully online systems, where there's literally no kill switch, no gate, nothing like this. There are many things that we can do, and that's what it looks like to be in a pre-scientific field.

It does come back to this pornography thing again a little bit, though. So you're saying that even professional ML researchers—many of them, in fact almost half of them—think that there is a very significant risk of existential harm. Average people just out there also think there is a significant risk of existential harm. And the average person—I mean, how are they basing that assessment?

I think it depends on the individual, of course, but I think there's a very common intuition: if there's something that's smarter than you and you don't control it, that's bad. It turns out that's a really good intuition. All things equal, no, you shouldn't build something smarter than you that you cannot control. Just the burden of evidence should be on why that's a good idea. Why should we give you this extraordinary license, this unusual license to take this completely unprecedented amount of risk with civilian lives that did not consent to be part of this? Why should we give you a license like this? It shouldn't be that you prove to me that I shouldn't do this.

The thing I'm wrestling with is that it sounds objective when we use these percentage numbers. Sayash Kapoor put a piece out about this in his AI Snake Oil blog, and he was saying that they're subjective probabilities. With the smoking thing, I'm not sure whether, at the time, there was hard evidence that it was killing people and giving people cancer. Maybe there wasn't. But isn't it a little bit more speculative now that we think there's a 10% risk? How is that actionable?

And think about this: I moved in next to your house, where your kids live, and I went into my garage and started making napalm or brewing homemade pipe bombs. What is the probability that I will blow you up? You don't know. That's just your subjective probability. You're just being subjective about whether or not I'm going to blow up your children. You don't have an objective probability of how likely my pipe bombs are to blow up your children.

This is how criminal law works. This is how risk assessment works: there are no objective probabilities of how likely you are to have an accident, or how likely this will happen or that will happen. It's just that we use the best models and judgments that we have.

But this comes back to the legibility thing, right? If I threaten you with a gun, it's intelligible that I'm threatening your life. If I slowly poison a lake over the course of 50 years and there are all of these second-order effects, it's illegible, right? I feel like it's the same way with AGI, because you're basically saying that there's going to be a catastrophe: if we don't act now, there's going to be a catastrophe.

But all the other people out there are saying, “Well, wait a minute, guys. I'm building these amazing applications with these LLMs, and I'm doing all these cool things, and there's economic growth, and it's capitalism, and this is an amazing thing.” How do you explain to those people that this is actually very dangerous?

I think there's a very nice thing here: we're not in that world. We're in the world where the people building the systems—the ones at the frontier, like Demis, Sam, or Dario—have actually said that there are extinction risks. They actually signed a statement in public saying that there are extinction risks from AI and that it should be a global priority alongside pandemics and nuclear war. I don't understand why we'd grapple with a hypothetical that is grounded in nothing, given that this is not the world we live in.

We have Geoffrey Hinton, who has left Google specifically to talk about these risks. We have Yoshua Bengio, who talks about those risks. So there are a few people who disagree. The point is not that this is unanimous, but the point is that we're not in a world where people working on this are just completely denying everything about it.

You mentioned Dario Amodei. He put out a piece recently called Machines of Loving Grace, and that felt like a bit of a U-turn because he was much more upfront about the risk from AI earlier on. I can't help but get the feeling that now they've got so much investment, you have to say the right thing.

And on the other side of the spectrum, you've got people like Marc Andreessen, and he put out this techno-capitalist manifesto. In your compendium, Connor, you actually wrote some profiles about all of the different reasons why people behave the way they do. Maybe bring in those 2 manifesto pieces, but also, how does it fit into these profiles?

6. Ideology Drives The AI Race

On the individual level, or close to the individual level, I think it's sometimes useful to think about 5 major ideologies that are currently involved in the AI race in particular. The oldest kind that's pushing for this is what we call the utopists. These are people who, for various reasons, think that transhuman utopia is both desirable and achievable, and in fact, they should be the ones to do it. So this underlies a lot of your Darios, your Demises, your effective altruists, and stuff like this.

Another view of this, which is the Andreessen view and seems similar to this, but I think is kind of different, is the accelerationist view. What they're saying is, “More growth, more good. More technology, more good.” So we just make more technology and put it there, and then good. Accelerationists don't even try to be king of the world. They're just like, “We just build all the technology and open-source it, and then good.”

There are other factors involved in this as well. Another big one is Big Tech. For them, AI is just another form of power, another form of gain. Nothing particular out of the usual there. And, of course, there's also just a bunch of opportunists who are just kind of in it because it's hype, because crypto is not the thing anymore, so they have to go somewhere. They're like, “This is the thing.”

Okay. And essentially, all of these folks believe that whoever gets to AGI first will win?

It depends, right? Everyone thinks this to different degrees. A lot of the accelerationists believe that the person who gets to it first has to be nobody. You have to open-source it. And the utopists think, “Well, if I get AGI, everything good.”

Big Tech, I don't think, really thinks that way at all. It's just power, power, power, power, power. And the opportunists, I think, don't think at all. I think it's just whatever makes them money. I don't think they believe things.

But people like Andreessen, even Satya the other day, he said, “My benchmark of progress is growth.” Undeniably, over the last 100 years or so, there's been at least a correlation between growth and increased standards of living and whatnot. So it's a reasonable inference, even if possibly an incorrect one. What would you say to those people?

Many things to say. The first one is just that fertility has been crashing for the last few decades, even though we still had growth. So I think the model is pretty bad. There are many things that have worsened even though we still had growth. I don't think this is an actual intellectual position. I think this is much more driven by ideology.

I'm interested in labor market disruptions as well. And I know that's a bit more of a near-term risk. But I think it was Buckminster Fuller, and quite a few people, as you've just been alluding to, said that when we have this technology, we won't need to work very much anymore.

But now we're in an even weirder stage, right? Because we have the prospect of the value of human labor going to zero, because AGI will be able to do everything. And then we've got this notion of, well, do we need universal basic income? Are we heading towards a form of zombie capitalism, where people don't have anything to do and people have no purpose in life?

So I think there are 3 main outcomes that different people will expect. I think one is extinction risk. This is the one that Connor and I expect is the main line on our current course of action. Of course, we do things because we think it can be changed, but right now it's the main line.

The other one is that we have somewhat controllable, somewhat superhuman intelligence. And the last one is the one that you just mentioned, which is post-scarcity. The truth is, our science of economics is not good enough to manage this.

Basically, the 2 proposals that I often hear are, one, communism—people being like, “Yep, the state will just dictate what everyone has, and the state manages everything,” and so on and so forth. The other one is UBI, or universal basic income. UBI in practice means the AI companies produce everything, you tax them, I don't know, 90%, and you redistribute this as income to people.

Then people have some market dynamics, where we have strong enough regulation that AI companies do not just strangle people. Have you lived on Earth in the last few decades? The last few decades are the decades where Big Tech has not been regulated at all. With social media, they just could do whatever the fuck they wanted. Newspapers are super regulated. Radio, super regulated. TV, super regulated. Social media: much more power, algorithm-mediated, no regulation whatsoever. With AI, no regulation whatsoever. Voluntary commitments.

So do you think that in 5 years, you'll have the political power, when they have the entire economic power, to be like, “Well, now we'll tax you and we'll make you give money to everyone”? How do you have a good economy when you have a few companies that have overwhelming power and you do not have leverage through your population? Or rather, the population has no leverage over you, and you just get lobbied by them, and you've already failed for 15 years?

I mean, Yanis Varoufakis used the term techno-feudalism, where the new capital is GPUs, and we have this insane concentration of power. The weird thing to me is that a lot of accelerationists kind of accept that this is the future, yet they still welcome it. Why is that?

Ideologies are a way of thinking where you just want a simple solution, and you're trying to justify why this simple solution works. You're not looking at the world and saying, “From the current world, what is my portfolio of solutions that I must approach and refine?” It's backward thinking. You start from the solution and then try to justify it as much as possible.

I think there are other reasons. The biggest one is just a big misunderstanding of humanist and Western values. The West has triumphed because it built strong institutions. The reason why you can have nice competition that is productive and so on and so forth is because we built an entire apparatus that was meant for this. That's the entire point of liberal humanism, the 19th-century doctrine, and so on and so forth.

If you don't have rules, you don't have competition. You get the state of nature, you get street fights, you get Somalia. These are not nice places. People prefer watching MMA, and people prefer the West for doing business. So you need a very strong rule of law. You need very strong institutions that will enforce contracts, internalize the externalities—the negative externalities—and make sure that you can compete, that you will not cripple your opponents, that you will not inflict harms on everyone else, and that the rules will be such that when you compete, you do something beautiful, you do something productive, and things like this.

This understanding, I think, has been eroded by massive FUD campaigns and massive lobbying campaigns from VCs and techno-optimist VCs, where they just FUD everything and finance anyone who will say those things. Not to disparage the spokespeople—I don't think they matter much. I think they're mostly driven by power and ideology.

But for all the people following them, I think those people just forgot what made humanism great, what made Western institutions great, and what made our constitutions last for 250 years. It was that we actually managed to strike a pretty good balance between fierce competition and a strong frame that leverages this.

Yeah, just to play devil's advocate a little bit: the libertarians also want to have laws, particularly around the protection of their private property and many other things. But they would say—because I understand that the power lens is a great lens to understand things—a lot of libertarians don't have power, and they say, “We need to have the natural world distributed. It's robust, and open source is a good thing because we'll build AIs that can counterbalance each other, and the system will be better overall because we'll have this kind of freedom of enterprise and private capital, and so on.” What's your response to that?

7. Chaos Is The Default

Nature is not a beautiful balance. It's red in tooth and claw. Nature is full of parasites and violence and cannibalism. It's terrible. Being a wild animal is terrible.

The idea that nature, or existing in nature, is this nice, beautiful thing where everything is good and you have everything taken care of, is such a privileged, first-world-problem view of reality. It's like growing up in a sheltered suburb, which is an incredibly complex system optimized to give you as much comfort as possible, and you're like, “Wow, I don't…”

I'm basically a wild animal. Look at me. See? The world's easy. Why is everyone complaining? Many of these people come from a very privileged background, right? Many of these libertarians are not hard-scrabble people who were born in third-world countries and fought their way up or whatever. Some of them are, and respect to them, right? But many of them are suburban white kids. Let's be very clear here.

And this is not a coincidence. This is not a coincidence. Because the truth is that if you talk to many people who've been in bad regimes, who've been in failed states or whatever, they care a lot about laws. They care a lot about order, way more than Westerners or libertarians would. Often, many of these cultures will have extremely strict—too strict, maybe—rules about law and order and obedience and authority and whatever, and this is also not a coincidence. It's because chaos is terrible. Chaos is truly, truly, truly awful.

I think this is the fundamental difference between me and these libertarians: deep down, I think chaos is bad. The state of nature is entropy, chaos, and death. Everything that is not that is an exception that we must fight for, something that we must build, and that is very, very hard. There's a saying that there's no such thing as chaotic good. I think this is true.

I think there are 3 alignments: lawful good, lawful evil, and chaotic. Those are the only 3 that exist. If you're chaotic, if chaos exists, if there are no rules, you can't be good. It's too chaotic. Systems will just decohere. If you have order, you can have evil or you can have good. Order can obviously be evil, right? But you can't have nonordered good.

As I like to say, libertarians are like house cats, completely dependent on a system they neither understand nor appreciate. Sorry to be harsh here, but this is truly what I think. For me, humanist values and Western institutions are an incredible anomaly. This is not the default state of the universe, and we can lose it. Then that's it.

I think you're advocating here for ethical humanism, and they might argue that that's quite anthropocentric. What you're saying here is that we shouldn't rely on the natural order of things. You're saying that we have some kind of privileged status, and we have created systems of governance and institutions and so on which have served us very well. If we just let the system rip, if we trust the void god of entropy, as you so beautifully articulated, Connor, then everything's going to go to hell in a handbasket.

I think this is a pretty core observation. We think that what's important is for humans, or to some extent sentient beings, to flourish and to reduce suffering. We think that everyone deserves to pursue their happiness and to be granted the ability to actually reach it.

I was at a conference and had a conversation with a guy from San Francisco who was batshit crazy. The guy was in a very bad social circle, and at the end of the conversation he was like, “Oh yeah, I do not care for children. And even the children close to me, I don't feel more affection toward them than toward a counterfactual non-sentient being. And I think that to the extent that we privilege the happiness of existing humans and it postpones AGI, that's bad because we're preventing trillions of nonhuman happy beings from existing. So fuck the preferences of existing humans, and also fuck them because they have no power, so they don't matter.”

After that, a San Francisco girl came to me and was like, “Oh, what did you talk about?” I was like, “Well, I was talking about how I dislike anti-humanist ideologies in San Francisco.” And then she was like, “Oh, humanism. Is this the thing where people want to live in human bodies forever?”

I think this is basically the understanding a lot of those people have of humanism. They don't understand human values. They literally do not have an understanding of human values. I would not call this anthropocentric. This is stupid. There is no one else. There's only us and a couple of other sentient beings. So either you care for the people and beings that are here, or you care for nonexistent things.

So, Connor, you said in The AI Safety Compendium that we've got this problem that the foxes are guarding the henhouse. What I mean by that is, Dario went on about this, and he said, “Oh yeah, well, we can self-regulate. We've got all of these AI Safety Levels. At ASL-3, we do this,” and OpenAI has similar things. That seems like a good thing, right?

8. Self Regulation Is Not Safety

I've recently been talking to the top petroleum engineers at ExxonMobil, and they have assured me that their climate-change readiness and preparedness framework will ensure that their new oil-drilling well will totally not be a problem. They know a lot more about oil than I do. I mean, they are petroleum engineers, right? So they must know a lot more about this than I do. They probably know more about climate change than I do. I'm no climate engineer. So what's the problem, then?

Yeah, but the thing is, on the surface, I agree that they might suddenly change tack. OpenAI might say next month, “Wait a minute, boys. We're going to start charging $4,000 a month for this,” and now normal people can't afford the technology anymore.

Right now, looking at what they're doing, they do seem to red-team. When you use ChatGPT, it seems aligned to some extent. It won't do illegal stuff, and it seems like they are genuinely making an effort.

We could say that ExxonMobil is making an effort to stop climate change. There are genuine arguments to be made here. They probably spend millions of dollars on climate-change prevention. They finance scientists, and they reduce their emissions in various ways and whatever, right? So should we therefore be like, “All right. Climate change is not a problem anymore. ExxonMobil's on the case”?

Why would you a priori even listen to this? Why would you even entertain this? Where is the cynicism here? Where is the journalism here? If someone says, “Hello, my job is to lie to you,” and you're like, “Well, this seems like a trustworthy guy. I should hear him out,” you should be like, “No. Why would I listen to this guy?”

Right now, there is a framing of alignment that has been done by the people racing, which is basically PR washing. It's like, alignment means my users like my product, which is completely unrelated to how AI alignment was coined. Alignment was about aligning AGI or superhuman intelligences to human values, such that when they act out in the world, it's in the best interests of people.

This has been a PR campaign for basically the last 10 years, and right now, when people ask this type of question, it shows that they won. It goes even deeper. OpenAI did a 1984 by talking about the original alignment problem as “superalignment.” No, it's just alignment. That's what it meant. Having nice chatbots was not alignment.

A big subproblem of alignment is: tomorrow, you have a super-powerful system that has more power than the rest of the world combined, and that is now the global dictator. What do you have this global dictator do? Right now, we don't have any answer to that. Our best answers are something like a Western constitution, possibly with a few bells and whistles. But even then, if we were like, “This is now the world government forever,” we'd be super afraid, because we simply do not have a good answer to that question.

This is the general alignment problem: how do you make a complex system that is powerful aligned with human values? This has so many subproblems that we haven't solved yet, like studying complex systems, studying systems more powerful than humans—which ties back into the discussion of intelligence and the continuum—and thinking about how you can make human values more objective. It involves governance, public-choice theory, game theory, sociology, psychology, and many other fields.

It's not even that we haven't solved them. It's just that we haven't scienced them yet. We do not have standard definitions, objective measures, or systematic indicators of progress for any of those fields. So when you talk about how we've made chat models that people like, I'm like, “But this is not solving any of the problems I care about. This is not solving any of the worrying shit.”

If tomorrow you have a system that is more powerful than the rest of the world, what is the type of stuff that you want it to do? We don't feel like we know meaningfully more about morals, public-choice theory, governance, institutions, or any of those pretty core topics than we did 5 years ago, when GPT-3 started, or 10 years ago, when we had the boom of deep learning.

If tomorrow Anthropic, OpenAI, and DeepMind start making progress on all of those questions, and we're like, “Whoa, those are tractable solutions. Let's integrate them into our governance structure,” and we start feeling more confident about the deployment of more and more powerful systems, then, wow, this will be real progress.

And we will see it in real life. We will feel more confident when more powerful systems are being developed.

The thing is, I know many alignment folks who work in the large labs, and they're very bright. They're brilliant people. But there is a kind of safety washing where we've been gaslit into thinking that friendly chatbots equal alignment. In the compendium, you basically said that it's really hard.

It's a moral and philosophical challenge as well as just a technical challenge. The current approaches are not on track. There are no automatic fixes. It's a high-stakes gamble. And you proposed that we need to have a kind of Manhattan Project to fix this problem.

9. Alignment Needs A Manhattan Project

Solving alignment for superintelligence would mean taking all of the problems that people solve in their day-to-day lives, solving all of them with software with no bugs, and producing a world that is worth living in. And I'm like, “Holy shit.” If you can't even put together a Manhattan Project for this, if you can't even put together all of our greatest scientists working on this for a generation, you're not going to make it. You're not even trying.

We want to understand intelligence. We want to understand coordination. We want to understand alignment. These are obviously problems that are extremely relevant to our species' long-term happiness and to having a good outcome, but they're also very dangerous. To address those problems, you need to have an institutional solution.

You need to have an institution, a group, a project, whatever, that can possess the knowledge to build an unaligned ASI and then not do that. This is the important property that you need. If we were talking about how we get to a good future with aligned ASI, on the path to getting to an aligned ASI, way before that, you get to a point where you can build unaligned ASI. And you have to then not do that.

Currently, our civilization does not have the ability to not do this. This is why we die. Could we build an institution that is cut off from the rest of the world, has super-high cybersecurity, is extremely cautious, and is properly staffed by our greatest geniuses and philosophers, and have them take decades or even centuries to work on this problem? Could that work? Maybe. That's the kind of thing where I'm like, all right, that could work. This seems like the minimum thing you would try.

I want to get to the question of why alignment is so difficult. We need to get to the complexity here. Naively, you might just think, “I've read Isaac Asimov,” and he was painting this picture of how we can use these rules to basically guard the behavior of AI robots: do this; don't do that. Why would that not work in practice?

10. Simple Rules Cannot Align Superintelligence

I think basically dumb ideas are ideas that refuse to engage with the complexity of the world. Here, if you do engage with the complexity and you're like, “What is a set of rules that I would feel confident about if a super-powerful system had those rules?” I think it's a very good way to think about the problems.

If tomorrow we have a world government, what is the constitution? What is the set of laws that we want this world government to have? I don't think we have a good answer to that. If we had a good answer to this, I would be much more optimistic about solving alignment now, not in general. I think we can solve it in general, although we have a tight deadline, but specifically now, with the current deadline.

If we had this answer, let's say tomorrow we have a super-powerful system. What is the set of rules where, if we're guaranteed that the system follows those rules, we feel good about it, the only constraint being that we understand the rules? You cannot ask it, “Just do whatever is good.” You have to actually understand the rules.

So one of the laws of robotics—do not directly, physically harm people—we can understand that one. That's one of the problems. This is just genuinely hard. The second one is that, actually, the first problem is trivial. You can just say, “Do nothing.” Boom. Solved. But this is not enough.

One of the big things is that you want to avoid bad things from happening. You still want a system that can deal with the other risks that are here. For instance, someone else building misaligned ASI is the most common one. But you could also say World War III. You could also say bioweapons.

Your system must not do nothing. It must deeply interact with human society in a way that will radically change its future. That's why we want ASI. What is a set of rules for a system that is so powerful that it can radically alter the course of humanity? What is a set of rules where you're confident that the system following those rules ends us in a nice place, not with someone else building an unaligned ASI?

So your system cannot just do nothing, or be super-limited, or just do cancer research, and so on and so forth. Once you have those rules in a way that is reliably legible to us, is not using words like “agency,” where we all disagree on the definition, but is more like “directly physically harm people to this level,” then it's not an objective physical thing, but it's still very legible. We can assume that the system will be able to evaluate it. Once we have those rules, then there's the technical problem of having the system follow those rules.

The good thing about rules is legibility. This harks back to symbolic AI, right? There were folks who tried to design AI systems using hard-coded rules of knowledge. It was great because it was intelligible, right? But the problem is, it's brittle, right? How do you define a chair? It's just like the blind men and the elephant, where all of the men have different perspectives of a bigger whole truth.

So the world is a very gnarly, complex place. And it turns out that there is no consistency in this. So then, if we want to have a framework of governance with any precision, we start galaxy-braining ourselves. We say, “Well, maybe we should look at the intention, the consequence, the function, the dynamics, the behavior.” In all of these different things, we're trying to make surface contact with reality, and they all have different trade-offs.

This is why I said, if you have a system that has this level of granularity, where you're allowed to use such concepts and your rules invoke such concepts, I'm still hopeful. If we had a constitution that uses concepts up to that level of non-objectivity, and we're confident that this constitution, if it becomes a constitution of the world government, leads to a good future, I will be much, much more hopeful for alignment.

It doesn't mean that technical alignment is a nothing burger because, for instance, the chair example or the direct-harm example, you can always go for edge cases. And so, if you do not protect yourself against edge cases, you can get adversarial attacks, and so on and so forth.

So you will want some technical AI safety that says, “Well, I only look at the mode of the distributions. If I get into edge cases, I will avoid combining many edge cases,” or, “I will be much more conservative,” and this type of stuff. But I think this is a much more tractable problem. If we have the correct rules, now your job is to make an AI follow them reliably and not get trolled. I'm much more optimistic.

Yes, but of course, they still need friction with reality. Something that just came to my mind is that I guess there are different forms of alignment. Similar to the real world, you can control people through motivation, with a carrot and a stick. You can place guardrails, so you can say, “If you go out of bounds, then something bad happens,” or even just guards in general.

I can put someone in prison and say, “You're not physically allowed to move out of these bounds.” Do those ring true to you? I mean, how many approaches are there to aligning these systems?

AGI alignment is just a special case of general alignment. It's like: how do you have a complex system and do what you want? The original meaning of the word alignment is often what would now be called intent alignment: you build a system that wants to do the right thing.

There are other forms of alignment. A common confusion that people have around alignment is that they look at GPT and they're like, “Well, look, it understands human values.” This is mostly irrelevant. It's not completely irrelevant, but understanding is not the same as wanting.

We don't exactly know how a future AGI system will make decisions. How will an ASI's decision-making system work? I don't know. I think it will have something in common with current systems, but many things will be very different.

How will you causally interact with that system or modify or shape that system in a way that consistently produces outcomes that you would endorse morally? I think this is the question. There are degradations of this, such as, for example, corrigibility. Corrigibility is this idea of building systems that allow themselves to be modified or shut down.

There's stuff like control. A system will do exactly as told, even if it's bad for you. This is a simpler version of alignment, because alignment is more enlightened to a certain degree. As for how exactly to achieve these things, it depends on the level of abstraction we're talking about.

I don't think an AGI system is going to be one coherent blob on one specific computer chip, the same way that I don't think thinking of a human as one specific blob of neurons in one specific brain is necessarily the best way to predict what they will do.

If you want to get humans not to do certain things, or to do other things, you usually have to think at higher levels of abstraction. You have to think about laws, groups, norms, culture, physical threats, policing, and stuff like this. I think which of these methods work, and which are even options, is super context-dependent. When we talk about how we enforce norms on humans, we use different concepts than when we talk about enforcing invariance on computer programs, right? Because we have different tools, and we're trying to achieve different things.

I think we don't know what the right abstractions are to talk about AGI or ASI in general. A lot of the concepts we use to talk about humans—when we talk about policing, incentives, threats, or whatever—how many of these will generalize to ASI systems? I think it's unclear. I think we don't know. There's no economics or microeconomics of superintelligence. There's no public choice theory of superintelligence, et cetera. I think it could be developed, but we don't have it currently.

I would say we don't have it at the human level. For alignment, which is having a system act in a way that is aligned with human values, we don't have a reliable way to do this. You mentioned a carrot and a stick. This doesn't get people to act in a way that is aligned with human values. This is closer to control, and still a limited form of control. Principal-agent problems, incentive problems, misaligned incentive problems, perverse incentives—all this type of stuff. So this is not even a solved problem.

There is alignment. You have a degradation, which is corrigibility, which is having systems that can be corrected. One of the great things about Western constitutions is that they can be changed. They plan for referendums. They plan for laws that can be changed. This is what they're about. Whereas if you think about a lot of historical religions or ideologies, they're set in stone. So by construction, they don't have the corrigibility property. They're meant to be fixed forever.

We usually don't go for alignment because we know we don't have it. In practice, the things that work are not like that. The things that work are more corrigible. Even when we pass laws, we do not expect them to be aligned with human values. We more expect trials and experiments.

If you think about it, the main focus of constitutions is not even on corrigibility. It's on what are usually called boundedness. Just make sure your system doesn't have too much power. This is the entire spirit behind democracy, republics, checks and balances, having representatives, many different representatives, decentralization, having a mayor, departments, councils, regional councils, many ministers, and so on and so forth. It's much more a form of boundedness, where we make sure that each system is limited enough so that none can take too much power, than actual corrigibility, and much less so alignment.

If you think about markets, the way we do markets is with strict boundedness. We're not aligning people with the common good. We're just creating conditions—very harsh conditions—such that when people compete there, we get some things that are aligned with good. But this takes a lot of effort, and sometimes this fails. Right now, we do not have a great theory of either of those. An example is growth and yet fertility crash. So even if we still have growth, fertility crashes. So what gives?

Another example is big tech. We did boundedness. We're like, “We'll split different powers. We have antitrust laws. We have all this type of stuff.” And yet, for the past 15 years, you have big tech that can just do whatever they want with all the crazy new tech—social media, social media feeds, AI—and not be regulated in any way whatsoever. This is both a failure of boundedness and of corrigibility. We cannot correct them. They do not let themselves be corrected. When you start talking about regulation, they lobby hard; they FUD. This is anti-corrigibility.

So if we had a system that was resilient to all of those failure modes at the human level, we could start thinking about alignment. I assume that if we had this, we would have a grounded enough understanding of alignment that we could start talking about superintelligence alignment. So it's not just making sure that current companies don't fuck us over. It's making sure that if you had a company that had a billion times its current power, it still works and doesn't fuck us over.

When someone is like, “Well, alignment is just an engineering problem, and we're making tractable progress on it,” I'm like, “Where is the tractable progress?” If you have solutions to these types of problems, tell us, and the institutions will be much better. Even smaller problems like corrigibility, boundedness, and so on and so forth have direct applications. It's just that we suck at them.

We don't have a level of alignment that is as refined as the way our institutions work and are aligned. We don't have a level of corrigibility, much less alignment, that is refined enough that, reliably, when our institutions veer off, we know they can course-correct. We don't have a level of boundedness such that you don't get one of the richest men on Earth working with one of the most politically powerful men on Earth and just colluding.

One thing I want to get to with the power thing, though, is that it becomes so abstract. This is the way a lot of companies and governments start using such abstract language like power, safety, and so on. It doesn't mean anything. If we were to regulate against power, we would have to use proxies along the lines that you suggested: how many ministers do I have, or how many people do I have? As soon as you start using proxies, then you have the Goodhart's law problem.

This is why alignment is hard. We're just not good at this. If you ask me, “Hey, Gabe, how do you design a constitution that does not fall prey to Goodhart's law?” I'm like, “Hey, man. People say I'm smart, but I'm just me. I'm not at that level.” I think this is what I mean when I say, “Owning where you are.” Collectively, we're just not that good. We don't know. The best thing to do from my point of view, given that we don't know, is, 1, stay alive, and 2, start experimenting, build your knowledge, and iterate.

We suck at this. We should iterate much more. It's one of the typical things about governance that no one wants to iterate. All ideologies are like, “Just do my thing. Remove all regulations. Kill all capitalists. Kick out all immigrants,” and so on and so forth. You don't even need this. You can experiment.

For the past 15 to 20 years, France has had a thing in its constitution where you can experiment. It has enshrined in its constitution the right to test a law locally and, if it works out well, extend it. From this point of view, if you think about what 21st-century governance would look like, embracing the complexity of things and our current state of knowledge, it would involve many more things like this. It would involve experimenting in a bounded way. You see what happens. You draw lessons from that.

You try to take those different problems, ground them, build standard definitions that are more objective than what we have, build objective indicators, reliable measures of progress, and scientific methods that can deal with a reality that is more complex than just physical systems or reproducible systems like simulations and things like this. This is a whole scientific program that should be undertaken.

There's also this problem of legibility to the demos. During COVID, do you remember? It was quite an interesting time. We had all of these scientists coming on TV, and they were giving us very high-resolution information about the COVID pandemic and so on. The problem with high-resolution information is that it's inconsistent and illegible. People didn't understand it, and people simply couldn't make sense of it. That's why, in a democratic system, we have this kind of hierarchy, right? We have the experts at the top, who are at a high resolution, maybe, maybe not. And then—

It's more technocracy than democracy, but—

Well, but you appreciate the point that it's beyond the cognitive horizon of most people to actually understand how this works. These people are busy, they're working, they've got stuff to do. So maybe we should have an elite group of experts who understand things at a higher resolution, and they should institute governance and so on. What do you think?

The people at the top are not supposed to be experts. They're supposed to be representatives. As representatives, when there are things that are about a domain of expertise, they're not supposed to take sides. They're supposed to take a consensus. And if there isn't a consensus, they're supposed to take a portfolio of approaches that takes into account the different beliefs of the different experts, which is very different from, I don't know, having a scientist as minister of science who takes his own opinions. I think this is a very technocratic view of democracy, which is not the one that has worked in the West historically.

My point, though, is that you are describing this epically inscrutable alignment challenge.

And I'm guessing that only a small group of people would be able to wrestle with that. Is that fair?

I think this is a very good question, and I think it's very important. For instance, this is a part where I disagree a lot with people like Eliezer Yudkowsky. Eliezer talks a lot about making people smarter and things like this. I personally do not think this is the bottleneck.

I think the biggest bottlenecks—I will name 3—are traumas, compulsions, and trolls. Traumas are things that people want to avoid. So, for instance, in the important problems, you have to do things that you dislike. For instance, talking to people. A lot of researchers or engineers hate talking to people, and this is causally why they discard governance and policy.

A lot of the things that are necessary to solve alignment involve talking to people. You'll need to do experiments with real-world people and real-world groups. You'll need to talk with people who disagree with you, and you will have to not assume immediately that they're stupid. In practice, I have found this to be a much bigger bottleneck than, "Oh, I would have loved it if this 130-IQ person had 60 extra IQ points," or whatever.

So that's one: traumas. The other one is the opposite. It's compulsions. You have a lot of smart people who are extremely compulsive. We had people who came to work on safety with a specific approach in mind. We did some of that, and we're like, "Well, the approach didn't work. Let's change." And then the person feels bad because they just wanted to work on this approach, not necessarily on the best one.

This is what I mean by compulsion. It's the opposite of a trauma. A trauma pushes you away; a compulsion pulls you toward. And the last one is trolls. Ideologies are great trolls. Modern ideologies that want to simplify everything will always tell you that a huge chunk of your human values are wrong and that you should discard them. The same goes for cults and things like this.

These things aren't even your compulsions. You're not even drawn to them. It's just that they fuck you up. In practice, I found that the strongest bottlenecks are people with trauma, people with compulsions, and people getting trolled. I think the solution to this is not higher IQ.

With people who are smart enough to reliably follow rules of various complexity—which I think includes at least 2% to 5% of the population—you can create more advanced scientific methods and more advanced constitutions that they can reason about. Through those methods, they can solve those problems. I think the bottleneck when you design such a system is how to avoid people being traumatized and guided by their trauma, guided by their compulsion, and trolled.

This is, for instance, a big thing about objectivity. When we introduce objectivity in science, it's not that objective things are the only things that matter. There are many, many other things that matter. It's just that when you have something that is objective, it's much harder to get trolled. You don't have bickering about definitions; it's just objective.

When you have something that you can touch, someone cannot lie to you. You can just touch it yourself. A big part of the scientific method is not about saying that interpersonal dynamics, interactive dynamics, and so on do not exist. It's about finding methods where, when we follow those methods, we do not get trolled.

I think this is a much bigger bottleneck to solving those problems than higher IQ or having 100 experts gather together and so on and so forth. I think this is a wrong view of science, which is an artifact of our current system, where the best that we can do is create a sandbox for startups and labs, give them enough money—both in the public sector, with research labs, and in private industry, with VCs—and hope that there are some who win and do epic shit without any theory for why.

I mean, maybe we should move on to your proposals a little bit. But you guys are basically proposing that we need to have this grassroots movement. And it seems—just being honest—it seems ambitious, right? Because it's a collective-action problem, and there's this legibility problem, and we need to build strong governance and institutions along the lines that you've been proposing. This seems intractable.

Hard or intractable?

Well, you tell me again.

Well, I think it's hard. I don't think it's intractable. Otherwise, I wouldn't be doing this, now, would I? From my point of view, saving the world was never going to be easy. And if you thought it was, welcome to the real world. The real world is actually hard.

We have to wrestle with the fact that the real world is complex. The real world is actually hard. Any solution that gets around that is not an actual solution. All these ideologies that simplify the world into some simple thing—"All you need to do is X. All you need to do is more growth. All you need is open source everything. All you need to do is communism," whatever—are fake and wrong. All of them.

It's because the world is actually complex. It's not infinitely complex. If it was infinitely complex, we wouldn't even have physics, right? Everything would just... There would be no people, right? It's in fact not infinitely complex. We have in fact built many great things.

We're in this slightly underheated warehouse right now, but at least it's mostly dry. We live in an industrial society. We have medicine. We have mostly quite safe streets. We live in London, and despite the reputation, it's mostly fine. And these things are not a given.

It's the same thing as we were talking about before, about how chaos is the default. I think we could have had the same conversation a billion years ago, when we were just atoms. "Why do you think that coherent cell walls are possible? And that seems really hard. Look at all these forces in the universe. And you're saying there's supposed to be a cell wall that can hold these things at bay? That seems unlikely. That seems intractable."

And my answer would have been, "I don't know. Why are we even talking here? We're molecules."

In the modern world, the way I see things is that there are actually tractable ways to make progress. One of the things is just that we are not currently in World War III. Gabriel was talking about at least 2% to 5% of the population. In my opinion, honestly, more people are more than smart enough to understand these problems. I think the percentage is actually higher than this, personally.

I think there is a difference between understanding the problem and making progress on the problem.

That's very fair. So, 2% to 5% who can make progress on the problem, which is a lot. This is a lot of people, right? And in terms of understanding the problem, I think most people can understand the problem to the degree necessary for democracy to function.

The whole point of democracy, as Gabriel was talking about here, is that democracy is a system where not everyone has to be able to solve the problem. That's the beauty of the system. If 50% can at least understand the problem, and 2% can solve it, that's okay. You still solve the problem. And that's really great.

There is nothing physically stopping us from just stopping, right? Just don't build AGI. Governments tomorrow physically could just go to all the people and say, "Knock it off. This is now illegal." Now, whether it will actually happen, that is a very different question. And I'm sure Gabriel has a lot to say about what the next proximal steps one would take toward that.

But I want to make it very clear that there are worlds where it is actually not tractable, where there are concrete physical barriers. This cannot be done. For example, worlds in which ASI already exists. I think there's some small probability that somewhere in a server, a proto-ASI has already been spun up, and we just don't know it yet. I think it could be the case. A 1% or 2% chance, right? If that is the world we currently live in, it's too late.

How do we win? And then, when I say "win," I don't necessarily mean a yes-or-no binary. There might be gradations of winning. But how do we win?

11. Institutions Can Still Save Us

I think the biggest shot that we have at winning is to leverage our current institutions, basically. For instance, if we're at war—if World War III on AI had already started—I would actually be pessimistic about the prospects of winning. I think it would go from tractable but hard to quite intractable. It's still physically possible; it's just that I don't see what I could do.

What I mean is specifically that we should leverage the fact that we're at peace right now. There is some trust between states, even though different people are trying to speed-run losing that trust in many different ways. You want to win. To win, you need power. So you try to accumulate power.

The problem is that this is a negative-sum game. If everyone does this, what happens is that everyone fights and competes. You have possibly 1 winner left still alive at the end, and possibly, with the resources that are left—which can be trust, money, living people, and whatever—they might enact the win.

But it's not even guaranteed that they have enough to enact their personal win. So I find this theory of change really bad. The reason why I say it's negative-sum is that zero-sum games are just redistributions. If you have to sink resources into redistributing, this becomes a negative-sum game.

That's basically how I see DeepMind, how I see OpenAI, how I see Anthropic, and how I see xAI. All of them are competing to be the one who builds AGI. You can give them good intent or not; it doesn't matter. What matters is that their current theory of change, their current plan, is to race to be the first. And this is an extremely negative-sum game.

Another type of theory of change is that if you want something, you can realize that you're human. It likely means that others want the same thing. So instead of trying to get power to accumulate to you, you try to get power to accumulate to everyone who wants the same thing—which, in the case of winning, meaning surviving and not dying from ASI, captures pretty much fucking everyone.

Given that you have everyone who wants this, you have a massive stream. You have the truth behind you, and so on and so forth, and you should think of all the ways you can leverage this. I talk to a bunch of people in AI safety, and many of them tell me something like, “Oh, but we cannot do this, Gabe. The institutions are broken.” Okay, what do you mean the institutions are broken?

They tell me, “Oh, the politicians don't understand anything about extinction risk. They're so fucking dumb. They don't know anything about AI,” and so on and so forth. I'm like, “But have you talked to them? Have you taught them?” And the answer is usually no. Then I'm like, “Well, it's not the institution that is broken. It's you.”

The institution is predicated on the experts going to the representatives, explaining the thing to the representatives, and making the case to the representatives. That's what a republic and democracy are. It's not a technocracy. It's not the experts who make the decision. It's not just having a few people make the decision. It's not just a representative regime; it's a representative democracy.

So you still must have the people who know come to the representative and give them the information. It's not because you're a representative that you have the science. At some point, someone must tell you the information. The opportunity here is so big that with Control AI, the nonprofit, over December and January—and I think the beginning of February too—we just cold-emailed all MPs and lords.

I forgot what the response rate was, but in the end, we got 60 meetings with them over a couple of months. When we tell someone else in AI safety that we got 60 meetings with members of Parliament, they're like, “Wow, how did you do this? We didn't know you were super-networkers,” and so on and so forth. It's like, “Bro, we just cold-emailed everyone. There's no network, no warm introduction, and so on. It's the coldest email possible.”

Then we just made the case for extinction risk to the best of our understanding for 30 minutes to an hour, depending on the meeting. Twenty of them—so, like, a third, a 33% hit rate—supported a statement: “Extinction risks are talked about by experts like Geoffrey Hinton, Yoshua Bengio, Sam Altman, and so on and so forth, and there should be binding regulation in the UK to curtail them.”

This is 20 MPs, while a lot of people would prefer playing House of Cards–type shit, machinations, shooting each other, and so on and so forth, just to get one MP to promise that possibly, in a few years, they might say something for them. I think that right now people are way too cynical for no reason, and they have not even tried.

I think Gabe is even underplaying this. This is only one example of a thing that Gabe, Control AI, and I have seen over and over again.

When we proposed this project, which was to get elected leaders to sign a statement that includes the word “extinction”—it's very clear—we talked to seasoned lobbyists, PR firms, and real experts, and they all unanimously said the same thing: this is impossible. It cannot be done. Maybe if you take out the word “extinction,” you can get a couple of people, but even then you're going to have to network, pay to play, or whatever. Otherwise, it's impossible. Don't even try; you're wasting your time.

This was complete bullshit. It's not perfect, no. There are many things that did turn out to be true, but many of them were not, and no one has done the obvious thing of actually doing the republican, democratic thing of going to every one of our representatives and making the case to the best of our abilities.

Okay, but what is the pushback? Why do they not want to use the word “extinction”?

It depends on the person. Again, the hit rate for something that wasn't optimized at all, without giving them anything in exchange—like a training session on AI—just making the case and that's it, was 33%. If you think about any sales process, you never have a conversion rate of 33%. That's already ludicrously high.

From my point of view, the main reason why it's not higher is just that we haven't optimized much yet. We've just done the dumbest shit possible, which is to go to the politicians, cold-email them, schedule a meeting with the one who responds, and make the honest case. Treat them as a fellow human being who can think and make decisions by themselves.

The case was not perfect. Many things were not perfect. Sometimes they're like, “Let me see with my whip,” and they don't come back to you, and so on and so forth. There are many different things, but still, we think this should be scaled up in many ways.

We should go for draft bills instead of just statements. A bunch of them were interested in actual draft bills to see what the regulation would look like. I think this should be scaled even within the UK—not only to legislators, but also to executive people, intellectuals, academics, and journalists.

This just requires a lot of effort to go to everyone and explain the situation to them to the best of our ability. Obviously, we think this should be done beyond the UK—in the US, the EU, and other relevant jurisdictions and polities in general.

Right now, the main thing that makes me optimistic is that whenever I hear someone who does this, they always give me super-positive results. Not only in AI, but in general. We do have functioning institutions. People are well-educated, especially representatives, and so on and so forth.

We have the best education that we've had since the beginning of human history. Any high schooler knows more than any scientist did 1,000 years ago. To the extent that your plan and your theory of change do not leverage this, and are just a negative-sum game, yes, you'll be fucked, and we'll all be fucked.

But to the extent that we leverage those actual great institutions—what made the modern world work, what led to the great things that we in the West benefit from, and so on and so forth—I think that if we leverage those institutions, we'll get great results.

Right now, the main thing that we want to do is basically extend this proof of concept, make it legible, and help other people and other organizations who want to do similar things do the same. I assume there will be bottlenecks, but I don't think the bottlenecks will be intractable. I think the bottlenecks will look more like needing more people to participate.

But there are gradations of winning. Around 30%—that's a lot—and that's talking about institutions, elected representatives, and so on. There's also the matter of just normal people, and that may even be higher. But what does it mean when you scale that number up? What barriers will there be? Will you find dynamics where, when you get to 50%, 60%, or 70%, you have adversaries and forces that push against you?

I think it's the other way around. I think the adversaries are the default. They're already adversaries. There are already many adversaries; they already exist. They've been adversarial. They've spent the last 15 years acting adversarially.

Alignment doesn't mean alignment anymore. Voluntary commitments are the peak of what states have in mind. The international summits have been interrupted, and the latest one has been moved from the AI Safety Summit to the AI Action Summit. They're already racing as hard as they can, and so on and so forth.

I think what we see are the worst dynamics that we can see. It's when no one fights back. It's when no one actually tries to use the good methods. It's when even the good guys are using the bad methods.

Anthropic is racing, and Anthropic has the support of effective altruism. Holden Karnofsky, the CEO of Open Philanthropy, the main effective altruist organization, joined Anthropic in January. Effective altruists are a key example of this. Instead of building grassroots support and making the case to everyone, they instead did entryism. They tried to enter various organizations and take power from there, which is the typical Trotskyist handbook.

If we actually try to leverage humanist methods, leverage our institutions, and so on and so forth, we have made so much more progress technologically and education-wise that we're now at a point where there is so much that can be done.

Imagine, I don't know, one of the Founding Fathers, or, if you want to go outside of the Western tradition, imagine the Buddha having access to the internet right now. All of them would be like, “What the fuck?” We had in mind projects that would be across generations, but we can actually solve it all now. The Buddha wouldn't talk about reincarnation. The Founding Fathers wouldn't talk about doing things over many generations and not hoping to actually solve the problem and things like this. They'd be like, “Holy fuck, we can do it. Let's go. Let's fucking go.”

I think the main thing to do is to leverage our institutions, leverage the good values that we have, leverage the education that people do have, and leverage our current information and communication technologies. If we actually do this, and if we have people go talk to their representatives and their local intellectual elites—whether they are academics, journalists, experts, or whatever—I think we can actually get meaningful change.

I think there are many obstacles on the way that have the shape of traumas, compulsions, and trolls, trolls being the biggest ones. For instance, a lot of the nice people don't want to say the true things. They've been taught that saying the true things is cringe or not strategic. A lot of the good people have been taught to lie, and so I think there are a lot of habits to unlearn. Our institutions are pretty powerful. We can pass laws. We do have constitutions, countries, and nation-states. If tomorrow we just want to ban AGI, we can block all of the AGI companies from one day to the next.

Yeah, but we started off by saying the collective action problem is not intractable, possibly, but very hard. We have these race dynamics because the winner takes all. And then we spoke about the spectrum of winning, right? It's not binary, but you need to at least saturate a threshold, or you need to find some consensus in order to make this work. Also, we live in a world where there are insane amounts of misinformation, disinformation, and so on. So it just seems like there are so many countervailing forces that you're fighting against here.

I don't think they matter that much. I think that, despite all of this misinformation and disinformation, we still have polls and surveys where people are really for basically any type of regulation against AI. So despite all of this, we're still there. This is what I mean when I say the adversaries have adversed for the last 15 years. Despite this, they're still only at that point. Despite this, people still see things.

This is why I think that if we leverage our institutions, it only gets better from now on. I think the way it gets worse is if we do nothing: the race dynamics mean that the US forces every other country to militarize, basically. At some point, the tensions are so high, and we don't have any international treaty to alleviate this through diplomacy, economic sanctions, or whatever, that it starts becoming military. At that point, you get either skirmishes or attacks, which I already think means it's getting too late, or possibly World War III. At that point, we're truly fucked.

But right now, that's not the world I see. In the current world, there is still a lot that can be done despite all the dynamics that you've mentioned. The current experiments are not before; they're after. The more we do it, the easier it gets, not the harder. The adversaries have already adversed. That's why the situation is so bad. They have done this for the last 15 years. Now, if we fight back, I expect it only gets better.

It's not really a collective action problem. We talk about dynamics like the prisoner's dilemma, where you need everyone to do the thing at first. But you can easily transform a prisoner's dilemma situation into one where people can unilaterally do things. I think there are a bunch of things that can be done here.

The first one is for each country to have an internal plan to deal with things. If the US tomorrow lets one of its companies build existentially risky technology, whether it is weapons of mass destruction or just an unaligned ASI, I think each country should have an actual contingency in case they see this happening with ambiguous proof. You don't need coordination to have an internal plan.

Another thing is kill switches. You want to have a kill switch on all of your clusters. We talk about runaway AI, and we're like, “We can just turn it off.” Well, implement the mechanisms to turn it off. Each country should have the president press a red button, and all the clusters are shut down for 15 minutes until we can see what's up. This should be tried every 3 to 6 months, and the clusters are not shut down. The APIs are still running when this happens. They should be fine. This is something you can do unilaterally. It's just having more control over what happens within your own territory.

And finally, for the elephant in the room, which is the international treaty: the nice thing with international treaties is that you can put whatever you want into them. Of course, as an individual country, you will not want to just sign it unilaterally and bind yourself unilaterally. You know what you can do? You can put in the treaty that it only takes effect when countries representing 35% of GDP and 35% of the world's population are represented. GDP gives you competitive pressure, so that you can play against others, and world population gives you legitimacy. This is something you can do. Coincidentally, I'm writing a brief and policy paper featuring this.

But I think you don't need to go for collective action problems. That's the beauty of technology in general. In a world where we had never thought about game theory, it's hard to come up with such a treaty that does not take effect until you have critical thresholds and things like this, because we're not even thinking in those terms. You just do the naive treaty, and it fails. This is what I mean when I say we can do better than the Founding Fathers, that we can do better than the Enlightenment humanists and things like this. We have all of this understanding.

We're in a strictly more scienced, more knowledgeful, and better technology-wise situation than they were. This is why I'm truly convinced that with their shit, they managed to build the current world, the current modern world. With our shit and their shit combined, imagine what we can do. This is what I mean. In practice, this is what it means to have access to technology. All countries can just meet every month over a Zoom call and talk about AI.

This was not tractable 100 years ago. You cannot ask representatives of all countries to meet every month. We can now. There are many things like this in coordination that are now possible, that I think we should do, and that I think are not collective action problems. If you don't want to join the worldwide call, just don't. Enjoy not having confidential information, sure. I think we should do it. I think to the extent that we do more of it, things will get better.

This is true at all levels. It's true at the country level, at the world level, at the citizen level, and at the local expert and local elite level. I think we should just play that game more, because whenever we have played it in the past, we have won more. Even today, when we play it, we get good things. Coincidentally, when you play the opposite game of just gaining more power, things go more to shit, and it's not a coincidence.

I think a lot of people do feel that way. I think a lot of people do understand that the reason things go well for everyone is when we act in a way that makes things go well for everyone. I think everyone has a native understanding that if you try to act in a way that makes things shit for everyone else, like getting power for yourself to trample others because you don't agree with them, it makes things shit for others. Everyone understands that when others do this, it makes things shit for you. I think those are very, very intuitive things for people in the 21st century.

A thousand years ago, you needed religions, but I think now we're at a point where we have truly understood those dynamics. We're well past humanism. Humanism needed a lot of morals to substitute for understanding. Now we have this actual understanding. We understand why, when people are being selfish and there is no frame for that selfishness, things go worse. We could even write formal models of it. Now we have economics and game theory.

I think that, armed with this understanding, you should just play the winning game. Don't try to trick your brain. Don't get trolled. Don't get into the FUD. Just play the winning game, and the more we play the winning game, the more we win. Is 10 people enough? I don't think so. Do we need a billion people? I don't think so. I think the magic number might just be between 10,000 and 1,000,000, and it's both a lot, but it's also very little.

Imagine us in Control AI. That's 10 people. Imagine 1,000 to 10 to 100,000 Control AIs, all contacting MPs, all contacting experts, and all having just a 33% success rate without any coordination, without anyone else, without optimization. Imagine if they all do this.

It is, but it's still a very complex collective action problem, and you guys are advocating basically to ban AI development, which is going to have a lot of pushback.

It already has, and not all AI development.

But I might say that there are many parts of your proposal which a lot of people would get on board with. You mentioned the kill switch, the firewall. That's a very good idea. We should firewall every single country, every single region. When bad stuff happens, we need to cut it off.

If you pitch it at that level, you could make a lot of your proposals go through because no one's going to say that's a bad idea. But anyway, just to you, Connor, maybe final thoughts: you want to make AI illegal.

I want to make ASI illegal.

ASI.

Unregulated, untrustworthy, non-consensual ASI.

So you wouldn't ban large language models?

Not necessarily. If they are precursors to ASI, yes.

But they usually could be.

I think we should not have open-source GPT-5 because that's way too close for comfort. The fog of war is way too big for me to be comfortable having a GPT-5 system be open-sourced. Will this happen? Probably, and I think that's very bad.

I think this is a sign of us not doing the obvious things. This is actually an important question that I think people should think about—the audience, you at home. Why is there no kill switch? Why do companies constantly get hacked? Why do we constantly lose all of our personal information and nothing fucking happens? What the fuck is going on?

We know how to build secure systems. This is a thing that can be done. Why is it not happening? Why would you expect it to be any different when we're dealing with systems that are way harder to build? The answer is, “No, I don't expect it to.”

Twenty years ago, we should have already been building way more secure internet protocols, way more secure hardware, and way more secure firewalls and emergency switches. If we had gone back, and if I was in charge, I would have built a much more secure internet than the one we're on right now.

How is it that we still click on links, and then some JavaScript overflows our Chrome, and then the whole thing is pwned forever? What the fuck is going on? Who the fuck designed this shit? I think these are actually important, and we don't have time to go all the way into it, but it's important to think about why this happened.

To get back to your important question, there's going to be pushback against ASI being banned. I think a lot of this pushback is from an extreme minority, a minority with extremely vested interests, which is the tech companies who want to build ASI and are okay with killing everyone. There are people who are okay with murdering other people. I don't care about that. We vote them out and we say, “Nope, you cannot murder people. In fact, it is illegal for you to do so. And if you try, we throw you in jail.” That's what democracy is, right?

Can I just challenge that a tiny bit? On Twitter, maybe I'm in a bubble, but it seems like everybody loves AI.

I think you're in a bubble. I think there's no sane person on Twitter, including myself. Go to any median person in any middle-income or high-income country and ask them what they think about transhumanism. What do you think the hit rate is between them saying, “Yeah, it seems like a good idea. We should accelerate towards that”? Do you actually expect any of them to say, “Yeah, that sounds like a good idea”? It's an extremely small minority of extremely weird people.

To be clear, I think it should be legal to be weird. I think weird people are great. I'm pretty weird, right? I think it should be legal to be weird and have weird beliefs as long as you don't harm other people. I'm okay with some people, some philosophers, talking about long-term futurist counterfactual ASI, uploads, whatever. Knock yourselves out. Have fun. I hope you're having a good time.

What is not okay is if they then do things which actually threaten human lives, which is what's happening right now. This is the thing we need to focus on: there are actually people who are threatening human lives with their actions. There has to be democratic oversight in this. People have a stake in their own lives, their own continued existence, and the amount of threat their society is exposed to. People have a stake in this.

We have done polling. The numbers are overwhelming and bipartisan. There are some people who are like, “Twenty percent chance I die? Who cares? Let's go.” There are some people who are like this, right? I think they should get a vote too. I do think they should be allowed to vote for this position.

But they're going to get outvoted, and they know this, which is why they are doing lobbying and blackmail and threats and propaganda and FUD, et cetera, because they can't win a fair fight.

Going back to what you were saying earlier, and what Gabe was talking about as well, why do we think this is tractable at all? There's a very, very important thing that we have going for us, and there are worlds in which this is not true. If this wasn't true, I wouldn't feel optimistic. The fact is that this is actually true: people don't want ASI to kill them.

There are worlds in which this is different. There are very strange worlds where, for some reason, most people are like Silicon Valley EA people—70% of the world's population. If that was the case, all right, fuck. We're actually fucked, right?

But we actually asked. Sometimes I feel like I'm living in The Truman Show, where people tell me, “Oh, don't go over there. There's nothing there.” And I'm like, “Have you checked?” And they're like, “Well, no, I haven't checked, but don't go there. There's nothing there. There's nothing there for you, bro. Don't even look. It doesn't matter.”

I'm like, “I'm going to go check.” “No, no, no. You're wasting your time. Strategically, you can't talk to politicians. You just can't.” And then I go over there and talk to the politicians. I'm like, “Huh, wow, okay. Actually, there's a bunch of... There's a thing here.”

This has happened to me so often in my life. I feel so gaslit by all these techno-libertarian people I grew up with, where I'm just like, “Well, no, I actually talk to normal people. I actually did polling. I actually did focus groups. I actually talk to politicians, and their model of the world is factually wrong.”

This is what we must leverage. This is the thing that still gives me hope that this is tractable. If I had checked and the techno-libertarians had been right, I don't know. That would be a very different world than the one we live in.

And this is a conversation we had. I pushed for this for actual years.

Gabe is one of the people who got me to look through The Truman Show and actually go and ask. I remember when I first met Gabe, I was often telling him, “No, Gabe, we can't talk to politicians. It's impossible. They won't listen.” He's like, “Have you tried?” And I'm like, “Well, no, I haven't.”

Then it was back and forth and back and forth and back and forth. Eventually, I did talk to politicians. Some of them are idiots, sure, of course, but it turns out many of them are actually not, or at least they care and they're trying to figure things out, and they need help. Then I look at San Francisco, and I look at the techno-libertarians, and I'm like, “You fucking lied to me.”

Connor, Gabe, thank you so much for joining us today. It's been great.

Oh, pleasure.

Pleasure.