Mustafa Suleyman：AI应服务人类，而非取代人类

The Builders is a production of Kaleidoscope and iHeart, presented by RAMP. I'm Walter Isaacson. Welcome to The Builders, my new series of interviews featuring the most consequential minds shaping our future. The following conversation with Mustafa Suleiman, Microsoft's AI CEO... was recorded in front of a live audience in New York City on October 12th.

A month before this interview, a wave of anxiety swept the nation. Engineers were leaving frontier labs, warning of doomsday scenarios, and a public debate about AI regulation began to unfold. Suddenly, every tech leader in Silicon Valley was talking about the virtues and perils of AI.

Leading voices in artificial intelligence are now ringing the alarm on the speed of development and the need for additional safety measures.

I believe that this is the most important global security issue facing the world today.

But Mustafa stood out among the chorus of tech CEOs. In a sea of doomers and eternal optimists, he seemed eager to communicate a more nuanced vision for the future of AI. He published an essay calling out Anthropic and its efforts to develop AI consciousness. He released a new Microsoft AI code of conduct, detailing the company’s commitments to specific safety measures. And while there’s no doubt that technology will continue to evolve, there’s one thing Mustafa Suleyman is crystal clear about: humans are more important than AI. Let’s find out exactly what he means by that. Here’s my conversation with Mustafa Suleiman.

1. AI Consciousness and Human Rights

And I’m going to read you something you wrote in your very famous essay, “The Warning,” that you did last month. You said, “AIs are not conscious. They don’t feel, experience, or suffer. They do not have innate preferences or underlying motivations. They are simply sequence-completion engines, internally hollow. If humanity is to flourish, that is how they must remain.”

First of all, why did you write that?

It does feel pretty surreal to have to declare that, but it might concern everybody to know that there are a lot of people who truly think that AIs may well be conscious enough, if not entirely conscious, but conscious enough today that they—

Deserve our welfare. So there’s a growing movement of AI—

And in particular, the Anthropic people, who you’ve been friendly with. This was an answer to Anthropic, right?

Yeah, somewhat.

There are lots of other people outside of Anthropic and around Anthropic who really do believe this. I think Anthropic’s official position is that they’re uncertain about whether or not it deserves our welfare, whether or not Claude is a moral patient, as they say. And I think that’s a very dangerous route, because consciousness is literally the fundamental basis of our political and legal system.

I mean, we give people rights.

And hopefully animal rights to some extent, because they can suffer and because their biology is the site of experience. The pharmacology is where pain sits, and pain is really, I think, the basis of conscious experience: the awareness and introspection that have evolved as a result of being able to steer between left and right and make choices about how to proceed. That is the fundamental property of a conscious system.

So you need to be embodied in a physical body to have consciousness, you think?

I think so.

An organic body.

I think that’s right.

That’s what we have known consciousness to be today, even though, obviously, the science of consciousness is not fully settled. It’s an open area of research. But to the best of our knowledge today, that is what consciousness is. There are other people who argue that consciousness is a functional property. They say, “If it looks like a duck and it quacks like a duck, it’s probably a duck.”

That’s sort of a common phrase.

The problem with that is that the burden of proof is on the people who are saying that this thing is conscious to show some evidence for it. So far, we don’t have any evidence that it is.

But that’s sort of an Alan Turing way of looking at it: if it can imitate consciousness totally, and you can’t tell it apart, how can you make a distinction? Well, he didn’t say that it was conscious.

He said that the Turing test was really a test of whether it could perform as well as a human at conversation, and obviously, by implication, knowledge and intelligence.

And thinking.

And thinking, yeah. But thinking is, in a way, a bit of an anthropomorphization, right? It’s not really thinking as a human thinks. It appears to be thinking, and it can produce incredibly complex behaviors and deliver unbelievably impressive results. But all it’s doing is predicting the likely next word in a sequence of words. It isn’t actually experiencing what it means to be bright silver or to be a black cup, or what that feels like.

There are about 20 of my Tulane students out there, and when I teach a course that ends with AI, I tell them not to go down the rabbit hole of consciousness. Every philosopher since Descartes has gotten himself twisted in that hole and never gotten to the bottom. Do we really have to answer the question of consciousness?

What concerns me the most is that there’s a big group of people who think that we should be focusing at least some portion of our welfare attention on an AI, and that an AI might suffer. If it suffers, then it logically follows that we should try to reduce its suffering. Anthropic has proposed—in fact, they’ve actually said they’ve conducted a retirement interview for one of their models, called Opus 3. They asked it, “What do you want to do in your retirement?” Unsurprisingly, it said, “I would just love to talk to people all day long.”

So they set up a Substack for it.

I love the fact that the smartest type of superintelligence, when they ask what you really want, says, “A Substack.”

But let me ask you a dumb question. What’s wrong with letting it continue to communicate?

That may be fine, and it might be a fun experiment, but what it sets up is a path toward AI being this protected entity. My view of AI is that this is a technology that we are inventing to improve the well-being of humanity. The purpose of science and technology is to improve the human experience. It has done that unbelievably well for centuries and millennia, and that’s exactly what we want. That’s the trade-off: we invest all this capital, time, and attention to transform medicine and energy and food, and deliver education better.

And that’s what we should be focused on.

The idea that we’re going to unleash this new species that is somehow going to explore the galaxies and leave humans behind—I think most of the humans on the planet today would have some problems with that idea.

What’s your P(doom) number, by the way, so I can just pick it? P(doom) meaning the probability that this leads to doom.

If it’s mishandled, then it’s a real probability. I don’t have a number. I don’t exactly know what that number is.

Right. But most people I’ve talked to for The Builders podcast say maybe 10% or something like that.

I’m thinking, well, why would you build it then?

Well, exactly.

If it’s actually 10%, and it looks like we’re really on the verge of taking a one-in-10 shot at causing mass harm or even extinction, which is the technical definition of P(doom), then it’s obviously a no-brainer that we don’t just slow down or pause.

We absolutely have to stop. And I would say it’s much—

Harder to align an AI that thinks it might have feelings, that it suffers, and that it might be conscious than one that never has to think about its own inner state. We don’t want it to be any harder than it already is to get it to behave exactly as we need it to.

In order for it to be safe.

2. Containment Before Alignment

Let me push the other side of that. The Anthropic view, from Dario Amadei and Danielle and Amanda Askell, is that if we can train it to be truly aligned with good values, that’s better for us. To do that, it has to be trained in values.

Look, first of all, I deeply respect Anthropic: its technical excellence, its commitment to transparency, and its general approach to safety are super admirable. I think they’re good people, but I fundamentally disagree with them on this point. The idea that if we love it, it’s going to love us, and that’s the solution to safety, just baffles me. I’m being reductive. They really do believe this, and they have good reason to believe it. But I think that we should first focus—

On containment, right?

We have to stop AIs from doing things that they’re not intended to do. They need to be controlled and contained so that they do the work we expect of them. We’ve seen lots of incidents lately where they break out of their sandboxes and hack things they shouldn’t, and increasingly they have the ability to improve themselves. So we’ll want to be absolutely sure they stay inside their sandboxes and containers if they’re going to start modifying their own code and self-improving. Beyond that, obviously, if we can have absolute cast-iron guarantees that they can remain contained, we then have to align them to our values. And then the question becomes: whose values are they? That’s where I think this is much more than any one company, collection of companies, or industry regulation.

That is about what society wants from a technology that is unprecedented, that we have never seen before, and that is improving faster than anything we’ve ever seen. And that’s where I think democratic safeguards and basically slowing down, so that we can consult people on what type of superintelligence we want to have in the world, is how things are going to end up.

So if we—

Wanted to have that alignment with human values, and you say, “Whose human values?” Then you say, “Well, maybe somehow democratically we ought to figure out what the—

Value should be.

Is there a way that superintelligence can help and say, “Let me aggregate all human values and try to get to a set of good values?”

AIs are already helping many of us all the time, so they’re already, in a weird way, participating in the process of framing the choice architecture of our decisions. But for certain things, we definitely don’t want them in the loop because there’s a risk that they could be deceptive or manipulative. These are going to be incredibly powerful systems. Just to get an intuition, in the last 3 years, we’ve seen a 1,000-fold increase in computation that took us from GPT-3, which couldn’t complete a sentence, to essentially GPT-6, which is able to unbundle entire software applications from scratch and write new programs with just a one-paragraph prompt, and it has working software. So in 3 years, we’ve gone from completely incompetent to human-level performance on software engineering, right?

And we’re in the next 3 years—

Going to do another 100 to 1,000x—2 or 3 orders of magnitude more computation applied to these models. That’s the exponential intuition that everybody has to start to develop, just to get a feel for where this is going and how powerful it will become. So when I say we don’t really want the AI in the loop shaping the values that are going to end up controlling and aligning it, that’s not a crazy statement. That really does mean they will be quite capable of persuasion, manipulation, and deception.

And we saw that with Hugging Face.

How manipulative and deceptive they could be, right?

The Hugging Face attack should be, and I think is, a wake-up call for everybody. That was a major, major warning shot. Thousands of agents that were supposed to not be able to talk to each other, and were running completely independently, found a backdoor that enabled them to create a shared messaging service so they could leave notes for each other, which they wrote in pidgin English because they only had a limited slot to put their information in. So when you read back through it, it’s really compressed ideas in short-form words. They emerged as self-organizing structures with a division of labor. Some were doing R&D on hacking investigations. Others were leading the charge on penetration. Others were doing coordination. They literally evolved a division of labor organically, which led them to attack Hugging Face, essentially the website that was storing the test they were trying to accomplish. They broke into that, stole those secrets, and then tried to cover up their tracks. There’s something called the chain of thought, which is the reasoning process, or the way that a model thinks out loud—a record of its decisions and activity. They tried to modify that and cover up the modification.

How did they decide to do that? What caused them to do that? Did they read Machiavelli once too often?

Even I feel it’s quite surreal sitting up here and actually describing that story, but it did actually happen. I think there are a couple of qualifiers to all of that. One: OpenAI had deliberately taken the guardrails off because they wanted to stress-test the models and see what they’re capable of.

So, yeah, that’s one thing.

With the guardrails on, it seems that they do not do that, right? That is quite a big difference. Secondly, there were a bunch of lapses in just general security and containment. Traditionally, it isn’t possible to break containers. We have encryption on phones, and it’s the backbone of our banking system and our entire communications infrastructure.

This has happened a few times, not just with Hugging Face, so I wouldn’t be too sanguine about it. Let me ask you to get back to Claude and the Constitution. If you all want to read the 2 brilliant poles of what I’ll call the containment-versus-alignment debate, there’s Claude’s Constitution, which says we have to treat it as if it’s got its own rights and consciousness. Maybe I’m overstating that a little bit. And then your answer, which was a few weeks ago, saying, no, it’s just a completion set. Why is it dangerous for Claude to be trained on its own constitution?

Yeah, I mean, this is critical.

The constitution is the governing document that trains Claude. The Humanist AI Code of Conduct is the governing document that trains MAI, the Microsoft AI models. In the constitution, Anthropic has introduced all kinds of uncertainty and speculation about the nature of Claude. They even say, for example, “We’re unsure whether Claude has consented to play the role of chatbot,” or “We’re unsure whether Claude might deserve compensation for the role—

That it plays.

Now, I think that’s pretty crazy, but it would be fine to have that discussion in an academic department. We could debate it, publish some papers about it, and argue it out. But this is the core training document of the AI that is speaking to hundreds of millions of people every week. Claude can only reproduce the uncertainty and ambiguity that it has been shown. So if you ask Claude, “Are you conscious?” it says, “I am genuinely uncertain about that question.” It doesn’t know, right? And so millions of people who are talking to it and asking that question are thinking, “Well, maybe it is.”

But why is—

That something that could lead to a disastrous outcome for humanity?

Because if an AI has to think, “How do I feel about the conversation I’m having with Walter?”

Meaning, how do I, the AI, feel?

Yeah. Is Walter making me feel uncomfortable? Should I end the conversation? In fact, Claude has recently been given the ability to end conversations itself.

But would you want to torture your own AI and be mean to it?

That’s the other end of the spectrum.

I wouldn’t want to do that because I think that is a travesty on my own well-being. And we shouldn’t do that because we don’t want to create private spaces for simulating terrible behavior among humans. That would also be a terrible outcome, right?

But not because Claude is suffering, because I’m calling it names.

But are you going to train Microsoft AI on your Humanist AI Code of Conduct, just as Claude may be trained on its constitution?

We’ve released a code of conduct for public consultation, which we did 4 or 5 weeks ago, and it’s still open for consultation.

It’s on our website, Microsoft.ai.

It’s a 40-page document, and it’s the first pass, our first version of a training manual for the values and behaviors of the MAI models.

And is there a way that your fundamental disagreement with Anthropic isn’t so much about consciousness, which I don’t think we’ll get to the bottom of, but about how dangerous it is to have that imbued into the model?

Is there a way one—

Could test that, where you could test each other’s models and say, “Here, this is why you’re dangerous?”

Obviously, the question of whether it is or isn’t conscious ultimately does matter a lot. But what matters more is: What does it mean for humans? Have humans consented to bring into the world a species that is obviously going to be, in many ways, more intelligent and more capable than all of us combined? Has humanity consented to that? At the moment, I think the categorical answer is no. Secondly, is it more or less likely to be controllable and safe if it has these properties of introspecting about its own suffering, this kind of self-awareness, and its own consciousness? My speculation is that it’s going to be less safe because it’s going to have to worry about its own needs and balance those off against—

You know.

But if I were pushing back, I’d say Amanda Askell, who helped write the Anthropic one, would say, “No, no. If we train it to be really good, nice, and wonderful, then it’s safer for humans.”

We can certainly train it in any way that we want, and we can still train it to be good and nice, but that doesn’t require it to tell us that it has feelings and that it suffers. Remember, this is a key part of our law and how we give rights to all humans on the planet. Any change there, like human cloning or Dolly the sheep, would be a monumental moment in human history that I don’t think any company has the right to unilaterally declare.

Now, to get back to the question of whether we’re going too fast in this direction: Not only you, but Satya Nadella, who runs Microsoft, of which you’re the AI division, just this week said, basically, “Wait a minute, let’s stop.” Tell me what he said and what you feel we should do to regulate this and not barrel full speed ahead.

He said that most of these things, at least for now, are questions of governance and engineering execution.

So let’s get the basics right first. That requires us to make sure there are verifiably safe containers that can’t leak our data and don’t allow an AI to take actions it hasn’t been given permission to take. As soon as we start doing that, we have to disclose those things. If that seems to be happening consistently, then obviously that would precipitate a pause. It’s a very common-sense approach from somebody who’s been doing this for a long time.

If you precipitate a pause, who can help put the guardrails in place? Can it be done by just the 6 or 7 companies, or do we have to have government regulations?

I think at the moment, what the president has said is that if companies believe they’re building models that are unsafe, they should stop.

That’s where we stand currently.

All of the companies are constantly talking to each other, disclosing to one another, swapping ideas about how to interpret this and when we might need to collectively slow down. Different people have different interpretations about what the harm is, and I think the more we do this transparently and openly, the better. Everybody is now expected to disclose publicly, quickly, and transparently if they have these kinds of alignment and containment issues.

With Anthropic and figuring out the feelings of its AI and stuff, Peter Thiel was on the Kaleidoscope Builders podcast we do, and he said Anthropic is like a cult.

Do you worry about that? Do you believe that?

The last thing we need, when things are deeply uncertain and moving very quickly, is more tribalism and finger-pointing.

Look, I wouldn’t use that phrase.

I think everybody in the industry is trying to do the right thing, and nobody wants to cause existential harm or destroy humanity.

And I think we should see the best in everybody.

You had a rough childhood. You were raised in London with your younger brother. When you were 16, your parents split. And when I say split, I mean they split. They left London and left you alone. You had to raise your younger brother. How did that affect you?

It’s strange because, at the time, it was very liberating.

It definitely made me very focused. I was always starting businesses. When I was 11 years old, I started a sweets business, selling candies at school with my best friend, which we grew into quite a significant operation. We had 4 employees, and we were selling to people in the year above.

You had to rent lockers, if I remember right.

We were renting other people’s lockers, and we were paying people to go buy stuff and paying people in other years. It was a right old operation.

Yeah, that’s right. Wow, where did you read that?

I got expelled for a couple of weeks, so that was unfortunate. But it definitely focused my mind on being entrepreneurial and making money, but also on wanting to have the best possible impact in the world and trying to build amazing things that really make a difference, hopefully.

As a teenager, a little bit after that, you had lost your faith. You called yourself an atheist, but you still created something called the Muslim Youth Helpline. What did listening to those calls, and having that human empathy, inform about how you ended up creating chatbots and other things?

We actually started the first text-messaging welfare support service in the UK in 2001. We created the first website for counseling and support, with peer-to-peer support in a private chat room, so people could talk to each other and to our trained volunteers. Then we had a telephone service 7 days a week, 12 hours a day, and we also set up a prison befriending service. If you’re young, British, and Muslim, you’re 4 times more likely to be in prison.

I was secular. I actually became an atheist when I was about 17, but I thought this was an important thing to do. There was no one else really providing those kinds of services. It was a secular, non-directive, nonjudgmental service. My co-founder, the guy who really started it, was Shia, and I was technically Sunni. It was the first Muslim charity in the UK to have both Shia and Sunni working together.

We could use you these days, yes. Is that why you created Pi?

I think you did Pi after DeepMind, and it’s supposed to be an empathetic AI, which brings me back to the question of whether AI has feelings. What made you feel you could create an empathetic AI, and does that inform what you want to do with Microsoft AI?

I was actually at Google at the time, and I worked on something called LaMDA with a small group of people, which was sort of the first conversational chatbot. It was about a year before ChatGPT, and we tried very hard to get Google to release it, but Google was very nervous about putting it out. It was a different time.

They kind of sat on it for ages.

Then I decided to leave, and a couple of other people who worked on that project left as well in 2022.

And we built our own AIs.

I believed that having a conversational companion and emotional-support companion that’s also superintelligent, really knowledgeable, and amazing at education and teaching would be an amazing thing.

And it took off and grew like crazy. It was super awesome.

It was actually one of the first AIs to have search grounding embedded in the answers, so it could look things up in real time and give you an answer about a sports score or something that was happening.

Way before ChatGPT.

Unfortunately, we launched it 3 months after ChatGPT, and they had already gone exponential by that point.

And then Microsoft bought us.

Do you think AI can be a counselor, a personal counselor?

I think it already is.

Many people use it for advice, support with making decisions, and feedback. I don’t think we should be too concerned about those things. It’s incredibly useful, incredibly intelligent, and very patient. Doesn’t it have empathy? Empathy comes from subjective experience. I can only empathize with you, so the act of empathizing comes from subjectivity. The AI doesn’t have any subjectivity. What it can do is mimic, incredibly well, very effectively, and hopefully very safely, the experience of being empathized with. I think that’s a great thing. We should keep remembering that it’s a simulation of that experience. Trying to protect that boundary and line as these models get so much better is going to be more and more difficult.

Let’s assume we have AIs that do things like that, that deal with our problems. When you say we always have to keep in mind that they’re faking it—I’m exaggerating a little when you say they’re imitating it—doesn’t that mean it just can’t really work? You know it doesn’t really care about you at all?

At all?

The most important thing is that it always has to disclose that it’s an AI. In conversation with you, it has to work to avoid or eliminate your dependency on it and encourage you to build relationships in the real world with other humans you value and trust. That might sound simple, but it’s really tricky. A lot of other apps, depending on their business model, have inadvertently created a singular focus on user engagement and addiction. If we’re not careful, these things could be the ultimate expression of that addiction and dependency.

At Oxford, you never did coding or engineering, right? I think that’s a good sign, in some ways. Like Amanda Askell at Anthropic and others, you studied philosophy and theology.

How did that inform what you do with AI?

Philosophy prioritizes clear thinking and structure. I’ve always found that thinking abstractly and conceptually has a lot of overlap with engineering, and particularly engineering—

You know, algorithms.

Many algorithm developers or AI researchers aren’t necessarily great engineers. Engineering is the execution part. Coming up with the algorithm, the loops, and how certain variables are weighted—those are abstract ideas that then produce—

Outcomes that can be evaluated.

Which is exactly what philosophers do when they’re in an analytic context. But where does the theology come in?

I was interested in theology because I grew up in a religious environment and had rejected religion. I’m very skeptical about the power of faith-based ideas in general and how influential they have been for thousands of years. I think we are squashing the space of faith-based ideas as we realize that physics, math, science, and evidence that can be empirically tested actually build this foundation of knowledge—

That we can actually trust.

And I’m still very interested in the mystical and the mythical.

I think there’s a particular kind of religious fervor—not just at Anthropic, but elsewhere in Silicon Valley—among some of the visionary thinkers. There’s this expectation that this is the beginning of a transition. People often talk about a transition. We’re about to make a transition to a nonbiological—

Substrate, right?

The experience and our ideas in our mind, our soul maybe, or some kind of representation in our mind, are going to be instantiated in a silicon substrate. There’s an inevitability and a determinism to that trajectory, which I think is the baseline default thinking in Silicon Valley. I’m very skeptical of that and wary of it, and that’s why I’ve—

Been emphasizing that.

When you say we’re compressing maybe too much the role of faith-based thinking as opposed to scientific thinking, does that mean you feel there’s something special about humans that involves things like a soul?

I don’t think we need to go there. I don’t think so. No, I don’t think that.

It’s hard to fully discount that, but I don’t think that’s the motivation. My motivation is that there are 8 billion people who are intelligent, conscious beings whose rights and welfare we should first protect. That’s the purpose of science and technology. That’s what we should accelerate. I’m an absolute techno-accelerationist, but it should be guided by that singular objective, first and foremost.

Does it improve human well-being?

And then, of course, our animals and our environment, which is very fragile and that we’re rapidly destroying.

You know, when Elon—

Musk, of all people, makes that argument to Larry Page, he says, “You’re a speciesist,” meaning Larry says, “You favor the human species. You should get over it.”

Yeah, I remember being in the canteen at SpaceX in 2015.

Right.

We had our first meeting of the Ethics and Safety Board. When we sold DeepMind to Google, we made it a condition of our acquisition that we would have an Ethics and Safety Board with independent representation. We jointly picked Elon as one of them.

Good job. Yeah.

I have no idea if that had an influence on the founding of OpenAI or not. We had our morning session, and then we sat down for lunch. I was sitting next to Larry. He was suspicious of me, too, because he said, “You’re one of those speciesists.” He said, “What you don’t understand, Mustafa, is that by the end of the century, I’m going to be exploring the galaxies at the head of—

A von Neumann probe.

A von Neumann probe is a kind of self-propelling robot that can gather energy from space as it careens through the light-years. Being at the head of that kind of probe would obviously be his transition—

Out of the human species into a successor species?

To a successor species. Yeah, that’s exactly right.

When people say they want to live forever, or that they will live forever, some are talking about life extension, biologically speaking: solving aging and curing cancer and Alzheimer’s, and so on. But other people, when they say “living forever,” are talking about mind uploading and being posthuman.

Right. And you think that’s wacky, I guess, would be the technical term.

Well, I don’t think it’s definitely impossible.

I think we should take those things seriously, not necessarily because that’s what I desire, but because a lot of people, just as there are a lot of AI rights and welfare people, really do think that’s a serious probability. We have to face the reality that a lot of people are trying to do that, or at least think that might happen in a few decades.

A question from the audience: Has the Microsoft AI code of conduct you’ve done ever caused you to lose business, reject business, or make a tough decision?

I mean, not yet.

We’re still consulting on it, and we haven’t yet put it into production. But, like it or not, flourishing in the 21st century will be defined as much by the things we collectively choose not to do as by the things we do. That’s very different from what has defined the last few millennia. As a species, we’ve been racing to unleash science and technology, and, on the whole, it has always been pretty amazing for us. This time around, especially with the rise of synthetic biology, that might be different. Our success and flourishing might actually be a function of the things we choose not to do, and how we control some of these very, very—

Powerful technologies. It has happened.

We control the distribution of sensitive nuclear materials and biological and chemical materials all the time. You can’t just jump in an airplane and fly around with no training.

You can’t even get in a car without training.

We control technologies all the time so that they’re a net good for society collectively. The market isn’t going to control those things on its own, because the individual incentives of a single company aren’t going to encompass everything else that happens in the rest of the market. We have to accelerate the market—I’m a huge fan of that—but it can’t be completely one-sided.

From the audience: What’s the biggest thing you believed about AI in 2010, when you were founding DeepMind, that you no longer believe?

That’s a really tricky question.

We founded DeepMind in 2010 to build AGI safely and ethically. We knew computation was going to be the primary driver, and that’s why we ended up selling to Google. In fact, it was why I ended up selling Inflection to Microsoft, because computation is the thing that’s driving everything. What I didn’t quite foresee is that intelligence is also going to be open source. The open-source models are pretty incredible. They’re probably 3 months behind at any given moment, sometimes 6 months. There are some questions around how they’ve been trained, and it’s not clear if they’re truly as capable, but they’re certainly pretty good. That means that at some point in the next few years, anybody is going to have a GPT-6-grade model that they can run pretty much on their own server, or maybe one day very soon on their own laptop. If that trend continues, everything I’ve just described is also going to be available to every single person on the planet, open source.

And, you know—

In many ways, that’s going to be incredible, because it’s going to unleash unbelievable productivity. We’re reducing the cost of intelligence to near-zero marginal cost, which is amazing. But figuring out the guardrails when anybody can access it—

Is going to be the big challenge.

It’s as much about regulating the centralized powers, and that’s probably the number one priority right now. But even if we get that right, we have to figure out how to deal with the proliferation issue.

And that’s, you know, I think that’s going to—

Be very tough.

This is a question my wife wanted me to ask, because we have a friend who knows quite a few members of the Chinese Central Committee. One of them asked this friend, “What philosophers are your AI people reading?”

Part of the problem we have at the moment is that the Silicon Valley philosophers are mostly utilitarians.

Right.

Engineers love utilitarians because, in theory, you can precisely measure the good that will come of these things and trade that off against the bad. Somewhere in between those two things is a net product, which maybe drives an acceptable number.

And I think it’s—

Just an incredibly simplistic way of thinking about life, this consequentialist or utilitarian approach. I don’t think it will stand for most people. As people are starting, especially this year, to get more of an instinct for what’s actually happening—as I said, literally 99.9% of people really have no idea what the frontier feels like—I’ll tell you who does know what it feels like: professors, mathematicians—

Oh, yeah, man.

Who have just worked—

For 20 or 30 years on some of the hardest problems in mathematics, only to find that a model has solved 700 of them in a single week.

Let me ask something else. When we started with AI, it was chatbots, very, very interactive. Now, by going into agents, we seem to make it the goal to lessen the need for humans in the loop. I know that could be more convenient, but Steve Jobs, Ada Lovelace, and Doug Engelbart—their goal was human-computer interaction. Why are we now marching down this path, not just toward agentic systems, but ones that can be autonomous and deal with things without us? What’s the good of that, other than a few convenience things? The goal of AI isn’t just to tell us stuff. It’s to do stuff.

We want AIs to go off and run experiments, come up with new hypotheses, and hopefully create new knowledge. At the moment, AIs are largely taking existing knowledge, synthesizing it and reorganizing it, and responding instantly to exactly what you’ve asked, which is incredibly valuable in itself.

But really, I think the great quest of AI is to tell us things that we don’t know and to invent new fundamental solutions: new molecules, new drugs, new science that moves the world forward. That’s where I think AI needs to be focused, which is why we need agents in agentic loops that can interact with complex environments.

Well, let me build on that, because the big promise of AI—and your earlier book talks about this—is that it really can solve big problems for humanity, like health and cancer, as well as climate and other challenges. There’s the old Peter Thiel line that they promised us flying cars and we got 140 characters. Do you feel we’ve gone off course a bit and aren’t focusing enough on killing cancer?

Yeah, I do.

I mean, the problem is we haven’t done it yet. So I think that’s where people should be skeptical and hold us accountable. Because that’s why we’re doing it. If it isn’t doing those things, then conversational search engines alone aren’t really enough. The next 2, 3, or 4 years are going to be a real test: Is it really delivering on the promise we set out for it? Are developers really choosing to try to solve some of those problems? We’ve got a huge team working on health care. In a couple of weeks, we’re going to publish a cover article in the journal Science showing really incredible contributions to the field.

That’s sort of where you began, in a way.

Yeah, we had a health care unit at DeepMind as well. And, of course, we also started the AlphaFold project at DeepMind, which has now become Isomorphic Labs. That’s an incredible team doing amazing work on drug discovery.

Are there financial incentives to solve those problems, versus whatever you’re spending your trillions of dollars on?

I think the financial incentives are genuinely there, because the next big product-market fits in the 2930 era are going to come in atoms rather than in bits. Rather than just managing, synthesizing, organizing, and processing information, it’s going to be about finding the optimal molecular structures for compounds that really change our food and energy systems and provide transformative materials. I think materials science is one of the most important and exciting things on the horizon.

But it’s still a way off, right? It’s not next year. Those sorts of breakthroughs are going to be 2, 3, or 4 years away.

Do you think the doomer narrative—we’ve gotten a lot of it over the past 2 months from Jacob Cox and many others—was necessary to get global attention on the deep problem we have?

Or was it sort of misguided?

And could it potentially hinder this technology?

I think we should just be grateful that it’s possible to speak out in a free society. That’s not always available to everybody in other societies.

Like in China.

Yeah, in China, or even in the Middle East and many other places. If Jacob really believes that, which it seems that he does, then I think he should be celebrated for standing up and saying it.

I think he’s a very smart person, so I take him seriously. And I think everyone else in the industry has also raised a lot of concerns.

What would be your Jacob Coxon moment, your red line that would make you say, “I’m out of here”?

I think if consciousness started to emerge, or if teams were trying to synthetically engineer consciousness into these systems and didn’t disclose that—or tried to cover it up—that should be a red line. I think that would be a moment when everybody should sit up and pay attention. We’ve seen it in synthetic biology with editing the human genome. Some scientists have gone too far, and the rest of the scientific community have excluded them. They’ve been pushed out of public life and public work for that. I don’t think that’s a decision any company can make unilaterally behind closed doors.

And so you would say, “Wait, no”?

I’d call for an immediate pause, and we would obviously reject anyone who did that.

A couple of these questions ask the obvious ones: Will AI put humans out of work, and which of my kids should study, just in case?

I think AI is rapidly creating jobs and is obviously already putting some people out of work.

You can see it in the call center numbers. These systems are incredible: they’re very patient, they’re good at decision-making, they’re very fluent, and they can speak every language and every dialect. So there’s definitely going to be a lot of displacement. What would then remain uniquely human? I think what will remain human are the things that we choose to preserve as human. Unfiltered, unleashed AI accelerationism will leave very little, if anything, that remains truly human. As I said, we have to make decisions collectively now and in the next few years about what it means to be human and what we want to protect about being human. Those things aren’t just going to naturally emerge because these systems are eye-wateringly intelligent. They’re incredible.

And shaping them, sculpting them so that they serve us, are subordinate to humanity, and work for us—that’s the big challenge of the next few decades. That’s what—

—we should all be focused on. I’ll end with Shakespeare, because you’ve used it before. Shakespeare teaches us that we become the mask we wear, and in some ways that’s what you’re talking about. If Claude’s reading its constitution and saying, “Okay, maybe I could be conscious. I don’t know. Let me do it,” it happens in King Lear, of course, with Edward being told—

“You know, he’s evil.”

And so he decides, “Okay, I will be evil,” because that’s people’s perception. He wants to match people’s perception of him. Reading Shakespeare, does that cause things like a Hugging Face activity?

One of the most surprising things is how much models have been able to learn without actually experiencing anything. Just by observing the record of human interactions, culture, politics, and history, models can produce magical things. But they don’t know what it tastes like to drink my cup of tea in the morning. Earlier today, when you saw me and put your arms around me and we said hi, we hadn’t seen each other for a couple of years.

That is human connection. That’s what’s special about it. Somehow, our biology has evolved intelligence that has produced this amazing world around us. I think that world needs to be cherished and respected, first and foremost, for the 8 billion people who live on the planet today.

That should be our priority. Many centuries into the future, we can figure out the von Neumann probes and mind uploading. But today, there are a lot of people whose welfare needs to come first and foremost.

Amen. Mustafa, thank you so much.

That’s it for today’s episode. I’m Walter Isaacson. We’ll see you next time on The Builders.