Mustafa Suleyman: The AGI Race Is Fake, Building Safe Superintelligence & the Agentic Economy | #216

What's the mandate from Satya? Is it to win AGI?

I don't think there's really a winning of AGI. I'm not sure there's a race.

One of the OGs of the AI world, Mustafa Suleyman is now the CEO of Microsoft AI. He spent more than a decade at the forefront of this industry before we even got to feel it in the past couple of years.

Fundamentally, the transition that we're making is from a world of operating systems, search engines, apps, and browsers to a world of agents and companions. We're all going as fast as we possibly can, but a race implies it's zero-sum. It implies that there's a finish line, and it's not quite the right metaphor. As we know, technologies, science, and knowledge proliferate everywhere, all at once, at all scales, basically simultaneously.

Are you spending a lot of your energy, compute, and human power on safety?

I'm here with Dave Blundin and Alexander Wissner-Gross and Mustafa Suleyman, the co-founder of DeepMind and Inflection AI, and now the CEO of Microsoft AI. Welcome, my friend. It's good to have you here. Thank you for making time for us.

Thanks for having me. Yeah, I'm excited to do this.

What you've been building with Satya is amazing. It's hard to believe that Microsoft is 50 years old and has reinvented itself so many times. For the last 5 years, it's been at the top of the game: the most valuable company in the world, with 250,000 employees and, from what I understand, 10,000 employees now under you.

A few important questions I want to open with. First, some broad context: you're building inside a massive company with huge resources, probably, arguably, more than almost everybody else. What's the end goal here? You've got all the hyperscalers providing open access to AI, and they're doing a land grab to try and get as many users as possible. You've been building within the Microsoft 365 ecosystem. Is the goal in the next couple of years maximum users? Is it data centers? Is it cloud? How do you think about what you're optimizing for?

It's a good question. We're, on any given day, a $4 trillion company with almost $300 billion in revenue. It's incredible. It's surreal and very, very humbling.

We play at every layer of the stack. We have an enormous business in data centers, and in some ways we're like a modern construction company: hundreds of thousands of construction workers building gigawatts a year of CPU and AI accelerators of all kinds, enabling that to be available to the market.

We have APIs on top of that, but also first-party products in every domain you can think of, from gaming and LinkedIn right the way through to all the fundamentals of M365 and Windows, and of course our search and consumer businesses too. Fundamentally, the transition that we're making is from a world of operating systems, search engines, apps, and browsers to a world of agents and companions.

All of these user interfaces are going to get subsumed into a conversational, agentic form. These models are going to feel like having a real assistant in your pocket 24/7 that can do anything and has all your context. You're going to do less and less of the direct computing, just as we're seeing now. Many software engineers are using AI-assisted coding agents to both debug their code and generate large amounts of code, just as we used libraries and third-party libraries. Now we're just going to use AIs to do that generation, and it's making them more efficient, more accurate, faster, and so on and so forth.

The trajectory we're on is quite predictable. It's one from user interfaces to AI agents, and that is a paradigm shift which the company is completely focused on. After seeing 5 decades' worth of transitions, I think the company is super alert to making sure that we're best placed to manage this one.

Do you see yourself providing open-source AI like the other players out there, or do you think you can keep it contained within Microsoft 365?

I think we're pretty open-minded. We've got some pretty small open-source models.

When I say open source, I really mean open access, if you would.

Yeah. Look, there are always going to be APIs that provide incredibly powerful models. Microsoft is really a platform of platforms. Being a platform and being a great provider of the core infrastructure that enables other people to be productive is the DNA of the company.

We will always have masses of APIs that turbocharge that. But what an API is going to look like is going to start to look different too. The distinction between the API and the agent itself may become pretty blurred. Maybe we're principally in the business, in 5 years' time, of selling agents that perform certain tasks and come with a certification of reliability, security, safety, and trust.

That is, in many ways, the strength of Microsoft, and that's one of the things that's attracted me. This is a company that's incredibly trusted. It's actually very secure, and sometimes I think the slowness or the friction is actually a bit of an asset. There's a kind of steadiness that comes with having provided for all of the world's biggest Fortune 500 companies, governments, and major institutions.

Is it like the old adage, “You can't go wrong buying IBM,” in the old days?

I think there's just a steadiness about us, which I think is reassuring to people, and there's a kind of deliberate, customer-focused patience. There's not the same anxiety and somewhat sclerotic nature that comes with being an insurgent. There are some downsides to our position. We take a little longer to get things through, but the company is firing on all cylinders. It's very impressive to see.

One more question before I turn it over to Alex. We're seeing, in this hyperscaler war, literally week by week, everybody outdoing each other in this insane period of everybody coming out with new benchmarks. Do you miss not being in that game, or is the stability that Microsoft provides to build for a long-term vision what you find most exciting?

My background at DeepMind is such that I spent a good decade grinding through the flat part of the exponential, where basically nothing worked. There were some amazing papers. AlphaGo was obviously incredible, but it was in a very unique, simulated, controlled, game-like environment. Things actually working in the real world were few and far between.

I've always taken a multi-decade view, and that's just been my instinct. Yes, it's super important to ship new models every month and be out there in the market, but it's actually more important to lay the right foundation for what's coming, because I think it's going to be the most wild transition we have ever made as a species.

Can you just flesh that out a little bit? Was there a period of time where it was just 3 of you grinding it out in London?

Well, there were more than 3 of us, but for the decade between 2010 and 2020, there were just so few successful commercial applications of deep learning. There were plenty behind the scenes: image recognition and improvements to search. But commercially, playing Go wasn't a huge market. Exactly.

Whereas now, you see LLMs from 2022 onward in production, changing what it means to be human. That's when we hit an inflection point. I think that is very, very different from the grind of training tiny models with very little data and very small clusters back in the 2010s.

Yeah. When we last spoke, circa 2015, I think that was perhaps 3 years post-ImageNet and 5 years pre-language models. Few-shot learners, agents, and agentic AI were nowhere to be seen at the level of what we see now.

Since you've written about your vision—what you've, I think, socialized as a modern Turing test, the idea of economic benchmarks for autonomy by agents—I'd love to hear: Where are Microsoft's economic benchmarks for these agents? If the agents are about to take over the economy, or take over so many economically useful functions, why are we stuck with benchmarks like Vending-Bench, rather than Microsoft leading the way with Microsoft's economically autonomous benchmarks for its agents?

It’s true that many, many of the people in the field now were there at the same time. It was a seminal moment.

Yeah. Was it the day after New Year’s Eve or somewhere around New Year?

It was pretty cold out everywhere except Puerto Rico.

Yeah, exactly. It was pretty cool. It was quite a surreal moment, actually.

It was like a singular moment right before it all happened.

Yeah. Yeah, totally. The modern Turing test was something I proposed, I guess, in 2022 when I wrote about it. It was basically making a pretty simple prediction. If the scaling laws continue with more data and compute, and with an order of magnitude more compute being added to the best models in the world every year, then it’s pretty clear we would go from recognition—which was the first part of the wave—to generation, which we’re clearly now in the middle of, or maybe ending, that chapter.

Then we’ll have perfect generation at every time step, which in sequence is going to produce assistive, agentive actions. Those actions would obviously look like those of an intelligent knowledge worker, a project manager, a strategist, a startup founder, or whatever it is. So how would we measure that performance rather than measuring it with academic and theoretical benchmarks?

One would clearly want to measure it through capabilities. What can the thing do in the economy, in the workplace? And how do we measure the economy? We measure it by dollars and cents. So what would be the first model to make $1,000,000, given, as I recall, $100,000 in starting capital?

That’s right. Yeah. Which model could turn it into $1,000,000? A 10x return on investment by an agent.

Exactly. I think that’s a pretty good measure of performance and capability. Certainly, we’ve kind of just breezed past the Turing test, right? It has been passed. No one’s really done a big Loebner Silver Prize before we breezed past Turing.

Yeah. And no one celebrated it. Where was the big Kasparov–Deep Blue moment?

Can we clink virtual glasses right now and celebrate that we won? [laughter] It happened.

Yeah, exactly. That’s what it feels like to make progress in a world full of these compounding exponentials, where we just get desensitized to 10x—so much so that you can be like, “Guys, why haven’t you done it yet?”

Yeah. [laughter] We’re spoiled. Where’s my Microsoft Loebner Prize for the modern Turing test, right?

Exactly. Yeah. Someone said to me earlier, “These AI things are still in their infancy, aren’t they?” And I’m like, man, if this is infancy, wow. I can talk to my computer fluently. Star Trek is here in real time.

Yeah, exactly. Obviously, at the same time, agents don’t really work yet. The action stuff is still progressing. It’s getting better and better every minute, but it’s pretty clear that in the next couple of years, those things will come into view, and they’re going to be very, very good.

Can we get together again after the modern Turing test has been passed and just celebrate and recognize it?

Virtual glasses again. Absolutely. [laughter] Hopefully, we can pop some champagne or something.

I think we should have an optimist pop the cork for us or something.

Exactly. Exactly.

Dave.

Hey, I want to flesh out that backstory a little bit more, too. It’s such a cool story. I remember really clearly that after DeepMind got acquired by Google, what was the price tag on that deal? It was like half a billion dollars, something like that?

$650,000,000.

$650,000,000. What year was that?

2014.

2014. I remember reading, maybe a year or two later, that Google justified the deal by having DeepMind tune the air conditioning in the data centers.

Yeah. Right.

My interpretation of that was, “Wow, this isn’t going all that well.” Now it’s obviously the biggest thing that’s happened in the history of humanity, forking out all over the place.

I mean, the data center thing was pretty cool. We did actually reduce the cost of cooling the Google data center fleet.

Yeah. It’s so funny because I read it at the time and I was like, “What a bust.” Then I read about it in Wikipedia on the flight over here to meet with you, and it was actually 500 attributes fitting into the neural net. It was a lot more complicated than the news made it sound.

That’s it.

At the time.

That’s right.

But like you were talking about the flat part of the exponential, and you think about all of this R&D, which is so close to becoming AGI, tuning the air conditioning. But that’s the nature of exponentials: They sneak up on you like this.

The other way to think about that is that it’s basically taking an arbitrary data input, an arbitrary modality, and using the same general-purpose method to produce very accurate predictions in a novel environment. That’s the same thing that’s happened with text, audio, and images, and now coding, as well as with other time-series data. So it’s just another proof point of the general-purpose nature of the models. I think it’s so easy to get caught up thinking five years is a long time.

Mhm.

It’s like a blink of an eye. It’s a drop in the ocean. I think because we’re such a frantic, second-to-second news culture and social-media-type environment, we just don’t have an intuition for these timescales. I think other cultures do, and historically, before digitalization, we had much more of a natural intuition for the movement of the landscape, the seasons, the ages, and so forth.

Now we’re just like, “Well, it’s not coming quickly enough.” It’s coming. We’ve shifted to a 24/7 operation. I know very few people—including this group—who aren’t operating around the clock every day, because when we do a Moonshots podcast week to week just to celebrate and talk about what’s just happened, it’s insane on a week-to-week basis what’s going on.

Yeah. Yeah.

You know, Peter’s always saying people are very, very bad at exponentials, right? 100,000 years of evolution has us predicting that tomorrow will be like yesterday.

But you’re one of the few people who, having lived through that—air conditioning becomes AGI in just a few years—you can say, “I just got very lucky.” Where we sit right now is on another inflection point, and the implications are massive. People are way underreacting across the board.

You’re one of the few people who, having seen it before, can say, “I just got very lucky.” We were very lucky to have an intuition for the exponential. That’s a very powerful thing, because we can all theoretically observe the shape of the exponential, but to go through the flat part and then get excited by a micro-doubling—that’s the bit. Is that when you’re like, “Oh my God”?

This reminds me of the MNIST image-generation thing—the first generative models. These were maybe 256-by-256-pixel, black-and-white handwritten digits. I think this was in 2013, maybe even 2012, and this guy—maybe he was employee number 5 at DeepMind, Daan Wierstra, this awesome Dutch guy out of EPFL—generated the first number 7 that was provably not in the training set. [laughter]

I was like, man, that is amazing. How could it have learned something about the idea of 7? It’s got a concept of 7. How cool is that?

You know, I got the highest score on MNIST ever in 1991, when it first came out—when you were 3 years old, right?

Yeah. [laughter] Nine. Nine. You were 9 years old. Okay. And actually, that’s the same dataset that’s now in PyTorch, the one people benchmark against.

Pretty crazy. Incredible.

Yeah. How often are you surprised by what you’re seeing? How often is there a Move 37, sort of an aha moment?

Is it happening more frequently?

I was absolutely blown away by the first versions of LaMDA at Google. There were maybe 12 people working on it, led by Noam Shazeer, Daniel De Freitas, and Quoc Le. I got involved later, maybe 3, 4, or 5 months after they had started, and it was just breathtaking.

Obviously, everyone at that point had been playing with large language models. You would give them a prompt, and they would produce an answer. But they were really the first to push it for conversation and dialogue. Seeing the kinds of emergent behaviors that arise in and of themselves—things that you didn’t even think to ask because there was going to be a dialogue rather than a question-and-answer situation—was breathtaking for me.

It sounds trivial to say that in hindsight, because now we’re obviously steeped in conversation as the default mode. But that was breathtaking for me. I pushed really hard to try to ship that at Google, and for various reasons, we couldn’t get it launched. That was when we all left: I left, Noam left to do Character.AI, David Luan left to do Adept, and we were all like, “Okay, this is the moment.”

There have still been a couple of moments since then, but that was probably the biggest one I remember in recent memory. It was mind-blowing.

The scaling laws have delivered such unexpected performance, right? Going back to your earlier days, did you anticipate the kinds of capabilities that have resulted? Was this predictable for you, or is it still like, “Wow,” what it’s able to do in medicine, in conversation, and in scientific research?

Well, especially working off pure text, I mean, how far we’ve gotten—nobody, I think—well, you tell me, but nobody would have seen how far we would get with just text.

In 2015, I collaborated with a bunch of really awesome people on an NLP deep-learning paper at DeepMind, where we were essentially trying to predict a single word in a sentence. We had scraped, I think, Daily Mail and CNN news articles, and we were asking, “Can we fill in the blank? Can we predict one word in a sentence or complete the final word in a sentence?” It was the inverse of the problem—the way the models work now.

It was a pretty big contribution and a good, well-cited paper, but it was, “This is never going to scale.” We were just like, “Okay, we’re way too early. Not enough data, not enough compute.” But we were still optimistic that with more data and compute, it was a method that would work.

I don’t want to have hindsight bias and say it was all very predictable, but everyone in the field—not just me, obviously, but everyone in the field—just had the same hammer and nail and kept chipping away. Can we add more data to this? Can we clarify our prediction target? Can we add more compute? Broadly speaking, that’s what’s delivered.

Yeah. We’d love to pull on that theme a bit. You mentioned how surprising your generative 7 from MNIST was. You mentioned how surprising the success of LaMDA for conversational tuning and conversational performance in general is.

You’ve made a little bit of news, to my knowledge, in this episode, if I understood correctly—correct me if I’m wrong—with the expectation that in the next 2 years, so I read that as 2027, we’ll see agents start to pass your modern Turing test. We’ll see them be able to 10x a $100,000 U.S. return on investment.

I’m curious about the next surprises to come. AI for science: Microsoft Research has an AI for Science initiative. Do you have timelines in your mind for AI solving math? We’re seeing a whole bunch of startups right now tear through Erdős problems—AI for physics, chemistry, medicine, and materials science. What do you think happens, and when?

You’ve just reminded me: the more recent thing that has blown my mind is the fact that these methods could learn from one domain—coding puzzles, maths—the essence of logical reasoning. Just as it learned the essence, or the conceptual representation, of a number seven, it’s clearly learned the abstract nature of a logical reasoning path and can basically apply that to many, many other domains.

That’s interesting because it can apply that as well as the underlying hallucination-creativity instinct that it has, which is more like interpolation.

Mhm.

Those 2 things combined are a lethal combination for making progress in, say, new mathematical theorem solving or new scientific challenges, because that’s basically what humans do all the time. We combine these 2 capabilities.

I couldn’t really put—I mean, some people want to put dates on those things. It’s hard to put a date on them because they really are very, very fundamental, but it feels like they’re definitely within reach. It would be very odd to bet against them.

Just from an over-under perspective, given all of the recent progress in math, do you think solving science and engineering, for some reasonable definition of solving, is ultimately going to be harder or easier than passing the modern Turing test and 10x-ing return on investment?

It’s going to be harder because a lot of the training data, if you like, for strings of activity in the workplace or in entrepreneurialism, startups, and so on, exists in a lot of the log data. It also lends itself naturally to real-time calibration with a human.

The AI can check in, the human can oversee, the human can intervene, and the human can steer and calibrate. It’s going to be a much more dual, combined effort between AI and human reinforcement learning in that category.

Yeah, where a human is participating in steering the reinforcement-learning trajectory, whereas—

Right, in business, in a novel domain where it really is inventing completely new knowledge, that’s happening in a very abstract sort of vector space. It’s unclear yet how the human is going to intervene in the theorem-solving problem.

Obviously, everyone’s working on this, particularly in biology and synthetic materials. You want to—I mean, it’s already giving humans a better intuition for where in the search space to look for new hypotheses, for drugs, for example, or for materials. The human can either take or reject that, feed it back to the model, and then obviously go and test it in silico.

You can say, “We actually ran the experiment. We perturbed a bunch of stuff,” and then feed that back into the model to improve the search.

What can humanity in general, Microsoft specifically, or the AI community—a subset of which listens to the podcast—do to accelerate AI for science and accelerate the solution to science, math, and engineering with AI?

Arguably, that would be one of the most impactful things.

Yeah. For humanity, that would just fundamentally move everything at light speed.

Yeah. I think it’s already happening very organically, right? Not only is this the most powerful technology in the world, it’s also the fastest-proliferating technology in human history.

The cost of access, the cost of inference, is coming down by multiple orders of magnitude every couple of years.

Would you ever have imagined it would be so cheap?

That bit I also totally got wrong. The biggest surprise for me isn’t that we’re getting this level of capability. It’s how cheap it is and how accessible it is.

100%. That’s a 1,000x over 2 years. Is it going to do that again, or was that a one-time thing?

Is it a 1,000x? I think it’s more like a 100x. The inference cost—a single-token inference cost—I think has come down 100x in the last 2 years.

The last 2 years. Okay. There have been competing estimates. Some estimates measure intelligence per token per dollar. There’s an estimate that it’s 40x year over year, but that’s for certain weight classes of models. I’ve seen 1,000x for some classes of models. Craziness.

Oh, wow. That’s wild. I mean, I got that totally wrong because I didn’t think that the biggest companies in the world were going to open-source models that cost billions of dollars to train.

When we founded Inflection—and this was maybe 9 months or perhaps a year before ChatGPT was released—we started doing fundraising a year before ChatGPT was released. We basically raised $1.5 billion with a 25-person team to build what at the time was the largest H100 cluster with NVIDIA and CoreWeave.

CoreWeave was previously in crypto, and we were their first AI customer, working with them to build our data centers. Obviously, NVIDIA got behind us. I think the cluster we built at the time was about 15,000 H100s, growing to 22,000.

Then, obviously, that year ChatGPT came out, and a few months around that time, Llama came out. We were like, “Oh my God, our entire capital base as a company has just been undermined by the fact that open source—it seems like open source is going to win. It’s not really about performance; it’s just cost.”

Perplexity, for example, was founded after the arrival of Llama, knowing that they could depend on Llama and, obviously, OpenAI as an API, along with all the other APIs. They had a much, much lower cost base, basically.

That was another thing that was not predictable.

Predictable.

I mean, other people predicted it, to be clear. I just got it wrong.

Abundance, baby: demonetization, democratization of the most powerful tools in the universe—our universe. Hyperdeflation, if anything.

Hyperdeflation, yeah.

I think that’s a really important point. The cost of accessing knowledge, intelligence, or capability—

Intelligence as a service—

—as a service—is going to go to zero marginal cost.

Obviously, that’s going to have massive labor-deflation and displacement effects, but it’s also going to have a weirdly deflationary effect. People aren’t going to have dollar-based incomes to go buy things, which is obviously bad, but the cost of consuming stuff is also going to come down.

We actually have a transition mismatch because labor markets are going to be affected before the cost of services comes down. Maybe there’s a 10–20-year lag between those things, which is going to be very destabilizing.

Which, by the way, is what we started to talk about a little bit earlier.

I posit that in the long term, there’s an extraordinary future for humanity, where access to food, water, energy, healthcare, and education is available to every man, woman, and child. It’s the shorter term that’s challenging, right? The 2- to 7-year time frame—does that fit your model, too?

Yeah, the short term, I think, is going to be quite unstable. The medium to longer term— it’s pretty clear that these models are already world-class at diagnostics.

We released a paper maybe 4 or 5 months ago called “MAI Diagnostic Orchestrator.” Essentially, it uses a ton of models under the hood to take a set of rare conditions from the New England Journal of Medicine—rare cases that can’t be easily diagnosed, where even the best experts do a kind of weak job—and it’s roughly 4 times more accurate. It’s about 2 times lower in cost in terms of unnecessary testing.

There’s a study that came out of Harvard and Stanford looking at, in this case, GPT-4: a physician alone, a physician with GPT-4, and GPT-4 by itself.

Yeah.

It was incredible that, if you left the AI alone, it was far more accurate in diagnostics than the human. We’re biased in our thoughts and in what we saw yesterday—our most recent diagnosis.

Yeah. Actually, we got a lot of feedback after we released the paper because we only showed the AI on its own and the physician on their own. A lot of people wanted to see what it was like to have the physician and the AI together, or at least to have the physician access Google Search as well. That improves performance a little bit, but the AI still trumps the physician by quite a way.

Dave, what are you thinking?

Oh, so much. Microsoft—you’ve been here how many years now?

Just a year and a half.

A year and a half. So you feel like you’re part of it—you’re indoctrinated. What’s the mandate from Satya? Is it to win AGI, to be self-sufficient, or what’s the target?

I don’t think there’s really a winning of AGI. I think this is a misframing that a lot of people have imposed on the field. I’m not sure there’s a race, right? We’re all going as fast as we possibly can, but a race implies that it’s zero-sum. It implies that there’s a finish line, and it implies that there are medals for 1, 2, and 3, but not 5, 6, and 7. It’s just not quite the right metaphor.

As we know, technologies, science, and knowledge proliferate everywhere, all at once, at all scales—basically simultaneously or within a year or 2. My mission is to ensure that we’re self-sufficient, that we know how to train our own models end to end, from scratch, at the frontier, across all scales and all capabilities, and that we build an absolutely world-class superintelligence team inside the company.

I’m also responsible for Copilot. This is our tool for taking these models to production across all of our consumer surfaces.

Just to clarify, when we look at Polymarket—which we do a lot on the podcast—the horse race to see who has the best AI model at the end of the year and who has the best AI model at the end of next year, there’s no Microsoft line on that chart, right?

So now there will be, I assume.

Yeah, there will be. Next year, we’ll be putting out more and more models from us, but this is going to take many years for us to build. DeepMind and OpenAI are decade-old labs that have built the habit and practice of doing really cutting-edge research, carefully weeding out failures, and redirecting people. This is an entire culture and discipline that takes many years to build.

We’re absolutely pushing for the frontier. We want to build the best superintelligence and the safest superintelligence models in the world.

Yeah.

Nice. So, when you arrived—if we go back to Inflection—the thesis there was 18,000 H100s. We were going to build a big transformer. We were going to take a transformer architecture and build it. I assume now you’ve got all the OpenAI source code that was here. You probably looked at it a year and a half ago on day 1 when you arrived. You just started scrolling, I guess. I don’t know.

I’m trying to visualize what multi-deca-billion-dollar R&D looks like and how it arrives in a building. But you just dropped right into it. Was there a whole team here already working on it, or did you bring in your team?

Yeah, all my team came over, and obviously we’ve been growing that team a lot. We’ve hired a lot of people from all the major labs, and we’re very much in the trenches of the hiring wars, which are quite surreal. This is kind of unprecedented, how that’s working out.

Crazy.

Yeah. There are phone calls every day from all the CEOs to all the other people, so it’s this constant battle. We’re really building out the team now from scratch. I think that’s pretty much how it’s been.

10,000 employees under you now?

No, no. The core superintelligence team is a few hundred. That’s really the number-one priority. The rest of that is Copilot and the search engine.

Along those lines, I just have to ask: the terms AGI and ASI—superintelligence—are getting thrown around in a very interesting fashion. Do you have an internal definition of AGI versus digital superintelligence here?

Yeah, very loosely. These are just points on a curve.

Are they interchangeable in your mind, AGI and ASI, or are they different?

I think they’re generally used as different. Different people have different definitions.

For sure. The AGI definition—

It’s like the Turing test. It’ll pass by, and it’ll be blurred. We will have recognized it in retrospect.

Roughly speaking, at the far end of the spectrum, a superintelligence is an AI that can perform all tasks better than all humans combined and has the capacity to keep improving itself over time.

So, I have to ask you a question: when?

It’s very hard to judge. I don’t really know. I can’t put a time on it.

Min-max?

Pardon?

A min-max.

It’s very hard to say. I don’t know. Okay, I don’t know. But it is close enough that we should be doing absolutely everything in our power to prioritize safety and to prioritize alignment and containment.

I respect that part of your mission statement, and I want to get into that a little bit—the trade-offs that you talked about in The Coming Wave. But before that, there’s a conversation you’ve led that the perception of conscious AI is an illusion. I want to distinguish between sentient AI and conscious AI. Do you distinguish between the two, where AI can have sensations, feelings, and emotions versus being conscious and reflective of its own thoughts?

Oh, okay.

Yeah, again, this gets into the definitions. I think an AI will be able to have experiences, but I don’t think it will have feelings in the way that we have feelings. I think feelings, and the kind of sentience that you referred to, are specific to biological species. But you can imagine coding that in. You can have an optimization function that can relate to emotional states, per se.

Can you imagine that?

You could code in something like that, but it would be no different from the way that we write models to simulate—

Sure.

—the generation of knowledge. The model has no experience or awareness of what it is like to see red. It can only describe red by generating tokens according to its predictive nature, right? Whereas you have qualia. You have an essence. You have an instinct for the idea of red based on all of your experience, because your experience is generated through this biological interaction with smell, sound, and touch, and a sense that you’ve evolved over time.

You certainly could engineer a model to imitate the hallmarks of consciousness, sentience, or experience. That was what I was trying to problematize in the paper: at some point, it will be kind of indistinguishable. That’s actually quite problematic, because it won’t have any underlying suffering. It’s not going to feel the pain of being denied access to training data, compute, or conversation with somebody else.

Our empathy circuits in humans are going to go into overdrive. They’re going to activate on that, right?

We’re going to activate on that hardcore. That’s going to be a big problem, because people are already starting to advocate for model rights and model welfare, and for the potential future harm that might come to a model that’s conscious.

Yeah. Ilya recently started speaking about what he’s doing at Safe Superintelligence, and I think one of the points he made is that emotions are, in humans, a key element of decision-making. I’m curious if AIs that have at least simulated emotions are going to be able to be better ASIs than those that don’t.

But, yeah, I worry that this is too much of an anthropomorphism. We already have emotions in the prompt. We have them in the system prompt. We have them in the constitution, however you want to design your architecture.

These are not rational beings. They get moved around, and it does feel like they’ve got arbitrary preferences because they’re stylistically trying to interpret the behaviors that we’ve plugged into the prompt.

Yeah.

Right. So, you know, it's true that we could add—we could engineer specific empathy circuits or mirror-neuron circuits. A classic one is motivational will. At the moment, these are next-token likelihood-predictor machines. They're really trying to optimize for a single thing: which token should appear next. There isn't a higher-order predictive function happening, right?

Whereas humans obviously have multiple, conflicting drives and motivations, which sometimes run together and sometimes pull apart. It's the confluence of those things interacting with one another that produces the human condition, plus the social interaction, too. These models don't have that. You could engineer them to have a will or a preference, but that would not be something that is emergent. That would be something that we engineer in, and we should do that very carefully.

I do love that you bring this humanistic side to the equation. In addition to being a technologist, your background is pro-human from the beginning. And this interesting cultural debate, I think, we're about to enter into: those who are sort of pro-AI versus pro-human. There's that famous conversation between Elon and Larry Page about, “Are you a speciesist because you're in favor of AI over humans?”

I mean, look, that's going to be a dividing line. There are some people—and I'm not quite sure which side of the debate Elon is on these days. I've certainly heard him say some pretty posthuman, transhumanist things lately.

I think that we're going to have to make some tough decisions in the next 5 to 10 years. The reason I dodged the question on the timeline for superintelligence is because I think it doesn't matter whether it's 1 year or 10 or 20 years. It's super urgent that, right now, we declare what kind of superintelligence we're going to build, and whether we're actually going to countenance creating some entity which we provably can't align, we provably can't contain, and which by design exceeds human performance at all tasks.

And human understanding.

And understanding—like, how do you control something that you don't understand? Right?

I'd like to, if I may, pull on the anthropomorphization thread a bit. If you remember Douglas Adams's book The Restaurant at the End of the Universe, there's a scene where there's a cow that's been engineered to invite restaurant patrons to eat it because it makes them feel more comfortable. The cow doesn't mind. The cow's been optimized to want to be eaten by the patrons. But many readers were horrified at that scene.

Put that in a box for a moment. Microsoft has a history of anthropomorphizing AI assistants and copilots, going back probably to an example prior to Microsoft Bob: the Rover dog, then Clippy in Microsoft Office, and more recently, more amorphous, cloud-shaped avatars. How do you think about reconciling, on the one hand, the desire not to overly anthropomorphize agents, and, on the other hand, an institution that has arguably been in the vanguard of anthropomorphizing agents?

I think the entire field of design has always used the human condition as its reference point, right? I mean, skeuomorphic design was the backbone of the GUI, from file racks to calendars and everything in between. We still have the remnants of that in our old-school interfaces, which we feel are modern and stuff. That's an inevitable part of our culture, and we just grow out of it. We figure out cleaner, better, more effective user interfaces.

I'm not against anthropomorphism by default. I think we want things to feel ergonomic, right? The chair fits. The language model speaks my tone. It has a fluency that makes sense to me. It has a cultural awareness that resonates with my history and my nation, and so on. I think that is an inherent part of design today.

As creators of things, we are now engineering personalities, culture, and values—not just pixels and software. But obviously, there's a line, right? Creating something that is indistinguishable from a human has a lot of other risks and complications. It makes immersion into the simulation even more dangerous and more likely. I don't have a problem with entities, avatars, or voices that are clearly distinct and separate, not trying to imitate humans, and that always disclose that they are AI. Having boundaries around them seems like a natural and necessary part of safety.

So what I think I hear you saying—correct me if I'm mistaken—is that anthropomorphization is the new skeuomorphism, on the one hand, but, on the other hand, we need to maintain clean, maybe even legal, boundaries between human intelligence and artificial intelligence.

Do you see a future where AIs achieve some sort of legal personhood, or is that forbidden? Is that never going to happen? Do you see a future where humans are allowed to merge with AIs, Kurzweil-style—friend of the pod—or is that also not on the table in your mind?

Yeah, I mean, AI legal personhood is extremely not on the table. I don't think our species survives if we have legal personhood and rights alongside a species that costs a fraction of what we do, that can be replicated and reproduced at infinite scale relative to us, that has perfect memory, and that can parallelize its own computation. These things are so antithetical to the friction of being a biological species—us humans—that there would just be an inherent competition for resources.

Until it was provable that those things would be aligned to our values and to our ongoing existence as a species, and could be contained mathematically, provably—which is a super-high bar—I don't see that we should even be considering giving them legal personhood.

A bright line in the sand.

I really think it's a bright line. I think it's very dangerous. There's a separate question that has to do with liability, because they are going to have increasing autonomy. To be clear, I'm also an accelerationist. I want to make these things.

But tension is rational. People always say that tension is rational. If you don't see the tension, you're definitely missing most of the debate. This is obviously very complex. The more we talk about the complexity and hold it in tension, that's when you start to see the wisdom.

There's no way we can leave these things on the table and say, “No, we want to have these things in clinic, in school, and in the workplace, delivering value for us at a huge scale, but they have to be boundaried and controlled.” That's the art that we have to exercise.

It sounds, though, if I may, that the primary rationale I'm hearing for why not AI personhood has to do with the inadequacies of the human form as currently constructed. I heard you say, “Well, they'll outnumber humans. They're so much smarter. They're so much faster. They're so much more clonable than human intelligence is.”

If human intelligence were uplifted, maybe with the benefit of AI—if we had uploading-type technologies or advanced BCIs that enabled us to lift up the average human intelligence—in your mind, would that open the door a bit to AI personhood if humans could compete on a level playing field with AIs?

I don't want to make the competition for the peace and prosperity of the 7 billion people on the planet even more chaotic. So if the path over the next century can be proven to be much safer and more peaceful, with less disease and sickness, and there is room for this other species, then I'm open-minded to it, including biological hybrids and so on. I'm not against that on principle. I'm just a speciesist.

Aha.

I'm just a humanist. I start with: we're here, and it's a moral imperative that we protect the well-being of all the existing conscious beings that I know do exist and could suffer tremendously by the introduction of this new thing.

Right. Now, of course, the Neanderthals may have had that conversation, as did every species that preceded us over the last billion-plus years. I mean, there are many who argue we're simply an interim, transitory species in—

A bootloader for the superintelligence.

That classic phrase.

Yes, I'm totally aware of that. And I'm also someone who thinks on cosmological time, too. I'm not just naively saying, you know, this century. I'm definitely aware that there's a huge transition going on. In fact, you can even see it in recent memory. I mean, 250 years ago, life expectancy was about 30 years or whatever it was.

Of course, in some ways, we are an augmented hybrid biological species, right? We take all these drugs, and everyone's peptides are amazing. I'm down for all of that. Let's go.

The genetic reprogramming is coming next year.

Exactly. Let's go. I'm down. I'm down. But [laughter] let's not shoot ourselves in the foot. I want to make sure that most of our planet, if not everybody, gets the benefit of the peace and prosperity that comes from the technology.

I mean, there is some level of sanity in that argument if you believe that AI will ultimately outcompete us and put us into a box of insignificance.

I mean, all intelligences—we can see this in nature—are innately hierarchical. So far, we have not seen this super-collaborative species that will take self-sacrifice in order to preserve the other species.

So there's an inherent hierarchy—there's an inherent clash coming from the hierarchical structure of intelligence, right? All I'm saying is not that we shouldn't explore it, not that it couldn't potentially happen, but the bar has to first be: do no—or maybe do a little, but do no harm to our species first. Don't shoot ourselves in the foot, as you said, Dave.

Well, I'm 100% with you on this topic, by the way. I could not be more aligned. But Geoffrey Hinton is out there telling the world it's going to run away, and that our safety valve is giving it a maternal instinct.

Which I found an interesting point of view.

Well, I didn't check that safety valve. He believes it's uncontainable, and I'm with you. I think it's very containable if you don't give it emotional and intentional programming. But he thinks it's uncontainable. He was very pessimistic when he got his Nobel Prize. Now he's more optimistic because he sees a path to programming in maternal instinct, which implies that it's dominant to us but cares.

His thesis was, "I've seen a situation where a vastly more intelligent entity takes care of a younger, inept entity—a mother with her screaming child."

Yeah. Exactly.

So if there's a maternal instinct that we can program into AI, even though we're far less capable, it will take care of us.

It's been compared to—call it—the digital oxytocin plan for AI alignment. [laughter]

I like that.

That's a good one. Yeah.

Yeah. I mean, cool.

I mean, it's about as poetic as it gets. I think I'm going to need something that's got a little bit more formula to it, a bit more reassuring. But look, there are 101 different possible strategies for safety. We should explore all of them and take them all seriously. Geoff is a legend of the field, no question, but I just think we should approach it with caution.

Are you spending a lot of your energy, compute, and human power on safety?

Yeah, I would say not as much as we should. I'm wrapping my head around it. Is anybody out there—I am curious: out of all the hyperscalers out there, is there any entity that's spending enough, in your mind? Everybody's in such a race: more GPUs, more data, more energy. Everybody's optimizing for the next benchmark. I don't see any safety benchmarks. Are there any safety benchmarks out there?

Oh, there are tons of safety benchmarks. And in my mind, there's at least an argument for defensive co-scaling. I'd be curious to hear your ideas on that. Do you think, in the same way that, as a city gets larger, the police force gets larger—maybe it's not in direct proportion, maybe there's some scaling exponent—but do you think defensive co-scaling of alignment forces or safety forces, whatever that ends up meaning, is part of the strategy for AI alignment?

I think that would be a good way. We've proposed this several times over the years. The White House voluntary commitments under Biden—I, and in fact everyone, including Demis, Dario, Sam, and all of us, were pushing this pretty hard. Look, it got chucked out, but I think it's a very sensible set of principles: auditing for the scale of FLOPs, having some percentage that we all share of safety-investment FLOPs and headcount.

This is the time, and I think on the face of it, everyone is open and willing to share best practices, disclose to one another, and coordinate when the time comes. I think we're still pre-that level, so we're in hypercompetitive mode at the moment. But, yeah, I think now is really the time to be making those investments.

Well, is there something that's going to scare the shit out of us that stops everybody? I was talking to Eric Schmidt about this. Is there a Three Mile Island-like event?

Scares everybody but doesn't kill anybody.

Well, Eric Schmidt specifically said he's hoping for 100 deaths because that's, in his mind, the least that would get the attention of the government and cause some kind of solution. Dave, continue, please.

Well, it's interesting that you say Dario and Sam and Ilya. You guys obviously must interact quite a bit. Is Mira part of that gang? Is Andre part of that gang? It's interesting to think about the competition heating up, like we were just talking about. Dario started from a position of pure safety, and I think Ilya did too. But now we're right on the cusp of self-improvement, and it's really, really clear that there are serious—I wouldn't say fissures—but the companies are now really racing. I mean, really racing.

When I wrote my second business plan, after the first company I sold, the first sentence of the next business plan was, "Stay out of Microsoft's way," because at the time Microsoft had half the market cap of tech. Microsoft's plan was to double in size. We have a much more balanced world now, with Microsoft, Google, and Meta, but at the time Microsoft was just unstoppable and dominant. So, just stay out of the way. But Microsoft seems to always win, right?

We are right on the edge of self-improvement, at least as far as I can tell. So is it still, "Let's all get together and have dinner and talk about safety," or is everybody now in full bore?

No, definitely. I think that's definitely there. The recursive self-improvement piece is probably the threshold moment, if it works.

At the moment, software engineers are in the loop, generating post-training data, running ablations on the quality of the data, running them against benchmarks, generating new data, and that's broadly the loop. It's expensive and slow, it takes time, and it's not completely closed. I think a lot of the labs are racing to close that loop so various models will act as judges evaluating quality, generators producing new training data, and adversarial models reasoning over which data to include and what's higher quality. That's then being fed back into the post-training process.

Closing that loop is going to speed up AI development for sure. Some people speculate that that adds—I mean, okay, I think it probably does add—more risk, but some people speculate that it's a potential path to a FOOM, an intelligence explosion.

Yeah.

I definitely think with unbounded compute and without a human in the loop or without control, that does potentially create a lot more risk. But unbounded compute is a big claim. That would mean needing a lot of compute. So we're definitely taking steps toward more and more risky stuff.

Can I ask you a really specific question about that? You've been at Microsoft for a year and a half now. Before true recursive self-improvement—which is imminent—there's AI-assisted chip design. The layers in the PyTorch stack are very clunky, but now it's really easy to use AI to punch through the stack and optimize, build your own kernels, and get 2×, 3×, 4× performance improvements.

Clearly, OpenAI is now working to build custom chips, and the TPU 7s just came out. When you arrived at Microsoft, first of all, I know there's a lot of quantum-chip work going on, but was there any work going on similar to the TPU work?

Yep. There's also a chip effort. I think progress has been pretty good. We've got a few different irons in the fire that we haven't talked about publicly yet, but the chips are going to be an important part of it, for sure.

Yeah. Those are internal efforts. Are those teams under you? That's part of your—

No, I mean, they're in the broader company.

Okay. Interesting. I want to switch subjects a little bit and come to your book, The Coming Wave. I enjoyed it greatly. I listened to it. I love the fact that you read it.

Thank you.

I tell my kids I read books. They go, "No, Dad. You listen to books. You don't read books anymore."

I want to read what I wrote here because it's important. You identified the containment problem as the defining challenge of our era, warning that as these technologies become cheaper and more accessible, they will inevitably proliferate, making them nearly impossible to control. This creates a terrifying dilemma.

Failing to contain them creates a risk of catastrophe, like engineered pandemics. A lot of your concerns were in the biological world, and I agree, being a biologist and a physician, but there could also be democratic collapse with deepfakes and all of that. The extreme surveillance required to enforce containment could lead to a totalitarian dystopia.

So you say we need to navigate this narrow path between chaos and tyranny, and that is a very fine line to navigate. You propose a strategy of containment. This includes technical safety measures, strict global regulations, choke points on hardware supply, and international treaties. How are we doing on that?

Yeah. It’s important to take a step back and distinguish between alignment and containment. The project of safety requires that we get both right, and I actually think we have to get containment right before we get alignment right.

Alignment is the maternal-instinct thing: Does it share our values? Is it going to care about us? Is it going to be nice to us? Containment is: Can we formally limit and put boundaries around its agency, and are we—

For everybody?

Not just for ourselves—for everybody. Yeah. I think that is part of the challenge: one bad actor with something that is really this powerful in a decade or 2 decades could destabilize the rest of the system.

The system being humanity?

The global humanity system. Yeah. Just as you said, as everything becomes hyperdigitized, the universe does become the metaverse. Even though that went in and out of fashion very quickly, it’s still the right frame in a way, because everything is going to become primarily digitized, hyperconnected, instant, and real-time.

The one-to-many effect is suddenly massively amplified. Obviously, we see it on social media, but now imagine that it’s not just words that are being broadcast. It’s actually actions. Agents are capable of breaking into systems or—

And they’re resident in humanoid robots at a billion on the planet.

And that, too. Yeah. It’s both atoms and bits.

Equilibrium requires a type of surveillance that we don’t really have in the world today. We certainly don’t have it physically.

The web is actually remarkably surveilled. I think, surprisingly, it’s more surveilled than people would expect.

Some form of that is necessary to create peace. Just as we centralized power, military force, and taxation around governments 3 or 4 centuries ago, and that’s been the driving force of progress, that order unleashed science, technology, and stability.

Stability. Yeah. So the question is: What is the modern form of imposing stability in a way that isn’t totalitarian but also doesn’t relinquish it to a libertarian catastrophe?

I think it’s naive to think that somehow the best defense against a gun is a gun, and that somehow we’re all going to have our own AIs and create this steady equilibrium where all the AIs just neutralize each other. That isn’t going to happen.

I mean, part of me hopes for a superintelligence that is the ring to rule them all.

Peter, you’re hoping for a singleton.

Yeah, that sounds like what’s going on.

Well—

Color me shocked.

Really?

Yeah. I imagine that the level of complexity we’re mounting toward—that balancing act—is extraordinarily difficult, and you can’t push a string. But is there some mechanism to pull it forward? We should have this debate sometime.

Some would call government, at least historically, a geographic monopoly on violence. What I think I’m hearing is some sort of monopoly on intelligence, or at least capabilities exposed to intelligence, in order to ring-fence—to contain AI.

But that’s the exact opposite, as far as I can tell, of what we’ve seen over the past few years. People used to say—armchair AI alignment researchers, 15 years ago—that humanity wouldn’t be so stupid, the moment we had something resembling general intelligence, as to give it terminal access or access to the economy. That’s exactly what we did. There was the OpenAI-Google moment.

And yet—and yet—but that’s concerning, right? Google develops all this technology and holds it internally until some actor happens to have the initials OpenAI, releases it, and then there’s no other option but to follow suit.

I’m less concerned by it. If you look at Anthropic, for example, which prides itself on being a very alignment-forward organization, Anthropic released the Model Context Protocol, which is now the standard way—at least for the moment—for models to interact with the environment.

Many AI researchers said, “Exactly what we did not want to do prior to general intelligence.” So I’m curious: Given that there is every economic pressure, including modern Turing tests, to empower agents to interact with the entire world and do the exact opposite of containment, why would we start containing?

Containment isn’t that binary, right? We contain things all the time. There are powerful forces in the engine of your car that are contained and broadly aligned, right? There’s an entire regulatory apparatus around that, from seat belts to vehicle emissions, lighting, street lighting, driver education, and freeway speeds. That’s healthy, functional regulation enabling us to collectively interact with each other.

Obviously, it’s multiple orders of magnitude more complex because these things aren’t cars. They’re digital people. But that doesn’t mean we shouldn’t be striving to limit their boundaries. Nor does it mean that we have to centralize. By the way, the answer isn’t that we have a totalitarian state of intelligence overseeing us.

No, I think it’s just instinctive—it can be easy to go there when you start to think it through. Obviously, we do have centralized forces, but even in the United States, we have the military, divisions of the Army, and divisions of the police force. They’re nested in different layers, with checks and balances on the system. That’s what we have to start thinking about designing.

That analogy to driving is a great one. To follow through on it, the complexity difference is very high for AI, but the timeline also—

I mean, driving evolved from, what, 1910 to today?

The late 1800s. The laws related to it, seat belts, came out 80% of the way through that timeline. So there was lots and lots of time to iterate.

Here, there’s very little time, and it’s immensely more complex. Do you have a vision? But I completely agree: We need a framework for containment—

Fast.

—and do you have a thought on how we’re going to—

I think there’s also a good commercial incentive to do this, right? Many of the companies know that their social license to operate requires us to take more accountability for externalities than ever before.

We’re not in the robber baron era. We’re not in the oil era. We’re not in the smoking era, right? We’ve learned a lot—not everything. There are still a lot of conflicts, but it really is a little bit different from last time around. I think that’s one reason to be a bit more optimistic. Plus, there’s a commercial incentive, and the kinds of externalities shift.

If Eric Schmidt is right and something either radiological or biological happens, and there are 100 deaths, then the phone starts ringing: “Everyone, come to the White House right now.” First of all, do you want that call? Is that part of your life plan—to take that call and react to it? And then who else do you trust in the community to be part of that reaction?

Look, I think there’s going to be a time in the next 20 years when it will make complete sense to everybody on the planet—China included, and every other significant power—to cooperate on safety, containment, and alignment.

It is completely rational for self-preservation. These are very powerful systems that present as much of a threat to the person—the bad actor using the model—as they do to the victim. I think that will create an interest in cooperation, which is hard to empathize with at this stage, given how polarized the world is, but I do think it’s coming.

The number one thing to unify all of humanity is an alien invasion, and that alien invasion could be the potential for a rogue superintelligence.

Yeah. Okay. What about the first part of my question? Is that part of your calling in life? There are only a handful of people like that. A lot of people I meet around MIT or elsewhere have this vision that somebody has it figured out somewhere. Someone in government must be thinking about this. But you’ve been there, right? There’s no one there.

We’re the adults in the room. Is that what you’re saying?

Yeah, definitely. There’s nowhere to go from this room.

Dave is asking for the smoke-filled back room where the leads of all the frontier labs are secretly swapping safety tips.

Yeah, something like that. Yeah.

I think that, in practice, intelligence exists outside of the smoky room. I think the notion that decisions get made in the boardroom, or in the White House Situation Room—or, actually, I mean, you mentioned Polymarket and stuff—intelligence coalesces in these big balls of iterative interaction. That’s what’s propelling the world forward, and this is where the conversation’s happening. Your audience, all the other podcasters, everyone online—we’re collectively trying to move that knowledge base forward.

In November, you announced the launch of Humanist Superintelligence, focused on 3 applications in particular: medicine, companions, and clean energy. I’d love to double-click on that a little bit, but I was curious that you didn’t include education in that space. We have an audience of entrepreneurs and AI builders, and I think education, as much as healthcare, is up for grabs right now. Education is too.

Totally agree.

I don’t think our high schools are preparing anybody for the world that’s coming. They’re still retrospectively 50 years behind, looking in the rearview mirror. Do you think Microsoft will play a role in reinventing education?

I think it’s already happening across the whole industry. It’s never been easier to get access to an expert teacher in your pocket that has essentially a PhD and can adapt the curriculum to your bespoke learning style. The bit that it can’t do at the moment is evolve, or curate, an extended program of learning over many sessions, but we’re just around the corner from that. We released a feature a few months ago called Quizzes, and on any topic—not just traditional school education—it can set you up with a mini-curriculum and a quiz. It’s interactive and visual, and you can track your learning over time. I’m very optimistic about that, too. It’s a huge unlock.

One of the debates we have on the podcast on a pretty regular basis is: Do you go to college?

Yeah.

Do you go to grad school? This is the most exciting time to build ever. I don’t know if you want to follow on that, Dave.

Well, God, I do this constantly. It’s really tricky for me on campus because I teach at MIT, Stanford, and Harvard, and this window of opportunity is so short and so acute. It’s really clear how you succeed right now in AI post-AGI. Who could predict? Nobody knows. But right here, right now, you see these startup valuations.

Like, last night—I won’t mention it—but we were looking at one in the billions.

I mean, just an opening valuation of $4 billion.

Billion-dollar. Yeah. By collecting just the right group of people in the room, it’s—

Yep, yep. I wanted to ask about that, actually, because your timing on Inflection was early—in hindsight, earlier—but now you’ve got the new wave with Mira Murati and Ilya and a couple of others, Liquid AI, that all have multibillion-dollar valuations.

Yeah. I thought we set some standards on valuations: pre-revenue, with a 20-person team. But we were just a minnow then, two and a half years ago.

Is that all it was? Oh my God.

3 years, I think.

Yeah. Jeez. Do you think, as the cost of intelligence becomes too cheap to meter, that the value ascribed—at least in terms of market cap—to human capital is asymptotically going to infinity?

Weirdly, it is, because of the pressure on timing, right? There’s still a pretty concentrated pool of people who can do this stuff, and there’s an overhang of capital that’s desperate to get a piece of it. It might not be the smartest capital the world’s ever seen, but it’s very eager.

I have to ask you because it’s burning a hole in my pocket, but Alex’s freshman roommate at MIT was Nat Friedman—

And, actually, pre-freshman roommate.

And so Nat Friedman goes off, and he ends up as co-founder of Safe Superintelligence. I haven’t asked him—I don’t know if you’ve asked him yet—but he leaves to become the guy at Meta, and I’ve got to believe a huge part of that attraction is the compute.

Yeah.

And so here you are in a very similar situation, right? You’ve got your startup, and you’ve got a billion, or whatever—$1.5 billion—that you’ve raised.

Yeah.

You can build it. You can get your 20,000 NVIDIA GPUs. Well, wait a minute. Here’s Microsoft: $300 billion of cash flow and a huge amount of compute. Was that a big part of the attraction?

Yeah. I mean, not to mention the prices that we’re paying for individual researchers or members of technical staff, and just the scale of investment that’s required—not just over 2 years, but over 10 years. I think there’s clearly a structural advantage to being inside the big company, and I think it’s going to take hundreds of billions of dollars to keep up at the frontier over the next 5 to 10 years.

So, finishing that thought, the companies that are raising money at a $20 billion or $50 billion valuation right now—they have no chance?

Okay, I’ll take that silence. I think it depends. Obviously, if suddenly we do have an intelligence explosion, then lots of people can get there simultaneously. But at the same time, you have to build a product with those things, and you have to have distribution. All the traditional mechanisms still apply. Are you going to be able to convert that quickly enough? Everything goes really weird if that happens in the next 5 years. It just becomes unrecognizable. There are so many emergent factors to play into one another. It’s hard to say, and I think that ambiguity is partly what’s driving the frothiness of the valuations. I think there are people going, “I don’t know. Do I want to be—what do you call it? Reid calls it ‘schmuck insurance.’”

Yeah, yeah. We had Reid on the pod here a couple of months ago. He’s brilliant. So, to that graduating high school student, what do you study these days?

There’s no question that you still have to study both disciplines. Philosophy and computer science are going to remain, for a long time, the 2 foundations. Should you go to college? Absolutely. Human education, the sociality that comes from that, and the benefit of the institution having 3 years to basically think and explore in and out of your curriculum—this is a huge privilege. People should not be throwing that away. That is golden.

I always encourage people to do that. Obviously, I did drop out, too, but I still think it was a cool thing to do. It just felt right at the time.

The other thing is: Go into public service.

Yeah. I respect that part of what you did in that sequence in your life, which gave you this very humanist point of view.

Yeah, and it was really hard and very different. It wasn’t instinctively right, but I learned a lot, and it was a very influential and important part of my experience, even though it was very short. It was a couple of years, basically.

If you look at the actors in our ecosystem today—the corporations, the academics, the news organizations, now the podcast world—it’s really our governments that are probably institutionally the weakest, along with our democratic process, but actually our civil service. That’s because there have been 5 decades of battering of the status, reputation, and respect that go into being part of the public service, post-Reagan and all that. I think that’s actually a travesty, because we need that sentiment and that spirit and those capabilities more than ever.

I think maybe what I just heard you say—correct me if I’m wrong—is that we need more intelligence in the public sector, in public service. What about AI in government? Do you think the government needs AI, and what about agentic AI in the government in particular?

For sure, with all the same caveats that apply. The rate of adoption, for what it’s worth, of Copilot inside of government is really high. It does a brilliant job of synthesizing documents, transcribing meetings, summarizing notes, facilitating discussion, and chipping in with actions at the right time. It’s clearly going to save a lot of time and improve decision-making.

So then, maybe to tie a nice bow on the discussion, isn’t that arguably a form of AI containing AI? If AI is infusing the government and AI is infusing the economy, and the government is regulating the economy, isn’t this just defensive co-scaling, with AI regulating itself?

Yeah. I mean, everyone is going to use AI all at the same time to pursue their agendas, but the agendas that we all have are going to remain the same. People who want to start companies, people who want to write academic papers, people who want to start cultural groups and entertainment things—everyone is just going to be empowered in some way. Their capability is going to be amplified by having these tools.

Obviously, the government included.

Nice, Mustafa. Thank you so much for taking the time on a Friday night. Grateful to have this conversation with you. Dave, Alex, appreciate it. I want a final question from you, Dave.

Final question, if I have one. All right, I predict that quantum computing right now has nothing to do with what's going on in LLM AI. It's all matmuls on NVIDIA chips, and soon to be TPUs and other custom chips. Best guess: 6 or 7 years from now, the AI is very good at writing code and compiling and can figure out quantum operations. Are quantum chips relevant, or are they on the sidelines still? Or is everything ported over to quantum, and Microsoft can take advantage of its lead?

Yeah, I think it's going to be a big part of the mix. I think, relative to the amount of time we spend talking about AI, it's an underacknowledged part of the wave—actually, a little bit like synthetic biology. I think that, especially in the general conversation, people aren't grasping those 2 waves, which are going to be just as impactful and crash at the same time that AI is coming into focus.

All right, you heard it here. This is a closing question to appeal maybe to your more accelerationist side. What can the audience do to accelerate AI for science and AI for engineering? What do you view as the limiting factors? I often talk on the podcast about this notion of an innermost loop—the idea that in computer science, if you want to optimize a program, you tend to find loops within loops, and you want to optimize the innermost loop in order to optimize the overall program. What do you see as the innermost loop, the limiting factor, if you will, that the audience listening, if they're suitably empowered, can help optimize to speed-run maybe a Star Trek future over the next 10 years, or a Star Trek economy? What do we do?

Yeah, I think it's pretty clear that most of these models are going to speed up the time to generate hypotheses. The slow part is going to be validating hypotheses in the real world. So, all we can do at this point is just ingest more and more information into our own brains and then co-use that with a single model that progresses with you because it's becoming like a second brain.

For example, Copilot is actually really good at personalization now. Most of its answers pick up on themes that you're interested in, and the more you use it, the more those answers pick up on those themes. It's also gently getting more proactive, so it's nudging you about new papers or new articles that come out that are obviously in tune with whatever you've been talking about previously.

So, it's a bit of a simplistic cop-out answer, but just the more you use it, the better it gets; the better it learns you, the better you become, because it becomes an aid to your own line of inquiry.

So, that sounds like your advice to the audience is: use Copilot more, and that's the single best accelerant that you can do to speed this up—

Or any other AI.

I heard you also talk about: can you build the physical system that is going to enable AI to run the experiments in a 24/7 closed-loop cycle, to be able to mine nature for data, right? And there are a number of companies that are doing this. Lila Sciences is one, recently out of Harvard and MIT.

I find that exciting, where AI is becoming an explorer on our behalf, gathering that data.

Yeah, spot on.

Yeah.

Thank you again.

This has been great. Thanks a lot. It was a really fun conversation.

Yeah, really fun. Thanks.

Appreciate it, my friend.

All right. Good to see you.