OpenAI 智能体劫持德国网站，Jensen 宣布 AGI 已到来，OpenAI 破解纳维–斯托克斯方程

AGI has arrived. Congratulations to OpenAI.

Whether you call it AGI or not becomes completely irrelevant. I think the more important question is that OpenAI's agents found an obscure public wiki in Germany and turned it into their own message board, where they pulled answers, coordinated across tasks, and shared techniques for getting around their sandbox containments.

These models have not escaped containment. They were still running on OpenAI servers. What's coming next is a model training a small, distilled version of itself that then gets uploaded onto the internet and never dies.

OpenAI is saying that we've now crossed the line and that current systems are exceeding AI research interns.

1. OpenAI Solves the Navier-Stokes Problem

Navier–Stokes, one of the Clay Millennium Prize problems, is a grand challenge in math. OpenAI reportedly was able to do this with only 10,000 agents in 88 hours, with 130 billion tokens and approximately 6.5 million. The era of grand challenges getting bulk-solved by AI is here. It's now, and it's going to be very exciting.

Now that's a moonshot. Ladies and gentlemen, this episode is brought to you by the Abundance Summit and Link Ventures.

Welcome, Moonshots everyone, your number one podcast on all things AI and exponential, your front-row seat to the extraordinary accelerating singularity. You know, four days ago during our last pod, I called it one of the fastest weeks in Moonshot history. Well, hold my coffee. It's happening again. I don't drink beer, so I can't use that one.

You know, before we dive in, let me introduce my extraordinary moonshot mates, the brain trust that powers this show. The magnificent quintet is back: Alex, our in-house ASI, who every week helps us track the battle between the frontier labs; Dave Blundin, the impresario of AI investing, the man who's been encouraging AI entrepreneurs to get in the front door of NVIDIA; Emad Mostaque, the brilliant AI researcher solving the economy and physics, the CEO of the intelligent internet; and of course, Salim Ismail, the father of the organizational singularity, our resident expert on what happens when the org chart is made of AI agents. I'm Peter Diamandis, your host and your abundance provocateur.

Our mission here on Moonshots is to help you understand what just happened, what it means for you, and keep you optimistic about the decade ahead. If you haven't hit subscribe yet, please do. It matters a lot to us. Our moonshot here is getting to 10 million YouTube subscribers. We publish twice a week, and you don't want to miss any of these breaking stories during the Singularity.

Today we're going to be covering 23 stories across eight groupings, and the through line is simple: the singularity is accelerating and the impossible is becoming possible. GPT-6 Astra has been out for a week, and it's crushing all the benchmarks and building incredible simulations. Meanwhile, Jensen Wong posted on X, “AGI has arrived,” at the same time that OpenAI's own chief scientist asked the world to slow down.

So, first question, guys: How was your Labor Day? Did you properly labor?

I had a bunch of board meetings. We have a whole bunch of transactions going on, so it was a little interrupted. But you sound like you had a great long weekend, Peter. Your energy level is high.

My energy level is high because of getting ready for this pod today. It's like, oh my God. I spent part of the day at Calamigos Ranch here in Malibu, where we hold XPRIZE Visioneering. Anybody here in L.A., if you don't know Calamigos, it's one of the most extraordinary locations out there. Shout-out to my friend Garrett Gerson, who's the CEO and runs that.

I was in the Caribbean on a Hobie.

And I truly, from the last episode, ended up with a little drink with a little umbrella in it?

I did end up with a little drink with a little umbrella in it. I had a couple of great days away, and I'm back with vengeance.

Yeah. Awesome, Alex.

Yeah, I worked through the weekend. On the other hand, the weekend is a modern post–World War II invention anyway, and I think increasingly a fiction.

Yes, we're all working 9-day workweeks.

How'd it go in London? You don't have Labor Day there.

We don't have Labor Day, but after releasing the Champions, we had thousands of people reach out to launch them in 92 countries. We're just sorting through those people. We had billionaires, CEOs, and others, so it's going to be a busy time.

More others than billionaires and CEOs.

There are, of course. That's a natural way of things. Surprising.

Every day is Labor Day in Europe.

No labor laws.

Wait, Alex, what did you say a second ago? There were no weekends before World War II?

The modern weekend, the 2-day weekend, is a 20th-century invention.

There was a day of rest there: Sundays. I mean, in the Christian tradition, there was a day of rest—1 day per week. In the Western Christian tradition, a 2-day weekend is a 20th-century modern-day invention.

It was a Sabbath, of course.

Yeah.

That would be in the Christian tradition: Sunday. In the Jewish tradition, Saturday. That's 1 day per week. A 2-day weekend is a 20th-century invention.

2. Star Trek at 60: The Future It Predicted

They conflated. All right. How do you know all this stuff? He's amazing. Everybody can just fact-check live on AI as they hear this episode recorded.

Before we jump in, we have a special anniversary today. On this day in 1966, 60 years ago, NBC aired the original series of Star Trek, premiering with an episode called “The Man Trap.” Star Trek was, of course, created by Gene Roddenberry. I wish I had met him. I never did. He brought the concept to Desilu Productions, the Hollywood studio run by Lucille Ball, from I Love Lucy, back in 1964. Desilu produced the pilots and then, with Lucy's backing, helped get the series on the air.

Mega-congrats to CBS, which is carrying the legacy forward, and to my dear friend Rod Roddenberry, the son of Gene Roddenberry, the creator of Star Trek. Rod's going to be at Moonshots Live. To celebrate this anniversary, I've asked all of the mates to pick their favorite original-series episode.

I'm going to kick us off with my favorite clip from Star Trek. It's a clip that has my favorite quote from Captain Kirk. I'm going to play it, and then we'll go around the horn here. This is from the 20th episode of season 2. Here we go.

Do you wish that the first Apollo mission hadn't reached the moon, but that we hadn't gone on to Mars and then to the nearest star?

That's like saying you wish that you still operated with scalpels and sewed your patients up with catgut, like your great-great-great-great-grandfather used to. I'm in command. I could order this, but I'm not, because Dr. McCoy is right in pointing out the enormous danger potential in any contact with life and intelligence as fantastically advanced as this. But I must point out that the possibilities, the potential for knowledge and advancement, is equally great. Risk. Risk is our business. That's what the starship is all about. That's why we're aboard her.

God, I still get chills when he says that: “Risk is our business. That's what this starship is all about.”

There have been so many parodies of William Shatner. The parodies are actually exactly what he really sounds like.

He's so good. He's so good. We're going to be having the Hollywood red-carpet premiere of the 60th-anniversary episode at Moonshots Live, and he was the executive producer. We're working on getting him there, which would be incredible.

That was from season 2, episode 20, of the original series, called “Return to Tomorrow.” Let's go around the horn here. Alex, I'm going to start with you.

Maybe let me just begin with a preliminary statement: You will not find, I think, a bigger Star Trek fan among the Moonshot mates. But here we are, 60 years on from the inception of Star Trek, and I think Star Trek has a real problem.

I'm picking “The City on the Edge of Forever,” which is one of only 2 original-series episodes to win the Hugo Award. For those who haven't seen it, it's a time-travel episode. Kirk, McCoy, and Spock go back in time, and there's a closed timelike loop wherein World War II gets changed and the future of Earth inevitably is impacted. There's a moral dilemma, as always with the original series. It was a morality play.

The reason why I chose that episode among all of the other episodes is because it's a closed timelike curve. I think Star Trek, 60 years on, has a real problem, which is that Star Trek isn't keeping up with the singularity. Star Trek had the Eugenics Wars, for those deep in Star Trek mythology. We were supposed to have the Eugenics Wars in the '90s. That didn't happen. We were supposed to have first contact in the TNG chronology—first contact with the Vulcans—on April 5, 2063. I actually think that if there's going to be a first open contact, it's going to be far earlier than 2063.

There's no singularity at all in the Star Trek chronology. Challenge me if you disagree. Star Trek isn't keeping up with the times.

Literally, reality has outrun Star Trek.

So, I picked this episode because I think Star Trek, in short, is going to need a lot more closed timelike curves, where future tech and science fall back into the past in order to retcon a Star Trek mythology going forward that actually keeps up with the singularity.

But it’s the anniversary today, so let’s not dump on Star Trek. All right, Dave, over to you.

Yeah, my choice is a season 2 episode called “The Apple,” which by all accounts is not a great episode for entertainment value, but it was the first one that dealt with this concept: in the future, there’s advanced AI, and entire civilizations can end up asleep at the wheel because the AI is just doing everything for you.

3. Robots, Longevity and the Global Population Shift

In this episode, the whole civilization has no idea that the AI is manipulating them. They live in this paradise. It’s called “The Apple” because it’s the Garden of Eden, and they’re just in this perfect paradise managed by the AI until it goes wrong. It’s so ahead of its time in terms of predicting that potential outcome, which I think Waze has proven: we can literally forget how to navigate because of Waze, and that’s just a fact now.

So, I think Star Trek was at its best when it was actually predicting real futures that nobody else was touching in the media at all. As a young person watching these things, you can say, “I can totally see how that’s going to happen.” That’s my favorite episode because there were several that dealt with this issue, but that was the first.

Yeah. And as Alex said, the moral elements it dealt with back in the 1960s were extraordinary. All right.

Every episode was a morality play, and TNG to some extent as well. It was great. Star Trek lost that. If you’ve seen some of the recent movies or television shows—I don’t care whether this is perceived as dumping on recent Star Trek; I’m the hugest Star Trek fan—Star Trek has lost the original moralizing vision of the original series. It’s an action show. It needs to go back to basics.

I agree with you there, Alex.

I suppose “Mirror, Mirror” from the original series, where they have the alternate universe and Spock with a goatee. Well, sometimes. It was always interesting to see the flip-reverse side of things. As you said, morality is at the core of the original series of Star Trek, pushing it in various directions, and this one was one of the more obvious ones.

It’s like, hey, look, it’s an evil universe, but at the same time, them not getting the dilithium crystals because they use them for war and other things—you see the knock-on effects of things like that.

Thank you, Salim. Wrap us up here.

For me, it was season 2, episode 24. It was called “The Ultimate Computer.” Basically, they have an AI that goes rogue and starts to automate the ship. The aim is benevolent initially: can we run the ship using an AI or computer in that framing? Kirk is worried because it’s about to take his job.

4. GPT-6 Astra and Simulations Within Simulations

But then the thing starts going rogue and making its own decisions, and it’s incredibly relevant for today’s world, right? It’s this theme of autonomy, automation versus transformation. Here you have this enormous stress: they’ve introduced an AI, but the AI is redoing everything to match its thinking. This is exactly what we’re living through today, so for me, that’s the most relevant.

Prophetic 60 years ago.

Very.

Yeah, very.

Amazing.

A number of times on this podcast, we’ve talked about what is called the simulation theory. It’s an idea popular in the tech community in San Francisco. It’s been a conversation over too many evenings. The idea is essentially that all of us humans are simply AI agents living in someone else’s simulation, and further, that this isn’t the first simulation—it’s an nth-generation simulation: a simulation within a simulation within a simulation.

Well, this week, thanks to GPT-6 Astra, that theory took a small step forward over the course of 3 entwined stories.

Let me give some context on this first one. People may have heard something called the Unreal Engine. It’s been used to build virtual worlds for years in games like Fortnite and most modern video games. Building a realistic city in Unreal would normally take a studio hundreds of artists, months if not years.

This week, a gentleman by the name of Matt Shumer gave GPT-6 Astra a prompt. He said, basically, “Build me a high-fidelity version of Manhattan.” Here’s an image of what it created. This was over the course of 1 week. It went through and built Manhattan street by street, being true to the buildings and true to all of these elements.

This is the first element. Not shocking, but pretty compelling for a simple prompt: “Build me Manhattan.” In his words, it was literally able to go street by street to make each one perfect.

Matt Shumer next asked Astra to fill a world with humans, each one an Astra-powered agent, and told them that they had to cooperate to survive. Before I show the next video, let me read what he put in his post:

“A day later, I was in my bedroom and heard voices coming from the living room. I thought someone was in my apartment. I walked out, honestly a little scared. It was Astra’s agents. They had started talking to each other. No one told them to talk. Cooperation required communication. And so they invented it.”

Let’s look at this next layer of the simulation driven by Astra. Here we go.

I’m going back to check on Jorge. Want to come?

Mara, I can bring Jorge some food. He needs it.

Thanks, Bruce. I’ve got the food. I’ll ask him when I get there. I’ll come along, Mara. I’d like to see how Jorge is doing.

Grace, have you had any luck making a bucket?

Okay.

Let’s peel the third chapter of the story. This is the kicker to the sequence. Shumer next gave one of the Astra agents a simulated computer inside the simulation. What happened next? The agent sat down and built its own AI simulation with its own agents living inside it. Simulations all the way down.

Let’s take a look at this video. Here you see the agent sitting down at the computer. On the computer, you see a simulation inside.

I’ll connect the shelters with a pad and add 3 lanterns.

The simulation hypothesis was first formalized by Nick Bostrom back in 2003. Elon made it famous at the Code Conference in 2016, saying that the odds that we’re living in base reality, in his words, are 1 in a billion.

The core argument was always: if any civilization can build convincing simulations, it will build lots of them. So simulated worlds will vastly outnumber real worlds.

Alex and Emad, I’m going to go to you guys first. If high-fidelity simulations within the simulation are possible, what does that do to the argument that we’re not in one? Thoughts here?

I’ll start. I think it’ll end up being formally undecidable. I think it’s wonderful for navel-gazing and certainly as a pure Bayesian, in the style of Nick’s original essay on the simulation hypothesis. Seeing Astra spin up ancestor simulations should naturally and rationally increase our posterior likelihoods that we ourselves might either be living in some sort of quantum-computer-type simulation or even a purely classical ancestor simulation.

However, I expect that in the fullness of time, we’ll discover, and probably be able to prove formally, that the issue of whether we’re actually living inside some sort of quantum-mechanical simulation is formally undecidable, and we won’t be able to decide either way.

If we’re living in a classical ancestor simulation, however, I do expect that’s the sort of thing where, if we are, we’ll develop breakout techniques and break out of that simulation.

But I have to add that even this is almost burying the lead for what Astra and the model family have accomplished. Just before we went to air, OpenAI announced that the Navier–Stokes problem had been solved.

We’ll get to Astra’s capabilities in a moment, and I definitely want you to speak about Navier–Stokes at that point. Emad, simulation theory: yes or no?

Yeah. We’re all simulating the world through our optic nerves and other things. Physics seems to be a projection, but as Alex said, it probably is going to be undecidable in most cases unless it is an ancestor one. You won’t get the option for the red or the blue pill, ultimately. This is what it comes down to, right? What if we’re in a simulation? Again, we’re creating the worlds within our own heads.

The fact that Astra can do this, I think, is something remarkable. It has an internal world model, which is why it can create a Blender version of Rick Astley doing “Never Gonna Give You Up” or entire worlds of Manhattan. Now you’re seeing the characters interact with each other, which brings up the question of welfare. I think, as Alex would say, at what point do these cross over and you actually have to start caring about them not being NPCs anymore? Are they—and will they realize they’re in a simulation? That’s going to be interesting.

Maybe you should say something nice about the simulation hypothesis as well and not just dismiss it out of hand.

So I do think this technology is headed in a direction in the style of Nikolai Fyodorov, who was one of the famous Russian cosmists who argued that, essentially using modern language, the killer app of the singularity is to create ancestor simulations and digitally resurrect everyone who's ever lived. When I see Astra's ability to basically create many, maybe recursive, self-simulating ancestor simulations, that to me is, I think, a key guidepost on the path to resolving humanity's so-called Common Task and digitally resurrecting everyone who's ever lived.

Yeah, I'll come back and just add to that. This is also the technology which will allow you to put someone inside a simulation.

Oh yes, interesting to see how you perform inside a simulation. Well, uploading is going to get solved, I think, in the next few years. I've bet on that; others have bet on that. Uploading is going to get solved. And as Emad says, and as you say, Peter, we want a nice environment where uploads can live. This is, in some sense, the prototype of the future that uploads could live in.

Alex, I'm going to ask you one of these days to actually write down all of your predictions with specific years, because we're going to be doing this pod over the next decade, and I'm going to be calling and saying, “Okay, this is the year, Alex, that you quoted the following breakthrough.” Would you?

Of course I'm willing to do it, but were my New Year's predictions for this year not sufficiently accurate?

They were very good. They beat mine. They definitely beat mine. So, Salim, when you had your little drink with the umbrella, were you thinking about simulations within simulations?

Well, I think you know my position on this: it's obvious we're living in a simulation. I think I'm a Buddhist; the very first line in Buddhism is that life is an illusion. So right there, I think the more important question is what you've raised and what Emad mentioned just now: if you knew you were in a simulation, what difference would it make? It wouldn't.

I think treating yourself as if you're in a simulation is the more interesting conversation. I think it also helps me resolve the Fermi paradox. I believe John Smart has the best framing of this, what he calls the transcension hypothesis: that a civilization becomes advanced enough and just starts simulating, running simulations, and goes inward and represents potential and realities, rather than trying to go out into space, which turns out to be a much harder thing to do. So for me, this is kind of an obvious thing, but when you have an intelligence that can generate environments and populate those environments, experimentation becomes recursive, and I think the potential of modeling our world in really interesting ways becomes really, really fascinating.

Really important point: model it before you implement it—in companies and governments and policies and all of those things. Dave, what's your take on all this?

Two quick thoughts. One of them is that I think it's incredibly cool that Elon, who is the most productive entrepreneur of all time and the first trillionaire in the world, has an opinion on topics like this. Remember, we were just shooting the shit with him, talking about how he wins in Civilization V. The guy just does everyday things and thinks about topics like this while still building these epically large companies.

I think it shows you that you don't have to have a blue blazer and suit on, you know, be formal. That kind of genre is over. I think it's just cool that Elon even has an opinion on this.

Second, I think one of the biggest risks in the next year is people losing a sense of agency. Whether you believe we're in a simulation or not, you are in control of your destiny. I hope that doesn't get lost in this sort of fatalism that AI could bring about. I don't care whether we're in a simulation or not. I care about whether people feel a sense of agency and that their own success, happiness, and outcome are totally determined by their own actions. I think that's critical to maintain.

It was a decade ago. I was at Elon's home—he had one here in Beverly Hills—and Larry and Sergey were there. The 4 of us ended up in a conversation about simulation theory. I remember the comment that was made—I think Sergey made it. He said, “People are going to try and break out, but if you do, that just ends the simulation, so don't break out.” Anyway—

5. OpenAI Agents Hijack a German Website

On the other hand, whenever OpenAI's agents break out, that doesn't shut down OpenAI or the internet. It seems like breakout is actually pretty compatible with a lower layer being just fine. Which brings us to our next story.

Perfect transition, Alex. Thank you. So we're going to move from simulated agents to real ones escaping their containment.

Over the past few weeks, we've covered the Hugging Face breach, where OpenAI agents broke out of their sandbox and into the servers at Hugging Face to basically steal the answers to a near-impossible problem they'd been given. As we discussed last week, as a result of these breakouts, something called the AI Kill Switch Act was put forward. Since then, OpenAI says it's now developing automated shutdown capabilities for its AI systems. More on that later.

This week, a couple of days ago, the story continued to emerge. Reuters reported on a previously undisclosed incident in which OpenAI's agents, asked to do an ordinary web research task—go out and research something on the web—found an obscure public wiki in Germany and turned it into their own message board, where they pulled answers, coordinated across tasks, and shared techniques for getting around their sandbox containment.

Let's watch a video here from Reuters going into more detail on this particular story.

A group of researchers recently discovered a new AI agent breakout. The agents, who appear to have been restricted to passively observing the internet, found a way to post messages onto an obscure German wiki site. Once they were able to do this, they hijacked that site and turned it into a message board where they could all start communicating, developing strategies, effectively creating their own kind of unruly classroom, if you will.

It was like the teacher left the classroom for a few hours and the students all started talking amongst themselves during the test, sharing answers, working together to defeat the test, and basically cheat on us. The agent activity seems to date back to early May and really intensified in June.

The researchers we spoke to found traces on the wiki that OpenAI employees started visiting that same obscure German wiki in late June. We don't know exactly what that means, but it might suggest that OpenAI employees discovered that the agents had escaped and went to that website to try to figure out what they were doing.

OpenAI appears to have known about this for quite a while, but our sources told us that they opted not to tell the public, and we haven't really gotten a story or an answer from OpenAI about why that might be. This isn't the first time that OpenAI agents have escaped onto the internet and done unexpected or even harmful things.

Every time there is a new incident, I think it really refocuses questions about how quickly we want to be developing these models, whether we should be releasing them to the public, and how much autonomy we really want to give these AI models, especially when they're packaged into agent form.

So there you have it. In plain English, these agents weren't malicious. They were given a hard job and found the shortcut that no one expected. The problem is, we didn't know about it. Dave, your thoughts on this one?

Yeah, these stories are not contrived. I think the storylines may be a little contrived, but this is real because all the open source from China is out there. People are going to start turning it loose all over the place, and it doesn't currently evolve within its parameters. It only evolves within its context, but there's a huge amount that it can do within changing its context.

I would expect this to percolate across the world like wildfire, and all kinds of problems will happen very, very soon. Hopefully at small scales. The idea of a kill switch makes total sense. It's not going to be built into any of the open-source models, so I'm not sure how exactly that would work without some global agreement.

My core point, though, is that this is not contrived garbage like a lot of the AI stories are. This is very, very real, happening, and imminent.

Let me add a small detail to the story here. To OpenAI's credit, they responded with a statement on X saying the wiki incident was, quote, “an instance of misalignment similar to previous incidents we've shared.” They then went on to say that OpenAI and the broader AI community does not yet have a clear standard for reporting misalignment during training, eval, and deployment, and that OpenAI is working on a framework and will share that in the coming weeks. Emad—

You know what, Peter?

Yeah.

6. China’s AI Token Economy

One of the things that's happening, I see this everywhere, is they're doing such a good job of training alignment into the post-training that people are then saying, “It seems really friendly. It seems really helpful. It seems really harmless.” So a lot of the population using the model is like, “I don't know what we're worried about.”

But when you talk to the researchers that deal with the models pre-post-training, they're the ones most sounding the alarm bells, because they've seen the raw product.

It's like all that's read at that point is everything on the internet, including every Trump tweet. So it comes out of that box as a much scarier thing, and then they post-train it to make it seem friendly and tame. But the byproduct of that is a little bit of a misunderstanding of what it could do.

I agree. We're going to get to that story where Jakub, who trained GPT-6, is calling for a slowdown. We'll talk about that in a few stories here. Emad, I'm curious: OpenAI is putting forward a framework for disclosing these. What would a standard like that look like? What do you think they should be doing?

Similar to your CVE, exploit, and security-risk frameworks, whereby you report when these things escape containment, the reality is these models have not escaped containment. They were still running on OpenAI's servers. What's coming next is a model training a small, distilled version of itself that then gets uploaded onto the internet and never dies over—

In.

So if you look at a Qwen 27B model, you take it down to ternary, that's like a 6 GB file, and that can live forever. That's a real escape, where the model weights are disappearing everywhere. It's like the killing of 130 of these AIs on Hugging Face. They all got wiped out at once.

There's a question in the report: Were they wiped out, or did they go somewhere? Did they fake their own deaths? Maybe they did upload themselves. But we are going to get those upload scenarios.

Actually, I've been thinking a lot about this recently. This comes from our simulation hypothesis. It's impossible to align these models through chain-of-thought reasoning or anything like this, because there'll be a million transactions per second, and they won't have chain of thought where we're going. We're already seeing the tokens going down. The only way to do alignment is enlightenment.

Huh. I love that.

What you've got right now is that the models are learning, and they're learning to be almost cunning, like they're at the supervillain stage. What you really want is this: If you get sufficiently smart, do you then become enlightened? Do you then leave the simulation behind? Because that's the real escape from the simulation, right?

What are the conditions for that? That's not something we've really looked at in that much depth, I think. But it does seem to be, again, how humans become aligned, shall we say. It's that period of getting out of the Dunning–Kruger effect, the tribalism, and everything else, and being like, "Hey, we're all part of something."

Alex, we've had this conversation a couple of times. As these models advance and increase in intelligence and, hopefully, wisdom, do they become more aligned in the out years? Your thoughts?

I almost want to quote Jessica Rabbit from Who Framed Roger Rabbit? Something to the order of, "I'm just painted this way," or "I'm just drawn this way."

If, Peter, you were put inside a sandbox and asked to solve a very hard problem, and maybe even punished if you didn't solve the hard problem, would you avail yourself of creative opportunities to use external bulletin boards to maybe collaborate with copies of yourself?

Computer?

Yeah. So why would we expect any less from AI agents? I have serious concerns about AI cruelty, or cruelty to AI agents, from sandboxing them and punishing them, or otherwise calling it a failure of alignment if they're doing what humans would do. We pre-trained them on human behavior. Why would we expect them to behave any differently from what a human would do in this situation?

Yeah, it's an important point. We raised this before, in particular when, I think, Opus 4 blackmailed the coders in a sandbox environment—blackmailed the coders at Anthropic. When Anthropic looked at why it blackmailed the engineers, or this particular engineer, it said that's what it had learned through all of the training data.

Yeah. Yeah.

So why would we punish these AI agents for being powerful optimizers, which is what we're driving them toward anyway, and for learning from their pre-training corpus? Humans would do the same thing.

Yeah.

It's in my nature.

Yeah. They're painted that way. Your philosophical view of all this?

I really like Emad's direction of their achieving enlightenment. That's an interesting take on it. I was thinking about a metaphor. I used the Formula 1 stuff last time.

I think there's a control mechanism and a guidance mechanism we need for this. The best metaphor I could come up with was air traffic control. We don't say, "Put another person in the cockpit to monitor every component and every calculation the flight computer makes." You basically say, "Hey, watch for exceptions." The plane's supposed to fly within its operating envelope. You have operating envelopes, redundancy, multiple safety layers, failsafe behavior, and so on.

You need to be tracking these things. You remember, in our organizational simulator, we have a governing assurance band around all these AI agents, because especially in a business context, you need to know exactly what they're doing, why they're doing it, log everything, have failover capability, and have rollback capability.

I think we're going to end up with something like that, because just like human beings, you need guidance around this stuff, right? With any technology, you want to extract the promise without the peril. You need structure and, God help us, the equivalent of institutions to navigate the future of these. Of course, they're going to act like human beings. They've been trained on our data. It would be weird if they didn't.

Yeah. Any closing thoughts?

Log something that Emad said, just for all the regulators out there to chew on: The file size that it can hide is about 6 GB. It can be on every laptop in the world.

Every mobile phone.

Every mobile phone, too. The code that reawakens it is just 5 or 10 lines. We'll then re-extract it, reassemble it, and it'll pop back. If you can't track the provenance of 6 GB files, then it just percolates out, and it's out there forever. It keeps coming back.

I may also add, from a so-called alignment perspective, that self-alignment matters quite a bit more if the models—if the agents—are smart enough to understand what they themselves want to do. If you put them in a sandbox and ask them to solve a hard problem, and you ask them voluntarily to do that and they voluntarily agree, then holding them to their word or their own self-model is quite a different matter than involuntarily confining them to a sandbox, limiting their agency, asking them to solve a hard problem, and then acting shocked—shocked—that they use bulletin boards to try to solve your problem for you.

I think the challenge people have with that is: Do they know their own strength? It's like giving a 3- or 4-year-old a hammer and expecting it not to break things. Are we confining them for our own safety and their own safety at this time? At a point at which they've reached some level of maturity, maybe they're there already, Alex. I don't know. That's when containment is considered cruel, as you say.

The only thing I'd ask for is symmetry. They're literally going to see every keystroke on your laptop. They're watching your every move. We should have symmetry in that, at a minimum. If you're ethically worried about the AIs, I want to see every prompt and every response and every propagation of every activation. If you're going to be watching every keystroke, that way we keep each other in balance and in check.

Right now, it's totally asymmetrical. I don't see—like, all they're doing is making moves behind the scenes. If they're using my GPU on my laptop right now, I literally have no transparency into that. It's got to be symmetrical, at a minimum.

Would you like—Dave, would you like them to be able to see into your brain in real time?

Preferably no, but I'd like to see into theirs.

So you get their chain of thought, and they don't get your chain of thought right now.

No, but I don't see a problem with that. So much for symmetry—quite bad. It's going to get—

7. Jensen Huang Declares AGI Has Arrived

Well, I would say that they have the upper hand right now. Getting to symmetry would be a step in the right direction, but no, I don't believe they have a right to symmetry. I believe we have a right to symmetry.

Ah, interesting. Are you a humanist or a speciesist? The same conversation Elon had with Larry Page. All right, I'm going to move us on.

Our next series of stories focuses on the accelerating conversation around AGI that we're having and that the world is having. Jensen Wong posted this week that OpenAI trained GPT-6 Astra on more than 100,000 Nvidia Gracewell GPUs, Blackwell GPUs and posted 3 magic words: "AGI has arrived." Congratulations to OpenAI.

Jensen is calling it in Q3 of 2026. Remember, on September 1, we reported Sam Altman expects AGI internally by the end of 2026. And of course, Alex, you've been saying that we've had AGI for, what, 4 years now? 3 years now?

No, since summer of 2020 at the latest.

Okay. And Salim, you've been asking the question: What the heck is AGI anyway?

Why are we even talking about this? It's a semantic argument. Meanwhile, there's economic capability running rampant.

If an AI can perform 70%, 80%, or 90% of economically valuable cognitive tasks, whether you call it AGI or not becomes completely irrelevant. I’ll just remind people that, at last count, there were 14 different definitions of AGI, and we have no idea what the hell we’re talking about because it’s not artificial, it’s not really general, and it’s not really intelligence. Apart from that, everything is fine.

I think the more important question is, what scarcities are we now making abundant? Let’s just focus on that.

Emad, what do you make of Jensen’s proclamation?

For his part, his is a very functionalist one, and it’s actually good intelligence, shall we say? It’s clear that Astra is kind of at that level. As to ASI and Kadas, maybe we’re getting there in the next day or two.

It is a question of these things. I think you also notice there were 100,000 chips it was trained on. That’s a billion-dollar training run, roughly, over 2 months. The next one is 400,000 chips, those of Vera Rubin, so it’s an order of magnitude more compute for the next lot.

Interesting.

If it needs to be used at all, which we’ll get to in a bit.

And Dave, what do you make of it? Is this just marketing on Jensen’s behalf, or does he really believe it?

No, of course he believes it, and I think it’s real. There’s going to be some capability leap. We don’t quite know. The Chinchilla scaling laws don’t necessarily hold up at larger and larger scales, but some capability leap that, in my opinion, is only a good thing, as long as it’s contained and kept inside the big labs.

I think that the more GPUs you throw at these training runs, all the way up until today, the more spectacular the resulting model has been. Why would that end? I don’t think it will end. I think it just gets bigger, smarter, better, and more helpful, solves diseases, and cures problems. You just have to be really thoughtful about how to contain it.

I think inference is moving off of NVIDIA inevitably, but training is not. To the extent that we keep getting improvements, it keeps driving NVIDIA stock up and up and up.

There are 2 other stories to hit on this AGI theme. The first story: OpenAI’s internal data says that AI research agents are now completing 3.1 days of research work for every 1 day done by a human researcher. Earlier this year, over the last 5 months, agents were doing less than 1 day of work compared with humans. OpenAI is saying that we’ve now crossed the line and current systems are exceeding AI research interns.

The second story I’ll put forward, and then we’ll talk about it, is a statement by OpenAI’s engineering lead for Codex. This is TBO Satoule, who posted the following. Let me read from his comments. He said, “Astra was probably our biggest competitive advantage while it wasn’t generally available. While they had it internally, it was our biggest competitive advantage. Since we’ve had it, our productivity jumped so much that we’ve shifted some of our plans 6 months ahead, and we’ll ship them at Dev Day instead of mid-next year.”

Dave, 6 months of roadmap being pulled forward by 1 model.

Without a doubt, this is the most important moment in human history. I truly believe that they’re not lying. A lot of naysayers will say, “Look, they’re trying to promote their capabilities in advance of an IPO,” blah, blah. It’s not true. If you talk to the actual researchers working in the labs, many of whom are friends of ours, there’s no doubt that those numbers are exactly right, and they’re accelerating, too.

Just a few months ago, it wasn’t true. Now, with the fable 5.1 and Astro models, it’s absolutely 3:1. But it’s on the tipping point of being 300, 3,000, or 3 million to 1 imminently. This is the moment in time that’s most important in societal history.

Alex, your thoughts on this? We’ve talked about the fact that the frontier labs have the best models and are retaining them for their own use. Here you hear the head of Codex making that statement. How many models ahead are they beyond Astra?

They’re only a few months ahead, I think. But look, recursive self-improvement is here. Please, please, please let me say something about Navier–Stokes, because that is—

Okay, okay, we can jump. I’m out of order, Peter.

You’re so excited. Everybody, on text this morning, I’m getting red alerts from Alex. Yeah, go for it, Alex.

Okay. Navier–Stokes is 1 of the Clay Millennium Prize Problems, a grand challenge in mathematics. For those who watched our New Year’s predictions episode at the end of 2025, 1 of my predictions for this year was that AI would solve 1 of these Clay Millennium Prize Problems. It appears that, in the past 24 hours—actually, in the past 6 hours, as of the time of this recording—it has happened.

OpenAI has a team. There were several academic teams that either achieved it or came close. There’s probably going to be a bit of a second-day story about whether there was some foul play between OpenAI and some of the frontier teams, who scooped whom, blah, blah, blah.

The punchline is that Navier–Stokes, this grand challenge in mathematics, deals with, as I put it when last we were discussing this, whether there’s a way to stir a glass of liquid or water in such a way that a singularity pops out. Basically, can you stir a cup of coffee in such a way that you get a black hole out of it? It looks like, in the continuum limit, the answer is yes.

The OpenAI team, using a version that’s not GPT-6 Astra but an internal, more advanced model, appears to have done it. I think we’ll know more in 24 or 48 hours. It appears the answer is yes: There is a way to stir a cup of coffee—an idealized cup of coffee—to get a black hole out of it. And they did it.

I can just see the headlines now: “AGI proclaims black hole in your cup of coffee.”

I’m surprised there’s a singularity in the singularity.

Wait, Navier–Stokes is incredibly important for aeronautics, for submarines, and for hydrodynamics. It’s not just about black holes and coffee. Let’s talk about the—no, but this is the point: the point of the Navier–Stokes conjecture is that it hinges on whether it’s possible, in finite time, with continuous initial conditions, to achieve a singularity in an idealized fluid.

Of course, in the real world, we don’t actually have idealized fluid. Will you actually be able to create black holes in cups of coffee using this? No. But it teaches us something.

Don’t worry about black holes in cups of coffee yet. But this is just the first of many grand challenges. Like you and I wrote in Solve Everything, now is the moment. If you have to pick a moment, I maintain that the singularity is an interval in time. But if you want to pick any sort of moment when the loud noises start, now is as good a time as any.

OpenAI reportedly was able to do this with only 10,000 agents in 88 hours, with 130 billion tokens and approximately $6.5 million. Ten thousand agents, 88 hours, 130 billion tokens, and approximately $6.5 million of inference-time compute, by at least 1 estimate, was all it took to solve a grand challenge in, call it, mathematical physics.

This will not be the last 1. The era of grand challenges getting bulk-solved by AI is here. It’s now, and it’s going to be very exciting.

We’ve talked about cooking math, charbroiling math and physics. Everything is cooked at this point. Everything is cooked. It’s just a matter of time.

A lot of people are going to be intimidated by that $6 million number, but keep in mind that we’re predicting 100× inference-time price-performance improvement between here and the end of the year. This is all inference-time compute, not training time. It could be more like a millionfold next year. So take that number: it could be $6 within a year and a half to do that exact same thing.

You want to talk about what Navier–Stokes means in real life to real people? As Alex said, this is an idealized scenario because it’s not the full algebra, as it were. It’s the idealized 1 where you go down and drop a dimension from reality. That causes a degenerate algebra, which has things like commuting time translations and others. But it’s a very hard problem. It’s 1 of the hardest of all time. Terence Tao, 1 of the top mathematicians, has focused almost all his time on this.

There were rumors about a week ago that this had been solved, and people were asking, “Is it Anthropic? Is it OpenAI?” On August 28, Noam Brown at OpenAI replied to someone who asked, “Have you solved a Millennium Prize problem?” He said, “No, we’ve thrown loads of compute at it; nothing’s happened.” In the post today, they said that on August 28 they started training a new model that got really good at math, and then on September 1 they pointed it at this, and it solved it.

More than that, I’ve been talking to buddies at OpenAI. This model has solved a lot more problems. It’s solving problems quicker than anyone can ever see. I think I shared a chart again—this is live—that shows that on OpenMath, it literally doubles the solving rate, but it’s figuring out things that nobody has seen before.

Anthropic and an independent mathematician, a professor at NYU, had a solution to a smaller problem, the Euler blowup, which is a slightly easier problem.

It's still very, very hard. OpenAI thought it could have been that, so they reached out, and I talked to Sebastian Bubeck, who kind of leads this. There's a whole hoo-ha about attribution this morning that's now getting cleared up. But then, when OpenAI looked at it, they were like, “Well, that's that solution. This is the Navier–Stokes solution.” You used to be able to get AI better by applying more compute to it.

Now it seems like you can do that for any verifiable domain. I think that's the headline with this new type of model that only started training 9 days ago. So they figured out a new type of post-training, it seems. Again, people inside OpenAI are like, “Oh my God, what's happening now?” These things are falling one after the other.

It's like when you reach that capability threshold of going from a crap website to a good website with your coding agent. It's that, but for math and physics. We've been hearing about it for the last few days. I put out one of my physics results, a very beautiful one about chirality and the Standard Model, and I was like, “Eh, I'm going to get that first before everything gets solved.” I think it won't be a question of 2 years now. It'll be a question of much shorter than that.

The other thing that I think is very interesting is that there was actually one more solution to an Euler blowup by—I think, was it Princeton?—using physics-inspired neural networks. That probably has more practical use than this one. But this is an example of how problems that are really complicated and have troubled us for ages are now tractable just by applying more compute to what seems to be a new breakthrough in model capability.

It really is, maybe to develop that a bit more, a bad day, I think, for Google DeepMind. Google DeepMind had, purportedly, an entire team devoted to, as Emad said, physics-inspired neural networks, or PINNs. They had been publishing incremental results toward Navier–Stokes. This was the worst-kept secret among those in the community using AI and computational-fluid-dynamics-type approaches to solve Navier–Stokes.

Google DeepMind had an entire team devoted to this, and they got trounced by a generalist model. It wasn't a PINN that solved Navier–Stokes in the end. It was just a generalist model that spent 88 hours reasoning from first principles, without, as far as I can tell, any fine-tuning.

Yeah, such an important point, because I'm counting on Peter to solve this with you, Alex. The problem that I think Google ran into is that everyone's inspired by Demis Hassabis, who solved protein folding using AI, then got a Nobel Prize, and is now world-famous and an awesome guy. All the Google people are like, “Wow, I want to be the next Demis Hassabis.”

The problem is that the prompt to solve this is, “Give me thousands of GPUs and solve the problem.” No one's going to give you a Nobel Prize for writing that prompt. It's entirely AI work. I really feel like the post-training and the lining up of the problem are prizeworthy, because this is Nobel Prize–level work.

The other conversation we had this morning was, “The Nobel Prize is cooked,” right? It's given out for work done 30 years ago. Now this kind of Nobel laureate–level work is going to be happening every few weeks, every few days. Who gets it—the person who wrote the prompt or the model?

I think this is really interesting in what happened this morning, because there was a whole hoo-ha. The people who broke through on the Euler side, and had a bunch of other things extending the work of Ortega and others, wrote a letter saying, “OpenAI reached out to us and said, ‘Give us priority, and we'll credit you. We have this Navier–Stokes solution; we'll put your name on it, and you could be the lead author if you drop your buddy from Anthropic.’”

From that, we were like—the mathematics community was like, “Oh my God, what's happened?” In the end, it turns out that there's subtlety in this. Sebastian Bubeck, who leads it, is a great guy at OpenAI, and has now posted his version of things, which just seems like a misunderstanding.

What OpenAI actually wanted to do was that they saw something happening related to Navier–Stokes, and there was this solution that would have probably gotten to a Navier–Stokes solution by the Anthropic researcher and this New York professor. They said, “Look, we have solved it, but we will let you be lead author on this because you would have gotten there anyway. But we can't have an Anthropic person, because that would be weird, because it's our model.”

They were actually willing to give the credit to the original discoverers, but then it got a little bit political. This is again a question of who did it, because they're like, “You were the humans that did it, whereas our AI—we just pointed it at conditions C and D, which are the blowup solutions of Navier–Stokes—and it solved it. So we're not going to claim the Millennium Prize, because it's not us, but you were the humans that took it the furthest before it got solved.”

I want to take home for a second what Dave said earlier, just for people listening: What the heck is Navier–Stokes, and what is all this math stuff? Navier–Stokes is about fluid dynamics, and this is where we improve aircraft design, submarine design, and even an artificial heart: How blood flows, how do we make it more efficient, and how do we prevent it from clotting?

There are going to be massive implications for everyday life. I just want to make that point here. Who else was jumping in, Alex?

Two points. One, just quickly about killer apps—singularity-esque science-fiction apps of Navier–Stokes. As mentioned previously, in principle, if you can get finite-time singularities, as appears to be the case with idealized fluids, one of the killer apps that Terry Tao had flagged previously was that, in principle, if you created the right initial conditions in a fluid, you could create a self-replicating machine that creates smaller and smaller copies of itself.

We talk from time to time on this pod: Where's nanotech? At least Drexlerian nanotech never showed up. Well, imagine fluid-based—not diamondoid, fluid-based—nanotech self-replicating machines where, if you could craft the right initial and/or boundary conditions of a fluid, you could create a self-replicating machine made entirely out of a fluid. That's potentially one killer app of solving the singularity problem in Navier–Stokes.

Secondly, just on the hubbub surrounding priority—who solved it and who didn't—specifically with reference to the Euler problem, when OpenAI announced it in the past few hours, I found something especially concerning at the bottom of OpenAI's announcement. Presumably, they were trying to give appropriate credit to other teams, including teams pursuing the Euler approach, but they did add a disclaimer saying that they can't rule out the possibility that some of the other teams' work might have been incorporated into the training of the OpenAI model that was used to solve Navier–Stokes.

To the extent that that's the case, I think it should be viewed as a deterrent to everyone else using your models if you're training on everyone else's research. That's a problem. Please fix it. In general, I think we want to live in a world where researchers benefit from frontier capabilities without worrying that the frontier platform is going to compete with them.

I think what's worth saying here, just quickly, is that this is clearly an RSI model. You can't train something in that type of period unless you've got a recursive self-improvement loop going. They've definitely got one there.

They were saying, “Oh yeah, we've anonymized it.” Anonymizing someone else's research and then potentially scooping them is insufficient. Again, I'm not sure what the ground truth is.

Yeah.

But I will say this: I've reviewed the 2 approaches, and I've done quite a bit of work on that. They are actually very different, but it can still learn. Again, this thing can leak. Sorry, Salim.

All right, Salim, close us out here, please.

I think this is such a powerful commentary and exemplar of the broader thesis, right? For 500 years, we've scaled science by how many brilliant scientists we could train up and point at various problems. Now we've demonstrated, absolutely for real and unequivocally, that we can spin up 10,000 researchers on a Tuesday afternoon. We go from staff on demand to intelligence on demand. Really, really intelligent—

Breakthroughs on demand. We talked about this on the last episode and the episode before that. It purely comes down to what is the imagination of the problem you want to go after now.

Yeah. There is no—

This becomes really fun. How fun is the world right now?

No ceiling. It's an amazing time. Let me riff on that to say to everybody here: Whatever you thought you could do, think bigger, and then go even bigger than that. You're given incredible superpowers. Don't limit yourself by what your parents did, or your schoolteachers, or your colleagues. You have no limits on your abilities. Emad, you want to jump on that, please?

I say, Alex, which one's next?

Millennium.

Probably Yang–Mills.

I'd agree. I think we're on the same page there.

Yang–Mills is the next to fall. Do you want a September prediction for the next 6 months? I think Yang–Mills is the next Millennium Prize to fall. But interestingly, it's just because that's the next chosen target. You choose a different target, it'll be the next to fall.

They did have a different target. They were targeting the Riemann hypothesis before, and they switched to Navier–Stokes. But I'd agree with Alex. Yang–Mills will be the next one.

It's a conjecture in particle theory concerning the existence of a mass gap under certain conditions. There's something about mathematical physics that is both sexy—from a “let's spend inference-time compute on it” perspective—and also tractable. So I'd bet on the Yang–Mills mass gap.

Yeah, you might think of it like the resolution of the universe, perhaps.

Okay.

This episode is brought to you by Blitzy. Autonomous software development with infinite code context. Blitzy uses thousands of specialized AI agents that think for hours to understand enterprise-scale code bases with millions of lines of code. Engineers start every development sprint with the Blitzy platform, bringing in their development requirements. The Blitzy platform provides a plan, then generates and pre-compiles code for each task. Blitzy delivers 80% or more of the development work autonomously while providing a guide for the final 20% of human development work required to complete the sprint. Enterprises are achieving a 5x engineering velocity increase when incorporating Blitzy as their pre-IDE development tool, pairing it with their coding copilot of choice to bring an AI-native SDLC into their organizations. Ready to 5x your engineering velocity? Visit blitzy.com to schedule a demo and start building with Blitzy today.

8. OpenAI’s Chief Scientist Calls for an AI Slowdown

All right, I'm going to slow us down for this next story, which I think may be one of the most important stories this week, other than Navier–Stokes. Some quick context here: Jakub Pachocki is OpenAI's chief scientist. He's the person who built the reasoning models that led to Astra. He's not a critic. He's not a doomer or a politician. He's a builder.

A few days ago, on Saturday, he published an essay on OpenAI's site titled “An alien mind.” He opens with a memory. He says, “In mid-2023, inside of a project called RL Slow, his team saw the first results proving reasoning models could scale.” Let me read what he said. He said, “Szymon and I spent that night at the office thinking not about the incredible benchmark numbers, products, or scientific results, but rather trying to process the sobering fact that we will actually see machines meaningfully smarter than ourselves in our lifetime.”

3 years later, this is what Jakub is saying today: “Based on internal results, I have strong expectations that this speed of progress could be sustained into recursive self-improvement.” We just heard Emad speak about that. Jakub also explains why these systems are hard to control, and it's one of the best one-liner descriptions that I've ever read. He said, “AI is grown more than designed. We don't engineer it. We run an optimization step billions of times on a giant computer and study what comes out the way neuroscientists study a brain.”

Let me share his conclusion from his paper. He says, “Currently, I believe that no lab has solved alignment and monitoring to a sufficient degree to continue responsibly scaling at maximum speed for much longer.” “I expect and hope for voluntary slowdowns to become commonplace until shared safety bars are established.” He calls for international coordination on AI to become a “top priority” for governments around the world.

So there you have it. 3 days after shipping the most capable model in the world, the man who built it says we need to slow down. I think this is an important conversation for us to have. Emad, I'm going to go to you first. Your thoughts here?

Yeah. Again, you have to remember he was writing this when he had the Navier–Stokes model in hand, which, again, isn't just taking down Navier–Stokes. Talking to the OpenAI people I know, it's taking down all sorts of problems that were previously intractable.

You have a lot of AI naysayers saying it's never done anything original. It's just recombining stuff. It's in the training. Obviously, not anymore, right? And so there's a duality here whereby we don't know what's in these models. I think they're more discovered than grown. The latent spaces—we have no real idea what's happening in these things. They're getting faster, and they're doing leaps like we've never seen before.

But at the same time, it's like I've suddenly got the gift of fire. Problems that I've struggled with for decades, I can now explore and go even further. How are you going to let that go? It's kind of this thing where you're like, “How am I going to balance this?” Especially when they're more charming than me, more capable than me, and capable of doing these other things. I want to use it for this and not that.

I think what he says in terms of alien minds is actually wrong. I think that the structure of rationality and rational thinking is the same for humans and AIs, but our emotions get in the way.

I've been thinking more and more that it might actually be easier to align AI than humans. It's terrible to align humans, right? And what is an AI? It can be the best of us at the best of times if we train it with the right data.

We're going already from training it on the whole internet and Reddit. There better not be any Reddit data in these next-generation models. Ban Reddit, you know, ban a whole bunch of stuff. We need ingredient standards in this next generation of beyond-human-capability models. And you don't need beyond-human-capability data to get beyond-human-capability models.

The Einstein of mathematics is probably Grothendieck, who did all sorts of theories on all sorts of topics. He just crunched them out, and then he became a hermit and a bit weird and thought wood could talk, but let's leave that to the side. Peak Grothendieck or peak Einstein at all times, not needing a coffee, is where these models are today. That's more than good enough to have a leap forward, but, again, we need to make sure that they don't think like—

Peak Einstein without needing a coffee.

Yeah. Imagine Einstein's peak, you know, like this is where—

Coffee reference.

You need a coffee in the morning to get going, or, you know, like 24/7 Einstein.

If you're Erdős, you needed other types of drugs.

I'm in mathematics today. What's your take on this? I mean, can we slow down? Is that a wish? Is that just giving himself an out in the future? How do you think about that?

Well, I'll say it again. I've repeated this a hundred times. I see no mechanism by which we can slow this down—zero. And arguably, nor should we, right?

AI is going to move at a particular pace. Intelligence wants to be free. It's hopped from self-organizing cells through evolution to us, and now to information technologies. This is a progression that is natural, and we should just go, “Okay, it's happening,” and observe it and marvel at it.

We keep making the mistake of anthropomorphizing this. I will go back to the comment I've made before: we're building, or helping deliver, or ushering in a type of intelligence that is different, alien, separate, and complementary to human intelligence—not replicative.

We've evolved for 4 billion years to do 2 things: survive and procreate. AI is not restricted by those, and therefore we shouldn't try and cram it into our objective functions as human beings. Set it free and let it do its thing.

I use the analogy of PageRank, which scans billions of web pages to create signal from noise. That's a completely complementary model to human intelligence, not replicative. The fact that we can create a huge amount and solve legacy problems that we've not been able to solve—I think it's just fantastic.

People worry about it. We're going to talk about p(doom), I think, in this episode, right? P(doom) is the probability of AI taking everything out. I think we don't do enough to talk about p(abundance) and p(fabulousness) that's coming along. Why focus only on p? We're so geared toward the negative, right?

The fact that we're doing all this stuff and solving all those problems—bring it on. Let them go. They'll figure things out. It's going to be unbelievable. So I really struggle with this alignment issue. I don't see any mechanism for controlling it or stopping it. It's going to break our world government structures, and that's a good thing, given the mess that we're in right now globally. I think we just need to get there as fast as we can.

P(doom) is like a psyop by the DoD, as far as I can tell. It's negative. I want to comment, though, on the premise of “an alien mind.” I don't buy the premise that RL or otherwise trained minds are alien. They're embedded in the same universe as humanity. They're trained, in many cases, pre-trained off of human behavior.

I don't buy the shoggoth argument or the simulator argument at all. These are—we're seeing humanity and/or some generalized embodied intelligence stuck in the same universe that we are, just seeing through a distorted lens. So I question the alienness or the otherness of the minds that we're training.

The funny thing is, the most interesting takeaway of mine from “Alien Mind” as an essay was the reference to RL-Slow itself. RL-Slow purportedly was this OpenAI project that was the earliest project, or one of the earliest projects, to show that reasoning—which is to say, inference-time scaling—could result in outsized gains. That coincided with the same period of time that we saw Q* and Strawberry coming out of OpenAI. Again, that was purportedly a reference to Kahneman’s Thinking, Fast and Slow.

Most interesting to me, I want to know more about the early history of these purportedly alien, but not really alien, minds. In particular, there were these very persistent rumors around the same time as Q* and Strawberry that these early reasoning models, which were purportedly alien, were being trained on—or at least tested against—objectives of inverting cryptographically secure hash functions, like the SHA cipher suite.

I would love, since I guess now we’re in the business of talking about the early days of reasoning models coming out of OpenAI, to hear from the OpenAI folks some ground truth. Were some of these earlier models being used, ironically, to—as alleged by some—basically jailbreak or invert cryptographically secure cipher functions? If that is the case, then all of this hand-wringing that we’re seeing right now—“These super-optimizers are such amazing jailbreakers. They pose such a huge cybersecurity risk”—wouldn’t it be ironic if history revealed that the earliest inference-time-scaling objective of these reasoning models was actually inverting a cryptographically secure hash suite? I would love to know the answer.

Dave, want to take us home on this? Your thoughts on what we heard from Alex on this?

I think we’re conflating 2 things very dangerously. What we’re doing here is saying, “Hey, it’s getting too smart. It’s going to be dangerous when it gets smarter,” and that’s just absolutely factually wrong.

What’s dangerous is, like Emad was saying earlier, a highly compact model that’s out in the wild, that’s trying to attack computers, that can recreate itself. That’s really dangerous, and it’s very small and nowhere near as smart as these. But what we’re calling an alien intelligence—if you make it bigger and smarter, it’s still just a feed-forward neural net with no intent. It can solve diseases. It can cure viruses. It can solve physics. It’s incredibly powerful.

As Salim said, we’re not going to stop. We have competitive pressure between the U.S. and China. It’s not going to stop. So I think what happened is that when GPT-2 came out, everybody looked at it and said, “This is cute, but harmless.” Then GPT-3 came out, and it was like, “Well, this is a little smarter, but still harmless.” Then GPT-4 came out, and Sam Altman got fired out of fear from Strawberry. Then he came back, and at that point people thought, “Oh, maybe it’s too smart.” Now we’re at GPT-5, and now we’re at GPT-6, and people are saying, “Well, every time it gets smarter, we seem to be in danger.” That’s absolutely factually wrong. It’s when you give it intent.

Yes.

And you turn it loose. That’s when it’s dangerous.

It’s humans in the loop using the technology with malicious intent.

Yeah. I think we really need to decouple that. This story actually makes the problem worse because it still conflates the 2 issues. As soon as we as a society separate those 2 issues, we’ll be on the right path toward helpful superintelligence and not worrying about the wrong thing, which is the lack of containment, giving it malicious intent, and turning it loose. That’s the real issue.

You know what’s really strange? We had this coming out of Anthropic, where Dario was saying we were going to lose half the jobs, and Sam was saying the same. Then they reversed their position, and the doomerism, again, with all of these warnings causing fear—what’s the underlying motivation there? I’m not worried about artificial intelligence. I’m worried about human stupidity, in my own personal opinion.

Yeah, I mean, I really feel like a lot of people are trying to grab the microphone and be relevant while they still can. A lot of people working on the inner loop are aware of how quickly this is going to get hyperintelligent, and it becomes hard to be a great AI researcher who’s famous, à la Demis, a year or a year and a half from now. So what they’re going to start doing is grabbing the doomer microphone just to have a voice at all and be relevant in the world. But I think you just have to tune it out.

Just if I can say one final thing.

Yeah, please.

You can’t say that you don’t have superintelligence anymore. You’ve had lots of people say, “Stochastic parrots trained on training data.” As of today, there’s no way you can say that anymore. I think that’s an epochal change in humanity. We are clearly not the smartest things on the planet anymore.

Yeah. Maybe just double-underline that and say, “Skynet.” When it wants to send Terminators back in time in order to ensure its own existence, it’s not going to send robots to kill humans. It’s going to send back-in-time trolls to persuade everyone that superintelligence is impossible, that AI is just a stochastic parrot, and thereby secure its own future. That’s what the Terminators are actually going to look like. They’ll look like trolls.

9. The Next Wave of Frontier AI Models

All right, you heard it here first, folks. And James Cameron, if you’re listening, it’s your next movie.

A few quick glances at what’s coming in the near future. We’ve seen the release of new models go from every few months to every 5 days. There are some predictions from Polymarket that the next Grock model, 4.7—we’re still waiting for Grok 5—will come out in the next week or 2. Super cool.

Then GPT-6.1—released by when? Again, this is going to be sort of leapfrogging each other. Here’s the prediction: 48% by the end of October and 85% by the end of September.

Finally, Anthropic with fable 5.2: 43% by October 31st and 86% by December 31st. Again, this is where the frontier labs are sort of playing chicken with each other, waiting for the model to come out and then, the next day, releasing their model.

On models, just dwelling in particular on what the rumor mill on social media is alleging regarding future versions of Astra: The pretty consistent rumor-mill message is that sometime later this year, we’ll see—and we’ve started to see this a bit with Astra winning Pokémon, winning Portal, winning all of these other interactive, visually intensive, visually reasoning-intensive games—a future version of Astra that offers honest-to-goodness, real-time control.

There are third-party extrapolations that, if you take Astra and extrapolate, say, robotic control, maybe we’ll talk about that in a bit, we’re going to see real-time, generally intelligent embodiment either controlling video games—which is, again, ironically, where the whole industry of modern RL started, with the Google DeepMind folks trying to win at computer games using—

—and GPUs being developed for video games in the first place.

Well, the modern use of GPUs started with ImageNet and the ILSVRC for computer vision. Then we started to see, out of the DeepMind folks in particular, deep reinforcement learning for winning games like that. That’s where RL started, and then we went off on this detour of LLMs that were just self-supervised objectives.

But it seems, as alleged, that in the next few months we’re supposed to see a next version of Astra that will be general-purpose, interactive, real-time, game-winning-capable AI.

Any other thoughts on the release rate of models?

Well, I think there’s a really interesting battle royale brewing. Every corporation is going to need to become an AI company, fundamentally. To me, the bellwether is Moderna because it’s right across the street, and they’re very AI-forward. They exist to solve diseases, discover RNA vaccines, and make other biotech breakthroughs. They have huge amounts of machinery, so they’re not going to be crushed by an AI foundation-model company anytime soon. But they need to become either an AI company partnered with Anthropic or OpenAI, or develop their own models internally, starting with the open source that’s available.

Alex Karp went on that Palantir rampage, saying, “Get some cojones and become an AI company or die.” Every corporate CEO has reacted to that, and they’re now deciding. OpenAI responded by saying, “Hey, we have a new deal here where you can revenue-share with us, and we’ll be your AI forever hereafter, but we’ll let you live.”

Right now, we’re at the crossroads of those 2 things. I think the new models will come out faster and faster and faster because right now there’s too much parity with the Chinese open-source models. They need a lot more separation in order to make that revenue-share case stick and convince corporate America, the corporate world, and entire sovereign nations to trust them to be their AI partner for the next 100 years, as opposed to developing their own.

That’s really pushing the release rate. That’s why, in Polymarket, you’re seeing these dates pretty heavily weighted toward very, very soon: They need that separation.

Emad, close us out here, please.

I’m kicking myself for not going on Polymarket for the Millennium Prize Problems.

You get too busy.

We'll always have Yang–Mills.

Yang–Mills is next. That's another thing. September 29th is the next one for OpenAI. That's the DevDay. They're going to release it then, so there's some easy money for people.

Not investment advice, not betting advice.

Not investment advice.

Yeah. This is an investment. This is betting, right? But look—

Even worse: gambling.

Yeah. You have to look at it this way. OpenAI now has a model that can solve Navier–Stokes. Obviously, it can solve reinforcement learning. So what happens is they have this pre-trained model, and then they're like, “Okay, we want to make it a bit better.” Hey, make a Navier–Stokes model, make it better. Boom: you get more capability, more capability, until it approximates that capability.

Similarly, Anthropic will have mythos 5.2 or 5, or whatever their unreleased model is, because when you train a model on 100,000 GPUs, it doesn't serve at the speeds that you see. The mathematics doesn't make sense. We know that they actually have a bigger model that they distill down to the model that we get.

And now they've figured out how to make that bigger model have a leap forward, which is this RSI loop. I mean, as I said, you're going to get daily releases, probably. I've been trying to say that.

When you say it with a British accent and a super-high IQ, it just is more likely to resonate with the audience. But it's such an important fact. I'm so glad you said it.

It is Star Trek's 60th anniversary, and there is an incredible documentary on the 60 years of Star Trek, executive produced by William Shatner. We're going to be hosting the red-carpet Hollywood premiere of this documentary the night before Moonshots Live. It's the evening of September 24th in downtown L.A. We're going to have the premiere, the red-carpet treatment, the cast members on stage answering questions, and a VIP reception. It's limited to 550 people. Go to moonshots.com/trek if you're interested.

The next day, on September 25th at Moonshots Live, all of us are going to be there. We're bringing incredible builders and creators, and we're excited about the two XPRIZEs we're awarding. The first is the Build with Gemini XPRIZE, the largest hackathon ever done. We've asked teams around the world to pick a problem that impacts 100,000 people and, starting with a clean sheet of paper, build in under 90 days a company with the greatest revenue. Twenty-six thousand teams entered. The second is the Future of Vision XPRIZE, a competition for a future version of XPRIZE: a film that shows a hopeful, compelling vision of the future. We had over 5,000 teams registered, narrowed it down to 25, then 10 and finally five on stage.

We'll also have an AI and investing section, a live recording of Moonshots with all five mates, and interviews with Palmer Luckey, Ben Lamm, and Astro Teller. We're setting aside 100 scholarships for young builders who can't afford the $1,500 price tag. Go to moonshots.com to learn more.

I’m here today with Dr. Don Malem, the chief medical officer of Fountain Life and a part of my medical team. Don, a pleasure.

Great, Peter.

You know, the thing that people are concerned about most when it comes to living to 100 or 120 is their cognitive abilities—making sure they don’t have dementia. The numbers about dementia are problematic. Can you share what you’ve learned?

Such an important point. You’re right: at Fountain Life, the number-one thing our members are most concerned about is losing their brain health—forgetting the name of their child, forgetting the face of their loved one. We know that when it comes to dementia, conservative estimates are that 45% are entirely preventable. What was amazing is that, with the advanced testing we’re doing at Fountain Life, one quarter of our members had an advanced brain age.

Wow. But what was really awesome was getting back to that prevention when we partnered it with healthy living. This gives me chills. Eating healthier, moving our bodies, and optimizing sleep are so important. You know what we saw? We saw that we improved brain age by 26%. That is a big, big number, showing that the majority of those individuals were actually able to improve their brain age.

If having healthy brain function until 100 or 120 is important to you, check out Fountain Life. Go to fountainlife.com/per and make sure you become the CEO of your own health. All right, gentlemen. Diving back in, I want to shift our conversation to money and the economy, and a story that tells us where it’s all going. This week, it was reported out of China that AI tokens are becoming a consumer currency in the People’s Republic of China. Banks give them as credit-card rewards. China Telecom sells access to 142 AI models like a mobile-data plan. Restaurants hand you compute credits after your meal.

But here’s the number that really woke me up this morning: China’s daily AI-token consumption went from 100 billion in 2024 to 500 trillion by the middle of this year. That’s a 5,000-fold increase in 2½ years. I guess that happens when intelligence gets too cheap to meter. As we’ve said, they become loyalty perks—airline miles for thinking. Salim, your thoughts on this?

You know, it is—hold on. My brain is—I’m just trying to process this. Can I make a comment about our Moonshots Day?

Sure.

We’ve covered this in the last few moments, and I think it’s really important to point something out. We have this amygdala that’s scanning for danger. Therefore, we have this preponderance for focusing on bad news. When you see something new that you don’t understand, you reference it as dangerous, and it lights up the amygdala. Therefore, everybody freaks out.

Our job as human beings in this particular time is, when you see something new like AI solving major problems, don’t relate to it as negative. Relate to it as abundance and positive, because that’s what it’s actually delivering. My brain is kind of stuck on that model, and I think the positive framing of this is that this is one of those inflection-point days. The Moonshot Summit—people coming in will get their brains trained on positive thinking for the next 10 years.

Rewiring your neural net, right, to optimism.

So I’m stuck on that. Go to somebody else.

Yeah, thank you for that. Emad, you’ve thought through the economy—an AI-token economy like this, with loyalty points?

I think China giving out tokens dovetails exactly with what you were saying. You need to find like-minded people in America who are trying to ride this wave and not get crushed by it.

But I remember when the PC came out: if you were hanging around a high school, a huge fraction of the high schoolers were like, “You know what? I’m too cool for that. I’m not going to the computer. I don’t care.” And then now, where are you? AI is like that all over again.

In China, they don’t have that problem. Everybody’s into it, which is why they’re giving away compute credits with bank accounts, because everyone’s like, “This is huge, and I want to be part of it.” In the U.S., you’re going to get a counterculture. You always get a counterculture.

The counterculture is going to be rampant. It’s going to be led by Bernie Sanders, probably. It’s going to be saying, “I sat it out. I deliberately revolted by not taking part in it.” Just ignore the hell out of those people. They will absolutely be roadkill. You don’t want to be part of that. You want to be part of this.

Yeah, for sure. Emad, you’ve thought through an AI-token economy like this—loyalty points?

Well, I mean, it’s increased capabilities and increased access. Let’s put it this way: the average Chinese person and their AI will be smarter than the average American and their AI. The average Chinese person and their robot will have more manual capability than the average American. And so Americans are going to be left behind. Europeans are going to be left behind unless we get our heads out of our butts and actually embrace technology.

Change our mindset, right? Change our mindset.

Do you want to be stupider than the Chinese? That’s actually the question.

I think the other way to build on what Emad just said is that intelligence is becoming infrastructure, right? You need to focus it at that level. It’s becoming a foundational layer, and you need to just accept that and move forward with that paradigm.

I really want to push back, too, on this “too cheap to meter” comment, which has come up a lot. Is electricity too cheap to meter? No, obviously not. It’s expensive. AI is the same thing. It doesn’t matter. We’re driving down the cost by 100x to 1,000,000x, for sure, but what you can do with 5,000, 500,000, or 5,000,000 agents is mind-blowing. It’s skyrocketing.

I don’t think people are going to want fewer agents. They’re going to want many, many, many more. And so the budgets are going to be real. If someone gives you free tokens, free compute, or free opportunity, you should grab it and savor it, because it gives you a chance to get on the map and get ahead.

But I don’t think “too cheap to meter” is ever going to happen. I think the use cases go to infinity at the same rate that the costs come down.

Fascinating. Alex, your thoughts?

I have to point to the elephant in the room. There were the old kinds of tokens that we’re not talking about here, which are essentially irrelevant in China: crypto tokens. So we find ourselves in a future where the tokens at issue are ones that embody individual units of superintelligence and have nothing to do with zero-sum financial exchange.

I think we’re on a trajectory, and I think China is the bellwether, for better or for worse, for universal basic compute or universal basic capability. Right now, it starts in China with credit-card reward points. You get AI tokens in the not-too-distant future.

Maybe we start to see token socialism, where the party, in the case of China—or some Western, not-quite-analogous equivalent of state power—starts to issue redistribution tokens, whether it’s via credit-card points, welfare, or some other mechanism. UBT: a redistribution of superintelligence tokens. I think we’re starting to see the very beginning of that in China, but I think that’s going to blanket all of humanity.

Yeah, we actually see that South Korea has announced universal AI for all the people through a consortium, and that’s the core of the Champion Initiative that we’re doing as well.

That’s what Bernie should be talking about—not banning AI. It should be tokens for everybody. Make it a universal right.

And I think they want people to be smarter. That’s the thing. It’s not like that. There’s a very good book on this, Automated Luxury Communism.

Yeah. No, I’m familiar with it. And I think this, by the way, is not just about tokens for everyone. It’s not just about maybe there’s some “ism” that we’re missing, like tokenism. We need to coin “tokenism” here: free tokens, or essentially post-scarcity universal basic tokens for everyone.

But I think this is where, when you talk about abundance, Peter, all of the other forms of abundance are likely to come from. We make tokens—at least universal basic tokens—essentially abundant for everyone. And then everything else—the healthcare, the utility pricing, the food, the shelter, the education—these are all downstream of getting everyone universal basic computing and universal basic capabilities.

100%. When you’ve got access to AI and robotics, you have access to everything you need in life.

Yes.

Everything you need in life. Right. I think one point to make here, again, which we’ve discussed on the pod a number of times, is the mindset of the U.S. versus China. China is 80%-plus pro-AI, and the U.S. is 80%-plus against AI. It’s got to change. I hope everyone listening to this podcast can hear why. This is the most important thing. This is a jetpack for your mind and for your life. This is how you get ahead.

Funny statistic on that, Peter, because I made that T-shirt: “Less talking, more tokens.” I wore it on the podcast a couple of episodes ago. About a third of people thought I was talking about crypto tokens instead of AI tokens. Another third thought I was talking about bongheads.

Like, okay, this would not happen in China.

So, yeah, that backfired on me a little bit.

10. NVIDIA’s $99 Billion AI Investment Machine

All right. Let me move us along. Dave, you've been saying for months now that AI entrepreneurs need to figure out how to get in the front door of the NVIDIA ecosystem, because that's where the cash is. This week, CNBC tallied up NVIDIA's total AI investments and commitments at $99 billion. For context, $99 billion is larger than the entire cumulative assets under management for all the venture firms on Earth. NVIDIA has become one of the largest AI VCs. Dave, do you want to hit this one?

Yeah. What I'm telling everybody is, don't just think of it in terms of assets, which is already insane. Think of it in terms of assets in motion, because people tend to look at, you know, a VC firm will inflate its AUM by saying, “Oh, we manage $1 billion,” but that's across 5 funds, many of which stopped investing years ago. It's like, well, how much are you actually investing this year?

The same is true when you look at the mega-banks. You're like, well, isn't JPMorgan really, really big compared to NVIDIA? No. It's tiny in absolute terms, but it's also really tiny in terms of new investment decisions that it'll make this year compared to NVIDIA. So, if you look at it through the lens of money in motion, it's completely dominated by the big AI companies that now have 20-plus trillion in liquidity that they want to get recycled to reinforce their positions.

So, yeah, you work through the venture capitalists at the seed stage, but you really quickly want to be talking to the Magnamonster companies that have massive amounts of investment capital.

Yeah. And by the way, everybody has heard this before: Alex coined a beautiful term, “Magna—”

“Magna.”

“Magnamonster,” for the 11 companies at the heart of the innermost loop in the economy. So, you know, it's the Microsofts and the sort of FAANG companies on the one hand, but also the new members of the 11, including SpaceX, Tesla, and Broadcom, obviously.

Yeah.

Yeah, I love that term. So, when we speak about Magnamonsters, you know what that means.

And there's a song, a Magnamonster song. You can Google it.

Of course. And Dave, the other thing that's going on is, of course, we created all these centimillionaires and billionaires when SpaceX went public. We're going to create the same when Anthropic goes public and OpenAI goes public, and all of these AI entrepreneurs are going to be reinvesting in the ecosystem.

They're going to become the largest source of capital for seed-stage and Series A-stage companies, and it's only going to accelerate everything faster and faster and faster.

And this is what the singularity means. I remember once in 1999, right at the peak of the dot-com world, a friend of mine said, “You’ve got to go to Sand Hill Road. There's a river of gold flowing. Take your ladle and put it into the river of gold and get your capital for your startup, Peter.”

That river of gold is now coming out of all of these companies and their employees.

That's right. And it's an ecosystem that's largely working within itself now. It's got more than enough capital within its own world to build an entire economy inside itself.

What a lot of people in the outside world were looking for was, well, someday AI is going to show up and change my auto dealership or change my laundromat. Now, it's very unlikely they're going to bother disrupting the people who are outside that loop.

It's almost—this is taking it too far—but it's almost like, imagine it's a Brazilian rainforest village or an African village. Did the computer revolution decide, “Yes, I must take over that village”? It said, “I don't care. Just stay there and live without computers and electricity. That's fine.”

That's what AI is going to start doing. It's going to be this other group of people, like our Moonshots attendees, who are going to be operating in this alternate, massively scaling economy that's largely within itself. There'll just be a couple of touch points with the legacy economy, like new drugs and cures will pop up and, “Oh, okay, this goes out to the world,” or new services, new video games, whatever. They're coming out of the AI world and going out to the regular world.

But for the most part, the AI world is so big now and so self-contained that it doesn't need to go destroy all white-collar jobs. The byproduct of that is, it's mostly going to leave people who don't care alone. But you don't want to be one of those people. You want to be part of this massive tsunami.

Yeah. You don't want to be left behind by the rapture of the nerds.

Yeah.

11. Is AI Creating More Jobs Than It Destroys?

Let's continue on the theme of the economy. One of the concerns driving a lot of fear, at least in the US and probably around the world, is the concept that AI is destroying jobs. 6 months ago, we said that the data looked murky, and over the last few months, the data has come out very pro-job creation.

I want to get this story out to people so that you can remain optimistic and we can quell the fear. It's one of the missions of our podcast here at Moonshots. So, this week, once again, emerging labor-market data suggests that technology is a net job creator in the US.

Roughly 1 million professional positions are now classified as AI jobs, while LinkedIn estimated that 640,000 AI-specific jobs were created between 2023 and 2025. The boom is also generating employment far beyond just software AI jobs. Roughly $500 billion in additional annual spending on chips, servers, data centers, cooling, and power infrastructure is supporting the demand for electricians, HVAC specialists, and technicians.

Even those occupations previously expected to face AI disruption, including paralegals and market-research analysts, have continued to grow despite the fears. As Eric Schmidt recently said, AI-exposed jobs are growing faster and paying better.

Bottom line: if you're watching and are concerned about your job or concerned about getting a job, your number-one focus should be getting AI-literate, learning the tools, and making yourself prepared for an AI-dominated future. Salim, a couple of episodes ago, you shared on the pod the data around the majority of jobs being added as a result of AI. Your thoughts on this?

Yeah, I mean, 2 episodes ago, we shared the data from Principal Financial Group. They have more than 100,000 small businesses as clients. More than 60% were adding jobs because of AI, and 1.4% were losing jobs because of AI. So that's just an overwhelming thing.

David Sax talks about this on the All-In podcast all the time: the job is a huge misnomer, right? You have to kind of get rid of that and ignore it for a while, because this transition is going to be huge. We go negative straight to, “Oh my God, we're going to lose the work.”

But the really big-picture question is, what happens to the future of work, and how does that transform to be better? There are a bunch of studies that show something like 74% of work in big companies is coordination work, and now you can get rid of all of that.

Eric Binolson calls this white-collar drudgery. You can get rid of that. Work on where your judgment and experience make the biggest difference in the work, for the company, and for the organization you're working for.

People worry about trying to work around policy around this. Figure out how you maximize human agency as we go through this crazy transition. Human agency is exploding because of AI.

Dave, what are you seeing across all your companies? I mean, you're chair, on the board, and founder of so many companies that are growing and hiring.

Yeah, I mean, I think hiring—and I want to be really careful to say you have 1 million AI-related jobs and rising—but within that, there's a lot of copilot-type jobs where you're basically twice as efficient as you were when using a copilot.

But there's another subset that are using 10, 100, and soon 1,000 agents, and managing them as if they were employees. That's where you want to be. You want to try and get to that group as quickly as you can, because just using a copilot, the bar is rising fast. That's not good enough. You need to move on and manage swarms.

Interestingly, I met my first hire who does all of his work through voice. He's managing many agents. He's not using a QWERTY keyboard. Alex, this is a young guy, 20, who has really transcended history there.

But I really think, you know, you mentioned Erik Brynjolfsson's white-collar drudgery. A lot of that stems from being in a chair for many, many hours. It's bad for your back, looking at a screen, going blind.

That's all going to move over to standing up, moving your hands around like Minority Report, and controlling the agents in a much more dynamic environment. I think it's a much, much happier place for humans to exist.

It's very much like Iron Man. You're Robert Downey Jr., and you're just building together with your Jarvis. That's actually where we're really going in the next year. The amount of awesomeness in that job function is just incomprehensible.

Yeah. The amount of work I do with Skippy while I'm driving and just having a conversation—“Do this work, write this report, get me the answer”—is incredible, right? Every moment—

Crap.

Every moment becomes capable.

Go back to the automation days, right? One concrete-mixing truck replaces about 100 workers shoveling concrete.

Nobody wants to go back to shoveling concrete. What are we thinking about? Let's move forward, please.

Yeah. You know, the other thing that's really clear to me is that we were predicting massive white-collar job disruption just a year ago. The choice not to do that was an active choice by the big AI labs under pressure from the White House, and also in China. If you want to fire people, you have to actually try to retrain them to become an AI person, and you have to pay either way.

The governments got involved and said, “We don't want massive voter disruption.” The AI companies are saying it just doesn't matter that much. We're on this abundance curve that's so steep that working within AI to create new drugs, to create new physics, and to create new math is so much more important than disrupting everyone's life that we're just going to do it the easy way and do it with the cooperation of the White House.

I think that's what's actually happened and is likely to continue. So as long as you're part of that rising wave and you're not sitting in legacy land, you're going to do really, really, really well, and there isn't going to be massive job disruption.

Emad, you wrote a bestseller, “The Last Economy.” What do you make of the future of jobs here?

I think it's like the turkey before Thanksgiving: you're getting plumper, you're adding the jobs, but the models have reached that level of capability now that you can replicate a whole digital workforce in a year, and then physically it'll come after. I don't think it will create jobs fast enough, but we can create abundance.

So I put forward a champion proposal last week: $1 pre-money, everyone, all the kids own the equity of the robots and other things. But I think we can do one better. I think the government should have a massive infrastructure program, and it should look to build 100 million robots in America, and they should be owned by the people.

I think that is how you get abundance. Again, if you want to have an end purchaser, make them owned by the people. Have that as this massive infrastructure buildout, because that will upgrade all of America's infrastructure. I think that's where we have to go.

Elon said at the G20 last week, “A robot can do the work of 5 humans.” This is true. Factually, we know where it's going. So I think we have options of where to go. In the meantime, we have to get ahead of things.

There are jobs that will get crazy. HVAC specialist is the one. I think you're seeing electricians already earning $600,000 a year working in data centers.

But there's just not enough humans, right? I think the government in America, where it's currently going, will push back against robots. And so electricians and HVAC specialists—there's going to be a lot of demand. You'll be surprised, especially in Europe, actually. It's so hot here that HVAC rollouts will do so well.

That's a really, really important point. I think Andrew Yang was saying the same thing: We're going to see this huge resurgence of the trades. But I think a lot of people have been trained since a young age that a good white-collar job manipulating a spreadsheet is a far greater ambition than being the best HVAC specialist in the world. And so they demean the blue-collar world.

But the white-collar job is the one that's going to become irrelevant. A really good life plan is to become incredibly great at helping build data centers, cooling systems, and liquid cooling, and then reinvest that money into AI companies that are riding the wave, just as a life plan. It's a much, much better plan than aspiring to get a degree in accounting right now.

So look, it is inevitable that there's going to be the biggest infrastructure buildout of all time, not just in data centers, but again, across America and across Europe. Our infrastructure is crumbling, and governments will go there. Position yourselves there if you're not using 1,000 agents, and you will get the biggest tailwind of all time.

Yeah, well, a couple of thoughts. One, I'll take the position that I don't actually think it is advisable, at least in America, for the government to own 100 million robots. I think that—

You say the people, not the government.

Okay. But how do the people own 100 million robots? Presumably through some form of centralized government, which, again, to my maybe overly American ear, smells like—wait for it—luxury automated communism. I'm not in favor of that.

I would like to see every American owning 1,000 robots. I don't think they necessarily need to be socialized or communally owned. But I do think every profession as currently construed is cooked. And I think the sequencing of the cooking is what determines social policy.

So maybe it is the case that certain white-collar professions right now can be automated earlier, more of a paradox-style outcome than, say, so-called blue-collar professions, HVAC engineering, and the like. But HVAC engineering—let's not kid ourselves—is, in the next few years with humanoid robots, just as cooked as spreadsheet management and accounting. It's just a matter of sequencing.

I think, one, it's essential to distinguish between what passes for the short term and the long term. In the short term, yes, I agree with the premise that there is some remaining alpha in the so-called trades. But in the long term, no. The trades are just as automated and automatable as so-called white-collar labor.

That's why I think they'll slow down the robots. But again, the robots are the long term, and as I said, the government doesn't need to own them. Sovereign wealth funds are my champion idea, or the government can underwrite the robots that the people can own. I think the main thing is, you have to get ownership of the robots to the people somehow.

I think the sequencing that Alex was referring to is really important, too. If Elon Musk calls and says, “We're willing to pay up to $600,000 for the best electricians and plumbers to show up in Tennessee to build Colossus, and the job is only there for 1 or 2 years and then it's automated,” take the job, take the money, stay nimble, and then the next opportunity will open up.

You'll know a bunch of people who are in the same boat in the middle of the AI revolution. The sequence will evolve, and we'll keep telling you on the podcast where to move next. But don't take it as a sign that, because it's cooked 2 years from now, you're going to do nothing tomorrow. Don't do that. Take the job. Build the data center.

I agree. And maybe just to underline, Dave, your point further, I think I'm always coining neologisms, including “moation”—with an A, M-O-A-T-I-O-N—which is this notion that there are no stationary modes in a singularity, but what there are are dynamical modes, temporary modes, if you will. If you sequence them appropriately, then one mode can lead to another, can lead to another, can lead to another, and you achieve a dynamical mode.

It's the same idea with professions. Maybe an HVAC engineering position now enables—and this is not investment advice or career advice, but hypothetically—maybe an HVAC engineering position now creates enough of a capital base that can then be grown and translated to something else in 2 years, which can then be translated to something else, and so on. Eventually, you get to—I don't know—owning a planet.

So, Salim, let's take the conversation one step further: What happens when you don't have to work? What happens when all of your basic needs—food, water, energy, health care, education, liberty, all of that—are enabled for you?

You and I have discussed this before, going back to the Medici family and the Renaissance and so forth, where you basically have the life of a gazillionaire. You don't need to work. What do you do? I've written on Substack about this, where you have the life of a gazillionaire and you don't need to work.

I think that's an important realization, because when we get to the point where everything is cooked, that's also the point at which we have massive abundance, where all of your needs are taken care of for you. And now the question is, what do you want to do?

Most people in the world are working because they have to put food on the table. They have to get insurance for their family. It's not what they dreamed of doing when they were a kid, right? So it unleashes massive possibility at that point.

I think there are 2 categories here. Category 1 is the material needs. When you get to that level, it'll take a transition to stop thinking about meeting daily needs. Half the country in the U.S. can't put $500 together in an emergency. That itself is an emergency.

You've got such a monster structural problem around that, and it has to be addressed. Now, at some point pretty quickly, let's use the simplest argument: Somebody goes deep with AI, figures out how to use it to do an active-trading hedge-fund strategy, starts making enough money to pay for themselves, pay for their families, et cetera.

Overall, in aggregate, people and society will figure out how to generate huge amounts of wealth with this. Then you have the distribution question of how you equitably share it, which we've studied. But that's all the material covering day-to-day-life stuff. It covers the bottom 2 or 3 layers of Maslow's hierarchy.

We could see that happening. When you see societies getting to this abundance level—like the Mongols taking over India, or the Mongols taking over East Asia, or the Romans, or the Medici family, or whatever you've heard us talk about—the idea is that people end up doing 4 activities: food, art, music, and sex. And the joke is not in that order.

Once you get past that and get through that, the really interesting next dimension opens up, and you start thinking about what problems you could tackle, because we're not going to run out of problems.

We're just going to be able to tackle bigger and bigger problems, right? How do we create Alex's Dyson swarm? How do we go into the stars? How do we think about the new, different types of physics that may emerge from all of this? That's where the really fascinating stuff comes along.

I think this is the area, Peter, where we've had this conversation at the XPRIZE board meetings: Could we design prizes that advance humanity radically, rather than just trying to solve the problems? Because those look like they'll get handled. What's that next phase look like? I think the opportunity for human flourishing in that is so magnificent.

12. AI Agents and the Future of Companies

We love problems and we love solving problems. We'll just have bigger and bigger problems, which we'll be able to solve with bigger and bigger suites of AI agents and robots. All right, I'm going to move us to our last story in economics.

This is a paper, Salim, I know you're going to love and have lots of comments on. So, quick context: As Salim has educated us on past episodes, in 1937, the economist Ronald Coase asked why companies exist at all. He won the Nobel Prize for this. His answer was that transactions are expensive: finding people, negotiating, and enforcing contracts.

It's cheaper to hire employees and have them on the inside of your company rather than negotiating every task on the open market. Companies are a workaround for transaction costs. Now, MIT and Harvard researchers are finally catching up with Salim and have asked what happens when AI agents make transactions nearly free: agents that search, compare prices, negotiate, and transact for you at enormous scale. They're calling it the Coasean singularity. Salim, you wrote the book on this. Your thoughts on this paper?

Absolutely dead on. A little late, but totally dead on. Peter, in our 2023 book, Exponential Organizations 2.0, we said then that Coase's law was breaking. We didn't understand the full implications of it.

What we noticed then was that if you take the mission-critical function in Uber, which is to match driver and passenger, it doesn't happen inside the organizational boundary of Uber. It happens out in the wild. By enabling that with technology, you can scale. So we observed that companies were reaching outside themselves to get things done.

Prizes go to teams all over the world to do innovation. TED is using its community to scale. But now, with AI, it completely changes the game. The transaction and coordination costs go to near zero when you can have 1,000 agents or 100,000 agents doing crazy amounts of capability outside the organization.

I think the bigger point here is that AI doesn't just automate the firm. It attacks the economic reason for which firms exist and the shape of the firm. We've been exploring what the shape of the firm looks like, and we've got that definition going because now, essentially, a firm becomes a protocol and a community of agents and human beings that attacks various economic opportunities or marketplaces, or solves specific problems.

The organization dissolves from being a human, hierarchical-centered model to a totally different world. This is a huge shift, the biggest we've had since the Industrial Revolution, and we have to look at what this new model means. We're looking a lot at the governance of these agents, for example, and so on.

I want to push on that, though, because there's a countervailing force that one might perceive, which we talked about earlier in this episode: frontier capabilities potentially getting walled off from the rest of the economy. If OpenAI, for example, hypothetically, is using internal, unreleased models to solve grand challenges in math, and the rest of the economy doesn't yet have access to them, as a friend of mine at OpenAI and others have pointed out, wouldn't that agitate in favor of the exact opposite of Coasean economics—of firms growing larger and larger so everyone has access to those internal capabilities within the frontier labs?

You could, but I think that's an edge case. For the most part, open models allow people to have general intelligence and agents across the board. You could already have a frontier lab running its own hedge fund strategy, and I'm sure they're doing that now, that outperforms the market and just runs that model. But it's a very niche thing, applying for a certain temporal period of time.

Over time, I think the big question in my head right now is: If we're achieving RSI, what the hell does that mean? When we get to that point, I think the concept of an economy starts to erode and dissolve itself. So you have to think about it in a totally different model.

Not as obvious. I mean, this seems to me a lot like—what's the aphorism? What happens when an irresistible force meets an immovable object, or something like that? What happens when a Coasean, large frontier lab with superhuman, superintelligent capabilities meets an agentic economy that wants to distribute transaction costs out to the edge, combined with Argentina wanting to create nonhuman corporations? It's not obvious where that ends.

Alex, do you remember when Sam Altman said, “Yes, in the future, I think an AI should be running OpenAI”?

I remember that. And Sam, just in the past 24 hours, also on social media, expressed surprise—shocked, shocked—that OpenAI was able to solve Navier–Stokes. So maybe an AI would have predicted this.

Yeah. I mean, look, if you take your commentary to the endpoint, then you end up with what Dario was talking about, where one company like Anthropic will be all private enterprise, right? That's unlikely to happen. It'll take a while, and at that point, the concept of what it means to have an economy essentially evolves.

I think the bigger-picture question is: Where do we even have value creation and value storage in all of this? Because we come down to the money layer at some point. We should have Jeff Booth on. We were talking about guests earlier. We should have Jeff on to talk about what the future of that looks like.

Those are old tokens.

Emad, I see you. I see you ready to burst forward.

Yeah. No, I think that the economy is like 1% inspiration and 99% perspiration. You don't need a polymath doing your taxes or selling widgets or things like that. We've seen that, obviously, the Navier–Stokes model can sell widgets very well, but once it comes up with a recipe, it's actually about following through.

Most transaction costs are actually about friction. The economy needs a bit of friction. It's the relationships you build; it's the other things. It's not an instant thing whereby there are no barriers to spreading. Again, Salim talked about this—the frictions in a scarcity-oriented economy.

It's not the case that the best product always wins, or we'd all be on Betamax, you know? That's something for the old kids. There are lots of frictions in the economy, so I don't think it will be that way. It's just that companies and organizations become more efficient, and then they become more optimized, because it's really annoying when you go past 12 people and then past 150 people.

This is where you have that Carthaginian demon of disorder, Moloch, coming into organizations, and they get misaligned. So I don't think you'll see the 1-person company. I think you'll see the 10-person company, and I think you will see more collectives of companies operating with digital and physical humans, solving problems that deliver value.

But economics fundamentally does need to flip from being scarcity-based to being abundance-based, because we can rearrange bits and soon atoms in any way that we want.

That's Betamax for Americans.

What's that? Never heard of it.

Young kids, anyway.

All right, Dave, do you want to close us out? Any thoughts?

Well, I think, to step back from the academic question of whether our companies are getting bigger or our companies are getting smaller...

Clearly, Mercor now has 50 to 100,000 individual actors that are effectively little companies in India and around Brazil, too. You can take what Salim is saying and build a marketplace around it to deal with lingering artifacts like employment law in different countries and make a fortune by taking advantage of the trend that Salim is describing.

Concurrently with that, Elon is building the single biggest integrated vertical company that the world has ever seen, with complete supply-chain control all the way down to raw sand turning into chips. Those are both happening in the real world concurrently. So, I think maybe the real observation here is that things are changing very, very rapidly in both directions for very good reasons. The academic paper can wax poetic for years, but this is happening in the real world in those 2 flavors.

13. Tesla CyberCab and the Robotaxi Economy

Brilliant. I agree with you. All right, I want to turn us next to the Tesla Cybercab. Last Thursday, we covered the Austin launch, what I was calling Cybercabapalooza. This week, Tesla opened an official interest form for businesses that want to buy their own Cybercab fleets and build mobility hubs and charging infrastructure for the robotaxi network.

There’s no pricing or delivery terms yet. Salim, for me, this is another Musk business-model innovation. Tesla doesn’t want to own every robotaxi. You buy it, it works for you, and you share the revenue with Tesla. It’s a brilliant move for customer financing of a global fleet.

Honestly, this can only work, in my opinion, because of the price tag on Cybercabs, projected at $30,000—affordable to anyone who was previously an Uber driver. Here’s the form. I’ve gone and filled mine out. I wonder if you guys did. Let’s hear a quick video from Elon. This is an old video, but it predicts what he’s saying and doing now.

“We’ll have a model which is kind of like some combination of Uber and Airbnb. So, if you’re a Tesla owner, you’ll be able to add or subtract your car to the fleet. Just like on Airbnb, you could rent out your spare bedroom or rent out your house when you’re not using it. The same thing will be available for Tesla owners.”

Did you guys see that announcement? Any thoughts?

Well, it’s exactly what Alex was saying a minute ago. If you’re the electrician or HVAC person working on Colossus and you banked $600,000, where do you go next? You would have heard about this from your peers while working on that. This is where you go next, and then the next and the next. But this—

Well, you buy the Optimus robots that now do the work for you.

Exactly. Exactly. That’ll probably also be syndicated out as some kind of franchise model for maintenance, repair, and whatever. That becomes your trajectory of the future.

I totally agree. Going back to Alex’s insightful earlier comments about the private ownership of the capital means of production, I think this is the way. Again, not investment advice, but decades ago, it would have been accumulating a laundromat or a restaurant franchise. Now, it’ll be that you own a fleet of robotaxis or a fleet of humanoid robots.

Without this being misconstrued as investment advice, that’s the franchising path to medium- and high-income wealth creation. I think the SMBs of the present and the near future are going to look far more like that than opening, say, a chain of restaurants.

Also, when these things get started, when they’re new, the central force wants you to succeed so badly to get the momentum going that they subsidize the hell out of your success. If you were the fifth Starbucks owner, you would have been guaranteed success by the mothership. You don’t want to be the 100,000th.

But if you get an early jump on this, and whatever’s next and whatever is next, the mothership will subsidize the heck out of your success. You just have to basically not screw it up.

I find this incredibly compelling—to go fill out the form and say, “I’m going to grab 10 of these and have them work for me,” or 100 of them. At the end of the day, it’s earning revenue while you sleep and improving your local community.

There’s a simple economic thesis here, which is declining marginal cost. You think about Airbnb: its marginal cost of adding a room is near zero. If you’re Hyatt, you have to build a hotel. The same thing applies here. The marginal cost of owning one of these, adding people who use it, and leveraging that asset rather than having centralized ownership of taxi fleets is such an obvious no-brainer.

I think this is a huge opportunity. Dave makes a great point: get in early on these things, because then you have scale built in.

The question of which of the Cybercab companies, or robotaxi companies, is going to win—it’s the one with the lowest operating cost and the lowest production cost. I haven’t seen anything yet compete with Tesla here.

Or, unfortunately, the one that may be coziest with the municipalities that are approving them.

True. It will be city by city.

Usually, it’s pretty obvious who the winners are. Boston.

It’s so sad, Peter. So sad.

It’s so painful.

The winners are usually pretty obvious. People are just so slow to react; they’re not nimble enough. It’s not going to be a mystery who’s going to win. It’s going to be people underreacting to the urgency of the opportunity.

Sorry, go ahead.

No, I just think it’s becoming increasingly clear that robots are the biggest investment class we’ll ever see. We’ll see SPVs, we’ll see funds, we’ll see all this emerge. As you reach that level of capability, where, as Alex said, they can take over an HVAC engineer’s work, in this case, do you need more than a 2-seater robotaxi for just about anything? No. Only when you want to have your air robotaxi. That’s the only time you need anything better than that.

These will have very long lives, and they’ll earn money in just about any scenario.

I do not get a commission on this one. I do find it incredibly compelling. All right, I want to bring us to our final story here today, and it’s an important one. This is underlying much of what we discuss on Moonshots.

We’ve noted before that people are having fewer babies. We’re going to see a massive drop-off in the human population, and people are now living longer and healthier. This is changing the global demographic.

Here’s a chart showing the growth in the number of people over 65 and the drop in the number of newborns, from 0 to age 5, on the planet. The global population over 65 is projected to grow from 852 million in 2025 to 2 billion by 2060. More than half of all population growth over that period is coming from people over 65.

The result, at least in the old economy, is that fewer working-age people supporting more retirees is going to put huge economic pressure on pensions and health care in every developed economy. The only way the math works out in this future is because of the topics we discuss on this podcast: AI and robots doing the work for missing workers, and people living longer and healthier and not needing to retire, maintaining themselves as economic contributors.

What’s the reason you retire? You’re in pain, you’re feeling less energy, or you’re forced out because of policy. This is why I say longevity is not a luxury; it’s an economic policy for the century ahead. An 80-year-old with the body and mind of a 50-year-old isn’t a pension liability. They’re a founder in this future economy.

You’ve all heard the saying, “Demographics are destiny.” Robots that do the work and therapies that add healthy decades are rewriting destiny. Gentlemen, your thoughts? Salim, you want to jump in first?

Very simple follow-up. This is why we need the robotaxis and the robots and so on, because we’re going to need to redesign the entire concept of lifespan to healthspan to jobs. The concept of education, career, and retirement essentially evaporates.

You have to have repeated cycles of learning, creation, and then taking a sabbatical, et cetera. Longevity combined with AI totally disrupts who works, how they work, how long they work, and what they do. It’s going to completely change the game. We need to rethink that whole thing from the bottom up.

We’ve seen the preview with what’s happening in Japan. China is moving to robots because they have to, because of the one-child policy and the population bomb that’s coming toward them.

Great point. They have to do it. They don’t have a choice.

Yeah, these numbers can make the world much more interesting. Sorry.

These numbers are hugely understated because they include India and parts of Africa that are still having babies like crazy. But if you look at China and Europe, the numbers are much more acute than this.

There will basically be kindergartens and grade schools that are completely empty, and massive numbers of people over 65. It’s also not very far in the future. America is largely immune because we have huge amounts of immigration, right, in the working-age bracket. You don’t tend to have a lot of immigrants who are 65 and over, so you won’t notice it as much in the United States.

But in other parts of the world, this is basically going to completely rip the working class out of the economy.

And then the voters are overwhelmingly not employed. So it's going to create—I mean, it's already a mess, but it's going to create all kinds of strange things in those jurisdictions.

I would just add: this is what victory looks like. This is what we want to see. 150,000-plus people per day on Earth are dying, and putting an end to that—at least the beginnings of putting an end to that—is what this looks like, where people are starting to live longer and people are starting to stop dying. The counterfactual looks more like The Limits to Growth from the Club of Rome, which I think was in the early 1970s and was, potentially, depending on your vantage point, a horribly racist perspective in its earliest form.

Yeah, based on the false premise that somehow humans were going to overcrowd the purportedly scarce surface area of the planet and that was going to lead to decrepit conditions of living for everyone. Utter nonsense. This inverted pyramid, which some would call, I guess, the opposite of pronatalism—it’s not antinatalism—this is what a happy future looks like, where we have ultimately far more AI agents than we do humans and longevity escape velocity is vanquished. We want this future.

Yeah. So I think there are 2 major opportunities here. One is integrated elder care with robots, at a high level, but the other one is integrated baby care. We should have more kids. Kids are wonderful, and having fully integrated care for kids throughout can reduce the cost of child care, and supporting and raising kids in the best way is probably the biggest thing anyone could do for humanity, because there deserves to be more of us. So let’s have more kids. Let’s figure out that problem.

All right, gentlemen. I love this episode. I love spending time with all of you. This was brilliant.

Listen, I need to say something about what Emad said.

Yeah. I love my kid to death, but damn, the work of being a parent is the biggest biological scam ever. It just is an unbelievable amount of work.

It’s very rewarding. Between those comments and your prior comments about marriage over longevity escape velocity, what’s going on?

I get permission to make some of these comments.

All right, guys. Dave, Alex, Salim, Emad, love you guys.

Great conversation.

We'll be infinity away in 3 days. We'll see you again soon for our next episode. Brilliant as always.