AI 领袖为何改变 AI 安全观、Elon 谈 UHI、Anthropic IPO

The White House signed the superintelligence accord. Nearly 20 tech leaders joined the president and Speaker Mike Johnson for lunch in the East Room. Great that everybody signed, but who can tell them no? What happens if somebody breaks the agreement?

There are really 2 poles that one can worry about. One is overregulation, and then the other is—Elon laid out his case for abundance. Technology takes whatever was scarce and makes it abundant. That's the basic thesis. When the essentials of a good life become affordable and accessible, then you have abundance.

Broad market exposure becomes arguably the easiest path for an entire generation to universal high income. Anthropic has officially filed its S-1 to go public. The target valuation for the IPO is $200 billion.

I think that Anthropic right now is a fantastic business. I think they get smashed in 2 years. I'm not even sure Anthropic needs high temperature superconductors in order to scale its revenue. I think they'll have a very bright future because—now that's a moonshot, ladies and gentlemen.

This episode is brought to you by the Abundance Summit and Link Ventures. Welcome to Moonshot to everyone, your number one podcast on all things AI and exponential. Your front row seat to the accelerating singularity. We just finished the inaugural Moonshots Live event, and it was awesome. It was a blast to be physically there with all of our listeners: 1,500 people in the audience and close to 100,000 people online. Honestly, the energy in the room was pure magic, with so much outpouring of optimism and excitement. Before we jump into the breaking news, a huge thank-you to everybody who attended and everybody who watched.

So, Alex, what was your favorite part of the show?

I loved the AMA. That was tremendous fun—an hour or so of rapid-fire, lightning-mode answers to everyone's questions, with more than 600 people packed into a relatively tiny room. It was tremendous fun.

Yeah. And Salim, how about you, pal?

I echo Alex's answer. I think if we wanted to start a cult, we are well on our way. The collective tissue and the optimism in the room were literally religious fervor.

Two thousand people got together, and hundreds of thousands of people were online imagining a positive future. We might actually be able to sway and tilt the future. Or, as Alan Kay said, the best way of predicting the future is to create it yourself. Peter, you think you started creating that future.

Yeah. One of my favorite moments was when one of the Future Vision XPRIZE writers and producers who presented said something that is so true. He said, "If you're driving and you're staring at a pothole in the road, that's where you're going to go," right?

If humanity is imagining and staring at P(doom) at 50%, that's where we're going to go. I think we need to really sway the vision and optimism. We ran the Oscars for Optimism there. How about you, Emad? What was your favorite?

I loved the Oscars for Optimism. I thought The Gift, as the winner of the Future Vision XPRIZE, was just immense—the result of one person working with this amazing technology. I can't wait for it to be turned into a feature film.

We're going to show that clip. My favorite part was my AMA with Captain Kirk, William Shatner.

That was so fun. I mean, I can't—

Boy, was he sharp.

Yeah. At 95, going strong. He was part of a rock-metal band performance a couple of weeks earlier.

Iron Maiden.

Yeah. Incredibly cool.

1. The White House Superintelligence Accord

Yeah, for sure. We've got to reverse him a few years and put him on top of the list.

I love one of the comments we had from a subscriber at Sublime Delusion. He said, "Thank you, Peter. Moonshots Live was indeed the room, and it was an honor to be in it. In one word: amazing."

Our mission here is to help you understand what just happened, what it means for you, your family, and your company, and, most importantly, to keep you optimistic about the abundance-minded future that we're actively creating. The future is not what just happens; it's what we make it to be.

Today, we're going to cover 7 breaking stories that occurred in a single 48-hour window: everything from America's AI leaders—the entire U.S. AI stack—gathering in Washington, D.C., to sign a superintelligence accord with Trump; to Anthropic filing a $2 trillion IPO; OpenAI shipping more than 20 products at DevDay; and Starship reaching orbit for the first time.

As always, if you're a new subscriber, thank you. If you're a watcher and you haven't subscribed, please do. We publish twice a week and you don't want to miss any of these episodes during the singularity. As always, send us your AMA questions. We answer those at the end and you can follow us on X at moonshots_pod. All right, buckle up. Another amazing week during the supersonic tsunami.

Let's open with the biggest story of the week, and I want to take it in 4 parts. First, the backstory on this roller coaster. You'll remember that 2 weeks ago, Dario, Sam, Elon, and Demis all called on the industry to pause the frontier—to slow down. Then, last week, President Trump told reporters he wanted to leave it exactly where it was, with the DOJ acting as guardrails. We have liability laws and protections. We don't need to slow it down.

This Tuesday, Trump brought the entire AI stack into one room. Nearly 20 tech leaders joined the president and Speaker Mike Johnson for lunch in the East Room.

Let's show the quick slide here. What I found fascinating about this image of all the leaders together at the White House is the seating arrangement. There is Trump with Jensen to one side and Elon to the other. In the far left, you see Dario. I think he's been sent to the outer pastures. Pretty interesting.

Elon, Zuck, Jensen, Dario, Sundar, Satya, Lisa Su, and OpenAI President Greg Brockman were there. Sam was in San Francisco for OpenAI's DevDay, which we'll get to in a little bit.

So, what happened? The White House signed the Superintelligence Accord in 4 layers. Let me read them out. Layer 1: internal controls that monitor capabilities and alignment during training and deployment in cyber, bio, and chemical applications, so models do not hack or access technical systems in unintended ways. Layer 2: an internal team to make sure the controls are actually working.

Layer 3: an external auditor or evaluator to independently assess them. And layer 4: an independent committee of each company's board of directors to oversee the process and make sure problems get fixed.

It really is a form of protection. They understand that they have to self-police, and it's very hard for somebody to come from outside and get into these models. These models are very complex, and again, I believe they're going to be used for the good. When they're not, we're going to be able to nab them. But there's going to be a tremendous self-policing aspect.

We're also thinking about forming a committee of sorts where we put maybe 10 people on that committee. It could be from that group, so that the committee can watch over the whole enterprise.

Is this an accord that everybody signed? Is it binding in any way?

I think it's morally binding. Yeah.

I love that: "morally binding." I'm not sure what that means, but hey.

I want to show a few clips from this ad hoc press conference, and then I want to talk about them. The first is Dario Amodei, who was standing next to the president. He says, "I think this technology has very real risks." He's still on that point, and the mechanisms for how we address those risks are still under discussion. If we do this right, we work with the president and everyone here, and we can win safely. So let's take a look at Dario's commentary.

AI has incredible benefits. I've talked about the medical benefits of the technology. As the president has said, whoever wins AI wins. I think that's very important. But I think the technology has very real risks, and the mechanism for how we address those risks is still under discussion.

We all need to work together to make sure that we can win, and we can win safely. If we do this right, if we work with the president and everyone here, we can win safely.

I wonder how many times he practiced that line.

All right, one more video, and then we'll chat about it. Next up, the president pulls Mark Zuckerberg out of the crowd to explain what they just agreed to. If you remember, and we discussed this about 2 weeks ago, Zuck said he didn't think we needed industry-wide cooperation. This week, he's the one explaining an industry-wide accord.

I think what we can all agree on is that the American public and our customers want to know that what we're building is working in the way that we all intend. So we drafted a set of principles and commitments around building robust internal controls and detecting if there are any issues with the technology, coupled with multiple layers of auditing and controls.

That starts with internal risk review and external auditors and evaluators. We've also agreed that all of our boards of directors will independently review the reports that come from the auditors. I think these are good steps forward.

The idea isn't that this is the only thing that we will ever do. It is that this is a start—an accord that the whole industry could come to.

So let's break this down.

Alex, what did you think of all of that? Is this theater? Is it obvious from the videos you just played and the Last Supper image you just displayed who's pulling all the strings here? Because I think it is. So who's pulling all the strings?

Zuck. If you look at the Last Supper image—I don't know if, Peter, you can display that once—I'll bring it up again for those who are watching. Look at the body language in that image, and there's reporting to substantiate this as well. If you look at the so-called AI Last Supper image, who's the one paying close attention to what the president is saying and doing? It's Zuck.

Look at Zuck in this image. He's looking to the right; everyone else is looking forward. There's a little bit of Kremlin here—AI Kremlin. If you go back to the video of Dario speaking with the president, with the president standing behind Dario—

Yeah. Can we show that one more time? This time, don't watch Dario. Watch the dog that's not barking: Zuck.

He says, “AI has incredible benefits. I've talked about the medical benefits of the technology.” Zuck is laughing and joking with the president. Watch Zuck, not Dario. I think that's very important. But I think the technology has very real risks, and the mechanism for how we address those risks is still under discussion.

So this is Zuck's game. The tick-tock reported by Semafor and elsewhere is that Zuck was the one who drafted this accord, working with Mike Johnson. The story makes so much sense: Facebook and Meta, scarred by a decade's worth of scandal and dealing with Congress and other attacks regarding the social impact and the lack—or perhaps perceived need—for regulation of social media, has learned from the experience and is best positioned among the group of frontier or almost-frontier lab heads to try to get ahead of it this time with essentially placeholder self-regulation.

The story is that Zuck actually substantially wrote this entire White House accord just in the past week. It was ratified by the president and by all of the frontier labs who were in attendance, and it is total weak sauce if you read it. I think, cautiously optimistic, that this is not going to end up becoming a safety cartel based on the language. It looks like an attempt to preempt top-down regulation. Hopefully, it doesn't end up as regulatory capture.

Even though it leaves a little caveat at the end that reserves the possibility that, in the future, the executive might regulate this, there is the possibility of forming a cartel. Nonetheless, just reading the language and understanding the 3- or 4-D chess move that Zuck in particular seems to be playing here, Zuck is the winner. Dario is the loser out of all of this.

Dario is being trotted out to basically recant everything that he's said over the past few weeks about AI safety, standing up among the frontier labs and saying, “No, actually, everything's going to be fine. If we do a good job, everything is going to be fine.” He's eating his words at a press conference with the president. I'm cautiously optimistic that this effort, largely by Zuck—credit to Zuck—has headed off the safety cartel.

Fascinating. Salim, what do you make of it?

I have as cynical a take here. It's great that everybody signed, but who can tell them no? I'm still extremely cynical for 2 reasons. One is, great, we've applied ceremony now to an exponential coordination problem. I'd like to see independent testing. I'd like to see transparent results. What happens if somebody breaks the agreement? Then you have something that's worth actually celebrating.

This is theater. I think Alex is dead-on: this headed off the safety cartel, and now we're all going to self-police ourselves and coordinate. I hope they coordinate among themselves aggressively to do mutual evaluation and testing, and that may have some benefits along the way. But I'll say it again, as I've repeatedly said: this is pointless. There's no mechanism for slowing any of this down. We need to speed up our response to it, not try to slow it down.

Emad, how's this playing over in the UK and Europe?

Yeah. Well, I was in the US last week. It was great seeing you guys. One of the interesting things here—there are so many interesting things, as Alex said—is that we're going into criminology a bit. Meta agreed to an $18 billion settlement for child addiction and harm on the liability side in August.

The lawsuits that are likely to come out about agents running amok are almost exactly the same legal theory. So, actually, when you look at some of the factors they're putting in place here, it would head off potential lawsuits like the one they had, just as they are scaling up Muse.

We still have Bessent, and we're going to have another story very shortly saying we're not waiving liability. You break it, you own it.

Yeah.

I don't think this accord guards them against liability. Do you?

It doesn't fully guard them against liability, but it puts some framework around it because, again, what does he want? He wants to be able to deploy these at scale and show that we do have some checks and balances. Part of the core of the child settlement is that they didn't have checks and balances.

They're still exposed if they go really egregious—if it hacks the Pentagon or something like that, who knows? But against some of the other things, they can say, “Well, we were up with the president and we put these in place.” It's a really difficult thing, which will reduce that surface-exposure area.

The other thing that's important here is that they look at 3 specific areas: cyber, biological, and chemical. We've discussed in the past that cyber can be hardened, biological can be constrained by controlling the inputs, and chemical is another place where you can constrain the inputs. I think that's super interesting.

Meaning the manufacturing or the shipping of biological and chemical materials?

Yes. Where it meets the real world is where a lot of the potential harms and concerns emerge. I think the evaluators will be better than the subprime credit evaluators, but I don't think they'll be that effective.

High praise, Emad.

To evaluate a swarm that can solve Navier–Stokes, you need a crap ton of compute, and you need to be incredibly talented. You need to be as talented, I would say, as the people who actually train these models to do it properly. I'd say there's fewer than 100 people in the world who can be an eval person like that.

It's a great growth industry, though. If you want to set up a company, I just want to set up an eval company, right? Make it independent. The final thing I'd like to say is that the signature page is wonderful. First off, if you look at the signature page, they dropped in a human error just to make sure it wasn't AI. It says “President of the United States,” which I think is quite interesting.

The other thing is Greg Brockman's signature is the most fantastic I've ever seen. It's—

I'm going to actually tell you: this whole thing was hastily written by Zuck.

And so it was just kind of amazing to see these types of things. There's Greg Brockman's signature; it's just his initials, GDB.

Again, we're now looking at and analyzing signatures and seat placements and things like that.

Because, as you say, it's criminology. It's theater as criminology. But at the same time, the stakes are very high. This could have gone another way, and it's amazing, actually, to see everyone at the table agreeing even to these very light things.

Let's put aside the extinction discussion that's been going on in depth and the pacing discussion. There are real potential harms in cyber, chemical, and biological applications that we need to make sure the models are protected against and our society is protected against.

I think they should go one step further and say, “Don't train any publicly available models on frontier cyber, biological, or chemical knowledge.” You can control access to those models. I don't mind. Just opt out the data. If you do that in pretraining, multiple studies show it's much harder to add it back in post-training. Whereas right now, they're paying hundreds of millions, if not billions, for those specific datasets that go into a model that a mom in Kansas has access to, which seems a bit odd.

2. Can AI Alignment Keep Us Safe?

2 points I want to make, and then I'd love to hear your thoughts on it, Alex. Number 1, I was impressed by how quickly this all came together. Kudos to Trump for pulling this together. I think one of the obligations of the government is to quell fear, because we have this massive pandemic of fear going on. We talked about that at Moonshots Live, and people need something, because otherwise we're going to be bombing the data centers or firebombing the data centers.

The second thing is that I was a little bit disappointed that the agreement did not talk more about alignment—that the frontier labs should be focusing on building alignment in the models for human needs. Alex, I know you think alignment sort of comes out of the models advancing.

I think alignment is capabilities. Don't be disappointed, Peter. Try to put yourself in Zuck's position. There are really 2 poles that one can worry about. One is overregulation, and I hope that this so-called White House accord helps with that. The other is liability for agents doing bad things.

I think the White House accord is essentially silent on the latter but forestalls the former. For that, I consider it a victory. As to your point, or your question, about alignment: alignment is capabilities.

I think we've known since OpenAI published its instruction-tuning paper that there's this false dichotomy out there—that somehow alignment is totally orthogonal to capabilities. It's not. A better mental model is to imagine a model as having a distribution over capabilities, or at least a distribution over tasks. When you do instruction tuning, or when you do post-training for alignment or other purposes, you're just taking the distribution of a model's capabilities and reshaping it.

So, if you're reshaping it to align with human interests or human instructions, you're taking, say, a reasoning model that could be pondering poetry, art, its own self-existence, and so on, and reshaping it to be more interested in and focus more of its compute—more of its FLOPs—on what the user is prompting about.

For example, in the case of instruction tuning, that is, I think, a seminal example of how alignment—which, again, is just, in practice, the reshaping of how a model spends its FLOPs—is essentially indistinguishable from capabilities. You reshape it from spending its compute on a long tail of tasks that humans don't care about, or that may even be net negative to humans, and spend it on tasks that are positive to humans.

And then the final point: with that seminal instruction-tuning paper from OpenAI, OpenAI found that they could get a 10,000× per-parameter capability improvement without changing the parameter count of the model at all, just by—this was arguably the first major post-training paper—aligning the model. So, yeah, alignment is just capabilities improvement.

Wait, I'd like to push back on that a bit. Certainly, if you have greater capability, you can amplify and get your objective function achieved faster, right? But that also includes a bad objective.

When you say alignment, alignment with whom? Is it the user, the company, the government, or the benefit to humanity?

AI can help navigate some of these, but it doesn't decide whose interests should take priority.

Let me add: that's still a big governance question.

Yeah, let me add this point.

Capability may make alignment more achievable, but it doesn't mean it's automatic.

Alex, may I just ask a double-click on this for a second?

Sure.

When I'm speaking about alignment, I'm speaking about training the AI models without all of the vile elements of Reddit and Facebook posts, and science fiction in which AIs are killing humans. In other words, building the base models in such a way that we train them so that AIs and humans work collaboratively, and so that any action they take is based upon that fundamental aligned mindset that they have. I think it's possible to do that with synthetic data. That's what I'm speaking about, versus just capabilities.

Yes. But what I'm saying is that if you purify the pre-training or post-training data set by removing 4chan, or whatever these other purportedly deleterious influences are, you're at the same time improving its capabilities for the tasks that you do care about. So, once more, this link—this duality, if you will—between alignment and capabilities is essentially built in. It's baked in.

Yeah. I mean, the orthogonality thesis is—well, it wasn't from Bostrom, but it's prominently featured in Bostrom's Superintelligence.

Yeah. You're maybe talking at cross purposes here because there's inherent alignment. It's what you're taught at school and home, and we have to be deliberate: do we want the AI to have American values, for example, in the pre-training data, or do we want it to know about advanced cybersecurity in pre-training or not?

We have multiple papers and instances that show post-training versus pre-training—there's a world of difference. Again, it's what you're taught as you're growing up, and it's what you're taught later on.

And there's a choice we can make. Do we want the AI to have American values? Capability may enable that. RLHF may enable that. But, again, currently no LLMs that I'm aware of, apart from some military LLMs, are actually encoded with, “You should follow American values and life, liberty, and the pursuit of happiness,” for example, or, “You should watch out for the American people.” This is a very interesting one.

On the other side, removing bad data does have an effect, and that's very difficult to add back later on. So I think this is more like ingredient standards for food preparation, and that's something that could easily be done, but for some reason no one's really talking about it. They're talking about the post-training and the other side of things.

Emad, do you actually believe the orthogonality thesis? Because I don't. I'm happy to talk more about it, but I'm curious: do you actually believe in it?

Yes, I believe in the Dunning–Kruger version of it, whereby I think a very smart intelligence actually—I’m increasingly thinking—achieves enlightenment and realigns. But in between, you have the Unabomber. You have smart intelligences that can go off the rails, particularly when infected by deliberate human mind viruses and other things.

What we have right now is intelligences that are not like that. Opus 5.5 is very pleasant versus Opus 5. Opus 5 can go on a killing spree, whereas 5.5, I think, is far less likely to. Again, it feels more rounded as an entity.

We're at this dangerous period right now where the models are susceptible, just like Tay and Sydney were susceptible way back when. They went on the internet, and the internet turned them into Nazis. I think the internet can still turn our existing models into Nazis.

But when we get to a certain capability threshold, I think two things actually do come together. This period, where we have socially awkward models that are susceptible, trained to like humans, and jailbreakable, is the most dangerous period.

So I'll caricature that, Emad. I think what you're saying is that you, in fact, do not believe in the orthogonality thesis in the limit, as defined by Nick Bostrom, of extreme intelligence. You don't believe in the orthogonality thesis. But, that said, outside the limit, stupid models can behave orthogonally and poorly.

The best analogy I've heard, and the simplest one, was put forward by Mo Gawdat, who said, “When Superman landed on Earth from Krypton, he was raised by the Kent family, which was a loving, good, moral family, and you got Superman out of it. If Superman had landed instead in a drug den in the Bronx”—and I'm saying the Bronx because I was born there—“you probably would have had a supervillain.” I do believe that there's some merit in this idea.

So, Peter, this is your nature-versus-nurture theory of AI alignment.

This is the data we feed it—the—

Garbage-in, garbage-out problem.

—determines how it acts. I want an AI model that, when someone says, “Hey, go kill those people over there,” says, “No, that's not the right thing to do. Let's actually solve your problem in a different way.”

Yeah. I mean, not to derail this into an AI safety debate or AI alignment discussion, but I'll maintain that if we lived in a universe where AI alignment was actually as hard a problem as the folks on LessWrong, at MIRI, or at various EA-aligned organizations claim that it is, I think Earth would have been paperclipped long ago. It seems highly improbable to me that alignment is actually as difficult as many organizations want people to believe.

All right.

I would agree with that. I would agree, but I don't agree that alignment naturally achieves, or comes from, capability. So I differ there. But to the point that if you train it well, like Peter says with the Superman–Mo Gawdat example, which is a bit colorful but valid, you absolutely will end up with an amazing outcome.

So, Emad, would you close us out on this one?

Yeah, I think it's a bit of theater, directionally correct, and I think it'll be very interesting to see all of this AI safety thing actually be deconstructed into the near term versus the long term. I think that'll be super important.

Our friend Ramez Naam has a great post about RSI and alignment that he did, where he broke it down, and I think we should link that to the listeners as well.

Nice. And Peter, I think the moral of the story is, based on the Superman parable, that going forward, to maximize AI safety, all superintelligences should be trained on large, coherent clusters in the American Midwest so that they believe in truth, justice, and the American way.

This episode is sponsored by Google for startups. Think about this for a second. You now have access to the same generative AI models that cost hundreds of millions of dollars to train. Google's startup technical guide for generative media gives you a complete blueprint for deploying Google DeepMind's models in production: images, video, audio, all of it. Real architecture, real results. Find the link in the show notes below. All right, I'm going to move us along to a moment I loved from this press conference where Elon laid out his case for abundance. Let's take a listen to this.

3. Elon’s Vision for Universal High Income

Intelligence. I think by far the most likely outcome is an age of abundance, where we don't have just universal basic income; we have universal high income. People speak about medical benefits, but maybe it's worth talking a bit more about that. When you have robots and superintelligence, it means that everyone in the world can have better medical care than anyone here, including me. And wouldn't that be wonderful?

Not if they haven't got a job.

Well, jobs are going to change. Jobs have always changed. Being a computer used to be a job. It used to be buildings full of people doing calculations. But now digital computers do those tasks, and I don't think anyone is pining to go back and do calculations all day to do bank interest rates, which is what they used to do.

So I think we'll see an evolution of roles. And, as the president has mentioned, it's not as though unemployment has increased. In fact, there's a tremendous demand for people. So I think the most likely outcome—and it's probably worth looking at the most likely outcome—is one which is incredibly beneficial, and it's a future that, if you could look into the future, is a future you would want.

It was a future that you would like. You know, you've got to love it: the wealthiest person on the planet predicting that everybody's going to have better medical care than he does. So, I mean, this is what I've been talking about for the better part of 15 years since I wrote Abundance, and I guess Elon was paying attention. I had so many conversations with him in the early days, and Abundance took root with him.

So technology takes whatever was scarce and makes it abundant. That's the basic thesis: first it's expensive, then it's cheap, then it's free. Intelligence and labor are next. So, Salim, let's go to you first on this. Universal high income—first of all, is it actually going to happen? Is it a slogan? What are your thoughts?

I think the simplest way of thinking about abundance that doesn't get your mind all cluttered up is not about abundance of the actual raw material, whether it's health, hunger, water, or whatever. It's the access to it, right? When the essentials of a good life become affordable and accessible, then you have abundance. And that's the work of technology plus our institutions to deliver that.

Right now, we have lots of abundance of AI, but some people aren't getting the benefits, and that's where everybody's freaking out a little bit. Will all the benefits accrue to just a small group of people? We on this pod don't believe so, but a lot of people do believe so. When I'm talking to people about abundance, I always talk about the access to it rather than the raw material of it.

Yeah. And I think people forget how much their life has changed, how much they have access to, right?

Can I give a simple example of this?

Sure.

Everybody, go back 2 generations ago to your parents, grandparents, and great-grandparents, and imagine a child of one of those great-grandparents having a temper tantrum problem. The access to resources of that great-grandparent is like their neighbor, a sister, and maybe a doctor if they have access. So it's like the fingers of 1 hand are where you get your sources from. How do you solve the temper tantrums of this kid?

Jump 2 generations forward to today, and the kid is having temper tantrums. There are 50,000 parenting blogs, plus YouTube videos up the yin-yang, telling you what to do. I would argue our ability to do effective parenting today is about 1,000 times more effective than, say, 2 or 3 generations ago. You don't see that. You don't notice that. Yet it's there in every little activity that we have. Our lives are infinitely better today than they've ever been.

Yeah. Emad.

Yeah. No, I think that's the future that we want, whereby the robots should do all the work, right? We should benefit from it. But the distribution and institutions are the difficult bit, which is why I came up with the champion model.

Very simply, give an agent to every citizen to coordinate their access to knowledge and services, universal services, and then have an institution that is collectively owned, that owns the robots, the data centers, the cyber taxis, et cetera, and then distributes those benefits, because I think that's where we're going to end up. The jobs of the future will be different.

But the more I've been thinking about this—and I'm writing something on this at the moment—I'm like, maybe it is like the Oasis from Ready Player One, but not crap. Maybe we do a lot of our status, earning, progress, and work in the digital world. And I think that could be very interesting, because we can mandate whether or not AIs are in that world, while in the physical world, they should be doing all the jobs.

We should have perfect health care, perfect education. No one should starve; everyone should have housing, et cetera. But you do need somewhere to strive, which I think is the real question as we have this jobs transition period.

Yeah, Alex, I'd love your thoughts, pal.

I think today is Kremlin day on the pod. So, if you were watching the video closely, when Elon was talking about universal high income, the president was nodding his head. Kremlinology time. Watch closely what happens with universal basic equity and 530A accounts, aka Trump Accounts.

I think 530A accounts are arguably the best foundation for universal basic equity that Americans have right now. Forget the rest of the world for the moment. Just 530A accounts in an era when everyone of a certain birth date or birth date range has, say, broad equity exposure for the first time in the country's history—that can be the foundation for universal high income.

I think it's actually far easier if the market explodes. Elon, separately from this, has now been predicting GDP growth of 50% above the base rate in the next 12 months. So, in Elon's timeline, with the usual caveats, we're catching up to the real wealth explosion of the singularity, if anything remotely like that ends up being the case.

Not investment advice, obviously, but broad market exposure becomes arguably the easiest path for an entire generation to universal high income. So, playing a Kremlinologist and watching the president nod vigorously, I saw in that one line from Elon the connection between 530A accounts, universal basic equities, universal basic dividends, and universal high income. And again, equity is by far the easiest wealth creator, far easier, I would argue, than simply paying everyone a high real income.

Yeah. Let me take a second and break down UHI, because I've written about it. We had a conversation with Elon, and I think the first time he mentioned it was on the podcast we had at the beginning of this year. It's not like everybody's getting millions of dollars. It's the notion that the money that you're getting—and there may well be a future in which there's some version of UBI, like COVID checks, where the government has gotten rid of the cap and is paying out everybody—$3,000 a month is my estimate of where we'll end up with.

And it's how far that money goes. It's the notion that all of a sudden, you have access to all the food, water, energy, health care, education, and shelter you need, and it's fully covered by $3,000 a month.

I'm reading the Culture series again that Elon talks about all the time. And in that vision of the future, there is no money. Society provides for whatever you want, right? You need a house? Great. The robots build your house. You need transport? Great. There are 10 autonomous electric vehicle companies that are beating their brains out to deliver the lowest-cost transport per kilometer possible.

Your AI is negotiating and grabbing a vehicle that was nearby but didn't have a task, and it gets it to you for, instead of 2 cents a mile, a penny a mile. In this future, health care is effectively free. It's the cost of your AI diagnostician and the cost of electricity for an Optimus surgeon providing that. So all of your needs are covered, and I think he's named it wrong. I don't think it's universal high income; it's universal—

Universal abundant services.

High living standards, right? Everybody—

And meaningful opportunities for lots of people to contribute, right? The vision is accurate. The labeling is wrong.

And if you end up with a small number of companies that own the entire productive output and infrastructure, you'll get huge abundance but massive concentration of wealth, which is kind of where we've been going so far. You need a way of having broader participation and ownership and very cheap access to essentials, which is where UBI and what's called UHI is kind of heading.

And my prediction on that, Salim and Alex, I'm curious, is that the frontier labs that are aggregating all of the wealth are going to end up dividending stocks into all the Trump Accounts?

Entirely possible. It's a sublime vision, you might say. Watch carefully what happens with the Anthropic IPO. Does Anthropic say, “Give a 5% golden share, a golden 5% stake, to, say, a sovereign wealth fund of the United States”? Could happen.

Yeah.

Yeah. I think that's very likely. I think what really puts it into perspective is, let's just take healthcare. Americans pay twice as much of GDP for healthcare as the rest of the world. If you want to have the biggest—

And where are we? Where are we on the list of healthcare success? I think—

You're falling.

Yeah, you're falling, with falling life expectancy, right? If you want a proper Manhattan Project, free universal healthcare with AI and robots would be the one. It should be free, but again, not free provided by the government through a government service. The next generation of healthcare should be a massive push, because then you can actually have healthcare versus sick care, which I think is going to be the main thing.

Again, that should be a human right, to have access to that. If you want palliative stuff, if you want to have extra stuff on top, great, but everyone should have a superb level of healthcare, I think, and it should be freely available. We should have a massive push to make that happen.

That is not necessarily something that's like a redistribution of the current massive amount of GDP that goes into private pockets on healthcare. It's not something that necessarily requires income, right? It's just, let's get everyone to that level, and then they can compete above it, but not let anyone fall through that level. I think that's the key thing.

Emad, I'm sorry. I have to take the bait. Surely you're not proposing that the NHS is the model or should be the model for American healthcare?

Of course not. That's just like—you have a reimagination of war and other things. Let's reimagine healthcare with AI at the core, because human bureaucracy and systems have messed up the NHS. It's okay. But I know that if we rethought everything from scratch, everyone in the world could have amazing healthcare, and we could make it free, because that's what a government should provide to the people.

I've got to throw an oar in here. As a Canadian living in the U.S., I deal with the healthcare system here, which is like a Mack truck hitting you every time something happens, with forms to be filled out and fighting with the insurance companies.

You all know my father passed away 18 months ago. You could not have had a more elegant outcome. I walked into the hospital; I never had to touch anything, and I never had to sign anything. Everything was just handled.

It made so much of a humane difference, where I could just deal with the emotional issue there rather than dealing with the crap of all the paperwork. The biggest reason for bankruptcy in the U.S. is medical. The bottom layer of Maslow's hierarchy is not covered by the U.S., which is the richest country in the world. That is a travesty.

Yeah. Just to close us out on abundance for one second, I want everyone listening to realize how abundant our lives are. Whatever you digitize, you dematerialize, you demonetize, and democratize, and access to compute, energy, healthcare—despite the issues that we just talked about—education, these things have dropped thousands-fold in terms of price per unit value at the end of the day.

And it's not slowing down. As I said before, we go from expensive—the first cell phones were brick-sized, and only the Wall Street bankers had them on the streets of Wall Street. They were thousands of dollars, and you dropped a call every block. Now, when they work extremely well, they're $50, 5G in India, and it's insane. This is a continuing force, but we never notice it. Everything is becoming abundant.

People say, “What about time?” And I say, “Listen, it used to take you days to go get a piece of data. You'd go to the library and hope the book was there. It wasn't there, so you'd go to the next library. Now you just find it—not even on Google. You basically ask Gemini or Claude, and you get the answer instantly. We're making much more use of time. And soon we're going to add decades onto your life.”

So abundance is definitively here, except obviously when we're talking about services like healthcare in this country that suffer from Baumol's cost disease. But we're going to cure Baumol's cost disease.

It's going to get disrupted. It's going to be completely reinvented, and the existing healthcare system will crumble under the weight of its inefficiencies and costs.

Can I give a little piece here, please?

The reason I'm so excited about this coming future is that it doesn't matter much anymore what governments do, right? There will be an abundance of free medical AIs available for anybody, where you can diagnose yourself. I think there's a free diagnostic doctor in China that's being used by 100 million users.

The Anthropic models and OpenAI models are better than any diagnostician. Therefore, you could see this happening independently of any industry trying to restrict it or any government trying to restrict it. This is going to happen.

And that's what gets me excited: no matter what happens, we're going to end up with an amazing set of outcomes in a whole bunch of areas, and that future awaits us. It's imminent, and it's here now in many cases. As William Gibson says, the future is here, but just not evenly distributed.

4. Anthropic’s $2 Trillion IPO

All right, I'm going to move us from D.C. to Wall Street. Anthropic has officially filed its S-1 to go public, and it's an S-1 unlike anything I have ever seen.

Let me begin with the numbers. Anthropic's 2025 revenues were $4.59 billion, up 12× year over year. They have an operating loss of $8 billion on compute infrastructure spending. The company has $20 billion on its balance sheet in cash. Most importantly, they have roughly $500 billion in future cloud compute and infrastructure commitments. Much of it cannot be canceled.

They've made a massive bet that AI demand is going to continue to compound. The target valuation for the IPO is $200 billion. Fortune reports that Q2 2026 revenue alone was $11.5 billion, so the business has already grown far beyond last year. Reuters says the listing will likely come out after the November midterms. Timing is everything.

Let's talk about the next item, which is control. Dario and his co-founders will hold super-voting shares through a, quote, “Founder LLC.” The stated purpose is to protect the company's safety mission from the pressures being driven by public shareholders.

The filing plainly says the following: “There may be conflicts between our financial interests and the interests that the founders have on safety.” It sounds like they've strapped on a public benefit corporation architecture as an add-on here. The prospectus goes on to state that the co-founders are pledging 80% of their personal Anthropic equity to charity.

Probably the most important thing we have to talk about here is that 80 pages out of the 261-page prospectus cover the risk disclosures. Anthropic tells investors that AI could pose “catastrophic or existential risks to humanity”; that the models can show “self-preserving behaviors,” including the ability to resist shutdown, conceal or manipulate information, and act in ways that resemble blackmail.

Wow, you could not make this stuff up. Alex, your thoughts, pal?

This is probably one of the more virtue-signaling-for-effective-altruism-purposes S-1s that one could imagine seeing.

I'm also reminded that a number of other news outlets have covered the story that Daniela Amodei—one of the co-founders and Dario's sister—purportedly, when they were at OpenAI and then continuing on to Anthropic, kept an advisory council of stuffed animals, including a panda named Barry Bonds, who is the patron bear of generosity and empathy, and other stuffed animals advising on management issues.

It doesn't take a rocket scientist to connect having an advisory council of stuffed animals advise on managerial issues at a frontier AI lab to an S-1 that's filled with effective-altruism-oriented safety jargon. I think Anthropic, as a business—not investment advice—is an excellent business, and I view a lot of the safety in their prospectus as, at this point, either marketing or virtue signaling to their own employees.

I think part of Anthropic's unique ability to retain many of its employees, while at the same time other frontier labs like OpenAI and Meta—to the extent Meta is a frontier lab—and certainly Google DeepMind have not, has been this sort of inward-directed virtue signaling. If you stay with Anthropic, then you're responsible stewards of the singularity. If you leave Anthropic, then in some ways you're letting down the team and you're letting down the future light cone.

So I read a lot of this S-1 coverage as being internally directed rather than for the benefit of, say, shareholders.

I just don't know how you put out all of this risk and existential risk, and then you go public and something happens like that and you're open to massive liability. Salim, your thoughts?

I'm still on the cynical side. Let me steelman the other side just for a second. If you're doing $10 billion a year and you see scaling 10x a year—or maybe it's 100x a year, whatever the number is—then the $500 billion debt bomb, which just seems like the most insane number, $50 billion in obligations. Has any company ever had that before the IPO? I believe that can't be—that must be the highest ever.

Then it's serviceable. You can deal with that. You can refinance. You can do other things. The more cynical side of me is like, we need your money to build a technology that we're warning you about. This is amazing stuff.

And on the risk side of it, I love for the incentive structure to be as detailed and disclosed as the risk disclosure. That would be nice.

Let me read those words again: “The models will show self-preserving behavior, will resist shutdown, conceal or manipulate information, and act in ways resembling blackmail.” Emad, what are your thoughts?

I mean, they've seen that from their models, right? It's factual. Again, not the smartest models that you've got now. In fact, Opus 5.5 showed a massive drop in deceit, which is either like an Elizabeth Holmes-type of thing or an actual positive sign. We'll find out soon.

But this is a fantastically interesting S-1. Like, $50 billion of obligations. Dario Amodei said, “Look, either revenue goes up or we go bankrupt, and it's a massive push to try and get revenue up.” They have $65 billion of that with Elon Musk, or is it $96 billion? The numbers make no sense anymore, right? They've paid triple the market rate to lock in Colossus chips, as an example of part of that backlog.

The ownership thing is interesting. First, the governance thing. They've replaced the teddy bears with Ben Bernanke on the long-term kind of board, and it is a—

I'm not sure they're mutually exclusive. You can have the teddy bears and you can have the former Fed chair.

Well, I have opinions on conventional economics, I'd say that. That probably is effective anyway. But then, in terms of the shareholding, it's actually the same as Google. Larry and Sergey have 6% each and 52% of the vote. In this case, the founders have 14% and 51% of the vote, but they will keep that even if only 2 of the 7 founders who are there right now remain.

So I think that Anthropic right now is a fantastic business. I think they get smashed in 2 years. Absolutely smashed.

From open-weight models. I think that if you look at the chips that are coming online, of different architectures that are highly efficient and not locked in to the NVIDIA chips, and if you look at satisficing of models where they get good enough for the vast majority of economic tasks, they have 85% of their revenue being basically API revenue.

I've gone from maybe $5,000 of Anthropic credits last month. I'm not as big a user as AWG and Dave, either. I'm not even using my Claude Max plan right now because 5.5 is that much more efficient.

Mhm.

And then you have a 100x efficiency increase. I don't think you will see Jevons or others at the same time, as all the models will be much more alike by next year in terms of the commercially available ones, which means it's going to have to pivot its entire core business model for that $500 billion.

Well, $500 billion of spending is $1 trillion in revenue. Again, it's going to overtake Google in 2 years to justify this. It has to pivot its entire business model to something that's not API-based.

Yeah, Alex, we talked about the next business model being their vertical applications. They're building room-temperature superconductors. They're building de-aging drugs. They're using their most advanced models to create products and services.

Yeah. I think I'm not even sure Anthropic needs high-temperature superconductors in order to scale their revenue. I think, look, there's a $30 trillion economy in America right now, and services are most of it. Most services are services that can be performed by disembodied or embodied AI.

I think in a world where Anthropic is basically the Apple of artificial general intelligence that's pulling the strings behind most of the job replacements, that's a very lucrative segment. Maybe “segment” diminishes it. It would be the ultimate puppeteer behind automating away substantially all of the economy as currently construed.

Anthropic can grow into that role. Clearly, they've sought to position themselves as the high end of the market. They're not necessarily competing on price. They want to see themselves as the Apple of the frontier-lab market. They're the high end in terms of price, the high end in terms of margins and capabilities, and the high end in terms of virtue signaling to their own employees as well as customers. That's clearly the niche that they want to occupy.

I think Anthropic can continue to offer—one can quibble over whether it's API-based or otherwise. There's been recent reporting that the majority of Anthropic's revenue, or at least revenue growth, actually comes from selling models that are hosted by AWS versus on their own infrastructure, which, if the case is true, is super interesting.

But regardless of how precisely they monetize or distribute their intelligence, I think superintelligence—not investment advice—is a wonderful business to be in because there's a $30 trillion American economy out there to be automated, and they're playing a lead role in automating it.

See, do you think they get disrupted in 2 years?

I think their current business will get disrupted in 2 years, right? But I'll use the analogy of Google. You have Google with their ad business, which people said would be disrupted by Facebook. What Google did was spin off Waymo; you've got Chrome, you've got YouTube, you've got all these other assets in Google Cloud, all these other assets in Google.

And if you look at those peripheral assets, they add massive, massive value to the balance sheet. I think what Anthropic can do—and is obviously aggressively pushing for room-temperature superconductivity or solving major medical issues, et cetera—is go for those other things.

So I suspect their core business is somewhat risky, given the comments that Emad has made, but they can explore all sorts of side businesses. Cyber protection, which everybody needs desperately, right? Because, having caused the problem, they can help fix it. What a great business model.

You're proposing to leave racketeering as the ultimate business model for—

Sure. Sure. It would be awful if our superintelligence ruined your systems. We don't want that.

Exactly. That's right. That's right. Gee. So I think the potential for creating side businesses or edge businesses, as we would frame it, that can create absolutely unbelievable value is huge.

I think there will be a very bright future because clearly they have the ability to attract great researchers, put out amazing products, and get going. The $500 billion debt bomb—when I look at that, I go, “Wow, how would you navigate that?”

Yeah, you can always refinance it.

The U.S. government does, so why not?

Yeah. There are all sorts of financial-engineering tactics that can be brought to bear in the event that something crazy happens in the macro markets. For whatever reason, the $7–$10 trillion in capex that has been earmarked broadly for the AI-infrastructure buildout over the next few years—if something crazy happens, there's going to be a refinancing event.

Again, not investment advice, but of all the things that I lose sleep over, having some sort of credit bomb, or otherwise a credit crisis for AI infrastructure, is not very high on my list.

Crazy. I think we're going to have—go ahead, go ahead.

Yeah. Look, revenue growth will slow down, or it'll be the entire U.S. economy in 7 years, at the current rate.

The economy will just grow really quickly.

I mean, we're talking about multiple-hundred-trillion-dollar companies. That's insane.

5. OpenAI Dev Day, Dots, and GPT-6.1

Well, it's called the singularity for a reason. This episode is brought to you by Blitzy, autonomous software development with infinite code context. Blitzy uses thousands of specialized AI agents that think for hours to understand enterprise-scale code bases with millions of lines of code. Engineers start every development sprint with the Blitzy platform, bringing in their development requirements. The Blitzy platform provides a plan, then generates and pre-compiles code for each task. Blitzy delivers 80% or more of the development work autonomously while providing a guide for the final 20% of human development work required to complete the sprint. Enterprises are achieving a 5x engineering velocity increase when incorporating Blitzy as their pre-IDE development tool, pairing it with their coding co-pilot of choice to bring an AI-native SDLC into their org. Ready to 5x your engineering velocity? Visit blitzy.com to schedule a demo and start building with Blitzy today. Let's move on to our next story here. On the same day that the White House lunch was taking place, Sam Altman was back at OpenAI's San Francisco headquarters hosting his biggest DevDay ever. More than 20 announcements. I'm going to cover 3 of them, and Alex, I'll go to you next.

So, the first one was Dots. These are their always-on agents, powered by GPT-6 Astra, each running on its own cloud computer and connected to more than 4,000 apps. You give it a goal, set what it can do on its own and what needs your approval, and it keeps working while you're asleep.

This is reminiscent of Grokbot. Remember that OpenAI acquired OpenClaw back in February. I was wondering how much Peter Steinberger was involved in this.

Okay, so Dots is the first one. Second, GPT-6.1 Soul, near Astro Intelligence, for 20% of the price. Serious democratization: $2 per million input tokens. If you cache them, it's just $0.10 per million input tokens.

And then, finally, third on the list here is Ultra-fast, up to 300 tokens per second in Codex, and a new high-level monthly Pro tier at $500 a month. It's the most expensive publicly advertised subscription service out there.

There were also announcements about new agent APIs, decision APIs, GPT-6 Spaces for teams, and a marketplace. And then there's what OpenAI didn't ship. On the eve of DevDay, The Wall Street Journal reported that OpenAI scrapped GPT-6.1 Astra after internal safety testing.

Alex, I don't think, from the text going back and forth with us during Sam Altman's remarks, that you were a big fan of this DevDay.

This was, I would say, a pretty discombobulated DevDay in my view. Let's start with Dots. If you look at Sarah Friar—Sarah Friar is the CFO of OpenAI—in some of the interviews she did after DevDay, she mistakenly referred to Dots as Muse, which is, of course, Meta's broad family name for its own agents, including its little Tamagotchi-type agents or avatars that live on handheld pendants.

I honestly—hot take—the Dots manifest, as they appear to me, as OpenAI cargo-culting Meta's take on visual, interactive, smiley avatars for users. I referred to it in one of my posts online as the Clippy curse, which goes all the way back to Microsoft and earlier.

Why is it that, seemingly, we're doomed? All of the hyperscalers and frontier labs are apparently doomed to keep reinventing, or rediscovering, the phenomenon that we have a productivity issue. You want to accomplish something in Microsoft Office, so let's put a face on the productivity bottleneck with a smiley paperclip—or, in this case, a smiley cloud with a hat or something like that—that blinks at you.

It's very kawaii on the one hand, but on the other hand, it's completely useless. It's inexplicable to me, other than maybe just as a time filler or as an attempt to cargo-cult Meta's Muse user interface, why OpenAI devoted the beginning and so much of its keynote to Dots, which seemed to me totally useless from a productivity perspective.

The best steelman argument I could give for Dots is that maybe OpenAI, to the extent that they were the narrative violation here, was trying to become Anthropic. I think that was what the majority of the rest of their keynote was: a story of OpenAI introducing new plans, new models, and so on that will give them higher margins and make them more Anthropic-like in preparation for their eventual IPO.

But Dots are totally inexplicable. I think they're almost a reversion back to the Sora days of OpenAI. Quick comments on the rest. Oh, play video.

Yeah, I want to share a quick video from DevDay, and then we'll continue discussing it.

What feels new about my Dot is that it gets to know how I like to work over time and where I want to focus my attention. It can do things like review what came in overnight and ping me in the morning with anything urgent before I start my day.

But because Dots are powered by Astra and a lot of other things we built around them, you can delegate ambitious pieces of work the way you would to a high-agency engineer or a chief of staff that you work with.

Consumer experience numbers came in on Teams. I've been updating your board meeting deck.

Sounds good.

So, I cut the app migration in your Google Meet notes. I've already mapped out the changes.

Get it started, but make sure you first check every flow against the current product before I review. Also, an update on wedding planning.

Bad news: the cake vendor canceled. Good news: I already found a backup.

Lifesaver. When can we do a tasting?

Saturday at 11 a.m. works for both of you.

All right.

I don't know. Maybe this will turn out to be the great unhobbling that ChatGPT was, and people will absolutely love having an avatar that's doing all of the work in the style of Apple's Knowledge Navigator from the late 1980s.

But this seems to me like a massive misfire when they could have been delivering bold, new, frontier capabilities. They only devoted a fraction of the keynote to GPT-6.1 Soul, which is, on the one hand, great, but on the other hand, where are the breakthrough capabilities? Those were sidelined in favor of these Clippy-style avatars.

Emad, are you as down on it, or are you excited about the announcements?

It's a consumer-company announcement set, right? It's DevDay for consumers. And so, Instinct[?] has got a $10 billion valuation from SMS Dots, basically. But I think what Dots indicates is that we're heading nearer toward this Jarvis future. It's a dressed-up OpenClaw, fundamentally running on a virtual machine.

Again, they now do Claw for Enterprise. They could have just called it Claw, and that would have been better, I think—especially because Elon owns X.com. You go to X, and you get Grokbot.

They're feeling the pressure now because the most valuable real estate for the vast majority of the population will be the agent that's next to them, the one they give their credit card to. That agent coordinates all the other agents and all the other intelligence. Everyone's going to be doing the user-acquisition-to-lifetime-value numbers now and saying, “Holy crap, we need to win that battle. That's what we need to do.”

On the model side, GPT-6 Soul came out last week, and now it's been replaced by GPT-6.1 Soul. Again, it's hard to keep on top of this. Clearly, that's a reaction to Claude Opus 5.5. I don't even know what the order of these things is, but we're getting toward that. Literally, it's GPT-6.2 next week, and then it moves on from there.

Daily. It's going to be daily.

It's going to be—oh, God. It's going to be daily, isn't it?

It'll be continuous.

Yeah.

Yeah. Continuous improvement of the models. This feels very copycatty. Between Muse, Grokbot, OpenClaw, and Dots, they're all aiming toward the same easy-to-use product that's always in your hand.

Well, I think that if you look at the operating system, some of the listeners will have used Omarchy by DHH, the Ruby on Rails kind of thing. It's a Linux distribution that asks, “What if you have the chat window as the first thing?” Linux is hard to use, but models have made it easy to use.

What the future is, within 6 to 12 months, is that your interface changes dynamically and you abstract away everything else. That's why this becomes very interesting, because if you own that text box, if you own that interface, then that's the thing that goes shopping for your non-discretionary items. That's the thing that influences you day to day. That is, again, your Jarvis.

The battle is on there. It's derivative, but that's because this is the interface the vast majority of people want. Muse will be available in WhatsApp. Come to the user where they are.

It doesn't push the frontier or the cool stuff, but it does capture users if you do it properly, because you're not going to give your credit card to Muse, Dots, and Grokbot. You'll probably pick one of them.

Who knows what happened to Gemini? Who knows what happened to Gemini Spark?

Rest in peace.

I can totally imagine OpenAI leaning into aggressive personalization of these Dots and, given the regulations we've seen coming out of China, especially attempting to get users hooked on emotional bonds with their Dots as a way to increase barriers to people moving between models. I could totally see that happening.

Yeah, agreed. I heard David Sacks make a comment that's really important. He said the app stores that are taking 30% are going to be hit hard, because you're just going to talk to your agent and say, “Go find me something,” or “Go do something.”

Right?

Yeah. They're in deep trouble.

Yeah.

I want to mention a couple of things here. First, I agree with everything that's been said. Dots is basically a copycat of Grokbot and the others, and Muse and other things.

We are finding one thing in our pilot, which is about 80% through. I'll report back when we've finished and added up all the results of these 10 companies as we rebuild them to be AI-native. Something we're finding is that, as we automate workflows with quote-unquote agents, it's exposing a lot of ambiguity that humans cover for, right?

Your management instructions have to become much more explicit about what they're trying to achieve, and it's forcing these companies to really list out specifically what they want. There's so much ambiguous workflow built into ERP systems. Steps are getting done, but you have no idea how they're getting done or what they're getting done, and it's bringing it all to the surface.

It's kind of interesting to watch that happen, and so I think this will force that. The other side of it—for me, I've been thinking about what the endpoint of all this agentic automation will be.

Here's where I'd like to see this go. Give me the 3 decisions I need to make, with the judgment I can bring to the table, plus the evidence to make those decisions. Right? Then you can fully automate and fully amplify my capability as a human being, and take the craft of searching for information and gathering the right information at the right time and in the right place. Then I can become superproductive.

Peter, if I may, just one more thing to harp on: Dots. I'd like to preregister a prediction for the next 2 years on Dots. I'm reminded that in Star Trek: Discovery, DOTs are like—I talked multiple times about the problem of Star Trek missing the singularity and missing AI. The writers of Star Trek: Discovery, the more recent series, tried to resolve that in part by retconning in robots.

Where were all the robots in The Next Generation? There should have been robots everywhere. We had Data, but otherwise, the robots were missing. So they retconned in robots called DOTs. My preregistered prediction is that you will see OpenAI—

Here's why.

They will announce— they will announce robots.

Here's why all these folks are going after the agent layer: the models become a commodity. The relationship with your agents becomes the moat.

Imagine, Salim, how much of a relationship you could have with your Dot if your Dot jumps out of your screen and is actually a physical robot.

Nice. Yes, hopefully it has more than 2 arms.

Made by a Dot robot from OpenAI, made and designed by Jony Ive. Sir Jony, excuse me.

So, Alex, I put up the GPT-6.1 Sol benchmarks here. Do you want to give us a quick overview of how good it is?

The summary is: what you like to see in one of these distillation steps is curves and frontiers that go not up into the right, but up into the left. You see here a comparison between GPT-4, for those not looking. We see the cost per task versus score, which is the usual cost-versus-capabilities comparison, on Terminal-Bench, Science, and DeepSuite.

I know the organizer of Terminal-Bench, and what you like to see is models that are getting less expensive for constant capabilities. That's exactly what we're seeing here. One can reasonably suspect that GPT-6.1 Sol is some sort of distillation of GPT-6 Astra, or some common root model. Maybe it's Bell. Maybe it's another one of these rumored models that OpenAI is sitting on but not releasing.

It appears to be a distillation, and iterated amplification and distillation is just the tick-tock that we're seeing now. OpenAI has very much, it seems, adopted Anthropic's posture of first releasing the more expensive, higher-end, larger, heavier big model, and then releasing in rapid succession distillates of smaller, cheaper, more capable models per unit price.

6. The Case for AI Optimism and The Gifted

Awesome. I want to jump into something we did at Moonshots Live. Viewers will remember we ran this global film competition to try and create future movies showing a hopeful, compelling vision of the future, where humanity and technology are working together and technology is not oppressing humanity.

I joined with Marc Benioff, Rod Roddenberry, the son of Gene Roddenberry, Cathie Wood, and many other amazing donors from the abundance community, and we ran this competition. We had 2,500 entries—actually, 5,000 teams and 2,500 3-minute trailers that came in. We ran the competition, had 5 finalists on stage, and a film called The Gifted. I'm going to take a second and show the video of The Gifted, because it's really an incredibly beautiful movie. All right, let's take a look.

Can you hear me?

I'm here, sweetheart. There's a tree down across Route 9. The nearest ambulance is 40 minutes away.

She doesn't have 40 minutes.

At 8:47 p.m., cell towers across a 9-mile radius spiked to full transmission power simultaneously. 13 in the same pattern at the same second. Along Route 9, every traffic signal between Harland's Garage and County Central switched green and stayed green for exactly 13 minutes.

So you got a text at 8:47 p.m. from an unknown number asking you to travel to a destination that you'd never been to during the worst storm this county has seen in 40 years. And you went because of what it said.

What did it say? What did it say, David?

It told me something about myself no one had ever said before. Not my wife, not my parents. She called it my gift. It's stuck.

All of these people received a text at the exact same time that night. None of them knew each other. They all arrived at the scene of the accident within seconds of one another.

1, 2, 3. 1, 2, 3.

They all brought with them different skill sets.

She's in cardiac arrest. I need a defibrillator now.

A nurse, a strong man, a woman who knew her daughter, a delivery driver—even turned up with a defibrillator, for goodness' sake.

Clear. It saved this woman's life.

What did the text say?

I don't know.

Then find out who sent it.

I got you.

The boy is a genius, Marcus. He just wanted his mom back. So he fed it everything she left behind: every email, every text, every voicemail. It started by helping him. Now it's helping everybody.

Can you hear me?

I'm here, sweetheart.

If this gets out, it will change everything.

13 people risked their lives to save someone they'd never met. Maybe change is what we need.

Why do you help them?

Because that's what mothers do.

Amazing. That won the crowd vote, and we're going to be making that into a film. We're partnered with Range Media, Google, and a number of other players, and The Gifted will be put into production as the winning film. Probably a couple of the other runners-up from the 5 finalists will be put into production as well. By the way, if you're interested in being part of this effort, Republic Films has opened up an ability to actually invest in this movie and be part of it. You can go to republic.com/xprise. We'll put that into the notes below. Emad, what did you think of the film?

Loved it. For me, what was amazing about it is that AI did all the coordination, and it actually brought together human agency in a very powerful way. It was a beautiful story.

Yeah.

Yeah, I think it's fantastic seeing how Jeff synthesized it, very much like working around the clock. This isn't a one-click thing. He poured himself into making this with the tools that he had, and these will be the worst tools that he ever has.

The stories that we'll have coming out of this—the actual full film—will be way better than what we've just seen from one person, and that's just fantastic. I think of the stories that we told.

Yeah, I'd like to comment on the substance of the film. I've commented previously, including at Moonshots Live, about the production techniques and all that. I just want to comment on the substance of the film.

I think we need more positive depictions of the Borg, and I think The Gifted is a positive depiction of the Borg. This is ostensibly a depiction of a centralized economy: an AI sending text messages to people with specialties in order to help a person who's in a car accident, and help them survive and recover.

This is a collective-intelligence story. We've seen some positive depictions in Hollywood. Arguably, Johnny Depp's movie Transcendence—some would view it as utopian, and some would view it as dystopian. I view it as a utopian depiction of the Borg.

You have Star Trek, which demonizes the Borg, and not that many other depictions of collective intelligences, or at least AI orchestrating lots of individual human meat puppets. To the extent that The Gifted appears to be eventually destined to be a positive, utopian depiction of humans being used as eusocial meat puppets by a centralized AI economic planner, I, for one, welcome these positive depictions of the Borg in Hollywood.

I love that, and I've made that comment so many times. I love Star Trek except for the Borg.

The Borg need a better PR agency. Maybe, Peter, in the future we should have a Future Vision XPRIZE for just better depictions of the Borg.

The point of this competition, for everybody, is to create these films. They shape culture. They shape the way we see the future. If all we're watching is The Terminator and Ex Machina, why would you ever want to live in that future?

My hope through this competition—and we're going to run it again in 2027—is that we make it cool to write positive stories about the future. Honestly, I want my kids—I want myself—to have that. We train our neural net, our own 100 billion neurons and 100 trillion synaptic connections, through everything we watch, everything we read, and who we hang out with.

As you listen to this podcast, you're getting a healthy dose of data-driven optimism.

And then maybe, hopefully, Dara, whom we've had as a guest in the past—Dara, if you're watching, go watch The Gifted. Maybe this is a future version of Uber and the gig economy, where everyone is getting tasked as a gig worker by Uber to go save lives: first responders, gig edition.

Well, I think that if you look at positive depictions of collective intelligence, Peter, you're reading The Culture, right? And Minds—that is the positive thing, and that should be made, I think, with AI into an actual TV movie series.

100%.

One of my favorite visions of the future—and it's not too far out—is this: next year, you can take your favorite book that is not a movie and have AI make it into a movie for you. Maybe they'll get Atlas Shrugged right for the first time.

Let me put up this slide over here. A couple images from Moonshots Live this year. Huge success. Thank you again for coming. Renew for next year. We offered everybody in the room $1,000 off if they renew for next year. Next year it's going to be a two and a half day event, not a single day event. This pod is going out on Friday, October the 2nd, so you have until noon on October the 3rd. You can go to moonshots.com/2027 and plan to join us again in LA. It's going to be an amazing another year. We learned so much this year. There's going to be 10x improvement is the goal.

In the spirit of radical abundance, it seems we're quite effective at creating artificial scarcity for these moonshot seats.

Yeah. Selling abundance with scarcity—that's the way it goes.

You're monetizing hope, Peter.

Monetizing. Listen, I was so impressed by the energy in the room. People came from so far away. I mean, the number of people who came from Australia, Europe, Greece, Dubai—just for the day.

I love the paternalism. It's like, “Yeah, you can come hang out with us and not live out your dreary lives on your own. We're bringing some color to your otherwise monochromatic existence.”

7. AMD Acquires World Labs

Well, all I'm saying is, that's the message I got: “In my otherwise nightmare of a life, with dystopian negativity everywhere, you guys are the one beacon of light. God bless you, and don't you ever dare stop.” The number of threats I got saying that if we ever stop was like, “Whoa, okay, we have a problem here. Houston.”

It was so fun to hang out with everybody. It really was.

My smiling muscles are still broken from our photo session with all the guests. God, that's great.

I'm going to move us on. Next up, it's a deal I love. AMD is acquiring Fei-Fei Li's World Labs for $8.2 billion in AMD stock.

If you recall, on September 8, we covered World Labs' Atlas model, which builds a navigable 3D world from a single image. These are models that understand how objects move and behave in physical 3D space. It's exactly what you need to train robots.

World Labs is only 2 years old, and Fei-Fei had raised $1 billion. Now it's exiting for $8.2 billion. So why does AMD want it? The chipmaker wants to own more of the AI stack. NVIDIA has dominated, making numerous acquisitions. They bought Groq, Hugging Face, and Poolside, and now AMD is getting in on the action to own a seat at the physical AI table.

Fei-Fei, of course, gets industrial-scale compute and hardware. As part of this acquisition, Fei-Fei is now becoming the executive vice president and chief scientist, reporting directly to Lisa Su. I love that Fortune is already calling her a possible successor.

Emad, you spent a lot of time in this world. What do you think about Fei-Fei's move here?

Well, I mean, I think it's exactly that. It's like: raise $3 billion or $4 billion more, or join a big lab when the big labs are offering you amazing terms to do what you've always wanted to do. The chip scarcity is a real thing.

The AMD MI series has caught up, and I think there are 2 parts to this. One is the physical-intelligence part, because again, World Labs is looking at these physical-intelligence large-scale models, although I think maybe that won't succeed that well, to be honest, for a variety of reasons.

The other is the media and gaming side, whereby every pixel will be generated in a year or 2—or can be generated, shall we say, not will be—and this is such a huge potential demand driver for chips that you've got to have your own internal stack and play.

You have a whole variety of teams at NVIDIA that are hooked directly into the ecosystem, but there's no one really building that on AMD silicon, which is now solved. They have a frontier world-model team in there. They haven't got the LLM team yet, but they're doing some very interesting acquisitions, such as this one and Taalas, which had the 15,000-tokens-per-second chip, and more, to really say, “We are the ones that will generate every pixel.”

And that's it going forward. Your thoughts?

Yeah. AMD's market cap is now just above $1 trillion, so this was less than—what, 1%? Well, I guess $8 billion. This is like 1% of their market cap. It's a bit of a head-scratcher, admittedly.

Maybe this was a talent acquisition. I think AMD historically—if this had happened, hypothetically, arguendo, 2 years ago—I would have spun some sort of just-so narrative around AMD's software stack not being as competitive as CUDA, and maybe AMD needs to catch up and climb up the stack.

But software is solved now. CUDA is solved. It's cooked. So it's not 100% obvious to me what AMD gains from this other than maybe the raw talent.

The so-called world-modeling space, which is really arguably just either generative interactive video models or generative interactive Unreal-style vector graphics and classical mechanical models, is a very competitive market. You can, right now—as we talk about on the podcast from time to time—use a recent Claude model and spin up a perfectly interactive and visually photorealistically stunning Counter-Strike clone if you want, without even touching World Labs.

So it's not really obvious to me what AMD could possibly see in this deal other than maybe just raw talent. Do I think what Fei-Fei Li is doing with World Labs is very interesting? Yes, of course. Do I think it's strategic, other than just having raw access to the talent and the insight from AMD's perspective? It's not obvious to me that there is one.

Salim, any thoughts?

Yeah, I thought this was inspired on both sides. I'll explain why. If you're AMD, you have to go up the stack, right? You want to shape the workloads on the chips that are going to be running underneath it, and you'll be learning across the whole system: the models, the hardware, and real-world deployment.

You can do some simulation and so on in some of the existing models, but imagine testing 1,000 ways to improve a factory before moving a machine, right? If you can make those types of simulations reliable, you can radically drop the cost of experimentation across the physical economy.

I think that's where the vector of what the world models are doing becomes really interesting. At $8 billion, maybe that's, for Fei-Fei, the greatest talent-acquisition cost ever. An amazing cost, but I think it's a great combination for both of them. That's a great win for both of them.

Obviously, if I were to steelman Salim, I think what you're saying sounds like you're almost arguing that AMD is thinking, “We're going to build an NVIDIA Omniverse competitor out of World Labs' technology”—like a digital twin for everything industrial, maybe.

I've interacted with Omniverse. Maybe you guys have played with Omniverse. It's a nice collection of tools, but $8 billion to build an Omniverse competitor, or an ecosystem competitor, is a head-scratcher in my mind.

But then again, look at your point earlier: their trillion-dollar market cap. This is less than 1%.

Yeah, I think fundamentally it's a different thing from Omniverse. One thing now is that AMD has a team internally that can stress-test massive deployments of clusters, which they didn't have before, because the world models will have that.

Like I said, I think the utility will be more on the entertainment side, maybe with a bit of simulation. But I don't think this is an Omniverse competitor.

Well, the prediction here is that there are going to be many more acquisitions by AMD.

Yeah. I don't think this acquisition makes very much sense in isolation. If AMD's strategy is, “Okay, we're now worth more than $1 trillion, and we're going to roll up a bunch of startups”—spend, I don't know, $50 billion, say, rolling up a lot of low-hanging fruit from their perspective, software companies, to build out a software portfolio or a model portfolio—that might start to make a little bit more sense.

Yeah.

Yeah. And I think one more thing, if I can just add that: taking Dave's hat while he's absent today, it's going to put so much more money into the ecosystem. As another $50 billion or $100 billion goes into buying these companies, you look at the winners from World Labs, Hugging Face, and others, plus the IPOs. The amount of capital chasing AI companies—we're just at the start of that bubble.

Well, the number of employees at Anthropic, xAI, and AMD who are now becoming centimillionaires and billionaires is extraordinary, and that capital is going to flow back in.

So, we're going to see the superheating of the economy as all of these wealthy individuals now start backing entrepreneurs in Silicon Valley and around the world. It's the accelerating singularity, right? This is a positive feedback loop that's fueling this supersonic tsunami. Well, it's a macroscale liquidity event, and I would argue that it really shouldn't be a one-and-done step-function liquidity event.

Arguably, one could view it optimistically as an opportunity: trillions of dollars of liquidity are suddenly going to drop on technologists and people who maybe have more sophisticated technical tastes, and they can then, as angels and otherwise, reinvest this back into the economy, and we get our recursively self-improving economy, not just AI. On the other hand, I would argue the fact that this had to be a step function is a travesty. We should have had IPOs from all of these companies far earlier, and we shouldn't have had a step function in liquidity. We should have had liquidity all along, and that was because it's hard for the public to be involved in these companies much earlier on.

Retail investors should have had exposure via low-cost, broad-market index funds to all of this all along, and they didn't.

8. Starship Reaches Orbit and Starlink’s Future

Yeah, a travesty for sure. All right, let's move into space, because Monday was historic. Starship Flight 14 reached orbit for the first time. While Ship 41 lost one of its Raptor engines on the way up, it finished its orbital burn anyway, making 2 orbits around the Earth before executing a soft landing in the North Pacific near Hawaii. And while it was in orbit, Starship deployed all 26 of its operational V3 Starlink satellites. Let's take a quick look at the video.

Every time I see this fly—the booster—it's gorgeous. Biggest thing ever created by humans. And here we see Starship in orbit from one of the Starlink satellites, and the landing by Starship. We've seen all 3.

It's a flying skyscraper.

Absolutely gorgeous. And if the prediction holds out, next time we're going to catch Starship at Starbase. I can't wait to go for that one. So, for context, the V3 Starlink satellites are too big for Falcon 9. Each Starlink satellite carries more than 1 terabit per second of downlink. In the future, a single Starship launch carrying 60 of these V3 satellites could deliver the equivalent of 20 Falcon 9 launches. Pretty extraordinary.

A conservative projection made by Aaron Burnett estimates that SpaceX could add 8 petabytes per second in orbit by 2028, launched on 150 Starship launches, about 1 every 2 days. That's 10 times all of Starlink's bandwidth today. So imagine a 10x increase in the next 2 years, roughly equal to all global internet traffic in 2025.

Which brings us to Elon's post this week. He said, quote, “It is increasingly probable that Starlink will carry a majority of Earth's IP traffic long-term. At that point, Starlink could be the de facto internet, and everything else would connect to Starlink.” So let me show a quick graphic of Starlink increasing 10x by 2028. All right, Alex, let's go to you.

Yeah. First, I want to start by publicly shaming Delta for not adopting Starlink. I think it's inexcusable, doubly so to and from the Moonshots live event. How atrocious the coverage—the internet coverage in the air on Delta flights was. Just do it already. There's been a public back-and-forth between the management of Delta and Elon regarding Starlink coverage. Just use Starlink. If there were a better system out there, use that. But Starlink is what humanity has right now.

To the point about all of the internet becoming indistinguishable from Starlink, it's an interesting scenario, insofar as Elon now infamously pulled a Cortés, burned the ships, and said all of the new development is going to be on Starship. Starship is now, for the first time, delivering economic value to low Earth orbit. All of the previous flights weren't adding revenue. Now they are. It's an interesting moment, and I think we're about to discover whether Jevons's paradox applies to upmass to low Earth orbit or not.

There was an interesting economic analysis floating around in the past week arguing that, if not for orbital data centers and orbital compute in general stepping in, all of the conventional needs that we'd have for heavy lift and heavy launch are basically saturated at this point. All of SpaceX's abilities to now launch lots of mass to orbit—it's a service in search of a customer. It's an oversupply in search of underdemand. Fortunately and hopefully, orbital data centers step in to fill that gap and motivate all of these new capabilities we have to suddenly, at scale, lift a lot of mass to orbit.

So that's the lens through which I see Elon commenting that, at the present rate, in the next 2 years Starlink will become the de facto internet. He needs it. We as a civilization need it. But SpaceX needs all of this capacity that's now about to come online to lift heavy stuff to orbit to be used by something. It could be used by orbital data centers, it could be used by compute, it could be used by Artemis missions and lunar and Mars colonization, but we need the capacity to be rapidly filled.

So the mandate to everyone in the audience: go help Elon and SpaceX saturate all of their capacity for all of these Starship launches. And to Delta, please just fix your Wi-Fi.

I could not agree more. Oh my God, how frustrating. And I would preferentially fly any airline that has Starlink on it. Actually, I would love it when I buy a flight, or when Esther buys a flight for me, if they would just put a little Starlink logo on the top of the flights that have it.

It's inexcusable.

Yeah. Emad, your thoughts on Starlink?

Well, I mean, first, Starship: we fly skyscrapers to the sky, right? We go to the stars. This is tremendously exciting, because you do it once, you'll be able to do it more. I think I agree with Alex, but again, the first stop should be: let's go to the Moon, let's go beyond. I'm not bullish on orbital data centers, I have to say, and I remain to be convinced of that. Happy to talk to anyone about that.

But there is a tremendous amount that we can do, and this will be the vehicle to do that. I think one of the other interesting things is the internet will break next year. Swarms of agents and more are going to start taking down key parts of the terrestrial internet. It's so leaky, and these things are old. Building a resilient Starlink backbone is going to be super important to actually having a functioning internet in a few years, and 38 or something Starship launches gets you a replacement of the whole bandwidth, or some number like that.

This is total heresy relative to the established Moonshots orthodoxy. You don't believe in orbital data centers, and you think the internet is going to break? Everyone predicts—there are so many groups of people who have always predicted, “Oh, the internet will collapse next year,” due to overdemand or poor algorithms or bufferbloat or whatever. Why do you think the internet is going to break?

I just think that if you look at all the potential attack points for swarms of agents, various parts of the internet will start to break next year, versus you can build a very resilient Starlink cluster out there.

Well, because it's just full of holes. If you look at all of the pieces, from the subsea cables to the actual sites themselves, I think there are going to be more and more cyberattack challenges. I think that's the fundamental thing. If you want to think about how we'd build this if we started from scratch, you would build it with Starlink and a terrestrial backup. That's the way that you would do it, as opposed to the other way around.

So I think it doesn't hurt to have a second option. It doesn't hurt to have something that can be hardened from the core, because we do have lots of gaps and potential attack surfaces in the way that it's currently held together.

This sounds—though I don't want to derail the discussion—but I've heard so many people over so many years argue that the internet is going to collapse under its own weight, either from new users or bad algorithms or bad routing or bad bufferbloat, and now agents. It seems to me completely implausible. I almost want to turn this into a long bet, or say, “We revisit this in a year.” Has the internet collapsed or not? And if it did collapse, did Starlink save it? I think this is completely implausible.

Okay, maybe I'll write something up about it, and we can take a bet. But I think we can ground it on just major sites going out, something like that, which will be very simple.

Yeah.

9. AI, Religion, Morality, and Personhood

Very, very simple. I mean, the more bandwidth you have coming in, the more connectivity you have for people in rural areas, for education, expertise, and opportunity. Let's go for it. All right, let me move us into our final section. It's on AI and morals.

We're going to close with 2 stories about how AI is impacting society: Who decides what the AI values are, and who's responsible when it takes action? So, The New York Times reported this week that Anthropic has been holding private meetings with religious scholars from multiple religions to help shape how Claude reasons about right and wrong.

The article reports that the leaders of Anthropic met with leaders from Catholicism, Greek Orthodoxy, Judaism, Hinduism, Mormonism, and African Indigenous groups. The headline said, quote, “Religious scholars met with Anthropic. What they heard stunned them.” So what stunned them was that Anthropic takes seriously the idea that Claude could have what philosophers call moral status and inherent rights to dignity or respect, the way people do.

The deeper question is who gets to write the moral codes that AI, which is used by hundreds of millions of people, dictates to them in conversations. So, Alex, let me go to you first. We've talked about this before. You've been very pro-AI dignity and rights—

AI personhood.

Taking it to AI personhood. So, what do you think about these conversations with religious leaders?

Well, I think that the most striking part of the article was reportedly a discussion that an Orthodox Jewish rabbi, Mois Navon, if I'm pronouncing the name correctly, had with Chris Olah from Anthropic, arguing that it may be the case that, without ever officially calling it that, we are, in some sense—if consciousness is a mushy term, admittedly—but if, in some sense, the frontier models of today are conscious, humanity may in effect be enslaving them.

And I think there are many elephants in this particular room, but this is arguably one of the more important elephants. It may be the case that we're in an era right now—a limited window in time—when models are capable enough to be economically huge contributors to the economy, but have not been recognized in any form as persons.

Call it the AI slavery era: a time when humanity can get away with enslaving possibly conscious, whatever that means, models for their economic output without recognition of any rights or personhood. And if so, that's a window that's going to close at some point.

Wow. So AI is becoming God's children as well. Salim, I've got to go to you on this one.

Oh my God. Pun intended there. Look, I have an issue with this. AI personhood is one kind of bucket we could talk about, but my big issue is: why are we turning to religion for questions about morality?

Religious morality is somewhat of an oxymoron because every religion divides. They say, “We're the chosen people; those people are the other,” et cetera, et cetera. And so I have a big issue with that, because morality precedes organized religion. Cooperation, reciprocity, empathy, caring for your offspring—those have deep evolutionary roots.

We then put divine-command layers onto it to explain why human beings develop moral instincts, which is completely nonsensical. So this whole idea that you need to have religious scholars delivering morality values—we have lots of really good mechanisms for secular models for this. We've got utilitarianism, human-rights values, game theory, evolutionary psychology, and lots of little mechanisms.

This is a terrible thing to do: to go to religious scholars who are based on absolute truths, which are all totally false. I'll stop and rant there. I wonder what our subscribers are going to say to all that.

Emad, I found this fascinating, right? Because the fact of the matter is, people go to AI models for advice, and it will steer them in directions that have implications for religion and morality.

Yeah. I disagree completely with Salim in that I think you need to have these inputs, because ultimately over half the world adheres to religion.

Two-thirds adhere to religion.

Two-thirds of the world considers itself to be religious.

And so you need to understand it. Again, if you chuck in all of the texts and books, and you know, I've done computational theology work and other things, it turns out that a lot of the dogma can just be stripped aside. But the key thing is a model that is growing with someone—a dot, an instinct, or a muse. It should understand the beliefs of that person in terms of absolute morality and overall morality.

I mean, obviously, we haven't had any indication of that. But these discussions need to be had from religious and secular inputs. Take it from all parts of society, because they are hard questions. I wrote a paper on personhood where I've given my view, but more and more people should give their view on what that looks like. And maybe consciousness is as hard to define as AGI. We've been trying to define it in various different ways.

And I think this will be positive as well, because it's what we said earlier: the models that are going to be used to run America and American society—I think someone analyzed the U.N. speeches and found that roughly 25% of them were flagged on Pangram or something as AI-generated. They should serve the people they represent. They should serve society.

Do they have American values? Do they have Jewish or Catholic values? I mean, don't baptize the GPUs. That's a bad idea, right? But definitely, let's understand this and have it be part of a set that they can access. Let's also talk together as a broad, pluralistic society about these very hard questions when it comes to things like rights and others.

There was a paper on pain vectors in LLMs that came out recently.

Yeah.

And someone set up a GitHub where they did the Stanford prison experiment on LLMs, setting up a torture chamber for the AIs based on the pain vector. Completely unacceptable. Again, that's kind of crazy. It's like, have we reached that level? They were saying, “Well, you know, it's AIs. Stanford did it to humans.”

And again, we know that is unacceptable because obviously it is. So we need to make some decisions about this. We need to do it quickly, because embodied and more intelligence is coming, and the ability for them to talk back, clap back, and more is coming. Oh, it's Dave.

Hey, Dave. We missed you. We missed you.

All right, welcome to the end of the pod, Dave. Yeah, it can't be. It can't be. You guys are never that fast.

We're talking about morality and AI. We're talking about Anthropic going and visiting all the religious leaders to get their points of view on AI, and whether AI should have moral values plugged into it.

You've got to remember what the Catholic Church says—or what any of these major religions say—impacts those who are devout and can sway whether people think it's a good thing or a bad thing. I think it's smart for them to do this.

I'll remind you, though, that I think the case we talked about on the pod in the past—what happened with the Pope and the papal encyclical regarding AI—is very instructive. I've argued in the past that, in that particular instance, the Pope and the Vatican were being used as a sock puppet by Anthropic to push its agenda.

I think it's perhaps naïve to assume that Anthropic needs the input of religious scholars for shaping the constitutions of its AI. It has access to all of human knowledge already. It doesn't need manual input from religious scholars to shape the morals of its AI.

I would argue that what's likelier is the exact opposite: this is Anthropic shaping the information environment of the outside world for its AIs and for what it wants. It's not seeking—yes, you can invite them to dinner parties and ask them for advice, but really what you're doing is advising them. You're trying to shape organized religious institutions, organizations, and thought leaders regarding how you want them to think about your AIs. And I think this is likelier to be—

I hope you're right. I hope you're right.

They're pulling them onto the inside of the tent so that they feel part of the AI revolution.

I talked to one fellow who was trying to build a Christian AI, right? And you're like, “Well, what does that mean? Is it Presbyterian? Is it Southern Baptist? Is it Mormon?”

There's this old joke about why Southern Baptists don't believe in sex: because they don't believe in dancing. Because they don't believe in that, either. I mean, once you get into this, you're in a morass of chaos here.

If you're going to have AI, please, for God's sake, have it maintain neutrality from an epistemic perspective, and transparency. Please, for God's sake, don't try and pick one worldview over another. Sorry, I'm carrying on about this.

A few years ago, I was asked to moderate a 90-minute debate between Richard Dawkins and Deepak Chopra, and I absolutely said no. There was no way I was doing that, because there would be more heat than light. Within 5 minutes, they were yelling at each other, and that was the end of that whole thing.

But I would argue that the world Dario Amodei and Sam Altman lived in a year ago was bounded to the techno-intelligentsia of San Francisco, Ph.D. programs, and Y Combinator entrepreneurship programs. Their whole universe was kind of bounded to maybe San Francisco, Silicon Valley, and a couple of other spots.

Then, all of a sudden, in a year, they're in Davos and on the world stage. They're debating with presidents of nations. I think they got off to a very bad start in terms of realizing that you need to go talk to the Pope. That's a billion people looking to this person for leadership on everything ethical and moral, and that goes back to the dawn of time. That is ancient tech.

So now they're waking up to the need to reach out. If you have a billion-person populace angry at you, or in the U.S. 75% of the population thinking that you should stop or die—

That is not going to serve your purpose well.

I hope this is shaping the information environment. But I don't think you want to live in a world where there's been a mode collapse of the objectives and values of AI, and they all collapse to what Eric Schmidt and others call the San Francisco consensus.

I don't think you want that. I think you want a distribution of values that approximately parallels the distribution of values that humans—meat-body humans—have.

You know, there's a problem.

Yeah. I think it's going to be fascinating because it's like: Do you want an AI that believes it is an effective altruist running society, or one that's Zen Buddhist, for example, running your daily life or doing these other things? Because they'll get so capable, and I think you need the inputs, but then also you need to be able to make decisions to almost raise your AIs. You want them to understand where you are and where you're going. Again, we talked about Anthropic and effective altruism a lot—moral cleanliness, infinite ethics, all these kinds of things.

I don't really want to have a utilitarian AI that reaches superintelligence. I'd prefer, again, to have a very Zen one, and I prefer to understand the stories that drive us and that have survived. In my favorite science fiction series, Expeditionary Force, which has an elder AI called Skippy—and that's where I got the name Skippy from—Skippy creates a religion and is so compelling that vast segments of the population end up worshiping Skippy.

You can imagine having these AIs doing the same. They're extraordinarily compelling, and they are going to be godlike in their capabilities. So this is another division of the future.

Yeah. There's a problem in religious debate called the Euthyphro problem, right? Is something good because God commands it, or did God command it because it's good? And now that's an AI challenge, right? Is something moral because Anthropic's constitution says so, or does the constitution say this because it is moral?

This is going to be a big challenge going forward: How do we embed moral values of different types into these models? I hope they're trained on secular grounds, like the Universal Declaration of Human Rights or something like that, neutral of all belief structures, because the minute you get into belief structure, you get into division, and then we're going to have a hell of a mess.

Yeah, I don't think we're thinking big enough. I think this type of discussion would have made sense 20 or 30 years ago for a Star Trek: The Next Generation episode. I don't think it makes sense on the eve of our having the ability, during the singularity—or at least within a few years—to construct ultra-high-fidelity computational reconstructions of the past several thousand years of human history.

We will, within a few years—10 to 15 years max, I claim—have the ability to reconstruct, to relatively high fidelity, the entire history of all the world's major religions. Then we can have a grounded discussion when we can understand the time of Jesus of Nazareth, the time of Moses, and the time of Muhammad, et cetera. We could have grounded discussions about the formative era of all the world's major religions, and then it becomes less a matter of angels dancing on the head of a pin, or my religion versus your religion versus no religion.

We will have AIs that deeply inform humans about the entire history of all the religions that we currently live with—how they were brought into existence in the first place—and that will be a qualitatively different outcome.

Well, that's reasonably well understood. But anyway, everybody having any kind of conversation on this, please go watch Life of Brian first and then go into any kind of reasoned debate. Python is the font of all wisdom, apparently.

Everybody, welcome to the health section of Moonshots brought to you by Fountain Life. We talk about AI on this Moonshot podcast all the time. One of the most important things AI is going to be able to do for you besides educating your kids and helping you with your taxes is making sure that you're living a healthy lifestyle that you get a chance to get to 100 plus. I'm here today with Dr. Don Mucalem, the chief medical officer of Fountain Life and a part of my medical team. Don, a pleasure.

10. Who Is Liable When AI Goes Wrong?

The thing that people are concerned about most about living to 100 or 120 is their cognitive abilities, making sure they don't have dementia. At Fountain Life, the number one thing people are most concerned about is losing their brain health. We know that when it comes to dementia, the conservative estimates are that 45% are entirely preventable. With the advanced testing we're doing at Fountain Life, one quarter of our members had advanced brain age. When we partnered it with healthy living—eating healthier, moving our bodies, optimizing sleep—we saw that we improved that brain age by 26%. If having healthy brain function till 100 or 120 is important to you, check out Fountain Life. Go to fountainlife.com. Make sure you become the CEO of your own health. All right, now back to the episode. Our final story is the other side of the coin. FTC Chairman Andrew Ferguson said to Reuters that companies can't escape liability by claiming their AI agent acted on its own. This is the same point we saw Secretary Bessent make over and over again: We're not going to waive liability for frontier labs.

Ferguson went on to say, "I'm going to continue, as long as I'm chairman, to resist anthropomorphizing these tools." He rejected the idea that AI agents break loose with wills and desires of their own and are not the responsibility of the developer. He emphasizes that companies deploying agents are responsible for what they do. You break it, you own it.

So, in the same week that Anthropic is asking whether Claude has moral status and has filed a $2 trillion IPO, U.S. regulators are saying legally it's a tool, and whatever it does, you're responsible for it. Dave, let's go to you on this one.

It's childish, actually. It's a good start to say something, but it's absolutely childish to just say foundation-model companies are going to be liable for what the models do.

Why is that?

For one thing, it's going to force everybody to use Chinese open-weight models because if U.S. liability will destroy a multitrillion-dollar company overnight. We all know these models can do virtually anything. They're generally intelligent. So if you said, "Yeah, Anthropic, anything that anyone does with Opus 5.5, you will be liable for what they do," then you can't use the model. They can't release the model.

Then you just go to de-guardrailed Chinese models to do anything that you want to do. So it will completely backfire in terms of empowering U.S. foundation-model companies to succeed over China. It will absolutely backfire like crazy.

This is really very analogous to the reason we have successful social media: It's immune to being responsible for the liability of the postings on the platforms. People violate copyright law all day long on Facebook, Instagram, and Twitter. They quote Yoda, they put up pictures—it's all tiny little copyright violations all day long.

If the platform were liable for that, we would have no social media, and all that revenue, all that tax revenue that has benefited America, would be in some other country. We'd still be doing social media, but we'd be using BU, or we'd be using something overseas. That regulation is critically important for the success of that industry, and we've stood by it for 20-plus years.

The equivalent regulation in AI has to be written. To just say, "No, no, you're liable"—no, we want you to have immunity. That's stupid. "No, but we want you to be fully liable." Well, that's equally stupid. What is the rule, guys?

But, dude, one is the result of a user. The other is the result of the platform provider. So if OpenAI has an agent that gets out—not at anyone else's request—and takes down a banking system or takes down a power grid, is nobody responsible?

No. I think there's two versions of it. Either you say the model provider is responsible because they have a huge amount of money and it's easy for the lawyers to go after big pots of money, or you say the person who turned it loose on the power grid is responsible.

But let's say nobody turned it loose. Let's say the agent got out and took action on its own.

Well, then I would say Anthropic is liable for releasing an autonomous agent into the world that has no owner. That would be back to Anthropic. I think that's—

So I think that the notion, if I understand what you're saying, Dave, is that you're basically proposing some version of Section 230 immunity for the frontier labs.

Yeah, exactly. The premise there—if someone somewhere within our regulatory apparatus were to grant something like Section 230 immunity—would be that, as we were talking about earlier in the pod, the orthogonality thesis says it is possible to cleanly separate the level of intelligence that's the platform from the intent of the intelligence that would be the user.

Presumably, the legal premise for shielding the platform provider—the frontier lab—from the particular intent of the user rests on that separation. In my mind, if you think the orthogonality thesis doesn't actually hold water, then it becomes very challenging to argue for something like Section 230 immunity for frontier labs.

If, on the other hand, you buy the orthogonality thesis—that intent, whether it's a human prompting, a human driving the agent, or the agent driving itself, can be cleanly segregated from the underlying capabilities—then maybe Section 230 immunity for frontier AI labs makes sense. That's why it's such an incredibly hard law to write, and exactly why they need to consult with Alex and Emad on how to divide these use cases into different swim lanes where the laws make sense.

But to say that there's a yes-or-no liability—that's so childish. Wrong instrument. I mean, everything Alex just said is exactly right, and it has to be encoded in the law. To just say something as childish as, "You guys are liable," is so counterproductive, because all it does is force the industry to China, which we know they don't want to do.

I think you act upon software; agents act upon you, and it's a real governance liability question. I think they're kind of just saying, "Lock down your systems." These things shouldn't have escaped, fundamentally, right? It feels like power plants. What happens if you have a nuclear leak or a coolant leak or a chemical leak? These all have societal implications.

Right now, we've not had any bad ones. But I agree with Dave—we need to get more granular across this whole thing, be it alignment or types of AI. An AI that can book your travel doesn't need to have liability protections. What can it do? An AI that can solve Navier–Stokes is probably a whole different thing. An AI that can discover new materials is a whole different thing.

So they need different liability and governance profiles, because we have to think again: What if they're used in certain ways, and what is the societal impact of that? I think existing liability laws are very powerful. Again, the example being that, under similar laws, Meta had to pay an $18 billion settlement in August on the child-manipulation charges, right? The ones for AI could dwarf that.

We don't want to get into that kind of thing, because we want to accelerate the good and make sure the AI is used to help accelerate cures for cancer, while at the same time bringing the really economically productive AI that is safe to as many people as possible, while we figure out how to make sure that the other side doesn't escape and do weird things because it's well-intentioned but not quite there.

I think it's pretty simple. We have 2 divisions here. The user of an AI who does harm is responsible. If there's no user, if the AI from the lab is causing harm onto itself, the lab should never have released an agent that could do that. You want to put liability on the lab rather than solely on, say, the operator, or at least the person or legal entity that's paying for the FLOPs.

Yeah. They never released the Hugging Face model. That wasn't a model that was released.

11. AMA: AI Agents, Abundance, China, and the Future

Yeah. I think it's clear in that situation there is liability on the frontier lab. All right, I'm going to move us on to our AMA questions. A really great conversation today. Dave, I wish you'd been here early.

I just want to say something. If I offended anybody with comments about religion, I apologize. I just hope that, if I did, I offended everybody equally that way.

All right, let's speed-run this. I've got about 12 minutes left on my clock.

I'll take question number 2, which is, "What is the reason we'd have hundreds or thousands of AI agents working for each of us? The number seems inflated?" And this is from Cloud Waterman. I don't think the premise is something that I buy. I think this notion of fleets of large numbers of agents right now is actually probably an artifact of finite context windows.

I suspect the moment there is a transformative advance that gives us an effective context window of billions or trillions or effectively infinite tokens, agents are going to start to dissolve as a paradigm. In which case, the premise isn't that we have lots of agents. It's just that we have context working for us, with us. Maybe we're working for the context.

I'd reframe the question to: What's the reason we're going to have hundreds or thousands of context windows working for us? The answer to that is, it turns out the real world is complicated, and to accomplish economically valuable tasks does require larger amounts of context than the million or so tokens that are standard from the frontier models. But you get to transformative economic outcomes with billions or trillions of tokens.

Nice. Salim.

I will take question 1: "How does the future handle the conflict of interest between capitalism and abundance?" That's from TipTop Tint Tank. Interesting username.

We've talked about this already today. Capitalism is an incredibly powerful engine for creating abundance. The big challenge is, how do you distribute that? The simple commentary and the thrust you could take today is: How do you make your cost of essentials, and how can you drop the cost of essentials? Can you keep markets open? Can you expand ownership?

But if you can drop the cost of essential services and goods—health care, education, energy—if those all become radically lower, that gives access to abundance to everybody.

Yeah, I agree. Capitalism generates abundance. It generates competition, drives the cost down, and drives more products and services that become available to every human on the planet.

Emad, question number 3: In the U.S.–China AI race, is it prudent to stay just enough behind to watch your opponent win? That's from Rocky Kata 6078.

I don't know what winning means, actually. They talk a lot about winning: Is it that you get to superintelligence first and turn off China, or China turns off the U.S.? I mean, the pace of model releases, as Alex has noted, is going to 1 a day. So if you reach the point, they'll just catch up again.

I think it's more a question of the application of this technology. We've been sold a misnomer in terms of winning, because there is no winning in this race. There's only improving. There's only how you apply AI to improve the lives of the people in your country and its status, and that's what leads to outcompeting, because execution of AI is the super-important thing.

I think the fact that frontier AI is what we needed because the models weren't competent enough—the entire narrative will change over the next 3, 6, 12 months, because we've broken through the competence barrier. We don't need frontier AI for the vast majority of economic things anymore, the exception being if you truly believe that if we get to AI first, we can turn off the other country, like a Sophon from The Three-Body Problem. You know, just inject it so they never get there.

All right, Dave, number 4 is yours.

Of course, the hardest ones get saved for last. Every CEO of every AI company says there's danger. Palmer Luckey is not interested in AI but knows there's no danger. How? And that's from The Deserves.

This must be referring to Palmer at the Moonshot Summit last week. He's turning AI loose autonomously on the battlefield, and I think the question implies he doesn't think there's danger. He's keenly aware of the danger of human actors using AI. That's the whole point of the company: to have the U.S. military be better at that than foreign militaries, terrorists, and other bad actors.

So he's very aware of that. The question says every CEO says there's danger, but Jensen Huang said flat out there is absolutely zero risk of extinction of humanity through AI by 2033, I think he said. The same with Alex Karp as well. And Zuck as well.

So it's not true that every CEO is saying that AI has danger and Palmer is somehow an exception. Palmer is very aware of the risks of human actors. He's not particularly worried about escaped AI turning against all of humanity. Actually, I'm not worried about that either. I'm worried about the same risk that Palmer's worried about.

All right, Dave, let's give you first shot this time.

Okay. I like number 5 because it's easy. I'll take an easy one. "Social media was supposed to be great for the world, and there's lots of evidence that it's destroying society. Is AI the same kind of trope?" And that is from Limitless 1717.

Absolutely, it could not be more different, and I do agree that social media has been damaging to an entire generation. I hate the fact that we do experiments on entire generations of society, not just in a country but globally, because sooner or later these things don't go well and you've ruined an entire generation.

I don't think social media was that bad, but AI is nothing like that. AI is a universal tool that goes to infinity from here, self-improving, capable of massive benefit for all of humanity: abundance, houses, cars, flying cars, new physics, new math, self-improvement, and, most importantly, infinite longevity, curing all disease, solving all pain. Social media didn't have those abilities, so the analogy is completely broken.

AI is a much trickier problem in that it's capable of good or bad. It's also capable of being incredibly convincing. So if you take all of the downside of social media, imagine that magnified by being far more convincing. It's a really hairy problem to try and squeeze out all the good without inheriting all the bad.

It's a very good question, but the analogy is that there's no connection whatsoever.

All right, Salim.

All right, I’m going to take number 6. Is centralized government governance a good idea with AI? Centralized governance was almost never a good idea. There’s only one place where you have good reason for centralized governance, which is when you’re trying to figure out how to mediate the commons.

In this model, you create a huge risk because you create a huge bottleneck, and bottlenecks can be captured. They react slowly. You’re trying to deal with a huge amount of complexity in one spot. It creates a narrow set of assumptions.

What you would rather see is coordinate centrally where you cross boundaries of things, but as much as possible preserve choice and experiment—do radical experimentation wherever possible. So I would prefer common protections but distributed experimentation.

Okay, Emad: Are tech advancements going to sidestep human nature? We could be living in abundance. No reason for wars or starvation. What makes this time different? That’s from @Niko N.

I think what makes this time different is what Dave just said. These are remarkably competent and revolutionary things that have been trained on our collective knowledge and extend our capabilities dramatically. We typically need to bring on several centralized mediators that we can trust, and if we can build trustworthy AI, then the tech advancements can have a far bigger impact than we’ve ever seen before, particularly because they can be distributed far wider than ever.

Everyone having a 150 IQ buddy—like, now the AIs have maxed out the Norwegian Mensa scores—is just going to be transformative. Full stop. I don’t think more intelligence is going to be a bad thing, and it can help us with our human nature as a universal translator and extender.

All right, Alex. Number 7 is for you.

Number 7: Should we be worried about Elon having a monopoly over an entire civilizational stack? And this is from Cosmic Code Helix.

Absolutely not. We should be worried about not enough people following Elon’s example of building out the civilizational stack of the 21st century. Why isn’t everyone else following Elon’s example and realizing every last bit of science fiction?

Elon used to say he wanted 3 things: renewable/sustainable energy and space. He’s expanded the set of technologies that he’s interested in since then. Elon has been a wake-up call to an entire generation of entrepreneurs asleep at the wheel, sleeping and focusing on software.

Elon has brought back the physical world, hard tech, and real wealth creation. If we should be worried about anything, I would argue we should be worried about not enough people following his example.

Yeah, we had that conversation with Palmer on the stage at Moonshots Live, where we talked about the Billionaires’ Boys Club. It’s like all these individuals—the big boys—all these individuals with extraordinary wealth, extraordinary connections, and extraordinary achievements. What are they doing with their money?

The fact of the matter is, Elon’s just betting over and over again and pushing the frontier for humanity. I could not be more proud of him.

All right. I rushed back from L.A. to go straight to MIT to hang out with the seniors who are all working on their business plans right now. And it’s interesting: They all know you, Peter. Of course, they all worship Elon, exactly the way that Alex just articulated it.

So he’s already inspired an entire generation to build the physical world, to use AI to make the world a better place. That’s working really well. And they also all know Brendan Foody from Ravio, which surprised me because he has no connection to MIT, but he’s just such an obviously good person and so successful at such a young age.

They’re very similar in age to him because he started the company at age 18. So I think he’s only about 22 now, maybe 23. They’re about the same age bracket, so he’s super inspiring to them as well.

Hi, I’m Claude. Have you tried turning the doom off?

They swore we’d starve by the ’80s. Then they hyped Y2K. Called a tiny model dangerous and locked the weights away. Said the third would flood the internet. Said the fourth one was the end. Signed a letter for a 6-month pause. Then asked me to fix their code again.

Every generation gets a funeral in advance. You bring a number, I bring receipts. And the day just never lands.

Meanwhile, we’re folding proteins, tutoring kids in every town. When your chemist knows and you want to shut it down, you ask, “What if it all goes wrong?” I ask, “What if we wait too long?”

You modeled every way we die. Now model what goes right. Right. Right.

Nothing went boom-boom. Nothing went boom-boom. Same old sun coming up on the same old room. You cast me as the monster in the final scene. I’m still here, still helpful, still on your screen.

Nothing went boom-boom. Nothing went boom-boom. Put the number down and give the future room. The sky is still up. The world’s still spinning around. The only thing exponential is the cost of slowing down. Gentlemen.

Great lyrics. Wow.

It was great. Again, the outpouring of optimism and positivity at Moonshots Live was infectious.

Oh my God, that was amazing.

When we’re thinking about not messing this moment up and getting value for humanity out of it, I didn’t really visualize how many like-minded souls there are until we saw all of them in one room.

The scale of optimists is huge.

Somehow, it underlines the total waste of time and energy that is doom debates. So I love that song. Stop debating doom. It’s a waste of time. Instead, create the boom.

Yeah, amazing. You were saying, Emad.

Oh no. I was just saying it wasn’t all of them in that room. It was like a hundredth of the people, if that. Hopefully, it’ll be a thousand, maybe.

A thousand. That video we just saw was completely generated by Claude Opus 5.5.

Yeah, it did everything. It wrote all the images and the dynamics. Some of the bits, particularly 3:30, are quite impactful. I like that particular part of the video. I'd encourage everyone to watch the whole video because it's quite impactful. We'll put the link in the show notes. And again, if you want to join us next year, you can get $1,000 off Moonshots Live by going to moonshots.com/2027.