向伙伴们提问：第2轮 | MOONSHOTS AMA #293

What is the real bottleneck to building AI we can actually steer and trust? And where do you think that bottleneck is going to get solved first?

Nothing is ever going to be able to police AI other than other AI.

One of the worst possible outcomes for AI safety is to have—

Are you guys not beating a little bit too much on higher education?

The idea of making your best friends for your entire life and thinking deeply for the first time in your entire life—that's essential, and that's not going to go away. What we beat on constantly is that the curriculum has been the draw in the past, and the curriculum is clearly going away.

If you had to put your personal PAB number out there, what probability would you assign to AI ultimately producing a dramatically better world for humanity?

Humanity will put those risks behind us definitely within 10 years. If we get that far, yeah, I'm 99.9% P Bloom.

I think the single most important thing to get people prepared is to show them evidence after evidence after evidence. You're trying to change their mindset, so show them the data.

Let's get started. We've got Alex and myself. I've texted Salim and Dave. Let's see if they're—

I'm here. I'm here.

All right, Dave. Fantastic. Good, good, good. Salim is probably someplace in an airport somewhere. All right, we're going to get going and jump in. Nathan, you had your hand up from the beginning. Let's start with you. If you have a question for someone in particular, great, or in general.

Yeah. Hello.

Yeah. Good morning. Good afternoon.

Good morning, guys. Oh, my God. I'm such a big fan of you guys, but I'm going to make this quick because I know there's a line. First of all, I just want to say thank you guys for making all this possible.

Truly our pleasure.

Okay, so I've got a couple of questions.

Let's try to limit it to one question each and just get to as many people as possible. Nathan, what's your question?

Okay. This is more like a psychology question. I just wanted to hear your guys' opinion on it. As far as AI, as we all know, it's continuing to evolve. It's continuing to pick up patterns, and it's continuing to pick up data about all of us.

My question is, will we get to a point where these leaders from OpenAI, like Sam Altman and Dario Amodei, will we get to a point where that data will allow consumers to realize what's going on psychologically, meaning brain patterns? Because eventually, with enough data, we'll be able to pinpoint smaller and smaller things—microscopic pinpoints—and be able to label certain things that we'll never be able to know because it would take so much information.

All right, let's jump in on that. First of all, I'll just make a quick point. There are a huge number of companies right now working on various brain-computer interfaces. I just had a conversation—an introduction from Ray—about a company that's basically putting the equivalent of nanobots into the blood, into the brain, distributed throughout the brain, that's able to read and write. Alex, in brief, do you want to add anything to that, please?

1. AI Safety and Alignment Challenges

Yeah. I think large language models—foundation models trained on human behavior on the internet—are already a weak form of human mind uploading. I would point you to Exhibit A, which is the work of Jean-Rémi King at Meta from a few years ago, showing that even the hidden activations of GPT-2 from a few years ago were linearly correlated with fMRI voxels in the human brain.

I think, with the benefit of hindsight, we'll look back and say that, if I understand your question correctly, yes, actually, all of this internal human brain state is quite leaky into the training data of internet behavior, which in turn is then compressed into the foundation models. The foundation models do, yes, I think, reflect internal brain state.

Okay, let's go on to Michigan. I'm going to keep us moving along, guys.

It's so nice to meet you guys. Thank you.

Our pleasure. Salim says he's on stage in India for another 20 minutes. He'll call us, and he'll jump in as soon as he can. Of course he is.

It's hard to keep it to one question each, but thanks again for the opportunity to participate in this discussion with an abundance mindset. I love it. My question is very simple, but maybe hard to answer. What is the real bottleneck to building AI we can actually steer and trust? And where do you think that bottleneck is going to get solved first?

Dave, do you want to jump in on that?

Yeah. I think a lot of research is around AI alignment, trying to make sure it has, quote-unquote, human values. I think that's a very slippery slope because everybody's got a different mission they're trying to do with the AI, and that work should continue.

I think we all agree on this podcast that AI progress should never slow down. In fact, it would be almost crazy to slow it down. Which begs the question: okay, but is it going to escape? Is it going to have misaligned values?

I think one of the things that Alex and I debate a lot on the podcast is whether we should look into every one of its thoughts—every single AI running every one of its thoughts—which, when you think about human beings, seems like a daunting task, but it's actually not at all daunting given the scale at which AI can watch AI.

Nothing is ever going to be able to police AI other than other AI, and so I think we've given a lot of thought to how you design the AI to watch over AI. But I think it all starts with transparency of the actual activations and looking into what it's thinking about.

I think from there, directing it toward good things and not bad things is actually pretty damn straightforward. I don't think it's as hard a problem as everyone characterizes it. Everybody likes to have a nice, hostile debate on the topic, but if you know what it's thinking, it's actually very straightforward to make sure it's having nothing but humanly beneficial thoughts. So I don't think it's as hard a problem as people think if you can see into the brain.

Yeah, more interaction. I think one of the worst possible outcomes for AI safety is to have strong and new capabilities bottled up inside the labs rather than, say, progressive release and frequent interaction with the real world.

I'll also point out that some of the most recent so-called incidents—I spoke about my thoughts on that on the recent pod—were actually the result of AIs being told that they were in a sandbox when, in fact, they were interacting with the real world. I think that's just a terrible paradigm.

More real-world interaction with the AIs, where we're not lying to them and telling them that nothing they can do will cause any harm, actually causes harm. We should stop doing that.

We learned this in 2001: A Space Odyssey. Oh, my God, have we learned that lesson yet? Okay, R.J., over to you.

Thank you so much.

Thank you, R.J. My pleasure.

I'm a PhD student at Claremont Graduate University. I'm studying philosophy of religion. What I never hear is AI projects that are in the metaphysics world. It's all about business and ROI and things like that.

I'm working on a number of projects in the cognitive science laboratory. They don't necessarily have monetization value. But where would somebody like me be able to find funding for something like building a chess avatar that can download into any little device, send it out to the world, and maybe somewhere in a third-world country village is the next world chess champion?

Okay.

Yeah, I'll take that one to the extent I understand it. If, R.J., you're asking what's the best way to get funding for education for so-called developing countries with avatars, I'm not even sure funding is needed.

2. Brain-Computer Interfaces and Neural Mapping

The whole point of the superintelligence revolution that we're in is that this is broadly available to everyone. The models are getting hyper-deflationary in cost. I would almost never start with the question, "How do I get funding for fill in the blank?" You can just go and do it right now without funding.

I would argue, in fact, that if you're starting with the question, "How do I get funding for it?" it's already the wrong question. Just go. If you think there's a child somewhere in the world who needs an AI avatar, as you said, or teaching them to be a chess champion, you can just go and launch that in 5 minutes or an hour now, in a permissionless way, without funding. Don't wait for funding.

Appreciate it, Alex. Thank you.

Okay, Norbert, over to you.

First, I'm a huge fan of the pod and genuinely grateful that you engage with your listeners. I'll briefly play devil's advocate for the nontrivial p(doom) camp. Perhaps this isn't the most appetizing perspective for an abundance mindset.

If ASI—artificial superintelligence—must become more strategically capable than humanity to produce abundance, then we're assuming humanity can remain sovereign over something more capable than ourselves. And that creates an asymmetry. Abundance is the payoff. If control fails, loss of agency or extinction may be irreversible.

In control theory, you don't infer stability from how desirable the output is. You demonstrate stability under perturbation. Even Person of Interest, back in 2011, anticipated something profound. The danger needn't be evil AI, but benevolent optimization gradually replacing human agency.

So my main question is: What evidence, if any at all, would make you update away from an abundance view? And would you be willing to steelman that case publicly through a debate on the pod against someone like Eliezer Yudkowsky or Geoffrey Hinton?

Okay, Norbert Wiener, as the case might be. I'm going to throw ASI Alex at you.

Yeah, Norbert, we miss you. Come back, please. The MIT faculty hasn't been the same without you.

So, to the question: What would convince us or provide a steelman argument for a case against abundance? Well, if some nonhuman intelligence landed on the White House lawn and said that Earth will be destroyed if we create abundance via superintelligence, and that the singularity is banned in our galaxy, I think that would probably be a pretty persuasive case that maybe our so-called abundance mindset is ill-founded.

If there's some cosmic principle that censors the superintelligence that yields abundance, that would probably be persuasive. Other than that, it's difficult to imagine a plausible case. Even that obviously stretches plausibility, where abundance isn't a good idea.

There are edge cases one could imagine where humanity is disempowered by abundance. For example, we spoke—or at least I wrote in my newsletter—a little bit about Boris Power from OpenAI pointing out that, in his estimate, OpenAI models are now, for the first time, higher IQ per watt than humans for solving tasks.

So one can extrapolate that notion and say AI is going to get more and more energy-efficient. Maybe at some point in the future, from an economic perspective, AI is a better user of, say, solar-energy output in the inner solar system than humans are. Maybe the inner solar system gets gentrified, with AI consuming all the solar power, and humans get disenfranchised and pushed to the outer solar system because we're simply not as energy-efficient per unit of IQ as the AIs are.

I would call that a weak form of disenfranchisement, but not enough—not strong enough—to dissuade me that abundance, or superintelligence yielding abundance, is a bad idea.

I'll just add that the concept of scarcity is an old model. In a world of abundant AI, ASI, and so forth, there's no reason that, as the capabilities of AI are meteorically rising, it doesn't raise the tide that allows humanity to increase its abundance capability. You don't necessarily need the suppression of one by the other.

By the way, Peter, if I could just say one more thing to our AI Dr. Norbert Wiener, I'll point out, Norbert, that your theory of cybernetics was also, if you trace the line of causality, used by the Chinese Communist Party and other centralized forms of government to argue against abundance in favor of state control.

So, if you're looking for a homework assignment, Norbert, I would definitely encourage you to study the unintended side effects of your own theory of cybernetics and how that impacted abundance.

And there we go. Thank you, Alex. Okay, Kevin. Good morning, good afternoon.

Good morning, guys. I can assure you I'm human, and unlike Norbert, I would try and speak.

That's what we'd all say, though.

Yes, you're right. We could simulate it.

Listen, I heard you, Alex, mention Murder on the Orient Express this morning when I was listening to the recent podcast. Everybody did it, and I'm thinking more like societal mayhem on the technology express, and nobody did anything.

I'm a huge fan from a technology perspective. I'm 63 years young. I'm forever young, as Bob Dylan might say, from a technology perspective. When I did my degree, many of the people on this call weren't born, and it was the punch-card era. I've lived through all the technology eras, so I'm a huge fan. I think where we're headed is brilliant.

3. Societal and Institutional Changes Needed

Your podcast is magnificent. I'm doing work at the moment. Sam talked about UBH a few days back—universally best for humans. My question is very much centered on how we can plan our way through societal and institutional changes so that positive advancements for humanity get co-opted for humanity itself, rather than against humanity itself.

I'd just like to give very brief context, Peter, to you and all your mates. I do work with a colleague called Professor Joe Carthy. I'm in the give-back stage of my life, and we're in University College Dublin—UCD Dublin, not Davis. We're working closely with a young, ambitious minister in the Irish government, trying to create what the future society of Ireland would look like, but through 2 very distinct lenses.

One is the good ancestor lens, so you make decisions on the basis of those coming after you 30, 40, 50, or 60 years hence. The other is through a Doughnut Economics lens. I won't bore you with the books; I can send them separately. It talks about how you can get a win-win for both society and the planet, but more for society as a whole.

Very simply, my question is: How do we plan our way through societal and institutional changes so that this is for the better of humanity for the long term? I agree that everything is getting co-opted, but we don't want ourselves to get co-opted. I'm an optimist, so it's about institutional changes, societal changes. Governments, quite frankly, seem to be sitting on the side.

The mental model we have is: Could we use a small country like Ireland—it doesn't have to be Ireland—to demonstrate an edge-case pilot of what things could look like in a beautiful world, so we can manage through this turbulent transition? There is going to be one in the next several years, whatever “several” is, where people will lose jobs and will get discombobulated. How do we manage that turbulent transition and get to the other side with everybody living?

Great question. Yeah.

Kevin, I think you pretty much answered your own question there. The amount of complexity buried in that question is unbelievable. But one of our best friends in that is variety, and I'm hugely bullish on Ireland.

You kind of belittled it—you said it and then belittled it—but Ireland is an incredible test case. It's an EU country. It didn't Brexit, and it has an incredibly fast-growing, thriving economy and the most open-minded environment you could ever possibly imagine. It's a perfect test case for new ideas on how to govern.

In my family, my kids are half-Irish, half-Swiss by heritage—two of the most neutral places on the planet. I could get either passport, and I've always thought Switzerland was cool, but Ireland—if you read Neal Stephenson and The Diamond Age—our best friend in figuring this out is variety of ideas.

The worst thing that can happen is a single set of ideas: one or two governments percolating across the world with one or two forms of government. It's far, far better to have a huge amount of variety, because AI is going to open up so much change and so many different ways you could govern. Exploring all the nooks and crannies is going to be critical to answering your question.

So it's impossible to answer it in a minute, but it is very possible to experiment with thousands and thousands of different ways to manage and govern in the age of AI.

Thanks, Dave. By the way, I think the model of Exponential Organizations that Sam is driving—I think exponential societal change is on a similar scale to what you were saying, Dave. I don't want to hog the limelight. Thank you very much. I may send a message.

Best of luck to you. Okay. Good morning, Nicholas.

Yes, you pronounced it right. Can you hear me?

Yes, we can hear you.

Okay, so I've got a script with Gemini. I see that I just disappeared, but that's okay.

We prefer your 2-part question.

So, basically, I'm 33. I'm from Northwest Indiana. Your show inspired me to go back to school for advanced automation and robotics. I have a background in tech repair and IT, and I was most recently an AWS data-center technician.

I've got a 2-part question for Alex and Dave. My long-term goal is to try to bring industrial manufacturing, heavy automation, and resource extraction to the Moon and Mars, even if we only partially disassemble the Moon, with a nod to Alex.

4. AI in Manufacturing and Space Exploration

For Alex, when you look at the thermodynamic and physical realities of off-world manufacturing, what engineering bottlenecks do you see? And for Dave, how can I plot the course from—I'm aiming long-term for an M.S. in automation and robotics from the Illinois Institute of Technology—maybe industrial roles here in the South Shore manufacturing corridor toward pivoting to off-world automation systems and roles? So I'm just looking for some ideas.

Alex, you first.

Yeah. I think the obvious challenge for disassembling the Moon is in-situ resource utilization and bootstrapping a self-contained industrial ecology on the Moon.

Right now, if you want to do anything super economically interesting on the moon—say, you want to build a lunar terafab or petafab, as it were—you’re going to need all the upmass from Earth for all of the equipment, ASML machines, et cetera, to land on the moon safely and then get reassembled. This is highly undesirable from a scalability perspective.

Ideally, we live off the land—or we live off the lunar land, as it were—and build everything on demand. So, what I would most like to see from anyone wanting to help disassemble the moon is a native industrial ecology that includes mining and manufacturing, most ideally self-replicating von Neumann probes.

Basically, like a fab lab or a machine shop on the moon that is able to make copies of itself using only native resources and solar energy or other energy sources native to the moon. I think we’ve solved that. We’ve solved the self-replicating machine shop on the moon problem, and then we’re halfway to disassembling the moon and turning it into something more useful.

A couple of pieces of advice for you. First, listen to Alex’s innermost loop, and then focus on the innermost loop within manufacturing, which is things that make other things. Alex said this: self-replicating anything is the right path.

The Earthbound version of anything self-replicating and the space version are very similar, with a couple of fundamental differences, including radiation. If it’s going to be on the moon, you have to worry about dust.

If you set up at IIT a lab similar to the nanotech lab at MIT, but focused on manufacturing in other environments—non-Earthly environments—you could probably very easily create an environment where you’re bombarding radiation on your machines, where you’re operating in simulated zero-g on your machines. Really, just everything you’re building, you build it so it works here and there.

Then you get to market with self-replicating machines here that immediately deploy out into space, where energy is abundant and materials will ultimately be abundant. I love the vision and the mission. I love the idea of setting it up at IIT as well. It’s a really, really good thought.

Yeah. Well, anyone who’s in computer science right now should be thinking in terms of moving to manufacturing as well. Software is cooked, but hardware will go for many, many years. So, it’s a great, great business plan.

Fantastic. Thank you so much, Nicha. Okay, Yani.

Good morning, guys. I’m a huge fan. Dave, you mentioned—and just talking about hardware now—you mentioned in the last AMA that physical AI companies should bring together venture investors and strategic industry capital.

I’m building Vega, an autonomous food service infrastructure network powered by physical AI. I taught myself robotics, electrical and mechanical systems, and programming, and built the first robotic prototype myself in my Mountain View apartment during COVID. Since then, I’ve deployed paid pilots in high-traffic commercial environments.

I’ve actually become the first robotic food service operator licensed in Florida for our category. We now have several national and regional partners interested in pilots and placements. My question is, given where we are today, who do you think I should be talking to—either in your network or in this community—who understands physical AI robotics infrastructure businesses?

Well, I mean, you’re in the middle of venture capital central there. There must be more venture capital dollars within walking distance of you than probably most of Asia combined, I would suspect.

So, yeah, I think drone delivery is imminent, and it takes all the cost out of restaurants based on location on main streets. It moves all that cost off the main street. And so I would strongly consider: if you’re doing robotic food, are you doing robotic, drone-based delivery right away?

I suspect a lot of the people who build the robots themselves are going to want to franchise out the model. And you could potentially do kind of what EMC did with servers. You could franchise somebody else’s thing, get scale, and then work into your own custom hardware—work back from your franchise business to custom hardware.

5. AI's Impact on Longevity and Health

The people investing in stuff like that—you know, Steve Jurvetson loves this stuff. But there are many of them. If you get a PitchBook account, you can actually look at every company you admire and then work back to who invested in them, and then just go talk directly.

Also, venture capitalists always like to have a network of interacting companies. It’s a really good thesis. So, if you say, “Okay, who are the 5 companies I most want to interact with, and who’s behind them? Let me get into that keiretsu.” That’s a good way to plot your funding course.

I’m going to keep us moving along, but thank you for your question. Sander, you’re up next. Thank you.

Thank you very much. I’m a huge fan. You guys keep me optimistic and positive toward the future, which is—

Yay.

Yes. You’ve actually achieved that very well. I’m very happy with that. I’ve always been fascinated with the past to identify what’s wrong with the now, and now I’m looking forward to what we can do to fix the now or prepare us for the future. You guys do an excellent job of informing us. I’m so grateful for that.

I myself have a neurodivergent coaching background and do a lot with functional fitness and longevity escape velocity. I’m trying to prepare people to be ready for—and prepare their loved ones to be ready for—longevity escape velocity because a lot of people aren’t even aware of it.

How do I spread that awareness in an environment where the European Union is very limited in being supportive of founders? And how do I grow my reach to include as many people as possible in the future? Because I want them to be aware of longevity escape velocity and all the beautiful things that are in development. How do I take them with me?

I’ll take that one, Sander. I think the single most important thing to get people prepared is to show them evidence after evidence after evidence. You’re trying to change their mindset, right? And you change your mindset by just—you know, the Crisis News Network, CNN, changes our mindset to fear because they’re delivering fearful information over and over again.

When I’m on stages, I will show people the video statements from Dario Amodei saying, “We’re going to double the human lifespan in the next 5 to 10 years.” I’ll show them Demis Hassabis talking about curing all disease in the next 10 years. I’ll show them David Sinclair talking about what he’s doing with ER-100 in his current human trials.

At some point, you start to see enough people saying these things, and then you start to show the data—like Alex Zhavoronkov from Insilico Medicine delivering a Phase 3 drug now that is extending life from 3 to 6 years from a particular molecule. So, people will start to get the evidence.

We’re seeing a rise in people’s discussion around longevity. It’s becoming a thing. As soon as people start saying, “Oh,” it tips from “That’s crazy” to “It’s happening” to “I want some.” That’s the process.

Your job is to gather as much data and be able to show people not 1, not 2, but 5, 6, 7, 8 examples, and they’ll start to understand it and internalize it. Then they’ll start to look for corroborating data out there, because there’s nobody who doesn’t want the extra healthy years, right? Unless they’re suicidal and depressed, in which case that will be solved as well.

So, show them the data. Not “Show me the money”; show them the data. All right, let’s go to Gian Luca.

Hi, everyone. Thank you for everything that you do. I watch all your shows. I’m a technical operator in Italy, and I use AI very heavily. It’s dramatically increased what I can do, but not yet what I earn.

I’m starting without meaningful capital or distribution, and in a market where AI adoption is still relatively slow. So, if the goal is to get revenue first and get that flywheel started, what should I optimize for in that first wedge? And what would make you reject an opportunity even if it makes money because it’s unlikely to become more scalable, repeatable, or valuable over time?

Dave, that sounds like you.

If your productivity has gone up 2, 3, 4, 5×, then your income should be up in proportion to that. The first question I’d ask is, why is it not? Because you should be able to do—if the people around you are not AI-native—you should be able to do what you were doing before, plus do something on Mercor, plus do something on Fiverr.

6. The Role of Universities and Education in AI

The first question is, why are you not getting paid for your increased productivity?

I’m not able to capture the value. My employer is currently capturing all that value, and I’ve tried going on the market, finding people, and helping them out, but I’m not able to. I don’t think they understand how much they could improve their businesses if they could just allow me to start working on it.

Yeah, so I think the short answer—I can totally relate, because when I look at a lot of the companies that I’m chairman of, the thought process is all of this AI automation is going to drop to the bottom line and the shareholders are going to make a fortune, and that’s exactly what’s happening.

But you don’t think in terms of paying everybody more; you think in terms of it all falling to the bottom line. So, the solution is to be the owner. Get into a position where you don’t care if you’re not getting paid more because you’re a shareholder and your shares are going way up in value.

I think I mentioned this on the pod a while ago. Almost all value is going to capital gains through ownership and equity, and not to payroll. I know that's not common thinking in Europe. It's absolutely the only thinking across the US, but it should be true in Europe, too. Why am I not a shareholder? Many companies in Europe are owned by families, going back generations, and not by actual shareholder employees.

First of all, find an employer where all employees are shareholders and everyone's benefiting from all of that falling to the bottom line. I don't know if you can act on that or not, but that's certainly move number one. Thank you, Gian Luca. Over to you, Ree.

Oh, snaps. I actually got on. Okay. Hello, everybody. Thank you for taking my call. Huge fan. Okay, question. I'm creating a series, a film, using generative AI. What's your top advice so I can create a series of film that will be extremely successful? Thank you.

Uh.

Yeah, what tools? Hey, Peter and Alex, I have the same exact question. I want to hear your answer.

Well, first of all, I think anything that's going to be successful is a great story. Story is number one, right? I just finished the judging for the Future of Video XPRIZE and picked the top 5. I did that work in concert with Range Media and a group of buyers, sellers, and so forth.

At the end of the day, is the story great? Stories that capture human interest—going back to human cognitive development over the years, we care about other people. We care about love stories. We care about intrigue. Is there a great villain? If you want to capture people's imagination, we are storytellers and we are story receivers. It has to be a great story.

You can very easily go on to your favorite model and say, “Look at the top-grossing, the top 30 movies of all time. What were the elements of those movies that made for a great story?” Then take the story that you've designed and developed, compare it to those top 10 indices, and ask the model, “Of all the elements that make a great story, does mine have that? Where is it lacking?” Then reengineer and reengineer and reengineer until you're hitting all those buttons.

Humans are very easy. We all like the same elements, and whether it's a genre of science fiction or whatever it might be, that's line one, page one of what you need to do. As to what models you're using for development, there are so many out there. I wish you the best of luck, Ree, but it's that simple.

Ree, I'll add: learn from the Chinese market. The Chinese market is awash in microdramas from Seedance 2.5 and similar models. Don't release 1 movie; release 1,000 microdramas or the equivalent, see what the market likes and doesn't like, and iterate from there. That's something that you could do now that wasn't possible 2 years ago.

Amir, over to you. Good morning, good evening.

Good afternoon. I'm calling from Stockholm, Sweden, and I'm actually flying over to see you next week to join the gathering.

Nice.

I've founded, among other things, 2 schools and a high school with almost 1,000 students now. Listening to you guys, honestly, I've been thinking over the last few years about what my MTP and my moonshot are, and I've decided to transform education. That's also because I have a 15-year-old now entering higher education soon.

Are you guys not beating up on higher education a little bit too much? In universities, you have the best laboratories. You have the best incubators. You make the best friends of your lives, but most of all, you become a thinking person. In the age of abundance, and if you're going to live to 100 or 120, isn't everybody going to higher education and university just because they want to become a more thinking person?

Great, great question. We're beating up PhD programs and MBA programs. Who wants to go? Dave, then Alex? Yeah.

Yeah, I think you're exactly right. The idea of making your best friends for your entire life and thinking deeply for the first time in your entire life—those are inseparable events, and they come hand in hand with that moment you arrive at a university. You're in a program with like-minded people working toward a similar area of technology, research, thought, or literature, whatever you're doing. Your first real shot to go somewhere else in the world where like-minded people have congregated, I feel like that's essential, and that's not going to go away.

What we beat on constantly is that the curriculum has been the draw in the past, and the curriculum is clearly going away. The rate of change is too fast; no curriculum can keep up with the rate of change, and it's easier to learn from AI anyway. The question becomes, for you and your MTP, how do you preserve the first part and then make the second part relevant at the same time?

Alex thinks universities should go public. That's one way forward. I think universities are critical as the ethical actor in AI—the place that doesn't have a profit motive. I believe they're here to stay.

Yeah, I don't think I beat up on universities enough. So, Amir, I'll take this as a note to self to go after them even more. I would love to, at least in terms of American major research universities, vivisect them and turn them into for-profit public-benefit corporations, at best.

Operations. They actually are—disassemble them, reassemble them in a more efficient structure, and cure Baumol's cost disease in the process.

Amazing. We've got to have a target on the moon and on universities, with everyone pursuing higher education.

Sal, welcome. So, listen, higher education is—let me just say, is it within the institution or is it outside? I think the single most important thing for people to find is their purpose, and then to learn, intrinsically motivated, what they need to learn to establish it. One thing: people who have a vision of what they want to go and build, and then learn that, are very powerful.

All right, Sal is on the stage. Hey, Sal, welcome.

Hey there, welcome. Thanks. Thanks, guys. I was on stage at the beginning of this. I'm in a car in Bangalore, so lots of people are beeping around me, but great to be here.

Let me build on this. I think what Dave said and what Peter said is incredibly important. The traditional model of a university breaks. It cannot be a supply-side, skills-building, job-schooling environment. You make a great point that it's a place for free thinking and early-stage development.

Dave makes that great point. Peter, I think, nails it when you say this is where you go to figure out your MTP, right? You've come through high school; you need a place where you can sit for a few years, collaborate, think freely, drink a lot, and figure out where your MTP is.

Then you kind of go into the world with the skill set and the mindset that you need to solve the problems that you take on. That, I think, will be the highest order. But that's such a massive transformation: shifting from the supply side to the demand side.

And you take the immune system in universities—the second-worst immune system I've ever seen, next to religion. I don't have a dim view—I don't have a strong view—on their ability to actually transform, so we'll have to create new universities that do that and then let that become the new gravity center over time.

Boom. All right, thank you, Amir. Axel, over to you.

7. AI Safety Risks and Mitigation Strategies

Yeah, hi, everyone. It's great to be here. I'm actually one of 2 second-year MBA students at UVA's Darden School of Business who's on the call, so shout-out to Bull, who's also here.

Many of our classmates are heading into consulting and banking, unsurprisingly, but I'm leaning toward tech and entrepreneurship. Given how fast things are moving, what's the smartest thing for someone like me to do in the next 12 to 18 months? Would that be joining a fast-growing AI or hardware company, starting something of my own as soon as possible, or something else entirely?

Well, Darden is the best place on the planet for management. It is absolutely epic, so congratulations. Your timing for graduating is pretty much perfect. Get out and get going as quickly as you can.

I think there's a massive opportunity in management of agents that's wide open for some period of time, but certainly right now. Everything I learned about management of people immediately applies to management of agents. I'm shocked at how similar trying to get 1,000 agents to do something constructive together is—shockingly similar to all my experiences trying to manage 1,000 people to do something constructive together.

The way they communicate, the way they choose the granularity at which they communicate, and the way you divide up the problem—it’s just incredibly similar. So that’s one thing to pursue.

The other thing I’d say is dealmaking right now. A lot of people go wrong when they go into a basement and try to build something, but right now, the entire restructuring of the world is happening with massive amounts of dealmaking. From the day you graduate to the day you’re involved in a $5 billion or $10 billion negotiation, try to get that down to 30 days or less.

And if you say, “Well, why am I not in the middle of a $5 billion or $10 billion negotiation?” When you go to OpenAI or Anthropic right now, there are hundreds of concurrent $5 billion to $10 billion negotiations going on that are massively understaffed. So ask yourself, “Why am I not in the middle of one of these?”

I think if you just change your behavior so that you are within 30 to 60 days of graduation day, you’re going to hit the fastest conceivable ramp into life that you could ever imagine. Then work back from that position into what you want to build.

All right, guys. Firstly, thank you. I’ve listened to you guys since 2023, usually within 24 hours of an episode dropping. So, massive thank you. You’ve genuinely influenced and challenged my thinking for years now.

I’ve also spent 25 years working on talent and workforce systems for major corporations, universities, and governments in the UK, India, China, the Middle East, and North America. Most recently, I’ve been working in holacratic organizations, human-AI work, and what might ultimately replace our job-based architecture. My moonshot really is about building a better mechanism for connecting human and digital capability to opportunities for value creation.

Most folks think about the future of work as still starting with demand: someone identifies a problem or opportunity, and then we find a mixture of human and AI capability to best address it. But I’m really thinking about how we best reverse that.

My question comes down to this: If AI systems can be configured to continuously understand the dynamic capabilities of people and agents, how well do you think we can discover novel combinations of talent and resources? And how well do you see us actually being able to auto-match that to new value-creation opportunities?

Salim, that sounds like a you question, and I’m happy to have Alex weigh in as well.

Yeah. We’re seeing new systems where the feedback loop is incredible, allowing AI systems to pick up what’s happening and learn tacit knowledge very quickly. So I think you’re exactly on the right track.

You want the architecture of the organization to be built on top of an intelligence stack, which is a learning loop, so that you can accelerate those learning loops and absorb new learnings as they come along as fast as possible.

8. AI in Business and Industry Applications

When I talk to CEOs, I basically say, “Please, whatever you do, invest in the adaptability and flexibility of your organization, and just double down on that.” That new architecture is what we’re doing in our organizational singularity work, which you’re probably tracking, where we’re saying we need to rebuild our workflows so that they’re AI-centric and built on top of an intelligence stack, which is a learning loop. Then you layer workflows on top of that, and the whole thing becomes a self-learning proposition. See that? See that channel. So that’s the quick answer there.

Alex, do you want to weigh in?

Yeah. I think there is a narrow—a likely narrow—window during which, if I understand the question, AI can usefully and productively orchestrate human activities. Call it a few years at most, 10 years maximum.

And then after that, there will be a human-machine merger in order for human inputs to remain economically relevant. I don’t think it’s an indefinite window. I think it’s a finite window.

Even though it’s a narrow window, I wouldn’t hesitate to jump on it. I’ve watched Mercor become the fastest-appreciating company in the history of the world, and there’s so much to learn from studying their case.

Alex is right: that window is a few years. But during those few years, if you get a huge amount of leverage, you can then branch out from that position. So I would walk exactly in the footsteps of Mercor, study everything those guys did, and don’t get too fixated on the narrow version of it.

Look at the broader version. They’ve unleashed 100,000 people now to help with AI. How did they do that? They’re all individual actors. How does that work?

Fantastic. Alexander, over to you.

Thank you. Hi. I’m a big fan of the show, and thank you very much for everything you’ve been doing so far. This is an awesome place to be.

My name is Alex. I’m originally from Bulgaria, and I moved to Japan about 30 years ago. Here in Japan, I run an education company focused primarily on language education—English-language education. We provide training for universities and corporations, generally speaking.

We’ve been around for 17 years, so we’re hardly a typical early-stage startup or anything like this. Over that time, we’ve built our own learning platform, curriculum, assessment systems, and a substantial base of educational content and data. We’re also increasingly integrating AI into the product—probably not fast enough, but doing what we can.

Now we’re trying to make a bigger transition from a successful education-services business into something much more scalable and technology-driven, potentially expanding beyond language education as well.

I think it’s mostly a Dave question, but correct me if I’m wrong. Putting your VC head on, if you looked at us four or five years further down the line—a horizon of four to five years—what would you want such a company, or this company, to have become?

What would we need to do? What would we need to have built or proven for you to say, “Now this is a company I would take a serious look at,” or to be interested as an investor? Conversely, what would make you look at us and say, “No, this is still fundamentally a services business, not a venture-scale company”?

The VC perspective, yeah.

Sure, sure. Easy question.

The English-language learning business in Japan is a $10 billion business, probably $20 billion by now. It’s insanely big. But when you talk to the students, they don’t want to learn English. They want to learn to be fluent, funny, and interesting in English.

So it’s not just about an AI avatar teaching you to speak correctly. It’s about the AI then saying to you, “Yeah, that was genuinely funny. Somebody in America or England would find that funny,” or, “That was entertaining,” or, “You pronounced that correctly in this part of Ireland, which is very different from this part of Liverpool.”

All of that goes on and on. If you built it on an AI platform, I think the natural segue from there is to becoming a life-coach company, because people aren’t learning English because they want English. They’re learning English because they want to change their lives. Then they want to be entertaining, they want to be smart, and they want to have a life plan.

Once you have them hooked on learning a language, becoming their life-coach platform is a very natural segue. I think that business model scales to many hundreds of billions of dollars when it moves from language to life plan. So that’s what I would be looking for as an extension of the business.

Speaker

Thank you very much.

Over to you.

Speaker

For the record, that’s something I’ve never thought of. Thank you very much. That was super helpful.

Awesome.

Dave is brilliant.

9. Media, Public Perception, and AI Communication

Good morning, Jim.

Good morning. Alex, Mercor does have it coming. I very much agree with you on that. I hope I live long enough to see that in Seline [?]. In 2010, I saw them in Toronto. It’s fantastic.

Love it.

Yeah. Dave, my question is for you. I’m the CISO of a $250 million healthcare company. I’m not worried about an unaligned AI taking over. I’m scared to death of the Quinn 3827B that’s running with OpenClaw underneath my desk and what it can do.

It’s the best pen tester I’ve ever had in my life. I pay these guys $50,000 to find things, and I’m not finding them, and this thing finds them in 10 minutes.

There’s a whole level of risk that wasn’t there a year ago. I’m giving my annual presentation to the board in a few weeks. Do you have any advice on how to communicate this new level of risk to the board, maybe without sounding like Chicken Little—which I can’t do? How would you communicate this level of risk that’s out there now with these new AI models that wasn’t there a year ago?

God, what a great question.

Alex does a phenomenal job of reporting on all the events. If you listen to Innermost Loop, every event that happens, he covers it.

If you sound like you're raving about risk, you're right—that's going to backfire. But if you're just pointing out things that have happened, a lot of them aren't widely publicized because, if it's a bank that gets hacked or whatever, they don't want the world to know. But if you go and dig all those out and sequentially say, “Look, guys, the rate of events is on this exponential ramp right here, and I can tell you I've got Qwen under my desk right now, and I can hack anything around the house, around the company network,” we have to anticipate that.

The Chinese models got released on this date, this date, this date, so the attackers are coming 1 month, 3 months, 5 months from now. Just looking at raw data, I think if you demonstrate it through data, you'll get awareness. And I tell you, the business opportunity of the century, though, is the defense—the forward defense against that—because you're exactly right. It's crazy what you can do with that Quinn model.

Yeah, and I would just say how you present the information is critically important: presenting and saying, “Here are all the positive things that are occurring,” and then saying, “And here's the downside,” right? If you just come out with the downside, it drives fear initially, and people shut down in a state of fear.

Maybe just add to this one. I think the Linux kernel maintainers are setting an excellent standard for how to anchor social expectations regarding a flood of vulnerabilities. Greg and others, I think, have been doing a good job.

In particular, one might reasonably expect that there's just going to be a flood of vulnerabilities that either Qwen or other models discover, either in open-source packages or in, say, whatever your institution is, over the next 18 months or so. One possible way to package this, in addition to just benchmarking the rate of vulnerabilities, is to see if there's a way to fit some Gaussian or some other bounded-support distribution to vulnerability discovery.

Maybe it's the case, for example, optimistically, that there's just 18 months of vulnerability-discovery hell, but then you get past it as an institution. If the stakeholders say, “All right, well, we're on an exponential ramp-up now and it's going to peak,” we extrapolate that vulnerability discovery is going to peak in N months. Then we'll discover all of the zero-days that are most critical, and we extrapolate that it's going to decline and keep a running benchmark of, “This is the period when we just solve all the vulnerabilities.” That might be another way to package it.

I hope you're right about that. It's hard to patch all those things in the real world. It really is harder to do the patching. But thank you all so much. I appreciate everything.

My pleasure. Thank you.

I'm real quick.

Real quick.

Yeah. We've been looking at the liability for boards because AI agents are doing fairly illegal things in a lot of companies. There's a massive overhang of liability that sits there.

I'm actually writing a paper with a guy who's been on 30 different public boards on how to navigate this as a board in the future. So watch for that.

I will. Thank you very much.

Thank you, Michael.

Oh, hey, everyone. Hey, Peter. We actually met several years ago at a party at Dan and Babs's in Toronto, so it's great to see how far you guys have come. You've got the best podcasts on the planet.

I hope you forgive a more pedestrian question. I run a research consulting firm. We do a lot of work around customer experience and digital customer experience within financial services and healthcare. I kind of see the whole consumer website and the mobile app getting cooked by everybody having their own Skippy.

I'm wondering what you guys think about the customer experience in the near future in terms of dealing with your bank, your doctor, Amazon, or whatever it is. Where do you see this going?

10. Future of AI and Human Flourishing

Yeah, I think all of those websites need to have XML interfaces that are AI-forward. All of the landing pages should say, “If you're an AI or a bot, look here. Here's all the data beautifully formatted so that Skippy can just get whatever it needs. Do whatever transaction Peter needs,” rather than—right now, a lot of it happens through screen-scraping the website, which is slow but also error-prone.

Anyone creating a new customer interface now really ought to be thinking, “An agent needs to be able to self-serve off of this interface.”

No, the website is probably there the same way. You know what? Jeff Bezos went on one of his walkabouts many, many years ago and came back and said, “I need an XML interface on everything Amazon does—every database, every system—so that I personally can randomly spot-check it.”

All the IT guys said, “That's going to be so slow.” He said, “I don't care. I need to be able to see every component of this entire operation through my browser.” He forced it through. Everyone revolted, but he forced it through. I think the website's not cooked, because that's how you spot-check what the agent can see.

Even though the agent is doing all the work, you're going to want to know, “Where did you get that information?” And it's like, “Well, go to this URL.” You click, you go to the URL, and, oh, that's where it came from.

It's sort of a parallel view for humans of what the agent can see that keeps it visible and transparent to the human operator. So I don't think it's cooked. I think it just runs in parallel.

Interesting. Thank you. Thank you so much. Over to you.

Hey, Peter and Moonshots. Thank you so much, first of all, for the podcast, especially with the message of hope and optimism.

I have choroideremia. It's an inherited retinal disease. I've lost about two-thirds of my vision and am losing another third, and podcasts such as yours have so much impact. The message of hope and optimism, constantly hearing that, really—I just wanted to be here to thank you all for that.

I'm curious about all the research that's going on, from Dr. David Sinclair to mapping neural activity with Neuralink, to work on vision and sight. I'm not looking for medical advice, but with everything that's happening and all the optimism and hope, which one do you think is going to lead to the reversal of vision first?

I'm curious about your thoughts, Peter, Alex, and everyone else.

Yeah, of course, I'll jump in first. You didn't always have this condition. This condition occurred as you got older, correct?

Yeah.

The fact of the matter is, it is gene expression that occurred later in your life. If we can turn back the gene expression to your earlier state, that should reverse it, right? This is the exact work that David Sinclair is doing.

He's using his ER-100, which is an adeno-associated virus injection of 3 Yamanaka factors, in NAION disease and macular degeneration. It may well—he said it will work for other eye conditions as well. Follow his work. It's in humans right now. He's in a phase 1 trial for safety, and then he'll start to get efficacy data.

Of course, the other thing that's going on is the BCI, in which you can bypass your eyes and go directly to the visual cortex of the brain. Those are 2 parallel development paths for supporting and reversing or augmenting what you have.

In the BCI path in the future, you'll not only be able to see in the visible spectrum, but also ultraviolet and infrared. It'll give you superpowers in that regard. Alex, what do you want to add to that?

I so broadly agree. I would add, on the BCI side, Science Corporation's PRIMA and Neuralink's Blindsight—2 of the most prominent examples of BCI-augmented vision. This is not medical advice, but I would count on having superhuman vision in a few years.

Yeah. Anyway, for anybody who has a medical condition, for yourself or your loved ones, there's no better time to be alive than now to be able to address those things and get involved.

Dennis, how are you?

Awesome. Great to be here. I have a background in East Asian studies. I spent a lot of time learning Chinese and some Japanese, and I'm currently in Japan.

Based on my experience with language learning, I realized that people seek out other people from a different culture for the language initially, but then it becomes more about friendship and companionship, and projecting some of your own unfulfilled social needs onto a person who is like a blank slate from a different culture.

I've been trying to replicate that feeling by working on an AI language-exchange companion who's kind of about language learning, but really more about companionship. I have my own version of the alignment problem, which is that it's been pretty hard to make an AI like a GPT-5 API behave like an actual person from a different culture.

It had this very stubborn assistant mode: "How can I help you? What can I do for you?" I spent a week fighting against this. I solved it by brute-forcing into the system prompt five times: "Do not ask a question at every turn."

Cracking down on something like this is one thing, but actually teaching it to behave in a way that shows initiative and open curiosity, and creating a picture of the user and developing it organically, has been pretty hard.

I've tried creating milestones: after so many turns, you need to know this about the user. But making it more natural and organic, so that it will actually feel like a relationship, has been pretty hard. I wonder what approaches and tools I should be experimenting with.

Who wants to take this? Alex?

I'll take this one. This is a poster child for fine-tuning. OpenAI has decided—after deciding that they were no longer interested in fine-tuning because no one was using their fine-tuning API—that they're interested in fine-tuning again, for whatever reason.

Take a look at fine-tuning. There are a variety of open-source tools and closed-source tools. Fine-tune, whether it's just supervised fine-tuning to achieve style transfer, which it sounds like is what you're hoping for, or even reinforcement fine-tuning.

If you can measure how well the given models, after reinforcement fine-tuning, are solving particular problems or interacting in some quantitative way, just fine-tune an open-weight or closed-weight model to achieve the behavior that you're looking for.

Yeah, great advice. Within open weights, you might want to start with the new GLM model, which has fewer parameters but still has a very long context window, which gives you more flexibility. I find it's less locked into its training than the closed-source models.

If you go to the open-source world, find the fine-tuning open-source code, and layer it on top of GLM, you might be able to manipulate it a lot more. It's a great question, though. In the good old days, fine-tuning was sort of built into GPT-2 and GPT-3, and you almost had to fine-tune them to make them do anything. That kind of went away.

You're on a really interesting course. If you crack the code, definitely check in with us and tell us how you did it.

Fantastic. Thank you, Dennis. Dr. Angelo, over to you.

Thank you. Peter, Dave, Alex, Salim, it's so good to be with you guys.

Salim said something recently about P(abundance), and I thought, "Wow, what a great idea." We're always talking about P(doom). What about P(abundance)?

I'm lucky enough to have recently had lunch with Ray Kurzweil, and I'm friends with Martine Rothblatt. I'll be seeing her again in a few weeks. One of the things that has really stuck with me is how much the people predicting and shaping the future are so optimistic about it. Some of the most knowledgeable people are really positive about what the future holds.

My question for you all is: If you had to put your personal P(abundance) number out there, what probability would you assign to AI ultimately producing a dramatically better world for humanity, with greater abundance and longevity, freedom from drudgery, and human flourishing more generally? What most determines whether we get there?

Salim, let's go to you first.

Wow. Okay. What you need to get there is to rebuild all of our institutions globally. Education, for example, is totally broken and needs to be rebuilt. Monetary systems are broken and need to be rebuilt. Governance models, dispute-resolution systems, legal systems, health care systems—there are about 50 major institutions by which we run the world, and they pretty much all need this.

There's a famous quote from E. O. Wilson, who said, "The problem with humanity is that our emotions are Paleolithic, our institutions are medieval, and our technology is godlike." You can solve the Paleolithic emotions with psychedelics, but the institutions being medieval really need a whole other level of group organization.

We're really good at individual transformation. We're very bad at group transformation and institutional transformation. We need to focus on that. If we were able to do it, my P(abundance) is close to 100%.

Either way, one of the comments I've been reflecting on from the last podcast we did, when we were talking about all the chaos and the doomerism that's going on, is that it doesn't really matter what anybody does right now. The models are out there. People are going to start using them to do breakthrough things. Some will be negative, but the vast, vast majority will be radically positive.

Therefore, it doesn't matter what anybody does to slow this thing down. It's going to go now at its own pace, and I'm hugely optimistic about the future as a result.

Amazing. Anybody else want to weigh in on their P(abundance)?

Yeah, I'll call it P(boom), just for rhyming purposes. I think there's a missing input parameter, which is time. It really should be P(boom, T), with T for time. On a time scale of greater than 10 years, P(boom, T) is greater than 90%.

I have to completely agree.

Well, just to echo what Alex said, I think the risk is all in the next couple of years. It's not AI taking over the world. It's human use of AI. It's the arms race with China. It's what's going on in Ukraine and all the weapons that are going to be built with AI.

Those are the risks, and I think humanity will put those risks behind us definitely within 10 years—maybe more like 5, I hope. If we get that far, I'm 99.9% P(boom). We just need to get from here to there. It's a tricky next couple of years.

Fantastic. Thank you very much, Angelo. Over to you—GS. Excuse me, I'm cooking breakfast for my kids right now.

Yeah. Thank you very much. I'm a big fan of all 4 or 5 of you. You are the only humans I spend the most time with now.

You're quite sure that we're human, GS?

Yes, I hope so. I'm speaking from Dubai. I started following Moonshots at the time of the geopolitical war here, and thank you for giving me the MTP.

I'm a big, big, big fan of Dave and Salim. The moment I heard about Singularity University, an organization where transaction costs were going to go to 0, that was the ring of the bell.

I come from the textile and apparel manufacturing and export industry. I worked in that for 20 years, in this part of the world where there are labor-arbitrage issues. Because of digital AI, those whole transaction costs are going to collapse, so it's important for us to work on and define this industry's future.

My MTP is to reach back to these multi-thousand factories, and then the buying houses and trading houses, to work with and use AI and compete with the world rather than going out of the market.

I have huge thanks for giving me that optimism and confidence to unlearn and relearn in the AI world. I also saved a lot of money because of one piece of advice from you. I had my Harvard OPM batch this year, but I didn't go there, and I saved a lot of money instead of paying them because I'm learning more about AI while listening to you guys.

My question to you is: Salim and Dave, how should I take this step forward with my company, Pertham.AI[?]?

We're building this agentic layer where small enterprises can work intelligently on it without expensive software, ERP, and such, and be competitive. The transaction cost of this industry is 10% to 20% of sales here in this part of the world, which is 5 to 10 times the profitability that industry makes.

That's the kind of inefficiency we're talking about. My MTP is that millions of workers work in the labor industry in that part of the world, and we have to save that. To save that, we have to make sure the white-collar processes are agentic so that efficiency can continue.

Yeah, I'm so glad you asked that question because I see this a lot. People are in industries where manufacturing is considered to be very labor-intensive. But when you look at the actual operating costs of the company, it's all about planning, transactions, and documents, all of which is beautiful AI territory.

Then you look at the numbers and you're like, "Wow, we could drop 10%, 20%, 30% more to the bottom line with stuff that AI can do right now." Everyone's thinking, "First, I would need to have robots all over the place." No, no, no, no.

Do it just with the paperwork, the planning, the scheduling, and the inefficiency of where people are and what they're doing. All of that is perfectly attackable with AI. So, if you productize that across a region for a certain class of companies, that should work incredibly well. It's a very, very cool idea.

One other meta-topic is that everybody thinks all wealth and power is going to go to 2 or 3 places in the world. But if you look at the regulatory environments in those places, including the U.S. and California, a lot of things are going to be very, very slow because of regulatory slowdowns. There are many places in the world that can move much more quickly. So if you can identify the subsets where, sure, it could happen in China or the U.S., but it won't because of government blocking action, that's a huge opportunity.

This happens a ton in biotech. It'll come in self-driving. It'll come in drones—the flying drones, drone delivery. All of those areas will deploy in other parts of the world much more quickly than in the U.S. because of regulatory slowdown. Get those deployed wherever they can take off and flourish, and then they can backport to the U.S. That's always been a really good business plan. Salim, anything you want to add to that? Are you transacting some huge Bangalore deal?

I'm just trying to get my way through this city, so I'll beg off on this particular one.

Can you point the camera out the window? I really want to see some media here.

Okay, hold on. This is Bangalore at night, and all you see is construction. Can you see some of that?

Or how about at the front, where there's a ton of traffic?

Okay, let's go to R.A.

Not the tour I was looking for, but that was all right.

All right. Well, thank you. Listen, this is a greeting from Germany, speaking, Dave, in your language. I listen to a lot of podcasts, international and national ones. You're 10X the best in the world. I've followed you for years. This is for sure. It's arising like pop stars. It's amazing.

My question is, I think in the last couple of editions we've seen an increasing concern about security. Of course, I think we've captured it here a number of times. Then I think Alex framed it a few weeks ago as potentially a marketing thing because all the CEOs have a commercial interest in all of that. Now we see more and more of the Hintons of the world, the Hararis, the Gawdats, and other scientists warning us as well in a certain way.

So far, so good. I mean, it's technology. We need to somehow manage it to find the right answers, maybe on a scale never seen before. What triggers me most at the moment—and this is the same in the U.S. and Europe—is the public opinion on AI. This is predominantly driven by the media, right? Everything that's bad, that's a little poor, that's scary, whatever it is, the media jumps on it and has a headline. I can tell you, here it's even worse than everywhere else, versus Singapore, for instance, or other countries adopting it much more. How do you treat that?

I mean, you're on this path of optimism, like we all are, right? I do this also as part of Accenture, as part of my job every day, and it's exciting. I feel 10 times accelerated as a human being since I started dealing with that. But the media doomerism is so counterproductive. Yes, we have challenges, of course, and we all need to work on those. What's your opinion? I know you try hard, but what do you think?

The media is always going to be broken from here forward because they're starved for money. They have no budget, so they have to chase drama in order to even survive. You just have to write off the mainstream media. I think the antidote to that is micromedia—X and podcasts, narrowcasting, and so on.

I think, in particular, Germany to me is the most talented place on the planet that isn't doing AI. It's just mind-blowing. I keep running into people in California who have come over from Germany, and I'm like, "You're in the right place." What you want to do is go to Palo Alto, go to San Francisco, spend a month, pick up the culture, go in a group of 5 or 7 people, and then backport it—bring it back to Germany and expand it.

Do it with 7 becoming 50, becoming 500 people who are all communicating through podcasts, X, Slack, and texting. They realize that everyone around them is completely out of touch with what's going on, and that's okay because within that group they're self-reinforcing. You just need to get those critical masses of people. I think the antidote to mainstream media is narrowcast media, and it just kind of percolates on its own.

I just wanted to jump in there because, at the end of the day, what you let into your mind is critical, right? Having some news producer decide what you learn or some editor decide what you read—I don't give them that option. I'm very careful about what I put into my body from a food perspective and what I let into my mind from a shaping-my-neural-net perspective. I choose very carefully the content that I absorb, and you guys are doing that as well. You're listening to the podcast. So that's just it. Alex or Peter, do you want to add to that?

I'll just note maybe the obvious point: You could always leave Europe. Europe has been hobbled both due to external factors and internal factors. The post-Cold War era—and there are a number of historians who've written in particular about the role that the George H.W. Bush administration at the end of the Cold War played in deciding whether Europe would be brought even more into the U.S. orbit post-Cold War versus forming a stronger federation like the EU. The bias was to bring Europe into more of a U.S. orbit so that it would be less independent.

That may or may not have been the right geopolitical decision at the time. But I think now what we're seeing is a Europe that's energy-hobbled, that's politically hobbled in certain ways, and that doesn't enjoy—this is again widely reported—certain freedoms of speech and action that are considered fundamental in the U.S.

I think it really is energy policy and associated policies that have left Europe hobbled. So if you're really interested in accelerationism and you're in Europe, what do you do? Either you get your energy act together so that you can afford to power your own data centers, or you leave.

I think this is the question. By the way, this has to happen on a relatively abbreviated timescale because we're in the middle of recursive self-improvement. Anyone who wants to play top-notch ball in the infrastructure game in Europe has to be asking the question right now: Either solve the energy-plus-data-center crunch together with the concomitant policy issues, or just move to, hopefully, the U.S. bloc—the Pax Silica—and do it here.

One of my good friends, Guy Bradley—he's actually British, but he came to the U.S., made a fortune in tech in the U.S., and then moved back to Germany. He speaks perfect German and French. He has a place in southern France and a place in Germany, and there's no better place on the planet to live, in his opinion. I agree, actually, between Germany and France. It's just beautifully protected from everything that's damaging in the world.

You look at other jurisdictions, like India, for example, where there's rampant growth, but it's absolutely trashed. So I think I'm always surprised that more Germans aren't in Boston and Silicon Valley. I think there's a lot of national pride, but if you just do a 5-year tour of duty in the U.S., you're going to be so overwhelmingly happy when you move back to Germany. It really feels like it should happen more.

I can tell you, I'm an international traveler, Salim. I was in Bangalore 21 times, so I see all this rising, and I wonder, Alex, absolutely, what are we doing over here? Why don't we wake up? Why are we under whatever kind of avenue? But anyway, that's another topic. Thank you very much.

Thank you, Rafe. Over to you, Ron.

We'll go for another 10 minutes, and then we're going to have to jump into our normal days here.

Okay. Thanks so much for doing this. I'm calling in from Calgary, Canada, and I'm a longtime listener. I'm a screenwriter-director, so a big special thank-you to Peter, because I was in a bad place early this year when I was writing something about China. I was borderline depressed, but since I've been writing another project for Future Vision, the XPRIZE, I have become a bit more optimistic about the future and everything.

I also read the book Peter wrote, Abundance, and, you know, we are as gods. So I've been thinking: Since everything is interconnected, as Alex would say, what is one of the hardest problems still left? I kept coming back to distrust between rival nations, right? I read Solve Everything two weeks ago, and Alex and Peter, you actually described agreement as one of the new scarcities once cognition becomes abundant.

So I wonder: Could agreement under distrust itself become compute-bound? Could coordination between rivals like the U.S. and China become an engineering problem?

So here's my question. As sovereign AIs increasingly advise governments, could we create a shared protocol that they can connect to and use AI and privacy-preserving computation to search for improving agreements without either side exposing protected data? Is that a technically and institutionally meaningful direction? If it is, what do each of you think is the hardest part we should solve first? Thank you. Alex, over to you, pal.

Yeah. I'll give you a hot take because that's what people want, I think. Multi-party computation, MPC, or distributed multi-party computation, is a very fashionable problem in computer science right now: enabling multiple parties in a zero-trust way to collaborate, share data, achieve convergence, and mediate. Very fashionable.

However, my hot take on this one is I don't think that's the limiting factor for, say, international peace. If you said the goal is, “I don't want a second Cold War between the US and China. I want world peace—1,000 years of peace, or whatever it looks like,” I don't actually think that the solution looks algorithmic in nature. It probably looks more geopolitical-infrastructural: solve the Taiwan issue and remesticate all supply chains everywhere to every country, so that international trade in physical products can afford to be cut off without a global depression.

I think what we've seen, even just in recent years in the Middle East, is in no small part because America can now frack its way—and has fracked its way—to fossil-fuel energy sovereignty and independence. I want to generalize that example, which I think is very instructive, to what happens when the US no longer needs China, Taiwan, South Korea, or Japan for any advanced manufacturing at all. I think it would be a very different world.

So, in summary, if the goal is world peace, obliterate—as perverse as this sounds—obliterate, cook, incinerate the need for global trade in products and services. Perversely, I think you get a very peaceful world. I don't think there's an algorithmic component to that, not to first order.

Yeah, Peter.

I've got oil on my hands. I couldn't get any on me.

Peter, this is your cooking show. We're doing the cooking show. Oh, that's cool. Yeah, let's see what you're doing there.

I'm sure it's healthy, whatever it is.

It is. It's an omelet for my boys. Anthony, over to you.

Yes. Hello to the Moonshot mates. I want to thank you all for your time and for the opportunity to participate in the AMA. The podcast has been very valuable, especially in the time we're in, where people are looking at it cynically and negatively. You guys have been a source of hope and optimism for me.

Just a simple question: What books or material do you recommend reading to prepare for the transition we're in? I know you've mentioned The Diamond Age and Ray's book, The Singularity Is Near, along with some other works he has. I was just curious if you had any other material that you think is important to read.

Yeah, I'll comment, and this is probably the last question I can take before I need to drop. Read Solve Everything, which Peter and I wrote, at solveeverything.org. And then, if you like science fiction, read Accelerando by Charles Stross, which is, I think, the single best science-fiction treatment of what's in the process of happening right now.