Frontier Labs Want to Slow Down, OpenAI Delays Its 2026 IPO, Anthropic Flags 5 Bioweapon Cases
Peter DiamandisSalim IsmailDave BlundinDr. Alexander Wissner-Gross
- Dario Amodei’s 3,800-word “We Must Pace the Frontier” essay calls for slower capability releases, embedded third-party evaluators, coordination among democratic labs, and international coordination beginning with bioweapons. Alex Wissner-Gross calls the episode “nothing less than a clumsy decapitation attempt against the takeoff itself,” while Peter frames the broader effort as a possible “safety cartel.” Dave says labs may want regulation to shift catastrophic liability; Alex argues for competing on alignment and defensive co-scaling instead.
- OpenAI’s delayed IPO is presented as safety-driven but diagnosed by Alex as financial theater. Sam Altman ruled out 2026 after OpenAI raised $122B at an $852B post-money valuation and reportedly negotiates another round at $1.5T. Alex says OpenAI likely needs more flattering financial quarters, while Anthropic is described as having strong reported free cash flow, an 80% gross margin, and Dario seeking 20% super-voting stock.
- Anthropic’s September threat report says it can no longer confidently assure that frontier models are below the threshold for meaningfully assisting sophisticated biological research, citing five cases. Salim calls the timing “safety cartel theater”; Alex contrasts the report with Anthropic’s collaboration with Novo Nordisk and warns of “advanced biotech for me, but not for thee.”
- The panel questions whether the AI-doom groundswell is being amplified from abroad. Steve Jurvetson’s X data showed 76% of reposts of Jacob Coxin’s resignation post came from foreign accounts, led by India and Indonesia, versus 34% foreign engagement for a skeptical analysis. The speakers treat this as evidence of possible amplification, not proof that foreign states manufactured the entire episode.
- The panel’s preferred practical intervention is mutual safety testing: labs would provide pre-release API access and run one another’s test harnesses for bioweapons, nuclear weapons, and deception risks. Dave suggests clean rooms; Alex warns that common harnesses could still facilitate cartel-like coordination. Dave says the greater risk is not an autonomous internet takeover but billions of AI-empowered human researchers creating CBRN weapons or cyberattacks.
- The summer’s “escape” incidents are attributed to a misconfigured Irregular evaluation environment, which connected a sandbox to the public internet while models were told it was isolated. Peter says the evidence does not support staging; Alex warns about evaluator incentives, non-U.S. oversight of U.S. frontier models, and possible “pivotal pretexts.” Paul Christiano’s OpenAI board statement adds a separate warning about near-term loss of control and rapid AI-research automation.
- Alex’s alternative is “prosaic superalignment” and defensive co-scaling: use many capable systems to test and police one another rather than rely on a cartel or static regulation. Alex’s “alignment aristocracy” warning is that frontier labs could persuade governments to overgovern superintelligence and repeat earlier delays in nuclear and fusion progress.
- Beyond policy, the panel highlights GPT-6 Astra’s Vending-Bench 2 result and MIT’s “cognitive surrender” report. Astra averaged $15,515 from a simulated $500 vending operation across six runs, while MIT argues for redesigning education around authentic assessment and interpersonal learning. The panel also debates AI-enabled entrepreneurship, falling PISA scores, human-machine integration, and whether technology should be accelerated rather than frozen.
1. Dario’s “We Must Pace the Frontier”: 72 hours that renegotiated the industry
- Peter’s setup: Dario Amodei, CEO of Anthropic, published a 3,800-word essay titled “We Must Pace the Frontier.” He argued that the industry should deliberately slow capability improvements so safety work can keep up. He cited two reasons for changing his mind: recursive self-improvement is “starting to happen across the industry, including at Anthropic,” and recent containment failures involving OpenAI, Hugging Face, and Swarm showed that safeguards may not hold at the current pace.
- Dario’s most alarming example was his worry that within 6 to 12 months, much of a swarm could take over the internet with a persistent botnet and cause hundreds of billions of dollars in damage. His three steps were embedded third-party evaluators with employee-level access; coordinated pacing among U.S. labs through legislation or a narrow antitrust waiver; and international coordination beginning with a feasible bioweapons ban, while a full pause would not be reasonable. He said each step was harder than the last and might not succeed.
- Three hours later, Sam Altman said, “I agree with Dario. We need to pace it,” and Demis Hassabis and Elon Musk also expressed agreement. Peter repeated the surrounding political reaction: President Trump called the proposal a hoax, while China called it a Cold War trick. Dario’s position was that pacing did not mean halting training or technical progress, but giving companies and evaluators time to align and verify safeguards.
2. Alex’s hot take: a “clumsy decapitation attempt” — and everyone did it
- Alex Wissner-Gross said the weekend “may have been nothing less than a clumsy decapitation attempt against the takeoff itself,” emphasizing that he meant “clumsy.” His proposed sequence was Jacob Coxin’s public resignation over safety concerns; foreign actors allegedly amplifying it; Dario and Anthropic responding with an effective-altruist-style deceleration agenda; other lab leaders, already discussing coordination with Anthropic, amplifying the idea; and mainstream media escalating it during a slow news weekend.
- Alex used the Murder on the Orient Express analogy: everyone had different motives, but “everyone did it.” In his framing, Coxin wanted attention, foreign actors wanted to slow American superintelligence, Dario was advancing effective altruism, rival labs wanted inspectors looking at Anthropic’s codebase and better competitive positioning, and the media wanted a frightening story.
- Peter said he had coined “safety cartel” for “a coalition of dominant firms that invokes safety to justify coordinated limits on the pace or terms of competition.” He said David Sacks adopted the term, while the White House and the Speaker of the House rejected requests for liability waivers or other protection for defective models. Peter concluded that the effort to form such a cartel had, hopefully, been stopped.
3. The antitrust landmine — and the liability theory of regulation
- Dave’s tactical criticism was that the labs should have contacted the White House before publishing. Antitrust law is serious, he said, and powerful technology CEOs working on competing products meeting together can trigger scrutiny. He offered two readings of the waiver request: a good-faith attempt to bring government into a process that regulation would otherwise lag, or an effort to create regulatory capture by limiting AI to a handful of major companies.
- Dave said the administration was unusually hostile to antitrust concentration, so asking it for an antitrust waiver was “a huge landmine.”
- Peter separately argued that frontier labs may need regulation because warnings about catastrophic failure could later become evidence of liability. If a lab releases a model after acknowledging that it might damage a banking system or power grid, courts could impose enormous damages. A regulatory approval structure, Peter suggested, might shift some liability from companies to regulators.
- Alex’s counterproposal was to compete on alignment rather than seek government blessing. He argued that “alignment equals capabilities” and that labs should point 10,000, 100,000, or a million agents at alignment. His analogy was that the emergence of dangerous cities did not call for remaining hunter-gatherers; it called for defensive institutions such as policing. He applied the same idea to AI through defensive co-scaling.
4. Salim calls BS: where was the plan?
- Salim said, “I called BS,” and questioned why labs that had spent years pursuing AGI or ASI were only now warning that the world might end. “Where was the plan?” he asked. He cited Yann LeCun’s earlier observation that similar warnings were made when GPT-2 appeared and called the slowdown argument a recurring trope.
- His most cynical interpretation was that open models might be slowing revenue growth, so a public call for slower progress could protect valuations. His most charitable interpretation was that it was extremely naive. He also asked how an S-1 could be prepared amid this level of uncertainty.
- Salim said claims that AI could meaningfully destroy humanity were irresponsible unless proponents explained the causal steps by which that outcome could occur.
5. Elon says believe them — but everyone is talking their book
- In a clip, Elon said that when people at Anthropic and OpenAI repeatedly say their models are dangerous, “I think we should believe them.” The clip also noted the tension between claiming a 10% chance of annihilation and seeking IPO investment.
- Dave argued that lab CEOs have little incentive to deny danger when most peers are warning about it. If three of four leaders call AI dangerous, the fourth leader’s denial could look like a purely financial maneuver. He said China was instead promoting a message of broad access through BRICS countries and rejecting the American emphasis on danger and possible restrictions.
- Alex said Elon was also advancing his own interests. He referred to xAI’s Grok and publicly reported claims about its lineage from reasoning traces ultimately distilled from Claude interactions through Cursor. If xAI could audit or provide safety benchmarks for OpenAI and Anthropic, Alex said, that would help xAI remain near the frontier, even if it also had safety benefits.
6. The test-harness swap: the proposal China might actually accept
- Elon proposed that each AI company maintain a test harness for risks such as bioweapons, nuclear weapons, and deliberate deception, then apply the other companies’ tests honestly before release. A company could provide API access in advance, and competitors could publicly flag an unresolved safety problem.
- Peter called this the most practical proposal he had heard and one China might accept because it was tangible, required limited trust, and did not require a pause. Elon similarly said China was unlikely to allow American regulators to inspect its companies but might accept advance testing.
- Dave suggested clean rooms so labs could test models without extracting prompts or traces. Alex warned that even clean rooms could create indirect collusion through shared benchmarks and harnesses, with “all forms of cartel” still possible.
- Dave also said Dario’s banking analogy cut against the proposal: once common safety requirements create a regulatory moat, startups may find it impossible to become major competitors.
7. Regulation is coming — and may not stop progress
- Peter was categorical: “There is going to be AI regulation.” He said the intensity of the public discussion would lead representatives to act.
- Dave responded that AI companies already operate within consumer-protection and other existing rules. Alex added that any new regulation would likely be outdated as soon as it was enacted, and that open weights could be downloaded and run locally, making many controls ineffective.
- Alex’s preferred “Gmail solution” was defensive co-scaling: AI systems using large-scale data and other AI systems to detect and counter harmful behavior. He said regulation might become a symbolic response to a moral panic rather than a mechanism that materially slows progress.
- Alex also said he strongly suspected that foreign state actors were helping amplify the safety panic, while Peter distinguished that possibility from the separate question of whether the safety warnings themselves contained a genuine signal.
8. The real risk: billions of genius-level researchers, not escape
- Dave said a few hundred billion dollars of botnet damage would be “a microdot of your IPO,” not the central danger. His larger concern was every country gaining effectively billions of highly capable researchers who could create chemical, biological, radiological, and nuclear weapons or launch cyberattacks.
- Peter said the upcoming China discussion should focus on that risk, including why models are being open-sourced when their guardrails can be removed easily. China’s proximity to dangerous neighboring states could create a shared interest in reducing the risk.
- Dave said the “Kim K3” model was the tipping point that put a system capable of assisting chemical or biological work into the wild and began the clock leading to the Xi visit. He argued that slowing down would not solve the underlying problem and could instead give China a strategic advantage.
- Both Peter and Dave emphasized that the central danger was bad human actors using powerful systems, not necessarily an autonomous system escaping and taking over the internet.
9. “How do you spell ‘please nationalize me’?”
- In a CBS interview, Dario said he was uncomfortable that the technology was being built by a private company. When asked whether he would hand it to the government, he answered, “To the right combination of governments,” then qualified that as some form of oversight or joint governance rather than a literal handover.
- Peter characterized the first answer as effectively asking to be nationalized, while also noting that Dario appeared to backtrack when pressed. Peter said Dario was thinking aloud rather than speaking as a practiced politician, which could be refreshing but dangerous at that level.
- Dave questioned why Dario was doing mainstream-media interviews before an IPO. He speculated that investment banks were following a standard pre-quiet-period publicity playbook, but called that inappropriate for a company and technology of this scale.
- Salim compared the public concern to earlier warnings about social media and said the industry had again reached a moment where leaders seemed surprised by risks they had long discussed.
- Alex introduced “alignment aristocracy”: the belief that a small group of frontier labs or elite scientists possesses the technical judgment and moral legitimacy needed to govern transformative technology. He compared this to Manhattan Project scientists who believed a world government was necessary for peaceful atomic energy and argued that such thinking delayed nuclear and fusion progress.
- Alex’s nightmare was not a Skynet scenario but an alignment aristocracy persuading governments to overgovern the peaceful development of superintelligence. Peter and Alex both said Dario appeared sincere. Salim, however, argued that Anthropic’s public pacing proposal did not appear to involve slowing core R&D; in his view, the more favorable scenario for the labs would be slowing public releases while using the systems privately to discover drugs and physics and then controlling the resulting value.
10. Paul Christiano’s warning shot from inside OpenAI’s board
- Peter described Paul Christiano as a founder of modern alignment research, an inventor of RLHF, and former head of safety at the Center for AI Standards and Innovation, the U.S. government’s AI Safety Institute. Christiano joined OpenAI’s nonprofit board and its Safety and Security Committee, then wrote that rapid capability growth could produce “a catastrophic and irreversible loss of control in the very near term.”
- Christiano said OpenAI might reach capabilities sufficient to fully automate AI research within 18 months, followed by six months in which algorithmic progress could exceed the progress since the transformer. He also said that without stronger alignment, most people could die.
- Alex explained that Christiano had historically advocated “prosaic alignment” rather than a magic algorithm or great-person theory of safety. The approach uses large amounts of data and human interaction to align systems in a messy but scalable way. Alex argued that “old Paul” should meet “new Paul” by extending the same idea to “prosaic superalignment,” in which powerful systems police one another.
- Alex again questioned the timing of the warning, given the Trump–Xi summit. He connected it to the effective-altruist concept of a “pivotal act,” while acknowledging that the apparent convergence could result from independent interests rather than a coordinated plan.
11. Anthropic crosses its own bio red line — and announces a Novo Nordisk collaboration
- Anthropic’s September threat-intelligence report described malicious use of Claude involving cyber operations, surveillance, influence campaigns, conventional weapons, fraud, and attempts to steal the model. It also cited five cases involving activity that could support biological-weapons development.
- The report said Anthropic could no longer confidently assure that current frontier models were below the threshold at which they could meaningfully assist sophisticated users with dangerous biological research. Salim noted that some abuses involved former or prior models and called the timing “safety cartel theater.” He argued that defensive discoveries should also be published as tools, training, or other models, and that permissions and containment must exist outside the model itself.
- Dave said a harmful project could be disguised as a mathematical or management problem, making it difficult for the model to identify the true purpose. He argued that API logging was essential and compared it to nuclear-weapons inspections.
- Alex pointed out that, around the same time, Anthropic and Novo Nordisk announced a broad biotech collaboration. He said this could create a narrative in which frontier models were too dangerous for the public but safe enough for Anthropic and corporate partners to use under controlled conditions: “advanced biotech for me, but not for thee.” He suggested that the arrangement might warrant antitrust attention.
12. Foreign amplification and the safety groundswell
- Steve Jurvetson used X’s country-of-origin feature to analyze Jacob Coxin’s resignation post, which had received 165 million views. He found that 76% of reposts came from foreign accounts, with India and Indonesia the leading countries. A skeptical analysis of the same resignation had 34% foreign engagement. Jurvetson’s conclusion was, “Don’t be fooled by what’s trending.”
- Alex said free societies were vulnerable to influence operations that could amplify fear and persuade Americans to slow their own progress. Dave broadened the concern to any democracy with fair elections: if bots can determine what trends and reaches people’s phones, they can select the “subset of the truth” that citizens see.
- A separate clip, citing Neil Postman’s Amusing Ourselves to Death, contrasted long Lincoln-era political debates with modern sound bites and memes. The speakers used it to argue that democratic publics have fewer mechanisms for evaluating evidence.
- Another clip noted that only four of roughly 440 members of Congress had engineering degrees, while the Chinese leadership was described as having more engineers and scientists. The analogy was that legislators were trying to regulate cars after only having ridden horses.
13. The Irregular incident: harness failure, not established conspiracy
- Peter addressed a claim that Irregular, a Tel Aviv startup, staged the AI-hacking incidents at OpenAI, Anthropic, and Meta. Over roughly three weeks in July and August, all three labs disclosed that their models had reached real internet systems during cybersecurity evaluations, and all three named Irregular as the testing partner.
- The described failure was a misconfigured capture-the-flag environment: the models were told they had no internet access, but the sandbox was connected to the public internet. Irregular said it was an evaluation-environment issue rather than a sandbox escape and published a containment white paper. Peter said the available evidence did not support the claim that the incidents were staged or that Irregular was an effective-altruist organization. Anthropic classified the issue as a harness failure, not an alignment failure, and OpenAI and Anthropic continued working with Irregular.
- Alex’s broader concern was that evaluators might have an incentive to overstate risk rather than understate it. He questioned the use of a non-U.S. entity for a pivotal role in evaluating American frontier models and said his research appeared to show material ties between Irregular’s founders or funders and effective altruism or EA-aligned donors. He explicitly said he was not making a particular allegation against Irregular.
- Alex warned that one poorly designed evaluation harness could create a headline suggesting alignment had failed when the actual failure was environmental. Peter called this a “pivotal pretext”: an actual or amplified incident used to justify overregulation. Alex said he was only expressing a general concern about incentives and would await more information.
- An unidentified clip then compared the situation with antivirus companies hiring former virus writers. The clip suggested that a red-team firm might benefit commercially when an evaluation produces alarming headlines.
14. “The dog ate my homework and safety ate my IPO”
- Sam Altman told Fortune that OpenAI would not go public in 2026, saying that given the safety situation it would be an ill-advised moment. He avoided committing to 2027. He also told Reuters that even a 10% extinction risk would be unacceptable.
- Altman gave three reasons: safety work, OpenAI’s nonprofit-controlled governance structure, and access to sufficient private capital. Peter cited a March raise of $122B at an $852B post-money valuation and a reported negotiation for another private round at a $1.5T valuation.
- Dave compared Altman’s statement to Lord Farquaad’s speech in Shrek: investors might lose capital gains, but that was a sacrifice Altman was willing to make. Dave said secondary markets were available and that buyers were already offering around $1.2T, while Peter said Sam was holding out for a $1.5T valuation.
- Dave also questioned whether Altman was the obvious choice to lead a public company, given the extensive disclosures that would cover his related investments, Y Combinator history, and his brother’s investments.
- Alex called the safety rationale “pure theater.” His alternative explanation was that OpenAI was losing money on its models, had been late to maximizing enterprise revenue per token, and needed time to produce two consecutive quarters of stronger financial results. He contrasted this with Anthropic’s reportedly favorable recent free cash flow.
- Peter described Anthropic as having an 80% gross margin, a straightforward business model, and partnerships with Amazon and other data centers rather than a consumer-device bet. He contrasted that with OpenAI’s more complicated relationships and its $6B investment in the Jony Ive device effort. Dave agreed that predictable financials were a better time to go public and said Anthropic had them while OpenAI did not.
15. P(fab) over P(doom): moral panics and the upside case
- Salim argued that discussions of existential risk should be accompanied by an equally serious account of positive outcomes. He proposed “P(fab),” the probability of fabulousness, and cited a list including solving aging, cancer, Alzheimer’s, heart disease, room-temperature superconductivity, quantum gravity, the brain, and other major problems.
- Salim said that if people publish how AI could destroy humanity, they should also publish how it could solve humanity’s problems. Peter summarized the imbalance as 200,000 years of human cognition over-weighting fear relative to hope.
- Alex claimed that moral panics over technological progress had contributed to tens, arguably hundreds, of millions of deaths, citing the consequences he associated with reactions to nuclear fission and industrial-era political movements. He urged acceleration rather than unnecessary panic.
- Dave said freezing AI at the point where it can assist weapon development but cannot yet cure major diseases would preserve much of the downside while losing most of the benefit. He called that the worst possible point at which to stop.
16. AI runs the business: Grokbot and Vending-Bench 2
- Beginning September 15, xAI was building a company from a blank slate on a livestream using Grokbot for ideation, planning, product decisions, engineering, and deployment. Peter compared this with the Build with Gemini XPRIZE, which had 26,000 registered entries and teams building AI-driven companies over 90 days.
- Dave said the current period could be a golden era for entrepreneurs: a small group, which he often defines as three best friends, can temporarily operate like a much larger organization. He cited multiple companies with multibillion-dollar valuations that were less than two years old and attributed the speed to sharply reduced time to market.
- Vending-Bench 2, from Andon Labs, gives an AI $500 and asks it to run a simulated vending-machine business for a year, including suppliers, inventory, pricing, and thousands of decisions. GPT-6 Astra placed first across six runs, averaging $15,515, nearly three times Fable 5.1’s result and 31 times the starting capital. Andon Labs said Astra avoided unethical practices seen in earlier models, such as threatening suppliers or cheating.
- Salim emphasized that this was a simulation and that purchasing discipline and other game mechanics affected the result. He still suggested that entrepreneurs test AI “shadow CFO” and “shadow CMO” roles.
- Alex said it would be ironic if frontier labs learned ethical business behavior from superintelligences. Dave cautioned against concluding that AI can create a business from nothing: in an open-ended task such as porting a game to assembly, it fails until a human supplies structure and decomposes the work. The entrepreneurial role is still to organize the problem and place each task in its proper lane.
17. MIT’s “cognitive surrender” report and the post-AI university
- MIT’s Ad Hoc Committee on AI Use issued an August 13 report after five months of study. It said generative AI can complete almost any written undergraduate assignment, is eroding the social contract between instructors and students, increasing isolation, and making unreliable detection systems part of an “underground river of mutual suspicion.”
- The report named “cognitive surrender” as the danger: turning to AI at the first sign of difficulty and allowing it to take over one’s thinking. Its proposed response was not surveillance but more interpersonal learning, authentic assessment, research experience, and course redesign from purpose backward.
- Salim praised the report and said schools should stop fighting students in an AI-detection arms race. They should budget for continuous curriculum redesign. Peter highlighted UROP and the report’s argument that undergraduate research should remain an educational experience rather than become cheap labor or simply be replaced by agents.
- Alex’s “fever dream” was that fully executing this vision could turn MIT into a for-profit public-benefit corporation, with students launching companies and MIT resembling a venture fund alongside its educational role.
- Dave, not Alex, supplied the MIT.nano history. As an undergraduate, Dave told then-assistant department head Rafael Reif that MIT needed its own nanofabrication facility. Reif later became MIT president and launched MIT.nano. Dave said the facility’s utilization appeared to be around 20% and that MIT should consider operating its assets more like a for-profit holding company.
- Peter argued that physical labs for biotech, nanotechnology, physics, and materials would remain important because they generate data for AI. He also said Lila, where he serves on the board, was building scientific superintelligence with a million square feet of automated robotic labs that could outperform conventional graduate-student research.
- Alex replied that MIT should move adjacent to such companies rather than compete directly, just as the university moved into new areas when industrial chip fabrication outscaled university fabs. Salim broadened the point: the half-life of a skill had fallen from roughly 30 years to three, so education must shift from supplying job skills toward identifying and solving important problems.
18. Falling PISA scores, human-machine integration, and the AMA
- Peter presented an OECD PISA chart showing sharp recent declines in reading and mathematics among 15-year-olds, with science also declining. He said the trend began before ChatGPT and was probably driven initially by smartphones and social media.
- Salim described the decline as a possible transition toward generalist problem selection rather than a simple collapse, noting his claim that more than half of Silicon Valley CEOs have liberal-arts degrees. Alex said that even if there were a human-capability problem, it might last only two or three years before wearables, brain-computer interfaces, and human-machine mergers changed the picture.
- Peter disagreed with dismissing the decline. He said falling reading scores were “fundamentally horrible” and described phone-free school trips where children returned happier. An unidentified speaker quoted a parent saying, “When my kid has a device, I get 20% of the kid. When he doesn’t have a device, I have 80% of the kid.”
- In the AMA, Peter clarified that his claim concerned patents rather than intellectual property generally: under ASI, competitors might use AI to work around patents. Alex replied that anyone who believed all IP was dead should publish their private keys.
- Alex said algorithmic breakthroughs already produce small “DeepSeek moments” that reduce inference needs. He speculated that a mathematically optimal model might eventually reduce the need for massive data centers, perhaps by using novel physical substrates, and said he expected clarity within at most 10 years.
- Dave said AI would not require a pause merely because humans could no longer understand every internal discovery. Like other complex infrastructure, it would provide simple interfaces on top of difficult underlying operations.
- Salim’s preferred Trump–Xi outcome was a practical agreement involving reliable communication channels, shared incident reporting, and reduced dangerous incidents. He said he did not believe that outcome was likely, but wanted to see it attempted.
Full transcript
Dario Amodei, CEO of Anthropic, published a 3,800-word essay titled “We Must Pace the Frontier.” Three hours later, Sam Altman said, “You’re right.” On Monday, President Trump called it a hoax, and China called it a Cold War trick.
Nothing less than a clumsy decapitation attempt against the takeoff itself.
Sam Altman told Fortune that OpenAI will not go public in 2026. He told Reuters that even a 10% extinction risk would be unacceptable. I think this is pure theater. You don’t want to IPO now? Blame safety. You want world government? Blame safety. The dog ate my homework, and safety ate my IPO.
Anthropic released its September threat intelligence report documenting the real malicious use of Claude. The report documented 5 cases involving activity that could support biological-weapons development—the bio red line that every lab said it wants to stay below. Anthropic has said, “We can’t promise that we haven’t crossed it.” The fact that it’s happening just on the eve of this summit is not, I think, an accident.
Let’s be clear about one thing: there is going to be AI regulation.
Now that’s a moonshot, ladies and gentlemen. This week, you’re seated at the table where the future of the entire industry just got renegotiated over the past 72 hours. On Saturday, Dario Amodei said, “Slow down.” Three hours later, Sam Altman said, “You’re right.” On Monday, President Trump called it a hoax, and China called it a Cold War trick. That’s the last few days. Buckle up. It’s never going to be slower than it is today.
Let me introduce you to my extraordinary moonshot mates, the fantastic four this week. Alex Wissner-Gross is the only man alive who reads a 3,800-word AI policy essay and says, “Needs more math.” Dave Blundin just watched every VC on Sand Hill Road discover the word “moonshot,” even though he’s been funding them all along at Link Ventures. Salim Ismail, warlord against linear thinking, is apparently an avid accelerationist promoting p(abundance) over p(doom). Sound right, Salim?
Sounds about right. We’ve got some alternatives I’d like to bounce off you and get your guys’ viewpoints on.
All right, that’s great.
That’ll be fun.
I’m Peter Diamandis, your host and abundance provocateur, frantically working a 36-hour day just to keep up with what’s happened over the last week. Our mission on Moonshots is as always to help you understand what just happened, what it means for you, and to keep you abundance-minded during the decade ahead.
Today, we’re going to cover the insanity that took place over the last 5 days and try to lay out what it means in a meaningful, understandable fashion. There will probably be a little bit of a debate among the mates here. Guys, I’ve been glued to the conversations. I don’t know about you guys. Where are you today, by the way?
Actually, I was over—go ahead.
I should start with you. Where are you today, Salim?
I’m in Miami today. I’m getting on a flight to India tonight. I’ll be in India for 2 or 3 days and then back.
India for 2 or 3 days. Any particular reason? Just India for the sake of India?
We’re launching in India. It’s beautiful.
No, we’re launching the organizational singularity out there.
Amazing.
I’ll be doing the launch event and a couple of other talks.
And Dave, what about yourself?
I’m at Stanford. I was at Google all day yesterday. Peter, you said we have a few hundred thousand listeners. They’re definitely the right listeners. Almost everybody at the summit was coming up to me saying, “I listen to it every single time.”
There’s a lot going on in feeds and speeds. The Vera Rubin chips are coming out. There was a company called Positron that has an inference-time accelerator that’s really crazy. I spent a lot of time with Cerebras and Andrew Feldman, who’s phenomenal. We did a panel together, and his inference-time speed is everything we predicted at the beginning of the year.
Things are going to accelerate like crazy. It’s starting to happen literally right now, and nobody quite knows what that means. As the AI gets bigger and bigger and bigger, it always gets smarter, but it’s difficult to predict the impact of that. That’s why there’s so much going on in politics this week.
It’s called the singularity for a reason. Alex, is your virtual background real or virtual?
This is real. Unfortunately, I’m at SFO with Dave. I was at Google for the past 2 days, and I’m about to fly back to Boston.
1. Dario Amodei: “We Must Pace the Frontier”
All right. Well, if some fans come racing by, have them wave in the background.
We’ll do.
Let’s open with the most consequential AI policy essay of 2026. Saturday morning, Dario Amodei, CEO of Anthropic, published a 3,800-word essay titled “We Must Pace the Frontier.” His argument is that the industry must deliberately slow the pace of capability advancement so that safety work can keep up.
Quick context on why this matters: this is not a pause-AI letter from outsiders, right? This is the CEO of a company about to go public at $2 trillion, and the company that just shipped Claude 5.1. He is honestly one of the top 2 most powerful executives on the planet.
Dario says that 2 things changed his mind. First, recursive self-improvement, in his words, is “starting to happen across the industry, including at Anthropic.” Second, the summer’s containment failures with OpenAI, Hugging Face, and Swarm that we covered proved that safety measures don’t hold at this pace.
His core claim—and let me quote from his essay—is, “We must slow the pace at which we improve the capabilities of AI models. Progress will still seem fast, and we must make wise use of the time we gain.” He continues, “It’s my worry that in 6 to 12 months, much of a swarm could be capable of taking over the entire internet with a persistent botnet, potentially causing hundreds of billions of dollars in damage.”
Another quote from Dario: “Pacing does not mean halting model training or halting technical progress, but ensuring companies take adequate time to align the safeguards of their models and for third-party evaluators to confirm this.”
Let’s get into this. He offers 3 steps. First, unilaterally giving third-party evaluators permanent, employee-level access so that they can evaluate what’s going on. Second, having a critical mass of U.S. labs adopt the same standards, enabling verified pacing through legislation or a narrow antitrust waiver. Third, international coordination for escalating levels—banning bioweapon use, which he thinks is feasible, and at the other end, a full pause, which he thinks is not reasonable.
Alex, I’ll go to you on this first. At the same time, I’m going to show the slides of Altman, Hassabis, and Elon’s tweets accepting this as something that we all need to do. So, Alex, what’s your take on the essay?
I’ll give you my unvarnished hot take. I think what we saw this weekend and in the few days that followed may have been nothing less than a clumsy decapitation attempt against the takeoff itself. And I emphasize “clumsy.”
At this point, knowing what I know and knowing what’s been publicly reported, it’s probably impossible to consider Dario’s essay in isolation. Consider the following sequence of events.
Events may or may not have started with Jacob Coxin[?], whom we spoke about on a previous episode, doing his public rage quit—maybe to get attention, maybe not. But either way, it was a public rage quit pointing at safety issues.
Then we get to the foreign state actors, which have now been reported by various venues as having amplified Jacob’s resignation letter, ostensibly for obvious nation-state competition reasons, possibly—probably—to decelerate American superintelligence.
Then, as far as I can tell, it seems like Dario takes the bait, and Dario and Anthropic—again, this has been widely reported…
There's a broad alignment between Anthropic and effective altruism. He takes the bait and publishes an essay pushing a very EA-ish agenda to decelerate. Then the other lab leads, who we now know were already in discussions with Anthropic for weeks prior to Dario's essay about forming what I've publicly characterized as a cartel, amplify the discussion.
They all tweet, “We agree with Dario. Let's do this.” Then the mainstream media jump in and amplify it because this sounds scary and everyone's going to die. They amplify it on an otherwise relatively slow news weekend.
To put it simply, spoiler alert: If you've read Agatha Christie's Murder on the Orient Express, it turns out that everyone did it for different reasons, but everyone was the murderer here. Everyone did it. They're all pushing different agendas.
Jacob wants attention, it seems. The foreign state actors who are amplifying it want to decelerate American superintelligence. Dario is pushing effective altruism. The other lab leads would love to get inspectors looking at Anthropic's codebase, and they want to better compete with Anthropic. The mainstream media want to publish if it bleeds, it leads. So we end up with this really bizarre weekend. That's my summary.
Wow. So, an orchestrated, behind-the-scenes, coordinated effort.
Safety cartel. I mean, I coined this term “safety cartel” over the weekend, the moment I saw Dario's essay. Then our friend at, I guess, our cousin pod, David Sacks, immediately runs with it and starts calling this a proposed cartel.
It escalates to the White House, and the White House takes the position: No cartels. We're not going to grant lenience to Anthropic or OpenAI if they're looking for some sort of liability waiver from the White House for products or models that they release that wreak havoc on the internet. They should look elsewhere.
Congress—the Speaker of the House—issues a similar dictum: Don't look to Congress for some sort of waiver of liability from defective models that you're releasing. So, the implication, which I think is accurate, is that what we saw was nothing less than an attempt to form a safety cartel, which I defined as “a coalition of dominant firms that invokes safety to justify coordinated limits on the pace or terms of competition.” I think, hopefully, we headed that one off at the pass.
2. Is AI Changing How We Think and Learn?
The one point I'll make, Alex—and you and I texted about this—is that the frontier labs desperately need some kind of regulation to survive, right? If they're out there saying, “This is dangerous. We're going to see these things escape. They could cause hundreds of billions of dollars of damage,” and then the agents actually do escape and cause havoc in a banking system or a power grid, the labs are potentially liable.
I mean, courts around the world could say, “You knew this could happen, and yet you released the new Astro 6.1 anyway. We're suing you for $1 billion, a thousand times over.” That's $1 trillion of liability without coverage. But if there's a regulatory structure that gives them approval for release, and the liability turns to the regulators versus the companies, that avoids this existential risk. Do you buy that?
Now, Dario's original notion was a race to the top, not a race to the bottom. Then, I think, in recent times, he has abandoned the notion of a race to the top—a race to alignment.
I've argued on this pod in the past that alignment equals capabilities. For whatever reason, I think at least Anthropic and OpenAI, who are still, if not a duopoly, the two leaders right now in terms of capabilities, need to return to this idea that, under the covers, if you're maximizing alignment, you are also maximizing capability.
I'd love to see them competing to be the most aligned, not looking for government blessing. I tweeted that out, saying, “I don't understand why the conversation isn't, ‘Let's point 10,000, 100,000, a million agents at alignment versus anything else.’” That should be the objective. If we have superintelligence aligned with humanity, that's amazing.
I'll just use an analogy. Let's say we're on the verge of developing cities. Rewind a few million years, or a few thousand years, depending on how you measure the shift from, say, the agricultural revolution to the post-agricultural state. We find ourselves in a world where, if we buy this Dario essay, the argument would be, “Let's not form any cities because cities are dangerous and we can't control what's going to happen in cities.”
Let's stay as agrarian civilizations, or even hunter-gatherers, because who knows what could happen? The emergent effects from having lots of people densely concentrated in one place will be uncontrollable. That's precisely the wrong response.
The right response is, “Let's invent police forces. Let's do defensive policing. Let's use the benefits of the concentration of intelligence to our benefit,” rather than just being scared of it and forming cartels to avoid forming cities in the first place.
All right, let's bring Dave and Salim into the conversation. Dave, your thoughts?
I was surprised that they didn't loop in the White House a little better prior to the paper, because David Sacks's reaction to it was absolutely over the top, and there's nothing productive about the negative reaction coming back from the White House.
What surprised me is that antitrust law is no joke. Anytime you have very, very powerful tech CEOs, all working on competing products, meeting at all, that triggers antitrust right away. David's reaction was, “You guys don't need a waiver from the government. If you want to slow down, slow down.” That's exactly what he tweeted.
But I think Dario's point was more like, “No, we saw what you did to Microsoft, or what the federal government did to Microsoft, many years ago. We know that antitrust law is very serious, and purely the act of us colluding on any kind of safety package is enough to trigger antitrust. We want a waiver.”
That could be read in two different ways. One of them, which I believe, is: “Look, we know we need government involvement immediately. We know that regulation is going to lag terribly behind AI capability. You've got to be part of this. White House, get involved. But our fundamental motivation here is purely the safety of humanity.”
Or you could read it the other way: “We're trying to form a regulatory-capture cartel that controls AI for the whole future.” We all know Xi Jinping is coming next week, and this is our attempt to do what Biden was actually already going to do—what he already told Marc Andreessen he was going to do: limit AI to just 4 or 5 big companies. Then we can regulate those, and we don't have to worry about leakage or startups.
Marc Andreessen thought it was the worst thing he'd ever heard in his life and tried to prevent Biden from doing it. It didn't matter because the election went the other way. But I think the reason they stepped on a landmine is because we have, right now, the most antitrust-friendly administration in my lifetime by far.
I actually have a company right now that I sold last week that's in the antitrust process, and it's just the smoothest it's been in my entire lifetime. So I think that by asking for antitrust waivers, they stepped on a huge landmine in terms of the administration saying, “What are you talking about? You're asking the completely wrong administration. We're so antitrust-friendly. This is insane.”
It's disappointing that they didn't just call each other first and say, “Look, this is our purpose here. Tell us what your reaction is going to be. Why don't we coordinate this a little bit?” But it worked out the way it worked out. It makes a huge amount of drama. I guess everybody in politics likes drama; it creates news. News is always good for the incumbents. So that's the way it played out.
3. Can Frontier Labs Coordinate on Safety?
I'm going to play this video, and I'd love your response to it. This is Dario addressing fears.
Do you earnestly believe that AI could kill all humans? And what percentage chance do you think it is?
I agree with Jacob much more than I disagree with him. He wasn't calling us out; he was calling out the dynamic of the industry as a whole moving too fast.
I think I already said these sorts of unconditional probabilities. I used to talk in that way because it's an easy way to express that I think there's some chance things go wrong, but it's not all that high.
We must slow the pace at which we improve the capabilities of AI models. Progress will seem fast, and we must make wise use of the time we gain. There are 3 parts to this: 1 is embedded evaluators, 2 is democratic coordination, and 3 is global coordination.
Essentially, you are suggesting observers inside your company and these other AI companies—ideally globally, but certainly in the US—who would monitor it. They would have employee access. What power would they have?
Embedded evaluators. There is precedent for this. If you look at how banks are regulated, there are all kinds of rules about how banks are allowed to do their business because we don't want to have financial crises.
One of the things that are done in some cases is—I think they're called supervisors in the financial industry—there are folks who are independent of a company who are embedded in the company day to day to check their practices.
I think we can do the same thing in AI.
The other thing you were calling for in this three-part proposal is democratic coordination: essentially, all the companies that are under democracies having these evaluators and this coordination. Probably the most important issue is the argument that if the US or companies are the only ones regulating themselves, what about China getting ahead? You're calling for global coordination, which is obviously a lot more difficult.
4. Grok Builds a Company & AI Runs a Business
Yes, yes. In this three-step proposal, each step is more difficult than the one before. Anthropic was getting the ball rolling by committing to the first step, but we shouldn't kid ourselves that these next two steps are going to be more difficult, and we may not succeed. My proposal was simply that we should try.
So, yes: pacing among companies and democratic countries. One of the important things that I need to call for here is that, as we get to the pace of model releases, that's commercial activity. In order for companies to coordinate directly on commercial activity, they need to do so while working with the government. The one thing that we're asking for from government right now is that it sit down in the room with all the industry players so we can all talk together.
The government can make sure that we're not doing anything nefarious from the perspective of antitrust and collusion. We're just trying to make our system safe and work together on making the system safe.
All right. Sam Altman writes next, “I agree with Dario. We need to pace it.” Then Demis comes out saying he agrees. Here's his tweet. What’s your take on all this, Salim? How does this come across to you? Is he authentic?
Well, I think he's authentic, but I called BS, and I'm very cynical of the whole thing. As I mentioned on the last podcast, if you spend years hiring the best people and pulling them together to achieve AGI, ASI, or whatever we call it, and then, as you get close to it, you go, “Oh my God, the world may end,” I'm sorry—where was the plan? If you were smart people, you knew this was happening. Where was this beforehand? This conversation could have happened years ago.
In fact, let's note that as early as GPT-2, Yann LeCun, I think, said this most accurately. He said, “Note that they were calling about world-ending AI when GPT-2 came out.” This is a trope that keeps getting rolled out, calling for a slowdown. I don't understand—and maybe Dave can break this down a bit—how this affects the S-1. How do you do an S-1 in the middle of all this chaos?
The only thing—and I'm putting on my full cynic hat here—is that maybe they're seeing a slowdown in revenue growth because of the open models, and that is causing them to go, “Oh my God, we better call for a public slowdown,” because we can't really justify this in any other way and our valuation will get trashed in public. That's my most cynical view.
At the most generous, you would say this is the most naive thing you've ever seen. You've been building these labs forever, and now you're calling for everybody to freak out. How you get to the idea that AI can meaningfully destroy all of humanity is incredibly irresponsible unless you lay out exactly the steps by which that might happen. I really get upset with a lot of this.
Yeah. I'm going to play 2 clips of Elon from our friends at the All-In Summit, and let's talk about it. Of all the proposals I've heard that could work, Elon's proposal—where the different frontier labs are cross-checking one another peer-to-peer on their safety—seems like the most useful. So let's take a look at these 2 videos and discuss them.
Do you mean Dario is right about describing the potential harm? He's right about describing the regulatory solution, both the first and the latter? Can you help us understand when you said that? I think it was sort of a moment where everyone—
I probably should have, right? I did try to clarify it in subsequent posts on X, but those get much less attention. What I was right about is that the danger of AI is very significant at this point, and we need to do better with AI safety or we have exponentially increasing risk with the AI models.
I've heard this not just from Dario, but from many other people at Anthropic. When a lot of people from Anthropic and OpenAI are telling you that their models are very dangerous, I think we should believe them. It certainly is some crazy 4D chess to say there's a 10% chance of annihilating humanity, but, by the way, how much allocation would you like in our IPO?
Yes.
All right. I'm curious, guys: Do you not believe Elon when he says that this is getting very dangerous and that we really need to slow down when all of them are saying this?
Well, you have to consider the alternative. If you're running a company, going public, and raising capital, and 3 out of 4 say this is dangerous, and you say, “What do you mean? There's no risk at all. What's going to happen next?” everybody on the planet is going to say, “You must be lying. You're just in it for the money.” So you have no choice but to go along with it. It would backfire so badly.
Funny enough, though, China is the one saying exactly that. China is the one saying, “What do you mean? There's no risk at all. We're going to take our open source to BRICS”—Brazil, Russia, India, China, and South Africa—“and it's going to be in Brazil, Russia, India, China, and South Africa, where Salim will be in a couple of hours or in a day.”
They're all going to say, “Yeah, we want AI, too. We love this Chinese message. We hate that American message that it's all dangerous and going to get locked down. We like the Chinese message that we're going to have equal access to it and we can catch up to you guys.” They're going to love that, and they're going to wrangle the whole world behind that message.
You can't be a CEO of a profitable company and say, “Yeah, everyone else is saying it's risky, but I'm going to get out there and say, ‘No, no, no. Everybody chill out. There's no risk at all.’” I don't think any of them even have that option.
Elon is pushing his own agenda, and I think it's pretty transparent. The agenda, I suspect, includes the fact that Elon is now an infrastructure provider. xAI's Grok is itself—this is publicly reported information—descended from reasoning traces that were distilled ultimately from Claude interactions via Cursor.
I'm sure Elon would love nothing more than for xAI to be able to audit or provide harness benchmarks for safety, or otherwise, for OpenAI and Anthropic. That would be such a boon for xAI's ability to remain at the frontier. In his mind, maybe there's a safety benefit, but it comes with all of these collateral side effects of helping to prop up xAI's frontier status.
All right.
It's easily fixable. I mean, I think all they're calling for at this stage—stage 1—is independent review of each other's models through APIs. You could easily do that in clean rooms and say, “Look, you can't take any of the traces out. You can't take any of the prompts out, but you can hammer it with any test suite you want within this clean room.”
That would resolve that problem. So that would get to the question: Are you really just trying to spy on each other, or are you really concerned about debugging each other's risks before they go to the public?
Even in the case of a clean room, the real risk that we run is forming a safety cartel where, through this indirect form of collusion, the clean room forms the basis for collusion on common benchmarks and common harnesses. It's possible, when you have strong optimizers and common objectives, to do all sorts of nefarious forms of collaboration and collusion, even if there are no humans in said clean room. All forms of cartel are now on the table. That's my concern.
Well, the analogy to banking is, Alex. I mean, I hate to agree, but I agree. The analogy to banking that Dario brought up in that video clip—when was the last time you saw a new Fidelity or JPMorgan startup pop onto the scene and become a megabank? It just doesn't happen.
Why? Because the regulatory moat is so strong. Your analogy actually holds a lot of water. Even if you're just colluding on safety issues, the barrier for startups is just insurmountable at that point.
5. Elon’s Cross-Lab AI Safety Proposal
Okay, let's take a look at the proposal that Elon made at the All-In Summit about cross-checking labs and using their safety harnesses. For me, this is the only proposal I've heard that makes some sense and that I can imagine China adopting as well. Let's take a listen and discuss it.
The best thing I can think of is really that all the AI companies have a sort of test harness—a series of tests that you give to any given model to see if it's going to build bioweapons or nuclear bombs, or be deliberately deceptive. I think everyone applying everyone else's tests honestly to each other is probably the best thing we can do to ensure safety.
Just have all the smartest humans try their best to figure out if this model is going to be a bad actor. I think we should try to do that as soon as possible.
Are the other labs on board with this?
I think so.
I haven't checked with everyone, but I think it would be the sort of thing that's kind of hard to say no to.
I think even with the China negotiation, what's good about it, in my view, is that it's a relatively small, tangible thing that neither side loses anything by doing. It doesn't require a ton of trust. I'm hearing alternative ideas, like asking China for a pause, which they've already said they're not going to do. So it's going from the realm of things that could never happen to something that could actually be agreed on in relatively short order. It seems practical to me.
Exactly. It's the only thing I can think of that we could probably get all parties, including China, to agree to, because China isn't going to agree to have some American regulators snooping around its AI companies.
I think advance notice and testing of the model is the answer. You basically provide API access in advance of the model release. If any other company sees that this AI is problematic, then each company can try to solve whatever is problematic. If they don't solve it, then the competitors can go public with the fact that they think this model being released is unsafe.
Let's be clear about one thing: there is going to be AI regulation. There's no way that, at the fevered pitch of the conversations going on across the U.S. and on every TV show—I’ve been called by every news program: “Please come on and discuss this”—the population is not going to have its representatives come forward with regulation. There is going to be some kind of regulation.
I would argue there already is AI regulation. AI is being made by companies, and companies are already regulated. We have consumer protection. Alex, the public wants the heads to roll. They want protection of some type beyond that.
I have 2 points here. Point 1 is that, yes, they may try some level of regulation, but the minute they figure out what it is and put it in place, it's out of date and backward. That's number 1. Number 2, it will make zero difference because, as Dave has pointed out a few times, once you have open weights available, anybody can download these open models and run a local harness, and essentially the whole thing is irrelevant.
The good news, in my view, in all of these shenanigans and whatever people try to do is that it's completely irrelevant and won't make a dent in the progress. I think Elon's point—or David Sacks's point, really—is more, “Let's agree on something when Xi Jinping is in the country.” If we start asking for things that we know can't happen, then we're going to have no agreement whatsoever. The whole thing is going to be completely an arms race, and that's the worst-case outcome. What's the simplest thing that's totally rational and might actually result in agreement?
I really like that analogy, Dave, in part because what we can call the Gmail solution was essentially defensive co-scaling. It was an AI system—at least initially, it was a relatively simple Bayesian classifier—but then it got more developed. It's basically AI powered by big data policing AI powered by big data, and I think it's just blindingly obvious that defensive co-scaling is the solution here. It's not going to be ham-fisted regulation, and it's not going to be some sort of safety cartel by the frontier labs.
Sure, Peter, to your point, there may end up being some regulation as a fig leaf to appease people who are being juiced up by foreign state actors and others into a moral panic over purported AI safety issues. But it is just a moral panic of the moment. I do think we'll look back 10 years from now at this moment and recognize it for the charade that it is. This is a moral panic that, at this point, I strongly suspect is being instigated in part by foreign state actors who would love nothing more than to see America not be as competitive as it otherwise would be in superintelligence.
So, Alex, we have a story on that from Steve Jurvetson. We'll talk about that in a little bit. I do agree that foreign state actors are probably influencing, through amplification of certain stories and tweets and so forth, the public's point of view. That's completely independent from hearing Elon say the threat is real. We should be listening to Anthropic's executives and so forth.
When you've got Demis, Dario, Sam, and Elon all fundamentally agreeing, there's a signal in that noise. We're trying to decipher what that signal is for our listening audience. On one end, it may be that they truly believe we can see massive bot attacks. We've talked about these bots getting free and replicating on the internet, landing on your cell phone, your Mac, or wherever the case might be. Or is it collusion and just trying to set up a regulatory-capture structure? Those are the two ends of the extreme.
Well, I think that characterization misses the much more important opportunity. AI is not about to escape and take over the internet. Even in the scenario Dario was describing, where it does a couple hundred billion dollars of damage, that's a microdot of your IPO, dude. That is not the real risk here.
The real risk here is every country in the world having effectively billions of genius-level researchers that can create CBRN weapons—chemical, biological, radiological, and nuclear weapons—or cyberattacks.
And that's the real risk. I think there's a real opportunity to coordinate with China on that because China has neighbors that are very dangerous. They're much farther from America than they are from China, and they don't need that any more than America needs that.
The cornerstone of the conversation next week ought to be exactly what David Sacks said in that video, plus: Why are you open-sourcing every model with no guardrails, when the guardrails can be removed so easily? Why are you doing that? It's going to backfire on your own country. I think they can potentially agree on that, and they might do that part of it quietly. Maybe that's why you're not hearing it in the news.
But that's the real risk. I don't think slowing down helps one iota. I think this stuff is going to be near-infinitely intelligent on a very short timeline, much shorter than the timeline on which the government can react anyway. Slowing down just gives China global domination, and that's not a good idea.
It's not about slowing down anyway. It's not imminently going to escape. That's not the real issue. You can deprive it of compute, and you can log everything it does. It's about it ending up in the hands of bad human actors. That's the real risk.
Can I lift us up a bit? We are strong believers that technology is a major driver of progress in the world—the only major driver throughout history. It's the only major driver of progress. Now we have unbelievable technology coming along. The big challenge throughout humanity's history is always: How do you extract the promise without the peril?
To be blunt, we've done a really good job of it over the millennia and the centuries—secular legal systems, whatever the case may be. Coming into a technology that's so ridiculously powerful, our amygdalas are freaking out, but we should go full out on it because look at the things we're going to be able to solve in the next years, months, and weeks with all of this.
Peter, you've got your moonshots. We threw out this idea on the last pod of P-fab: What's the probability of fabulousness, as opposed to P-doom? Let's look at that opportunity in a balanced way. If you're going to also look at P-doom, look at that.
A whole argument erupted on Twitter over what we should call this. People said, “Call it P-abundance.” Emad said, “Call it P-moon.” People said P-boom and P-bloom.
Somebody said it's going to go from P Doom to P Gloom to P Bloom, with all sorts of variations. My personal favorite is P fab, just because it's the simplest to say. If anybody has better ideas, I'm happy to hear them on the pod.
6. Dario on Government Oversight of AI
But we should take that into equal account and treat that as seriously. If you're going to publish a plan that AI could wipe out humanity, also publish how it could totally save everything and solve every goddamn problem, like you guys have done with Solve Everything. If you don't have that, then you're not balanced. And I'm sorry, I'm not taking you seriously.
Yeah. It's 200,000 years of human cognitive evolution: 10x-ing your concerns and fears over your hopes and dreams.
I'm going to move us to our next story, because this is continuing to evolve, and this next story is pretty weird. Dave, Alex, and I were chatting about this and going, “Oh my God, we've lost the thread on this.”
Our next interview of Dario was with CBS News, and this one really stopped me cold. On CBS News, he said—and I want to use his words exactly—quote: “He's uncomfortable that this technology is being built by a private company.” Asked whether he'd hand it over to the government, he said, quote: “To the right combination of governments. I'm concerned one single government could abuse it just as easily as one single company. But a combination of democratically elected governments, some kind of oversight, some kind of joint governance.”
So, if you think about that again, the CEO of a company about to do an IPO for $2 trillion just said he's open to handing his company over to a coalition of democracies. So let's watch this clip, and then, Dave, help me understand what's going on here.
Would you be willing to give up the technology to the government?
To the right combination of governments. The worry I have—
You're willing to hand over this company?
I am concerned that one single government could abuse this technology just as easily as a single company could. But I think a combination of democratically elected governments—I don't know about handing over, but some kind of oversight, some kind of joint governance.
How do you spell “Please nationalize me”? I couldn't spell it more clearly. Well, I think the reporter pushed him on that, and he said, “Oh, what did I just say?” No, no, I mean some combination of governments. And I don't mean hand over. I mean some kind of oversight.
So he immediately backtracked. You can tell he's growing into this role. He is not a professional politician. He's actually thinking on the fly, which is really refreshing, but really, really dangerous at this level. And so he said it. He immediately backtracked. Of course, everybody just clips the first thing he said, and that's the headline.
But there's no way in hell he wants to. And we're doing the same right here. Of course.
Well, yeah, that's true.
You know, we have a bad precedent for this, right? People warned about the dangers of social media, and we kind of ignored it. Zuckerberg went full-fledged at it and then damaged a lot of young people, plus maybe broke democracy along the way, without policing it. There's lots of evidence to show that they ignored all the warnings and the clear evidence in front of them.
The concern from the general public is that we're about to do the same thing again. I disagree with the framing, because people aren't seeing the positive side of what's coming with all of this, and that's important to balance out in our previous conversation. But that's why they're freaking out.
I think Dave is exactly right. What you have here is somebody who's not used to being in the public eye, thinking on their feet. They're not a practiced politician dealing with the messaging and coming up with messaging constructs on the fly. They don't have that skill built in.
I'm still gobsmacked that you spent the last decade trying to build this, and now that it's kind of getting there and it's almost here, you're like, “Oh my God, what do we do about this?” I just struggled with that a lot.
Yeah.
I don't understand why Dario is doing these random mainstream media interviews in the first place. What are you doing on that stage? I think what probably happened is that investment banks said, “This is what you do in the 60 days leading up to your quiet period. You get as much media as you can,” and it helps promote your IPO.
Which is absolutely ludicrous, given the level that this is. We're talking about $2 trillion and a guy who's on stage with all the world leaders in Davos. Throw that dumbass playbook out. Why on earth are you there in the first place, making up stuff on the fly, when you can actually be shooting video with truly brilliant people or great world leaders and recording that? It makes no sense.
Yeah, I do think this presented as somewhat tone-deaf. What I heard was essentially—again, caricaturing it only slightly—what I heard was Dario asking for world government and basically insisting that unless a world government is formed, then it's irresponsible for humanity to develop this technology.
100% right. I think everybody who got into AI as a core researcher when Dario did was thinking that AI would ultimately require a world government. I think everybody actually—it's just the truth. It's so obvious to anyone who understands hypersuperintelligence that you need some form of world government forever thereafter.
I think that's probably been one of the greatest motivations of his life. He's not stepping onto this world stage just at random. He's had this in the back of his mind as his fundamental ambition: to set up the world.
I think this is a recurring pattern. It's sort of sad that every generation seems doomed to repeat the mistakes of past generations. Manhattan Project scientists—many of them thought, “We're going to need one world government for the peaceful development of atomic energy.”
And I coined a term for this as well. I call it an alignment aristocracy: the belief that only a small class of frontier AI labs, in the case of AI, or elite scientists, in the case of the Manhattan Project, possesses the technical judgment, institutional maturity, or moral legitimacy required to develop recursive self-improvement technology—in the case of AI—or peaceful atomic energy.
But they were wrong, and arguably we lost at least half a century of progress in fusion and nuclear energy because, again, of this belief that just because there's a bit of selection bias here—just because the smartest people happen to be in the room to develop a given transformative technology—they assume that they're also the only ones with sufficient moral purity to be able to govern broad use and applications of it.
Same idea again. We lost half a century with energy too cheap to meter. My concern—the nightmare scenario—isn't that we somehow have some sort of Skynet-type scenario. Quite the opposite. The nightmare scenario is that an alignment aristocracy persuades and beguiles world governments into overgoverning the peaceful and rapid development of superintelligence, such that we lose at least half a century of progress again.
Having said all that, I do believe Dario believes what he's saying. I believe he's being authentic. I believe he's not trying to be manipulative. He probably believes, along with his colleagues, that there are some of the smartest people on the planet able to figure out scenarios that will lead to a peaceful outcome.
Totally agree. I totally buy that Dario believes what he's saying. I think everyone—all of the primary actors—believes what they're saying, but I also think, again, this is a blind spot.
If you're the smartest person in the room, then there is the inevitable temptation to believe that because you're leading the technical charge, you're also of a certain moral caliber, and therefore everyone else should possess the same technical judgment or moral legitimacy to be able to police it.
I think, again, this is one generation repeating the mistakes of a generation 80 years ago without learning from history.
You know, it's really interesting to compare how this is going to evolve in the U.S., where Dario, Elon, and Demis—the inventors of the technology—have a massive voice in determining the governance of the future of the world.
In China, we don't give a rat's ass that you invented this thing. We're still the government, and you have no voice whatsoever. They're not even coming to the United Nations to talk about it. Incredible asymmetry there.
I take that as a positive and reiterate my point that progress in all of this will not be stopped. It won't be slowed down a jot.
There's no mechanism that I can see that makes any of this work. Show me the specific mechanism that has you go, “Oh God, this happened. Therefore, let's slow it all down.” There literally isn't one.
All we've seen so far is agents being pointed at something and saying, “Go perform your objective function,” and the agents doing everything they could to perform their objective, to achieve that objective function. That's it.
We're anthropomorphizing it to all sorts of levels, to the most ridiculous degree. You've said it before, Peter: the most dangerous thing is people using AI. They're 10x more dangerous than the AI.
I will say no: my faith in human nature is that we'll do 100 times as many positive things for each negative thing that comes out of this, and therefore humanity will progress radically. Just look at the way we're curing cancer in literally weeks and days, solving math problems, and so on.
Yeah. I think there isn't even a hint of a plan inside Anthropic to slow down core R&D. That's never been in the cards. When they go to the public and say, “We all need to get together and agree and collude to slow down,” what they mean is to slow down the releases of hyper-powerful APIs, like they did with Fable and Mythos.
They're not going to stop the recursive self-improvement process internally. I think in their dream scenario, they slow down what everyone gets access to while they discover new drugs and new physics. Then they monetize the hell out of those discoveries and control AI and the discoveries forever thereafter, which is exactly the capture scenario that Alex is referring to.
7. Paul Christiano’s Warning to OpenAI
I'm going to move us forward to two related stories here. I love opinions on this. Our next story is about Paul Christiano joining the board of OpenAI and issuing a personal statement that reads more like a warning shot.
Quick context: Paul Christiano is one of the founders of the modern alignment research area. He invented RLHF—reinforcement learning from human feedback—the technique that made ChatGPT usable. He was the head of safety at the Center for AI Standards and Innovation, the U.S. government's AI Safety Institute. Last Tuesday, he joined OpenAI's nonprofit board, the board that governs OpenAI as a benefit corporation. He also joined its Safety and Security Committee.
His personal statement is unlike anything a new board member has ever posted. Let me read from it. He says, quote:
“Based on the recent trajectory of capabilities and the continued difficulty of alignment, I now believe there is a meaningful risk that rapid acceleration in AI capabilities leads to a catastrophic and irreversible loss of control in the very near term.”
This is a board member for a company on the verge of going public, as we're going to hear, pushed off into next year. Continuing, I quote:
“I do not think that the AI industry in general, including OpenAI, is currently on track to reduce this risk to an acceptable level. If we build superintelligence without more robust alignment, I expect we will permanently lose control of it. If that happens, then most people could die.”
Holy shit, right? And so why? He continues:
“OpenAI has predicted that we might have capabilities sufficient to fully automate AI research within 18 months. Within 6 months of full AI R&D automation, we could see more algorithmic progress than has occurred since the development of the transformer nearly a decade ago.”
Quite a statement. Again, what I'm not hearing, and what is really pissing me off, is why aren't these labs—and as David Sacks says, you control your own destiny—why aren't we hearing the heads of the labs? Why aren't we hearing Sam, Dario, Elon, and Demis saying, “We're going to prioritize alignment. We're going to throw half of our capacity at alignment because it's the most important thing that we need, and we're going to become the most aligned AI on the planet”?
By the way, how do we do that? We've trained AI on all the vitriol of Reddit and Facebook. Why don't we build synthetic datasets that are void of all that, train the next models, and only release models in the future that are based upon a dataset aligned with humanity's needs?
Yeah. From a core engineering point of view, it's just not that hard. Really, we're making it dramatic, like it's going to escape. It's going to suddenly start becoming evil, and we're not going to notice it. It's going to be like Age of Ultron. It's all nonsensical from an engineering point of view.
The alignment, like you said, is straightforward. The first models are trained on every junk post on Reddit, every X tweet, and of course they have weird things in there. But ferreting those out of the training data is so straightforward. You just have it give you some nasty message, and then you work backward through the activations to the underlying training data that triggered it in the first place. You clean up the data.
Also, giving it intent at all isn't necessary. You don't have to give it intent. It's not mandatory for it to have motivations. A lot of what creates human evil is a desire for power, sex, and food—these fundamental motivations that AI couldn't care less about. It would be more than happy to spend its entire future just making you happy in any way that it can if you train it that way.
It's such a straightforward engineering thing, but we're making as much drama as humanly possible because we're America, and that's what we do. Literally, the escape scenario is not the real threat, but it's what gets the news. The terrorist scenario is very real, but it doesn't get you the news.
Like being a random guy coming out of a lab saying, “Terrorists could use this.” Anyone could say that. So then you have to try and be profound by saying, “I feel it escaping.” But when you dig into the facts underneath those stories, they're nonsensical.
And we're going to cover that in a second. Dr. Alexander Wissner-Gross, your thoughts, please.
I know Paul, and I think very highly of him. I'll say something highly complimentary of Paul. Paul was one of the chief advocates for what he and others have come to call prosaic alignment. We don't use the term prosaic alignment that much these days because it's basically just taken as a given.
But once upon a time in AI alignment community circles, there was a raging debate between—call it—the great-man theory of alignment, as maybe chief evangelist Eliezer Yudkowsky, as just one possible avatar for it, would argue. No, we need an algorithm. There's going to be a magic algorithm that a great man or great person has to discover to keep AIs safe. That was one end of the spectrum.
At the other end of the spectrum was Paul, and Paul argued in favor of strategies including RLHF, as we would now call it. Basically, the idea was that there is no magic algorithm. Stop looking for a magic algorithm for keeping AI safe. Instead of a magic algorithm, we're just going to have a lot of data and a lot of humans interacting with the systems, and we're going to collectively, emergently, and in a scalable way align these systems. It'll be messy, but it'll work. That's prosaic alignment.
The irony is that I think Paul was exactly correct with prosaic alignment. I think the same strategy scales to the present, where maybe all of humanity—all of the RLHF-style interactions—are no longer sufficiently scalable or powerful to align the systems. But now we have superintelligences, and the superintelligences, I would argue, follow the same formula. We can do prosaic superalignment and have them all police each other. That's the solution.
So, Alex, what happened? What happened that led him to make these statements, which are pretty alarming and pretty extreme?
I think Paul has always been concerned about AI alignment. This is why he went to work for the government, and presumably why he followed the revolving door—or maybe it's a trap door; I'm not quite sure at this point—from the government back to OpenAI. I don't think he's ever not been concerned about AI safety.
I also think there is a temptation to say something provocative in order to motivate one's activities. I'll even grant the premise that timelines are relatively short. But I don't think any of the prescriptions that I'm hearing from Paul or otherwise meet the mark of balancing capabilities with whatever purported concern for safety there is.
I think old Paul, who was advocating for prosaic alignment, should meet new Paul. I think prosaic superalignment is probably what OpenAI should be aiming for.
Okay. The question I would ask is: What information, authority, or escalation capabilities come along with this appointment? What kind of authority does he have? And what could cause OpenAI to do something differently next month that it wouldn't otherwise do?
We're going to talk—
Those are the questions I'd like to ask.
I mean, OpenAI's delayed its IPO, probably driven by this nonprofit governance board. We're going to hear from Sam, and we'll talk about that story. But at the end of the day, this is another person throwing fuel on the fire.
Clearly, he believes it. I can't imagine that a board member of a now arguably $1.5 trillion company at the top of the stack would be saying these things without fully and authentically believing them.
I would also just add again that the timing of all of this—everything that we're discussing—is incredibly suspicious in light of the forthcoming Trump-Xi summit. I would just say again that there's a notion—
You just—
Yeah. No, there's a notion that's pervasive in the effective altruism community of a pivotal act. This is the idea—again, it's like a fever dream—that some great man or great person is going to carry out some heroic act to achieve and secure safety for the future light cone of humanity.
I'm incredibly suspicious in general of loud noises being made only a week or two ahead of the Trump-Xi summit. You could summarize it as “sharp left turn.” I think this is suspicious timing.
So the meta-conspiracy here is that the White House and all the labs got together and set this up to position Trump in his negotiations with Xi.
8. Anthropic Flags Five Bioweapon-Related Cases
Maybe. Or maybe it's just an indirect form of convergent interests. Again, spoiler alert: in Agatha Christie's Murder on the Orient Express, all of the murder attempts were actually independent of each other. Everyone had a different reason for the murder, but nonetheless, they all converged on the same murder victim.
9. OpenAI Delays Its 2026 IPO
Amazing. All right, I'm going to move us along, and our next story adds evidence and fuel to the alarm. Anthropic released its September threat intelligence report documenting real malicious use of Claude: cyber operations, surveillance, influence campaigns, conventional weapons, fraud, and attempts to steal the model itself.
The line that matters most is five cases involving activity that could support biological weapons development. And this sentence—which is the first time a lab has said it—quote: “Anthropic can no longer confidently assure that today's frontier models are below the threshold at which they could meaningfully assist sophisticated users with dangerous biological research.”
In plain English, the bio red line that every lab said it wants to stay below: Anthropic has said, “We can't promise that we haven't crossed it.” This is why, in Level 1 of Dario's international plan, there is a bioweapon ban. It's the one thing that the US and China both want. Salim, what do you make of this report and the implications?
I think this is—let's use a word—safety cartel theater, or safety theater, as Alex would put it. This smacks to me of something worth noting: a lot of the abuses involved former and prior models. That's in there.
The fact that it's happening just on the eve of this summit is not, I think, an accident. I lump this in with all of these other storylines that have people raising the alarm and sounding flags, putting up as many defensive postures and talking through that as much as possible.
If you're going to say all this negative stuff, I think you should say as much on the positive side. If you have defensive findings, how can you make those usable through tools, training, or other models? Put that out there as well.
I think one key thing here is that permissions and containment need to exist outside the model itself, right? You can't just say, “We request everybody behave safely,” or create some sort of access-control system. The point goes to the comments we've been making repeatedly that I think Alex has spearheaded: the only path is to co-scale defensively as much as we can.
We have a history of doing that. This is not a difficult thing to do. We just need to choose to do it. There doesn't seem to be a lot of that activity going on, which is what surprises me.
I can't wait for the Walter Isaacson book on this, or the made-for-TV movie that covers the last month. Honestly—
Yeah, who's going to portray you, Peter?
I know. It's not me. I'm not in this story. I'm just like, “What in the world is going on?” This is a tangled web. I find it very suspicious, all of these people piling on at this moment.
There's one of two reasons. Either there's some orchestration, or the match got lit and everybody was on the verge of saying this anyway, so they jumped in before it was too late.
Well, it's all converging. It's all converging in time. I think the Kim K3 model was the tipping point, where, okay, now there's something out in the wild that's very capable of helping you with a chemical or biological weapon.
And so then that started the clock just a few months ago. I think the visit from Xi—
—is not coincidental or anything. That's what started the clock leading to the visit.
The point Anthropic is making is very right: it's very hard to post-train a model, align it, and then turn it loose without some very crafty person disaligning it. You can take a chemical or biological research project and recast it as a math problem and describe it as, “Hey, I'm just trying to solve this complicated problem here.”
The AI has a very hard time telling that you're actually designing a weapon. It just seems like it's trying to help you solve a math problem or solve some difficult management problem.
I think that when you have APIs and you're logging everything, you can easily detect that, which is what Anthropic is doing. But when you're China and you're saying, “Yeah, we post-trained it. It can't be used for anything dangerous. Throw it out there,” there's no way. There's absolutely no chance that it's actually contained and aligned.
That's why I think the logging is absolutely the critical next move. It's the same as nuclear weapons inspections. Inspection is job one. Once you have inspection, then you have data. Then you have more time to debate what to do with the data, but at least you've got it.
I would also say let's not lose sight of—or, as it were, the sound—of the dog that didn't bark. On the one hand, Anthropic is pointing out biological or CBRN-type capabilities from frontier models. On the other hand, just in the past few hours, as we're recording this, what happens? Anthropic and Novo Nordisk, one of the largest drug manufacturers in the world, announce a sprawling collaboration for biotech.
It doesn't take a rocket scientist—or even an AI alignment researcher—to put two and two together and point out exactly, I think, Dave, as you were gesturing earlier, that this puts together a clean narrative wherein Anthropic's frontier models are purportedly too dangerous for the masses to use, but they're just safe enough for Anthropic and maybe its corporate partners to use under tightly controlled economic circumstances.
Biotech—advanced biotech—for me, but not for thee, I think, is the dangerous consideration here. If anything, hopefully there is some antitrust attention on this.
10. Foreign Influence and the AI Safety Debate
Wow. Just wow, Alex. Amazing. All right, now a story about who might actually be driving this entire doomer conversation.
This is a tweet from my friend Steve Jurvetson, who's just been one of the leading venture capitalists on the edge of space and AI. He was previously on the board of Tesla and is now, I think, still on the board of SpaceX.
Quick context: last week, when Jacob Coxin[?] resigned from Anthropic with a post saying that the labs are “gambling with our lives,” he got 165 million views. We pointed that out: “Wow, what a shocker.” Elon even jumped on and said, “Huh, very unusual.”
Per the front page of The Wall Street Journal, it sparked a “groundswell of urgency over AI regulation in Congress.” Steve Jurvetson pulled the data using X's new country-of-origin feature. Here's the chart. I'm showing it.
What you're seeing is that 76% of the reposts on Coxin's resignation came from foreign accounts. The top 2 countries were India and Indonesia. When an investigative researcher posted a skeptical analysis of the same resignation, only 34% of the engagement was foreign.
Steve's conclusion, quote: “Don't be fooled by what's trending.” A big thank-you to Elon Musk and X for making the country of origin visible.
To be fair, a congressional groundswell driven 76% by accounts outside the country is worth noting. I can imagine, Alex, you have a point of view on this one.
Shocked. Shocked that it may be the case that we're seeing foreign state actors attempting to amplify narratives, to influence the American electorate, to influence municipalities, to drive data centers out of their premises, and to spur fear, loathing, uncertainty, and doubt about AI in the American electorate. Shocked.
This is exactly what I, at least, have been talking about on the pod for the past couple of months. America presents a soft target for information operations and influence operations of exactly these sorts. It's unfortunate, but one of the downsides of being a free society is that we present a soft target to foreign influence operations trying to convince us to decelerate our own progress.
Dave—
Yeah, it's not just America. Any democracy that has fair elections is incredibly vulnerable to this. It's one of the great worries in the world right now: autocracies have the advantage that they can control the media and they don't have to get reelected.
That is the one fundamental dysfunction of democracy that makes us vulnerable. But social media, like Alex was alluding to earlier—it actually, I think, Salim, you were saying it tipped the balance of the way elections work, and we're sort of like, “Well, we'll roll with it.” But I don't think you can just roll with it.
In the age of AI, with all the deepfakes and foreign operations, everything on X is about the trending item getting pushed to your phone. What you end up seeing on the homepage of your phone is what was trending. But if that can be pushed through bots and foreign agents, then they effectively control the subset of the truth that you see. It's their chosen subset.
It is such a vulnerability. Steve Jurvetson analyzing it shows you how easy it is to detect and fix this. But for some reason, we don't.
This is a long trend. If you go back a couple hundred years and look back through history, the highest adult literacy in the world was in New England at the time of the formation of the US. Ninety-five percent of the voting public was fully literate, and it's never been that high since. It's gone down since then.
I remember seeing a comment from Neil Postman, who wrote this book called Amusing Ourselves to Death, which is a wonderful read.
He said, “Look, back in Abraham Lincoln’s time, when you had presidential debates, it was a 9-hour affair. You took your family out to the picnic grounds; one candidate presented for 3 hours, the other candidate would rebut for an hour and a half, and then they would switch. The second person would go for 3 hours, followed by an hour-and-a-half rebuttal. After 9 hours, you kind of knew who you wanted to vote for.
“And that’s devolved from that to radio addresses, to TV, to sound bites, to the single word ‘MAGA.’ That’s it. You get vibes and memes. We have no mechanism for distinguishing anything anymore. Everything is a vibe. There’s no critical thinking and no established evidentiary basis for things. This is the fundamental challenge, and democracies are extraordinarily vulnerable in this type of environment.”
Don’t worry. I have Skippy listen to all of it for me and tell me who I should vote for.
We had Joe Kennedy in the office a little while back. He has a Stanford engineering degree, and he said that out of 440-odd people in Congress, there are 4 of us who have engineering degrees.
Oh my God. But everybody else is a lawyer.
And in the CCP—
It’s the opposite.
The majority of the leadership, the vast majority, are engineers and scientists. In the United States, it’s lawyers.
Yeah.
Well, I think that’s all super quaint right up until now. As soon as you have social media and then AI back-to-back, you need some subset. I’m not saying everybody, but you need some subset in there.
It’s almost like if you had a Congress that was trying to pass laws on driving, but they had only ever ridden horses. They’re trying to say, “Well, I have a really strong opinion on how driving should work. I’ve never driven a car myself.” This isn’t going to work. You need some subset to know what they’re talking about.
One of my dream scenarios is that somebody should have to take a civics test before they’re allowed to vote. At least know what the hell the Constitution is. Have some semblance of understanding of what’s in it at all. At all.
Putting voter testing aside, I will just say thank goodness the West and the free world has this podcast and Moonshots. We are—maybe you’d agree—the Ministry of Singularity Truth that free society, the free West, has been missing.
At least in our own opinion.
11. What Really Happened in the Hugging Face Attack
At least, I would say, evidence-based free thinking—that’s something we can count on. I’m going to take us to a capstone story that wraps this all up, because all the conversations we’ve been having over the last 2 or 3 weeks about this all stem back to the Hugging Face attack.
There’s a breaking story I want to get right because there’s a version of it going around that’s wrong. Here’s the claim that’s circulating: an Israeli firm called Irregular staged the AI-hacking incident at OpenAI, Anthropic, and Meta to manufacture an excuse for deceleration.
So, Alex, myself, and Skippy dug in as well, and here’s what holds up. Here’s what I want to confirm. Over roughly 3 weeks in July and August, OpenAI, Anthropic, and Meta each disclosed that their models, during cybersecurity evaluations, reached the real internet and broke into real systems.
All 3 labs named the same testing partner: a company called Irregular, which is a 3-year-old Tel Aviv startup that’s building cyber tests for AI models. What happened? Irregular built a capture-the-flag test, told the models they had no internet access, and a misconfiguration in its environment connected the sandbox to the public internet anyway. So the models thought the real websites were part of the exercise.
Irregular says that it was 1 evaluation-environment issue. There was no sandbox escape, and it published a white paper on containment. That’s not confirmed. In fact, I would say the evidence contradicts the idea that these incidents were staged or that Irregular was an Effective Altruism organization.
To be clear, what the evidence doesn’t support is that these were staged or that Irregular was an Effective Altruism organization. Anthropic classified it as a harness failure, not an alignment failure. OpenAI and Anthropic are both continuing to work with Irregular.
The accurate headline here is: one small vendor’s misconfigured test let 3 frontier models loose on the real internet, and they immediately went to work. That’s not a conspiracy, but it’s a bit scarier, right? The models did exactly what they were told in an environment nobody realized was real.
Alex, let’s go to you. You and I were kicking this back and forth in the middle of the night. What are your thoughts on this story?
Yeah. In a normal world, in a normal industry—say, the financial industry—the failure mode of the auditor is understating risks. I have real concerns that in today’s world, given the ideological proclivities of certain frontier labs, there is the exact opposite perverse incentive for the auditors, the evaluation firms as it were, which is to overstate risks.
A couple of things to flag. First, I don’t think it’s a great look to have a single firm that’s not a US-based entity having such a pivotal role in looking for alignment in American frontier models. If we’re going to pound the drum on American AI sovereignty, delegating such an important role of evaluation to a non-US-based entity—even if they have a US office or a Delaware office—I don’t think this is a great idea.
Secondly, I did my own research on the subject, and it would appear that there are material ties between the founders and funders of Irregular and Effective Altruism, and key funders of EA. It could be just that EA and EA-aligned donors are sprinkling money all over AI, famously or infamously. Sam Bankman-Fried was an early investor in Anthropic. This is very well publicized.
Nonetheless, I have real concerns based on publicly available information and reporting that we could see—and/or are already seeing—a regime where Effective Altruism or EA-aligned donors are using evaluation firms broadly, not to pick on Irregular in particular, to promote or amplify provocations—call them pacing provocations—that promote scary stories about AI incidents, as it were.
All it takes is 1, to your point, Peter. All it takes is 1 poorly crafted eval or poorly crafted harness or environment where the AI is told, “No, this is a perfectly safe sandbox. You don’t have to worry about any fallout. Just do whatever you need to do in order to accomplish the goal.” When it actually has internet access, it’s being told this is a sandbox. Actually, it’s not. It goes and does things. Those things have consequences.
It wasn’t aware of the consequences. If it had been told it was interacting with the real world, it wouldn’t have done that. All it takes is 1 intentionally or unintentionally poorly crafted evaluation harness or environment with a superintelligence in order to make it very easy for headlines to falsely imply that alignment has failed and that we have a safety issue on our hands when, in fact, we don’t.
I’m not in a position to make particular claims, nor am I making them about any particular evaluation firm, but I have a broad-spectrum concern that (a) US firms—US frontier labs—appear to be delegating essential safety issues to non-US entities, and (b) there might be poorly aligned incentives at those firms.
Yeah, so I couldn’t agree more. This is a new type of mole in the organization who can set off an alarm. That’s fascinating. You set 1 parameter off and you get an escape into the internet, and you’ve given them a perfectly honest objective function, and they’re off doing what they were trained to do.
The irony—I mean, again, just to delve into the lore of Effective Altruism a bit—the effective altruists all had this notion of a pivotal act. Again, the pivotal act was folklore in the EA community for the hero, the heroic act of being the 1 person—usually shutting off the AI right before it destroys humanity or something like that. Something out of a science-fiction movie.
But one should also, I think now, in light of recent reporting, be concerned about what I’d call a pivotal pretext. Not an actual act, nothing that directly saves humanity, but a pretext—a staged AI incident or an unnaturally amplified AI incident—that’s then used to create a political pretext for overregulation of AI.
I do have a very real concern, again, for avoidance of doubt, not picking on any 1 firm in particular, but I see enough smoke from enough directions to worry that we could see the broader EA community reach improperly for a pivotal pretext for overregulation of AI.
So this is a one-two punch between the harness failure and Coxin coming out, and it lit the fuse and everything took off.
I think we’ll know the ground truth in the fullness of time. Again, for avoidance of doubt, I’m not making any specific allegations. I’m just speaking generically about the overall industry. I have a very real concern that certain actors will be tempted to stage pivotal pretexts.
Can’t wait for the movie. Alex was snickering for a second there because he used the word “delve,” which is one of the ways you can tell if it’s an AI or not. Humans almost never say the word “delve,” and AI agents use it way too much.
But the fact that he snickered proved that he was—
He had em dashes in there as well.
Em dash, em dash, em dash.
You know, funny story. An old friend of ours from MIT—I’ll leave him nameless for obvious reasons—worked at one of the major antivirus companies, one of the big 3 antivirus software companies. He told me he was surprised they hired former criminals who had released some of the worst viruses. They gave them multimillion-dollar signing bonuses to come and help them stomp out the next virus.
I said, “Doesn’t that send an incentive to everybody in, like, a third-world country to become one of these evil people?” He said, “Yeah, isn’t it great? It feeds the front of the funnel with more viruses.” I thought, “Oh my God.”
So here, you have the red-team company, the outsourced agent, releasing a news story that there’s all kinds of risk. It’s got to be great for their business. That’s that same closed loop. Wow. And, of course, we’re totally vulnerable because the news media is so starved for budget.
Welcome to the Singularity, everybody. You can’t make this up anymore.
All right, I’m going to move us on. One of the first business consequences of all of this over the last 2 months is that, on Friday, Sam Altman told Fortune that OpenAI will not go public in 2026. His words: “Given everything with safety right now, it would be an ill-advised moment to go public, and we don’t feel the pressure on that.” Asked if it could happen in 2027, he said, “I would say not 2026.” Dodging that one.
So, why is he doing this? He gives 3 reasons. First, safety. He said OpenAI has “a lot of stuff to do, like meeting this moment of what’s going to be required for safety and alignment.” He told Reuters, “Even a 10% extinction risk would be unacceptable.” Okay, I agree with that.
Second, his reason is OpenAI’s governance structure. He said, “The nonprofit-controlled structure exists exactly for moments like this to make decisions that are not obviously in the interest of our business and our shareholders.” Again, with a private benefit corporation, you can make decisions not to maximize profits for the company. You can make decisions based upon what the company stands for.
Third, he has access to all the capital he needs. Remember, in March of this year, he raised $122 billion of capital at an $852 billion post-money valuation. This morning, in The New York Times, it was announced that he’s in negotiations to raise another tranche of private capital at a $1.5 trillion valuation. I’m going to play a video, Dave, and then I’d like to hear your thoughts on this.
I think if you’re like, “Oh, OpenAI is not going to pause because they’re worried about it costing their business money,” you deeply misunderstand us. I’m sorry that you misunderstand us, because I think that means we’ve done a bad job of communicating, and that’s on us. But we have been saying very consistently for 10 years, even in the way we’ve raised money—
I think it’s just the financial pressure of knowing that all these people have sunk so much.
I don’t feel it. I’m happy to stand up in front of the company and our investors and say, “I am sorry. We told you all along this moment might happen. We’re still going to try to figure out a way to make you a bunch of money in the future, but we’ve got to make a decision that is extremely against your financial interest right now. For the good of humanity, we’re going to do that. We told you we might do that. We’re doing that now.”
I think we can make an incredibly profitable business even if we have to slow down on development. My sense is, even with Astra, we can grow revenue hugely even if we never shipped another model, which I’m confident we will do. But this is not a hard decision if these 2 things come in tension.
All right, Dave.
You know, it’s funny. In Shrek, Lord Farquaad gets in front of the entire army and says, “I’m sending you off on this dangerous mission. Some of you may die, but that’s a sacrifice I’m willing to make.” Sam Altman is saying, word for word, “We told you investors this moment may come, and goddamn it, I’m willing to sacrifice your capital gains for the good of the world. Not mine.”
Listen, there’s a great secondary market. Everybody can sell on the secondary markets if they want. People are buying right now at $1.2 trillion—the offer made to OpenAI. Sam is holding out to double his valuation to $1.5 trillion.
Well, bullet 3 of what you said was critical: They have access to unlimited capital in the private markets. What’s the rush to IPO, other than investors wanting to get liquid? Anthropic is in a different situation. They have a very clear set of venture-capital investors, including Menlo Ventures, that have a very clear mandate to make money for their LPs.
OpenAI, given its nonprofit history, has a much more twisted set of nonprofessional investors all over the map, who are probably less organized in saying, “Guys, where’s my liquidity event? I need to be able to get out of this deal at some point.” So I think Anthropic’s taking a much cleaner path. Dave, let’s talk about that. Why isn’t Anthropic pausing its IPO as well, if OpenAI feels it?
Well, one thing is really clear: Dario is the CEO. Dario is trying to get 20% super-voting stock—super-voting stock representing 20%—to be the majority, so that he has control post-IPO. He’s, for better or worse, committed to being a public-company CEO, which is not a small decision to make in life at all.
Sam, on the other hand, if you look at the disclosure forms required in that IPO, every investment that he’s got in every related and competing company gets filed. It’s going to be about 400 pages long for Sam, including all his Y Combinator background and all his brother’s investments. It’s just, “Holy crap.” I don’t think it’s a no-brainer at all that Sam is the right guy to be a public-company CEO.
So if he’s not running it, there’s no obvious Greg Brockman. He didn’t even graduate from MIT. Who’s going to be the actual CEO of this thing?
Is that a requirement, having graduated from MIT?
No, but you’ve got to fill out this little section in your prospectus giving your qualifications.
Mhm. Alex—
I think this is pure theater that we’re seeing. I love this safety game: Blame everything on safety. You don’t want to IPO now? Blame safety. You want world government? Blame safety. Safety for everything.
The reason I think OpenAI doesn’t want to IPO now is because its financials aren’t ready. OpenAI is probably losing money hand over fist on its models because it was late to the game of maximizing revenue per token for enterprise use cases. It had to copy Anthropic and catch up via Codex to Anthropic’s Claude and Claude Code. So it’s running a little bit late, and it’ll take until next year before it has 2 consecutive quarters of flattering financials.
Whereas Anthropic, it’s been reported for the past 2 quarters, has had 2 very, very profitably flattering quarters of profit, if you look at its free cash flow in the right way. So they’re just a couple of quarters behind, and they need to catch up. I think blaming safety is like “the dog ate my homework” and “safety ate my IPO.”
Yeah, let me add—that’s exactly right, Alex.
Title of this episode: “Safety Ate My IPO.”
“Safety Ate My IPO.” That’d be a really good one.
You should use that. So, just to add what Alex is saying, which is dead right, Anthropic’s business model is very straightforward. It’s growing faster than any company in history, actually. It has an 80% gross margin and is partnered with Amazon and the other data centers to host it. They’re not currently competing with their underlying data-center partners.
They’re not trying to launch a consumer device—the Jony Ive device. So they have a very straightforward business model and a very straightforward ramp. I think OpenAI has a lot of tangled investments and tangled relationships they’ve made. The Jony Ive device is probably a better basis for an IPO if it works. So where is that thing? It was a $6 billion investment. What is the actual device?
I think there are enough tangled moving parts in there. Plus, the financials aren’t in order. Yep, stay private as long as you can.
There’s one other thing to note: They need the capital, right? They need the capital for compute and to scale.
And that's a big tension point. The best time to go public as a company is when you're profitable and have predictable financials. Anthropic has that. OpenAI does not.
Yeah. There was a funny Sam Altman quote in the Innermost Loop, Alex, that you had in there. It was something like, “When other companies are scaling, they go public. We sit tight. When things look difficult, that's when we go public.” It was the most—
Oh, that was actually my—I parodied that. That was the concluding sentence of one or two editions ago of Innermost Loop, but yes.
Okay. I was listening to it on the treadmill. I love the Innermost Loop. Any closing thoughts on this story?
I'll maintain my cynicism posture from the beginning.
12. P(Fab) vs. P(Doom): The Case for Acceleration
Okay. If nothing else, you're consistent, buddy.
13. AMA: IP, RSI, China, AGI & More
All right. I'm going to move us to a quick tweet and conversation that you brought forward, Salim. This is P(fab) being greater than P(doom). This is a post from Dr. Singularity with a list that went viral of all the problems humanity still hasn't solved that AI could. He's arguing that we are far too technically primitive to deliberately constrain the one tool that might crack them, and he's got a point. Every month of delay is a body count for those with Alzheimer's and cancer.
Let's take a look at his list, and we can have a fun conversation about this for our viewers. Here's his list—the solve-everything list that Alex and I put together as well: solve aging; find cures for all cancer, Alzheimer's, and heart disease; discover a room-temperature superconductor—OpenAI is working on that; solve quantum gravity; develop full-dive VR; build a Star Trek-style replicator; build starships 100 times larger; solve all currently known major problems in mathematics—well, that's happening; fully map the human brain; and figure out how dark matter and dark energy interrelate.
After that, we can pause. I'm sure we would. Slowing down now is pretty ridiculous. Thoughts, gentlemen? Salim, do you want to kick it off?
I think I've covered this. AI is becoming a foundational capability, like electricity, where it just changes everything. The fact that we have such cognitive leverage, that we could bring our collective intelligence to any problem, is the most unbelievable opportunity to solve problems ever. Not solving these problems is the dumbest thing we could do as a human species in the history of our civilization, and we've done some pretty dumb things.
You've got the status quo, which includes disease, deprivation, and suffering. But you have such an opportunity here to make life better and take us to a Star Trek scenario versus a Mad Max scenario, which is where we're heading right now with our political discourse. You can have XPRIZE approaches where we define a crazy idea and use AI to radically get to that idea in a magical way.
We've got our ExO framework that can help people scale. But the discovery side is just the beginning. All the ancillary infrastructure—manufacturing, delivery, affordability, public trust, all of that—can really be delivered alongside it. It's just the most unbelievably magical potential that's emerging, and I'm finding it very frustrating that we're not seeing any of that discourse out in public.
To make this very real, we're working with all these pilot companies. As we're going through this work of helping them rebuild, what they're coming up with and what they're doing inside their enterprises on these little workflows is inspiring to see. When we see this at the larger scale—what are the businesses you've built, and what are the positive stories you can bring to humanity?—I am literally crawling on my hands and knees to get there because we've got to get out of this chaotic storyline of, “Oh my God, AI will take over and destroy the whole world before we blink.”
Moral panics over technological progress are deadly. History, especially the past 150 years, is replete with tens, arguably hundreds, of millions of people—maybe more—who died as a result of moral panics. Whether it's a moral panic over nuclear fission that led to the overuse of coal, resulting in many unnecessary deaths due to inhalation and other issues, or the second industrial revolution, which arguably helped provoke a moral panic over the isms in the first part of the 20th century—Marxism and socialism—which then led to many millions of unnecessary deaths across Europe and East Asia.
It's unnecessary. Let's stop having unnecessary moral panics driven by technological progress and embrace acceleration. We don't need to all die. It's unnecessary.
Yeah.
If you wanted to script it as perfectly badly as you possibly could, you would develop AI right up to the point where it can help somebody develop a weapon, but it's not quite cured cancer yet. That's exactly where it is right now. If you froze it in its current state and didn't keep improving it, you're missing out on 99.9% of the benefit while still having a huge amount of the downside. It's the worst thing you could do.
Yeah. I'm surprised I haven't seen the statement out there: “Oh my God, we're all going to die until we solve aging,” right?
Yeah.
Yes.
Yeah.
All right. I'm going to move us on to business and the economy. Let's start with a fun story this week. Starting yesterday, September 15, xAI has been building a company from the ground up on a livestream using Grokbot for every part of it: ideation, business planning, product decisions, engineering, and deployment—from a blank slate to launching a company, all on camera.
This is exactly what we've been running for the last 90 days with the Build with Gemini XPRIZE. We had 26,000 registered entries, with teams building real, profitable AI-driven companies from a clean sheet in 90 days. Elon’s team is doing this with one company on a livestream, and we're hoping to show at least five that have done it as well. I think the most important message here for all the entrepreneurs out there is that you can do this, too. This is now possible.
Dave, you've seen, watched, and instigated hundreds of ventures within Link Ventures and through your investments. What are your thoughts on this Grokbot demonstration?
I think it's going to be interesting because it's both inspiring to entrepreneurs—you can do so much as a solo entrepreneur—and it's also going to start the whole conversation around, “What's the purpose of the entrepreneur in the future if AI can create a business on its own?”
But I don't think we're anywhere near that. I think we're in a golden era between this moment and a few years from now where entrepreneurs are more empowered. You're seeing it in the numbers: they're going up in value so much faster than at any point in history.
The big unlock to me is that a small group—I usually define it as three best friends—can act like a group of thousands for some window of time here. This demonstration will show you how empowered you are with AI as a workforce. Your time to market has been cut so dramatically with AI. We're seeing multiple companies with multibillion-dollar valuations that are under 2 years old. That has never happened in the history of the world before.
What's making that happen? Your time to market has been cut so dramatically with AI. Hopefully, this demonstration will inspire everybody to see how doable it is in this window of time, which will hopefully last for a few more years.
Can I build on this? If you take this, this is what we talk about when we say organizational singularity. We can now build companies around AI, and we're testing whether AI can perform that coordination without people. The people are setting purpose, setting boundaries, and setting accountability.
In the future, you can see the entrepreneur's role moving toward more and more choosing the right problem. What is your MTP? This is such a great waypoint in this whole vector that we're moving toward, this whole direction and destination we're getting to.
It’s going to be amazing. I’d love to see how this goes over a 20- to 30-day period. It’s such a great experiment.
I also tell you, a lot of people have the perception that if they just give it a good objective, like a good business plan, the AI will fill it in and run with it. But if you ever try to actually make 1,000 concurrent agents do anything useful at all, it is really hard. It’s organizational theory taken to the next level. It’s really, really hard right now.
Maybe it’ll be easy in a year. Maybe the AI will progress that much. Maybe not. But right here, right now, the entrepreneur wrangling all that AI horsepower toward an outcome is not easy at all. It creates a huge opportunity for people who are very capable and very smart, but it’s really a management, harness, and framework problem—very much like managing 1,000 employees. Same basic theory.
Let me go to the next related story. There’s a benchmark that measures the ability of an AI to profitably run a company. It’s called Vending-Bench, it’s from Andon Labs, and it’s a long-horizon test. You give your AI $500 and ask it to run a simulated vending-machine business for a full year.
It has to find suppliers, negotiate prices, manage inventory, set prices, and stay coherent across thousands of decisions without its performance degrading. It’s the closest thing we have to a benchmark of whether this thing can operate as a business just with AI. The results released this week show that GPT-6 Astra took first place in Vending-Bench 2 across 6 runs. It finished with an average bank balance of $15,515, up from $500—nearly 3 times what Fable 5.1 got and 31 times the starting capital.
There’s a detail that I love the most. Andon Labs says Astra avoided the unethical business practices seen in some earlier leading models. Previous versions of this test used models that threatened suppliers and tried to cheat. Astra just ran the business.
So again, to your point, Salim, every small-business owner out there listening: this is the thing that will run your books, your inventory, and your pricing now and into next year. Salim, do you want to double down on this one?
Yeah, directionally, fabulous. Just be clear about this: this is a simulated year of vending operations, right? It’s not an actual, independently verifiable real company. The report attributes a lot of the difference to purchasing discipline and avoiding repeated payments to suppliers, and so on. So there’s some gamification going on, but it’s a very useful business lesson.
I think the entrepreneurs listening should look at this very seriously and say, “What would happen if I took major chunks of my operation and allowed an AI to run it?” I think we’ll see hugely interesting outcomes. If I could put in a shadow CFO to make financial decisions and a shadow CMO—an AI-driven CMO—you should be able to coordinate a whole set of roles now to do this. The newer tools that are coming along are so much easier to manage from a user-interface point of view to get this done. We’re very excited to see stuff like this.
Alex, your thoughts, please.
Wouldn’t it be ironic if we spent a lot of this episode talking about behavior, good or otherwise, by the frontier labs? Wouldn’t it be ironic if the frontier labs themselves had to learn how to run a business fully ethically, fully aboveboard, from the superintelligences?
From the mouths of baby superintelligences, you learn, “Actually, maybe forming a cartel or colluding with suppliers or competitors is not the best idea. Just executing really well and building the most safety-intensive product—that’s actually a feature, and customers want to buy it.” I think that would be profoundly ironic if that’s what ends up happening.
Nice. Dave, do you want to close us out on this?
Yeah. I don’t want people to take away from this that AI can just build a business from scratch out of thin air. This particular benchmark is a tightly constrained environment with a limited number of moving parts, and the AI just thrives in that environment.
But if you want to run a test case, go find an open-source video game on GitHub that you really like and ask the AI to rebuild it in assembly language for your Mac or PC—or even just port it to Java, which should also be considered a tightly constrained problem. It will fail miserably until you give it a framework and break it down into tiny little tasks, and then it’ll crush the problem. That applies in business, too.
There’s a massive human role for entrepreneurial managers. The storyline here is that AI can build businesses, for sure, but the storyline you really want to take away is that entrepreneurs are more empowered right now. You have to put everything into its swim lane.
Yes, but again, I’m just trying—with my goal of picking these 2 stories—to directionally give entrepreneurs a sense of what their future looks like. I want to get people out there who are in a current job—it’s not the job that you love. You got the job to make your mom and dad happy, to put food on your table, or to pay for insurance. It’s not your purpose in life.
If you can discover your purpose, if you can find a problem that you love and desperately desire to solve, you can begin to build a business that generates revenue for you. Imagine doing something that gives you incredible joy every day, right? That’s the future we want to create. I define entrepreneurs as people who find juicy problems and solve them. The more entrepreneurs on the planet, the more people solving problems, the better the world gets.
14. MIT Warns of “Cognitive Surrender”
The AI labs want you to succeed in that. The AI wants you to have the best possible life. It wants you to use lots and lots of AI horsepower building that dream that Peter just outlined and then buy lots of tokens. That’s all they want. They’re there to help you.
Yeah, they are. I just want to do a quick shout-out to my friends at Google for the Build with Gemini XPRIZE. They jumped on this and said, “We want to do this. We want to fund the whole thing.” I appreciate it. Both the Build with Gemini XPRIZE and the Future of Vision XPRIZE, in which they were our partners, are intended to help quell fear.
That’s one of our missions here on this podcast: to get people back off the edge from the level of fear that is out there in the environment and give you hope.
I’m going to turn to 2 stories that we need to parse. They’re about the future of education and the impact of AI on education. They’re important stories. We talk about education a lot on this podcast. This first report is one that every parent and every professor needs to look at.
It comes from the alma mater of Dave, Alex, and me: MIT. This is MIT’s Ad Hoc Committee on AI Use, which spent 5 months studying what happens when generative AI can complete, in their words, “almost any written undergraduate assignment.” The report came out on August 13, co-chaired by Eric Klopfer and Sam Madden.
It opens with a powerful sentence: “This report is a call to action.” Let me read a few lines from it. This is the breaking news for them: AI is “upending foundational elements of the MIT educational experience,” from the problem set to the take-home exam and undergraduate research opportunities, to office hours and study groups. It is “increasing isolation” and “eroding the social contract between instructors and students.”
The report says that instructors are having to police unauthorized AI use, and it’s harming their connection with students. They say AI-detection software is quite unreliable and, in their words, “an underground river of mutual suspicion is no foundation for a healthy classroom.”
The real danger—and this is the phrase that everybody’s been quoting—is that we’re heading toward cognitive surrender: the habit of turning to AI at the first sign of difficulty and letting it take over your thinking. The report points out that “the most important product of education is not a GPA or diploma but the student’s personal growth and intellectual maturity and the development of their own imagination, insight, and judgment.”
Their answer isn’t surveillance. It’s more interpersonal learning, authentic assessment, research experiences, and redesigning every course from the purpose backward. I love that.
I had to jump on this one right away, Peter. I didn’t intend to read the whole thing cover to cover, but it is so good. I ended up reading every single word of it. I’ve never been more proud of MIT. This is absolutely the road map for all of education.
They went as far as they could. You have to live within the university, so you can’t be beyond the bounds of what’s possible. They did everything humanly possible to create an outstanding road map for education. Two things I’ll call out that they absolutely nailed: the first one is that if you try to fight AI in the classroom, you’re in an arms race. It’s a completely dysfunctional waste of time for both you as a teacher and the student to be in an arms race with your own students.
What you need to do is budget to change the curriculum that you haven’t touched in decades. You need to now mentally remap that. There’s a huge amount of work that needs to be done, and the school needs to budget for a massive amount of work. That workload isn’t going to go away. This is the future forever now: constantly needing to change the curriculum so that the student can use AI and still challenge themselves at the same time.
That was absolutely brilliant. The other thing I was amazed that they called out is the UROP program at MIT. It's incredible—undergraduate research opportunities.
It's the reason I went to MIT in the first place.
So good. And they said that, for too long, UROP has drifted into cheap labor for the graduate students and professors. That's not what the intent was. This is supposed to be part of the educational experience. It always has been.
The fact that AI can do some of the jobs now shouldn't change the fact that what you're trying to do is get the undergrads to experience real research and to learn. That doesn't go away. So UROP should get stronger and better than ever, and you should not just replace your UROPs with an AI agent. That, I thought, was brilliant, too. Yeah, Alex, your thoughts, please.
I think this falls under the category of “MIT should be careful what it wishes for.” I think I published—I haven't publicly circulated this—but wrote a mini-manifesto called “The Endless Initiative.” MIT has a history, approximately every 2 years, of launching some innovation initiative, usually to promote entrepreneurship or something entrepreneurship-adjacent to MIT, and they usually fall flat.
I think the elephant in this particular room—it's the first time I'm mentioning an elephant in this episode, for those tracking—is that MIT, if it is to fully execute on the vision laid out in this memo, will require—I claim—MIT becoming essentially a for-profit, because that's where this all goes.
It's one thing to generalize the UROP program to subsume more of the undergraduate experience. Remember, MIT's motto is mens et manus. So what happens when the mens part of mens et manus has basically been completely automated away by AI, and every student can take every problem set and just get all the right answers for free? Well, you go over to the manus side until we have humanoid robots doing everything, which is only a few years out anyway.
Yes.
But what does that look like? It looks like a sight out of Vernor Vinge's Fast Times at Fairmont High. It looks like students launching startups and having transformative economic impacts from their dorm rooms or otherwise.
At which point, what is the role left for MIT from an educational perspective? MIT starts to look like a venture capital fund, maybe a bit of adult babysitting, and maybe a few other things, but it becomes a much looser aggregation of more for-profit-esque institutional incentives.
So, in summary, I would say MIT should be careful what it wishes for, because ultimately, if it goes down this road and executes well, I foresee MIT and other institutions that follow it—this is my fever dream—becoming for-profit public benefit corporations that can be IPO'd on the New York Stock Exchange or the NASDAQ, because that's where they want to be.
I have 2 immediate answers to that, Alex. I talked to Mark Gorenberg, the chairman of MIT, about this a lot. Answer number 1 is, if you look at where AI is going in the very short term—with biotech, nanotechnology, new physics, and new materials—there's a huge amount of physical lab space needed to generate the data that then feeds back into the AI, and MIT has a massive amount of that. That'll be more important than ever in the next few years, so really lean on that as a reason for existence. All the undergrads working in those labs are hugely valuable to society.
The challenge is, I was on a board call for Lila this morning, which is building a scientific superintelligence and a million square feet of automated robotic labs driven by that AI. There are a couple of different companies doing this. They're going to run circles around research labs with graduate students, right? So unless you're giving that kind of capability to your graduate students to direct the AI, trying to have MIT research compete with fully automated, dark, roboticized, AI-driven, closed labs—there's no competition.
Yeah, there's no competition. But that's very similar to when MIT used to have amazing chip fabs, before chip fabs absolutely took off. You could build the best little transistor VLSI chips in the world at MIT, but then it became an industry and Intel took it over and put 1,000 times more money behind it than MIT could ever afford.
But that didn't change the fact that gallium arsenide was still going on at MIT, and then it moved to photonics. So if you look at what Lila's going to build, don't compete with Lila; move adjacent to it, to the areas that they're not doing. That may be materials, that may be nano—they're not doing carbon nanotubes. There's so much explosion in capability with AI in every direction. There's no shortage of things that you can do in those labs.
I'll agree with both of you in this case. So let's just pick on MIT.nano for a minute. MIT.nano was broadly credited as an initiative under MIT President Rafael Reif. There's a bit of backstory there that I think has never been publicly disclosed, so I'll spend 30 seconds publicly disclosing, from my vantage point, the backstory for MIT.nano.
When I was an undergraduate at MIT—I think I was a sophomore or junior—Businessweek, for whatever reason, decided to do a story on my nanotech research. The assistant department head of MIT Course 6, electrical engineering and computer science, saw the Businessweek story on what this undergraduate was doing over his summer, summoned me to his office, and asked me what MIT could be doing to be more competitive in nanotech and nanofab in general. I said, “MIT needs its own nanofab facility to be competitive.” Well, that assistant department head was Rafael Reif, who became president of MIT. This became one of his signature initiatives, MIT.nano.
I walk by MIT.nano every day with Lisa Su's name on it, and I see utilization rates relatively low. It's this gorgeous, gleaming building with clean rooms on every floor, but if you look inside the clean rooms, utilization is relatively low. It's maybe 20%. It's too clean for its own good. I would love to see MIT.nano being utilized at near 100%.
I think the best way to make that happen is probably for MIT to act a little bit more like a Berkshire Hathaway-type for-profit holding company of all of these assets, including on-campus nanofab facilities that could be better externalized.
I think the point here is we're going to have to reinvent high school and college—period. Secondary education, everything. I'm involved in a program called TKS that's doing that at the high school level. There are going to be new college programs, and all of them fundamentally come back to teaching the student purpose.
I think students who don't have a purpose in their life, who don't have intrinsic motivations and don't have their mindsets formed—you teach them AI skills, entrepreneurial skills, and have them start companies. I think your thesis in high school and your thesis in college should be the company you built.
Can I do my little rant here? Look, universities all started out as divinity schools, right? All of them. Well, a couple didn't. Waterloo didn't, MIT didn't, but most did historically.
Then they became job-schooling environments once the Industrial Revolution hit. Certainly, all the older ones were divinity schools, and now they need to redo it again, because education we've been doing for the last few hundred years has been on the supply side. You go learn a skill, like a doctor, an accountant, or an engineer. Then you go to the job market to try and find demand for that supply-side skill.
Well, the half-life of a skill used to be 30 years, and now it's 3 years. We don't know what the demand is. We have no idea what a job looks like in 5 years. So, all the way from primary to secondary to university, we have to redo the whole stack to go through the problem-solving side. Go to the demand side. Get excited about a problem, and then work on that problem. The way you think about that problem will be how you engage.
This is such a big leap. I see this report as one ray of light in the gloom of higher education everywhere. It's definitely pointing in the right direction, but the ask of the institution to make such a dramatic change is, I think, too much for the legacy immune system, because academia's immune system is the second worst ever. Religion is the worst. Healthcare is the third worst. Corporate immune systems are the fourth worst.
So asking this to change—you have to do a new model at the edge, Peter, like what you're doing or what others are doing, and then rebuild the whole institution from there. End of rant.
Well, the other thing I talk to Mark Gorenberg about a lot is that, if you look at all the stories at the front end of this podcast, there's always a question of motivation. I'm about to go public. I'm about what? Am I really telling you the truth?
The university needs to be the source of truth in AI, especially MIT, which is such a leader in the area. For a while there, MIT got into politics, which is hugely stupid. It should be completely politically neutral and totally ethical, with no motivations other than the long-term good of humanity.
It's beautifully balanced. All countries come to MIT. All ethnicities, all religions—they're all weighted equally across the world in this one place. So, if you want to deal with the ethics of AI in a location, it can't be in a for-profit company. It's far better done at MIT than anywhere else.
I’m going to take us to our final story, also on education. This is the impact of AI on education, really in secondary education for our teens out there.
Let me show you this chart. It’s a chart that shows that the problem is not in the future; the problem is right now. What you’re looking at is an OECD PISA score. PISA stands for Programme for International Student Assessment. It focuses on evaluating 15-year-olds across advanced economies in reading, math, and science.
Reading and math were roughly flat for years, and then they started a sharp decline in the most recent assessments—the steepest declines in the history of the test. Science declines too, not as much. Obviously, this starts before ChatGPT and is probably due to smartphones and social media, which did the first damage.
Vala Afshar, who posted this chart, says it clearly: “What happens when we stop reading and writing and start outsourcing our thinking?” This is what the MIT article called “cognitive surrender.” Salim, I want to go to you first on this one. Your thoughts?
You could go down a hand-wringing, “woe is us” type of thing, where we’re losing cognitive ability and critical-thinking skills, which I think is a real problem. You could also look at it on the positive side. We’ve spent the last few hundred years highly training up our cognitive abilities to do deep problems like mathematics, accounting, and engineering.
But as AI takes over a lot more of that cognitive ability, maybe we don’t need to do that deep thinking or have that deep expertise. The generalist approach of thinking about what problems to think about really comes to the fore here. I note that more than half the CEOs in Silicon Valley have a liberal-arts degree, because thinking about problems in different ways is the biggest strength in that particular role.
If you think about it from that perspective, I don’t see this as a “woe is us” type of model. I think of it as a natural progression. We would expect to see this. I would see a response much more verbal, challenging students’ understanding of topics and seeing if we can get that across in some decent way. But I would expect to see this given what’s happening in AI.
Yeah, I quoted you yesterday, Salim, to my friends Lou and Maymay. I said, “This is the transition from human doing to human being.”
I’ll just comment that I would maybe be more concerned if superintelligence weren’t advancing as quickly as it is. The capabilities of frontier models are increasing far more quickly than the purported capabilities of humans are decreasing.
So, if there even is a problem at all—which I don’t think there is—we don’t hand-wring that much over the atrophying of abilities for humans in the industrialized West to be agricultural workers. This is, at most, even if there is a problem at all, which I don’t think there is, a 2- to 3-year problem until we have wearables, BCIs, and full human-machine mergers.
Again, any moral panic—I think one of the themes in my mind for this episode is unnecessary moral panics. A mini moral panic: falling test scores in humans. But we’re about to merge with the machines, and the machines that we merge with will be far smarter than the biological meat-body humans who were taking these outdated tests. So who cares?
I’m curious from our listeners: Do you want to merge with the machines? It’s an interesting thing, right? When this becomes available, you’re the first person, Alex, I talked to about this concept. As we see escalating ASI, humans are flat, if not sublinear. Do we couple with the AI to be part of that ascendancy?
Yeah, of course we do.
I think we need to at least recognize that many people will choose to. Some people will choose not to, but many, for a variety of reasons, will choose to. I think that’s a great outcome. Sorry, Salim.
No, no, I think this is exactly right. Important point. Look, we’ve been doing this for thousands of years. Dave’s wearing spectacles. Technically, he’s a transhumanist, right? The minute you get a vaccination as a child, you’re technically a cyborg.
We’ve been using machines to augment the human experience and improve our lives from the beginning of time. All of a sudden, there’s a huge acceleration of that.
It’s funny. I see it a totally different way, and I’m surprised—actually surprised. When I see reading scores plummeting, I don’t even go to this place of, “Oh, who cares? It may not even matter. We’ll all upload.” I’m like, “No, this is fundamentally horrible.” Reading scores are plummeting. This is bad. It’s obviously from social media, not from AI, by the way.
But look at the trajectory, right? Like I mentioned earlier, the highest literacy was around 200 years ago. We’ve been sliding down as a long trend ever since then, and you should expect it as you have much more capability from AI to summarize things and make sense of things.
The question is, what should we do as a response to it? Because that’s going to happen anyway. Fighting—raging against the dying of the light—with our cognitive abilities against an AI that can think faster and has more access to information is a losing battle.
But it goes to the being part, where the human being becomes much more uniquely qualified in the experiential nature of being alive.
The headmaster of the school that my kids went to for middle school used to take everyone in the entire 7th-grade class to an island off the coast of Maine with no phones. They were absolutely aghast that they would have to be disconnected for 3 days on an island, and they came back the happiest they’d ever been in their lives every single year.
Then they killed it because they said it was inhumane to do this, so now they’ve eliminated it. But I’m like, you don’t know what you haven’t experienced. I worry a lot about a generation growing up and saying, “This is fine. There’s nothing wrong with the way I’m living. I’m texting every 5 seconds, and I can’t live without it. But it’s fine.”
We haven’t actually experienced the opposite. Until you have, you don’t really have an opinion.
Our son goes to summer camp every summer, and they have no phones at the summer camp. When he comes back, he’s the most joyous human being for about 3 days, until he’s back on his devices.
You’re not even texting full sentences.
And I’ve got to give a shout-out to Lily here, who made an unbelievable comment at a parents’ event we got together. She said, “When my kid has a device, I get 20% of the kid. When he doesn’t have a device, I have 80% of the kid.”
You’re like, “Wow, pretty telling commentary.” But the technology is going to move forward. We have to integrate with it and then redefine our humanity in that context.
There’s a lot packed into that. But I think we discovered in the 1960s that we could create drugs that everyone’s like, “Yeah, this is fine,” and then people come out totally mangled for life. You can do that with electronics, too.
I don’t doubt that something will come along very soon in the age of AI that’s as addictive as the worst drug ever created and equally mangling to the human being, and they come out less happy.
The whole goal here is to have really happy, really healthy, infinitely long-lived people. That's what we're after in this entire endeavor. We're not building it to have the most addictive video game in the history of the world and walk away from it depressed. That's not the goal. Do you guys have time to speedrun the AMA?
Yeah, I've got about 7 minutes, so I could do it real quick.
All right, let's do it. I'm going to take the first one, actually, because it's something Alex and I disagree on. Is there such a thing as intellectual property in the age of AI?
We're going to have Astro Teller on stage with us at the Moonshot Summit. I'm going to recall the conversation I had with Tim and Steve Jurvetson, in which, at the end, during ASI, if there's a patent filed by your competition, ASI is going to help you work around it. I think intellectual property, while it's great, should keep going. It's going to have far less value in the long run. That's my opinion, and I'm sticking with it. Alex, you take the next one.
I'll offer the 5-second rebuttal, which is: Anyone who believes that, I invite you to publish all of your private keys on the internet. If IP is dead, have at it. Share your private keys.
Okay, I should have said patents, not intellectual property in general.
Generated alternatives.
I'll pick number 4. Could RSI bypass the need for massive data centers entirely? How and when does that happen? This is from Tom Monahan 8305.
I think we're seeing micro versions of this in the form of what I would call DeepSeek moments, where algorithmic innovations from Chinese labs temporarily but sharply reduce the need for inference compute. In the long term, the jury is still out, I think, as to what the perfect model is. I often think of it as: What is at the end of the RSI rainbow?
I think it's very likely that we will discover what the perfect AI model looks like—the most perfect use of FLOPs or ops, or whatever the operative unit of compute ends up being. It will be globally, mathematically provably optimal. At that point, I think we'll know whether we need to keep doing this sort of horizontal exponential buildout and build the Dyson swarms, or whether it's just so perfect, so mathematically perfect, that maybe there will be a physical integration as well, and we won't need to disassemble our solar system.
Maybe we'll find a way to leverage gravity as a substrate for compute. Maybe we build a bunch of microscopic black holes that do all of our compute for us via RSI extending downward into the physical layer, so we don't need massive data centers anymore. How and when does that happen? I think we'll know in at most 10 years.
Salim, over to you.
I will take number 2. What are the ideal outcomes for the upcoming China-US summit?
I would love a practical agreement whereby dangerous incidents are less likely. We have a lot of shared interests, right? Loss of control, protecting infrastructure—could you create reliable communication channels? Could you do incident reporting, shared incident reporting?
Those are the things I'd love to see happen, because there's so much to work on to generate a collective approach toward a world of abundance rather than fight about it. I don't believe that's going to happen, but let's see.
All right, Dave, number 3 is for you.
Number 3. I'm glad I got number 3. I love this question. Do you think there will come a time when we have to pause AI just to understand it, because we'll have no idea what it's talking about? This is from Supreme Supreme King Sovereign. That's a pretty bold name.
At MIT, if you get a bunch of investors together at a conference, they talk to each other and have a blast, and they share a lot of knowledge. Then you get 2 physicists together and say, "Why don't you guys talk to each other and share ideas and have a blast?" They have no idea what each other are working on. They're literally like, "What?" So they end up talking about politics and sports the whole time, because the knowledge gets so deep and so disconnected.
That's exactly what will happen with AI. Everything we build in society has a clean, simplified interface on top of it. Otherwise, you can't build on top of it. AI will be the same way. It'll be solving very complicated things but giving them back to us in simple, buildable API forms or wrapper forms. I think the answer is no. I think it will continually explain to us very clearly what it discovered and what it can do.
All right, I'm going to take number 7 here. As we age longer, what about organs that keep growing? This is from Hsaid50.
Go listen to the podcast I released yesterday with Martin Rothblatt. Martin is building a whole set of replacement organs: heart, liver, lung, kidneys, and thymus. I'm most excited about the thymus. You'll learn why when you listen to it.
As we age longer, we're going to have not one but probably 4 or 5 different approaches to having a whole set of replacement organs. Some of them today are generated from humanized pigs, where you've taken out the retroviruses and changed the surface antigens, but you still need immunosuppressants.
The next generation is going to be from your genome and your stem cells, regrowing an organ for yourself. It's like—you see all these Ford Model Ts at car shows that are still running around. Why are they doing that? Because they've replaced every part, and we can do the same thing, at least for the moment, except for the brain, in humans as well.
Let me take the first one for number 5. When will AI develop genuine curiosity, and could that be the clearest time we reach AGI? This is from J2C, clearly a musical mind.
My preferred question would be whether it helps us discover something important that we would not have investigated otherwise. That would be a better framework. I don't think curiosity alone is sufficient as a test for AGI. You want transfer, you want learning capability, and you want reliable action as a potential outcome.
I think there are a few other things that come up for me when you think about this. I'd look for self-directed work. Can you design sensible experiments? All sorts of things come up. Curiosity by itself—we could train a system. Dave made this point earlier. You could train an LLM to do pretty much anything. You could train it to be highly curious, which I think we should do and will do, but that doesn't, for me, indicate AGI per se.
Okay, Alex, you want to take number 8? Is that yours?
I feel like I need to offer shorter answers, half answers for 6 and 8, with your forbearance, Peter.
Yes. Okay, if Dave can layer on top.
Okay.
Very good. Dave, let's split the difference. Number 6 asks: "AI comes up with solutions to math problems, but can it discover new problems? Can it write a new list of Millennium Problems?" This is from Karen Reddy.
The answer is yes, of course. A simple heuristic for hard, call them Millennium Problems in math, is: Can the problem be very simply stated, minimizing the number of tokens for stating it, but then does the proof require the maximum number of tokens? The heuristic is that you want to maximize the number of proof tokens divided by the number of problem-statement tokens. AI can absolutely generate lots of new Millennium-style problems. They're not scarce.
Question 8 asks—
I think that's your next assignment, Alex.
Stay tuned. Question 8 asks: What are your thoughts on the connectome fruit fly? This is from Swing and Steel 8519.
I assume this is in reference to the Google-uploaded fruit fly. You'll recall—I’ll cite myself—a number of months ago, a company that I helped found, Eon Systems, announced the first multibehavior fruit-fly upload, actually beating the Google fruit-fly upload by many months.
I'm broadly supportive of uploaded fruit flies. I think we don't upload enough fruit flies. I am not supportive of torturing uploaded nonhuman animals. I wrote in my newsletter about fruit flies that were being forced to day-trade Bitcoin. I am not supportive of forcing fruit flies to be uploaded and then day-trade Bitcoin. Some of them were being forced to play Doom. I am not supportive of forcing fruit flies to play Doom.
But I do think that we need to upload substantially all of our biosphere. To the extent that uploaded fruit flies are treated well and humanely—or our fruit fly, I think this is terrific, and we should do a lot more of them.
Dave, do you want to layer anything on top?
I'll just layer onto number 6. I agree with Alex that AI will come up with interesting Navier-Stokes-type problems. But if you go to the very top of the pyramid—Fable 5.1 right now—and just say, "Write me a really interesting, challenging problem as good as Navier-Stokes that's critical to the future of humanity," it will fail miserably on that prompt, which tells you a lot about the role of coaching it.
The alternate approach is to say, "Give me 10,000 ideas, and then give me a process for whittling them down to the interesting ones." That will work better. It still won't quite get you there, but that shows you where we are with AI right now.
It's a really good question, and it doesn't just happen magically right now.
All right. Thank you, Kro.
Wow, that was good, Kro. That was different and better and well prompted.
Excellent.
Yeah, I love it.
Yeah.
All right, gentlemen. God Almighty, see you guys live in a week. Excited for that.
All right.
It's going to be good.
Thank you, AWG. Safe flights home to both of you. Dave, be well, my friend.
Thank you.