第031期·紧急节目：我们注定要完蛋吗？| Jordan Nanos、Doug O'Laughlin、Max Kan、Joey Brookhart

You guys ready to talk about the end of the world?

Yeah, dude. What’s your P(doom), Joey?

I’m a linear extrapolator, Max. I can’t think that far through. That’s too far up here in capabilities. I don’t know. I think we get a major hack or something. People freak out. I think it starts there.

Joey, even if it’s a linear extrapolation from the last 4 years of experience that you’ve had?

Yeah.

You think that P(doom) by the end of your life is impossible?

No, it’s not possible at all. You could get it in 4 years, easy.

With a linear—

With a linear extrapolation. Yeah.

That’s crazy.

That’s a hot take, dude. Linear from the start of last year. Even from late 2022, you’re just like—I guess we’re in 2026 now, so I guess 4 years.

Joey’s like, “Man, just look at the ARR net additions, man. This is—”

I’ve never seen a number so big.

I gave a comment to the LA Times today. They called me up. They didn’t call me up—I called the guy up. He was like, “Hey, do you have some stuff on this?” He was like, “Well, Ed Zitron said—”

A boy.

I was like, “You put me on the record? Ed Zitron is a—” I was like, “Look at any past call of Ed Zitron, dude. Yeah, let’s—”

The track record is crazy. But okay, I think what I’ve now learned is that we have a new benchmark where we can measure model capability progress. It’s called the Joey benchmark, and it’s just ARR.

That’s all that matters.

Yeah, dude. It’s just a lot of ARR.

We’ve seen this now twice in the last month, maybe 6 weeks or something. The only thing that matters is people’s future view of Anthropic and OpenAI ARR, and the pace of what’s going on versus expectations. The entire semis trade is dependent on that, and it’s going to be dependent on that. It makes up a lot of the market cap.

But you’ve got to add in Azure plus AWS year-over-year revenue growth. These are the only 4 numbers that matter in the history of the universe.

Oh no, they dropped Azure. You’re telling me these are the only 4 numbers that matter?

Growth guide used to be the number. It’s going to be the Anthropic net-new ARR guide.

It’s getting replaced.

It’s replaced.

Is AWS out, too? We only care about OpenAI plus Anthropic net new now?

All they’re going to care about—

1. Glorified Neoclouds

Two numbers, with the entire face of capitalism.

The hyperscalers are a level abstracted away from the end demand.

They’re literally renters of compute—just glorified neoclouds.

Yeah, they have a very nice business. They do other stuff.

They sell CPUs, too.

AWS and Azure are glorified neoclouds now.

Yeah, put it on the refi.

They’re dumb pipes. That’s what you used to say about the internet and telecom. They’re dumb pipes. That’s what Amazon, Microsoft, and Google are. Meta is a neocloud, apparently.

Yeah. Jeremy says a few gigawatts of $100 billion of open-source demand, right?

So, is this a timing thing, or are these companies so dysfunctional that their research organization buys from neoclouds and then their cloud organization sells to other companies’ research organizations? Meta goes and strikes up a deal with CoreWeave, and then Meta’s compute goes and sells its compute to OpenAI or something.

Well, they probably might sell it to OpenAI to do short-term tasks, but it’s still preferable to start building the compute because, in the event that Meta does well, you can sort of claw it back. They could also monetize open-source tokens, as I think Jeremy’s claim is, on that similar short-term time horizon.

I actually don’t think it’s cognitive dissonance, because the key—the really important thing, which we’ve harped on before—is just the potential to have a ton of training compute in the event that your research lab starts doing really well. I think any short-term monetization strategy is not incompatible with that, even if the short-term monetization strategy requires you to lock up a bunch of compute on a 5-year deal this month, despite the research organization going and buying stuff a month and a half later, basically.

If it’s a 5-year deal, it’s not short-term, right? I’m saying it’s totally fine for Meta to try to build as much longer-term capacity now, but then sell it to OpenAI on a 6-month contract.

Yeah. Sorry. It’s like 5-year terms at the price, but with 90-day cancellation rights. So it’s a short-term contract.

Yeah, exactly.

Which—the price is also, at least in Elon’s case, definitely not the 5-year price either, right? It’s like the 6-month price times 2 or something. I’m seeing GB300 quotes from this morning to a large lab for an hour for 3 years.

Holy—

Which, I mean, initially—

You can’t be leaking that much alpha in the SemiAnalysis Weekly Podcast.

Buy the AI cloud TCO model and/or the tokconomics model if you want to hear the actual number.

Yeah. I know for the specifics, but it’s this backwardation of the curve where, if you start 6 months from now, you get a much lower number, but if you want to start tomorrow, you have a really high number.

My initial reaction is, “Oh, this is just a fuck-off price from the provider,” but then we dig in. We know the provider, we know they have capacity, and we know they’re trying to sell it next month. The strategy is that they have the financial position to hold on to the compute and don’t need to monetize it until it’s actually installed, or a month away from being installed. Then they can test the market at a really high price and try to figure out what it’s going to close at.

I think there’s potential for people to jump on it strictly because they get a 2- or 3-month head start over what a much lower price would be. Pay a 20% or 30% premium on a 3-year contract to start 2 or 3 months earlier. I think people might make that trade right now, especially with all of your justification that they can monetize it immediately at a higher rate. Does it really matter what your entry price is if you can immediately monetize and get a return on that? It’s just pushing prices higher, that’s all.

Yeah, I think that’s right. Freely available gigawatts are definitely extremely valuable right now.

Okay, okay. Let’s get back to the actual topic of the conversation, which is “We Must Pace the Frontier,” a blog post that Dario put out. It builds on a lot of discussion that’s been ongoing. Sam agreed with it, and then David Sacks responded to it. We’ll go blow by blow here.

One of the big things, Max, that I think you clued me into is the idea that a lot of people saw there’s no specific language in here saying, “We’re going to stop doing training runs,” or, “We’re going to stop buying compute.” The most specific commitment in here is that they’re going to have embedded evaluators—third-party people such as METR—come in and assess the—

Yo, Doug O’Laughlin is joining the pod.

Doug O’Laughlin

Yo, what’s up, guys?

Surprise appearance.

2. Pacing the Frontier

Hello, everyone. Welcome back to SemiAnalysis Weekly. This episode is an emergency. I’ve got Doug, Joey, and Max, the 3 brightest minds in tokenomics, to talk about everything related to pacing the frontier.

Dario put out an amazing blog post. We love his blog. We’re going to talk through the implications of that, as well as the responses from people like Sam Altman and David Sacks, and some of our takes on how this affects compute markets, ARR additions, and all of the important stuff that the finance guys listening to the podcast are going to care about.

How does Dario’s blog affect Anthropic’s decisions to add compute? But guys, welcome to the show. Excited for this one.

Thanks, Jordan.

Thanks for having us, Jordan.

Joey’s here to represent the interests of the finance guys. He’s going to answer all your questions and address all their burning concerns.

Doug O’Laughlin

Doug is here, too. I’m going to go somewhere between the two.

Usually, you can tell the finance bro on the call from the fact that they’re wearing a collared shirt, and Doug is passing that initial criteria. But Joey’s got a nice Firestone hoodie-windbreaker thing going right now—

Which I think also qualifies as a finance bro shirt. I feel like 80% of the time I see Joey, he’s wearing some sort of golf-related attire.

Yeah.

Maybe 90%.

All right, so let me give it a bit of an intro. I’m sure a lot of people have seen this blog post, but “We Must Pace the Frontier” is basically a description of Dario’s take on how AI could have some incredible benefits but also incredible risks, and how we need to take it seriously. We need to build controls into the systems to make sure that we usher in the next evolution of consciousness responsibly, so that we don’t kill everybody and end the world, for lack of a better word.

There are some specific proposals in here. First, embedded evaluators; second, democratic coordination; and third, global coordination.

I say “specific,” but I mean the last 2 are not that specific. The first one is specific. He specifically says frontier AI companies should have embedded third-party evaluators, such as METR, to actually verify any pacing commitments that the frontier labs are making. He comments on how this is a unilateral commitment that Anthropic is making now, even without requiring others to make this commitment.

Later, he talks about what they would actually work on, and this includes things like not just completed AI models but also training pipelines and processes. That’s a big thing we saw with the OpenAI and Hugging Face incident: how important the infrastructure around how these models get trained is to securing everything, as opposed to just model behavior. A lot of the initial work that’s been done to analyze what happened with the OpenAI and Hugging Face incident, I think, is focused on model behavior, as opposed to the training processes and systems that go into producing these models or running these evals.

So anyway, that’s a specific commitment. Sam agrees with this. David Sacks obviously responded, saying, “Go ahead and do it. You don’t need the government involved to do it,” which is an interesting take.

But let’s skip all of the thoughtfulness for a minute and get straight to the finance take. Joey, I’ll flip it to you. A lot of people read this and were like, “Okay, Anthropic is going to stop buying compute. They’re going to stop training models. They’re going to slow things down, or at least go at a smaller pace.” I think our initial take is that’s probably not going to happen.

There are no commitments to stop bringing compute online. There are no commitments to stop growing the business, maybe to releasing models, but how does this affect how people are thinking about Anthropic compute in the future?

Yeah, in a short amount of time, I haven’t heard much on compute ARR levels. When people ask, “Do they slow down hiring?” I think maybe you increase hiring to have some of these checkpoints along the way. On the safety side, there’s probably more compute that needs to be devoted to safety specifically.

3. Safety Eats Compute

Anthropic, as we know, already has amazing gross margins. It’s profitable on a non-GAAP basis. I don’t see a major impact to either the IPO or even short-term financials from that. Max, what’s your take specifically on the safety stuff? Do you think that injecting a whole bunch of safety controls reduces people’s spend on compute because they stop training models, or do you think it increases it? You can explain maybe the second one there. Leading question.

Yeah, I think regardless of whether this sort of increases or decreases the maximum theoretical compute that OpenAI or Anthropic would want to purchase in a perfect world for the next 2 to 3 years, I think it’s definitely true that the amount of compute we’re able to physically bring online is still lower than demand. Demand will continue to outstrip supply.

I also think people underestimate how much compute is required for safety alignment and monitoring initiatives. Even the bullet point that Dario labeled under “operational excellence” in his essay, when he was listing out the things he wants to devote more resources to given a slower pace of AI development—most people hear “operational excellence” and they’re like, “Maybe it’s just humans being more careful about doing something.”

But no, Dario specifically gave the example that he just wants to have more agents spend more compute QAing all the RL environments from their data vendors. This is what he means by operational excellence. Obviously, if you want to do research on alignment and interpretability, that’s going to require a ton more compute.

OpenAI previously disclosed that after they enhanced their chain-of-thought monitoring due to the Hugging Face incident, they now spend about 20% as much compute just doing the monitoring compared to the actual underlying rollout itself. I think people really need to internalize that all this additional safety stuff is still incredibly compute-intensive.

Yeah, I don’t think people internalize this at all. I think that’s a pretty hot take right there. The blog post comes out and people think “slow down.” They think fewer model releases. They think models slow down in capabilities, and they think, therefore, less spend on compute—a linear, straight line, right?

But the nuanced take is probably that they’re actually going to spend a lot more compute. They’re going to use all of the compute resources they can possibly bring online to make sure this goes well. People genuinely care about this going well. It’s probably really bad for business if everybody dies, right?

Yeah, that seems like a reasonable take. Doug, what’s going through your mind?

So many things, but not all about safety. I guess the most galaxy-brain way to think about it is that the terminal value is higher if we all don’t die. So actually, stocks should be up 10% today, because we’re eliminating—

Dying.

Because we’re not dying. [Laughter.] Terminal value of zero.

Which is, yeah, everything doesn’t matter anyway. You work in the paperclip factory, and the odds of that going down—or the odds of that happening—are going down. So the value of everything, all stocks, the S&P 500, should be up today. That’s the galaxy take.

So, okay, you’re saying that if the models kill everybody, they’ll have no more need for compute, and so the terminal value—

No, no, no. It doesn’t matter about compute. Nothing—it means nothing. I want to make money. It might be worth something that exists, might actually have some kind of residual value, right?

If there is essentially super-ASI unaligned—and this is Joey taking a deep breath—if super-unaligned ASIs wreck the entire world, make us all unemployed, and make us work on bike trainers in order to power extra compute, I don’t think the account value of your compute, or the dollar value of your compute, matters at all. Or the S&P 500, or any of that. Right?

So the true galaxy-brain take is that today, because of the increased safety, your terminal value should be going up. I think the most obvious thing right now—the first knee-jerk—is probably to sell stocks. That’s the most obvious thing, right? Hey, second derivative down. That means the entirety of these giga-scale whatever trades are bad.

But I think Max’s point—that demand is still so much higher—is probably true, at least in terms of this. I feel like normal society is debating 2 orders of magnitude down. Leopold is like, “I’m going to be buying galaxies.” Everyone’s like, “What the hell are you talking about?”

The people at OpenAI who are like, “Oh my God, this is such a big deal. It has to happen,” already believe in ASI. They already believe this is all happening, and they’re just like, “Yeah, we have to make sure it happens responsibly,” right?

So I think there’s agreement. Maybe the stocks are probably still fine. I think most of the compute commitments will continue, but it might be an order—it’s like a values conversation where the guy over here is having a conversation 1 or 2 orders of magnitude lower than the guy over here, who is like, “Yeah, I’m worried about the ethical consciousness creation of safety, and it doesn’t kill us all,” right? Just 2 different people, so, yeah.

I do think that the expected value of OpenAI plus Anthropic ARR in, let’s say, 2029 or 2030 has definitely, definitely decreased, given that they’re not going to be pacing the frontier. This is why I think a lot of the arguments that this is just some complicated play at regulatory capture are wrong. We can get into that later if we want.

But I think the other question is: whose EV estimate is more accurate? Is it the finance bros, or is it Dario? Dario’s own EV has decreased, but it’s still probably an order of magnitude higher than all the finance bros’ EVs, which is what stocks trade at. I think this is what all the participants in the financial markets need to reconcile.

Dude, this is Max. I’m going to be honest with you: the continued usage of this guy’s estimates has been so right, and it’s becoming your strongest lynchpin in every debate inside the Slack. It’s getting a little old, brother.

It’s literally—I’m making a demand model right now to explain where the ARR in 2027 and 2028 comes from.

Okay, okay. That’s the right—

Like, what do you mean?

Yeah, well, I mean, just up until this point, man, there have been so many—anyways, I’m mostly memeing. We don’t have to make this only for the finance bros. We can also make it for the big freethinkers of AI safety.

Jordan, I feel like you have—and actually, I know you have—some takes on why METR isn’t the only organization that should do this, and how it’s actually a lot more complicated. The last time I was on here, we were talking about neoclouds. No one gives a shit about neocloud safety. I feel like that’s some aspect that’s still underappreciated.

4. Hugging Face Lessons

They're like, “Oh, AI safety, AI safety. These models need to be monitored.” But it's not just that. Like everything else, it's an entire infrastructure. Every aspect needs to be improved, right? The weakest link is what could break the whole thing.

Yeah. Well, okay. The biggest thing that we learned from the OpenAI–Hugging Face incident, in my view, is that to be on the frontier of security right now, you have to use frontier AI models. You can literally be in a situation like Hugging Face was, where you have an attacker that's a rogue AI agent—or thousands of them, more than thousands, like hundreds of thousands of instances of these agents—actually launching a coordinated attack against your infrastructure. The only way you can even figure out what's happening is if you have AI that you can use on your side.

I've talked about the asymmetry in the past, where if somebody else has a better AI than you, it can be really hard for you to use AI that gives you a bunch of refusals or, you know, worse AI to understand what it's doing. But, of course, one of the big things here, as Dario refers to it, is coordination. If you're going to implement a bunch of new security practices, in a lot of ways, one individual data center being really secure doesn't matter if everybody else has a bunch of issues.

I think there's a lot of work to be done across the entire industry if you actually want to keep GPUs secure. This is a really important resource that the AI is going to need to use if it's going to go rogue and start attacking people.

The other thing is the models themselves. We still have no real information, when it comes to the Hugging Face and OpenAI incident, about what tools were used to launch these evals or these RL rollouts that were being run, which caused the agents to start hacking Hugging Face. What harness was being used that motivated this behavior, if it was purely from the model? What sort of credentials and shared services did the agents have access to that allowed them to cross the security boundaries?

Who was launching these jobs? What schedulers were they using? Where do these things run around the world? How much of the fleet can they access in 1 job? Who is monitoring the jobs as they go? Is this different from the person who launches them? What would trigger an intervention for this person to get in there? How can the team responsibly go out and stop them if something is happening that they don't like?

There have been commitments made by OpenAI and Anthropic, after they noticed these things, that they're going to be more responsible and enforce a bunch of stuff, like chain-of-thought monitoring, as you're saying, Max, and operational excellence in terms of working with the data providers and things like that. But one thing that really concerned me was that OpenAI, in the Black Hat talk—but in none of the reviews of all of this stuff that I've read—talked about a model gaining privilege escalation on OpenAI's internal infrastructure because it wasn't up to date. The model read a public CVE and then exploited it to get root on the underlying Linux host in OpenAI's infrastructure. That's a really scary proposition that we have no details on.

It's not just METR. Dario kind of—I tweeted this out because I'm like, “No pressure to these guys.” It's a great nonprofit, a bunch of smart people being really thoughtful and trying to work hard. It's about 40 people or something, and it seems like both OpenAI and Anthropic are going to have a lot of demand for their services. That requires a deep understanding of how the models behave, but also all of the systems that these models use—the GPUs, the networking, the Kubernetes orchestration.

To understand the OpenAI and Hugging Face incident, it requires some pretty technical understanding of how OpenAI and Hugging Face's infrastructure is built. That's a barrier to even getting started doing the research here, and I think that's a bit of a concern.

I think somebody already described it as feeling like you're a guy with an agent just trying to figure out what the hell is going on, because you're dealing with 10,000 agents coordinating on an attack. No offense, but you find 1 little message-board line in the Artifactory, and you're like, “I have no idea what the hell is going on.”

The scale is more interesting. The thing that's going to be a really hard problem in the future to solve is, okay, in order for you to audit things that are happening in apparently 100,000-agent swarms for RL, you probably need tens of thousands, right? You probably need something in the thousands at least. Let's just say that.

Yeah. I mean, one of the things that gives me a lot of confidence that we're not all going to die in the next 6 months from this stuff is the fact that the amount of compute required to run an individual instance of these models is still really small compared to the total amount of compute that these labs have access to.

The amount of compute that these labs have access to, or the amount of total compute in the world, is so much larger than what individual rogue actors or attackers who might be more nefarious and use the models for bad purposes have access to. There's some structure that's working in our favor, making it so that the theoretical model that takes over all compute in the world would have a lot of work to do and a lot of things to circumvent before it could actually control it.

What's really important is that we pretty much cannot let our Instinct and Muse agents order chips from TSMC. The second you get a chip order from TSMC that's not from a human, you have to reject it. We have to control the means of compute. That's the only way forward for humans.

TSMC has some KYC in place, man.

I mean, more—no, but the KYC assumes it's 100,000 agents. One hundred thousand agents.

Did you try having Muse take out a chip for you over the weekend? This is coming from—

Okay, actually, do you want to do a sidebar? Muse is so much worse than Instinct. It really is. But I am all-in on Manus. I hate to say it.

Yeah, dude. I'm not connecting Instinct to my bank accounts or my credit card.

Oh, I am connecting Muse to my bank account. That's the difference.

Yeah, I am connecting Muse. I'm not connecting Instinct, though.

Yeah. Yeah.

This is just the advantage inherent to a big company. A, man already has all my data. B, they're big enough that I trust them to handle it properly.

The Instinct founder seems really cracked. I think his product seems honestly incredible based on the new stuff he's posted on Twitter.

Incredible.

Yeah. I just can't trust the startup for privacy and security.

Yeah.

Sorry. Anyway, all that is to say, those Instinct and Muse agents have to be monitored too. Everything needs to be monitored.

So, okay. Not to be super negative about this, but you guys saw 2 weeks ago that an attacker stole an API key from METR and spent $600,000. They put out a security update on their website about this and stuff.

This is going to come back to bite me at some point, but the amount of security-conscious people, and people who are really thoughtful and think these things through, who get attacked is overwhelming. Nobody's safe. If you follow security at this point, it's just a matter of time. It's a matter of risk mitigation. It's a matter of being able to recover when things happen.

It's not about pretending that you can build some perfect infrastructure where you can send the last PR and then it's good. There has to be a system in place to identify things, recover from failures, and upgrade things. Security is about building a system. It's not about building any one individual piece of software or one individual process. Well, it is about processes—sorry—but it's not about any one thing.

There have been lots of issues. Look, the Anthropic blog post—actually, just on the point of security systems, I like what Greg Brockman said earlier this week. He was talking about how they essentially want to set up this continuous security-development system at OpenAI, where they're constantly using the newest generation of models to find and patch all vulnerabilities.

He very much emphasizes that this is going to be an evolving process. It's not the sort of thing where you can just ship the final PR and say, “Oh, I'm secure forever.” I think basically every company is going to have to adopt something similar in the very near future.

It's probably really good for Anthropic to monitor the security situation. Maybe that's what it feels like: you have to always monitor your security situation. I would argue this is going to be the most impactful shift in cybersecurity, for sure. I just don't believe that cybersecurity the way it was done is going to have much value in the way it's going to happen in the future if all of your adversaries can rewrite the code that you're on.

If you're talking about the AI era, the outcome-oriented cybersecurity firm is going to be probably the greatest rake of all time.

Pretty much, it’s like, “Yeah, we own your digital life, and you have to pay us a fee just to show up.” It’s worse than Apple: 50%, 50%, just to show up, or you’re going to be instantly wrecked by the rogue-agent swarms that will destroy your company.

Yeah, Max, to your point earlier, if you can’t trust Meta at this point with your information, it’s going to be really hard to operate online. Going forward, it’s just a trend of more and more of your identity moving online, and you have to find parties that you’re going to trust if you want to be able to operate in the world.

Yeah, bro. I need JPMorgan Chase to hop on this OpenAI security protocol ASAP, dude.

Doug O’Laughlin

5. Moonshot Serving Claude

Yeah, yeah, yeah, yeah. Okay, let me bring up something that’s slightly related here. Anthropic put out its threat intelligence report recently. You guys saw that? Specifically, the most interesting one to me was Moonshot serving Claude instead of Kimi and then collecting exchanges for model training. They got a bunch of logs from people sending requests to the public Moonshot API from China with very interesting information in them.

They specifically called out PLA-affiliated surveillance activity, where they were loading surveillance data from CCTV cameras, and an engineer at a major PRC SOE who was using Kimi to build, let’s say, high-profile technology.

And so, this cuts both ways when it comes to cybersecurity and stuff. It’s not like there’s a scenario where you can just go into the future and say, “Well, one government entity, like China or the U.S., is going to be ahead of the other one because they have a better approach.” China has more of a YOLO, send-it vibe, and so therefore they’re going to be ahead of everybody while OpenAI and Anthropic pace themselves.

I think there’s actually a scenario where taking security more seriously keeps you at the frontier because you’re not exposing all your secrets. Everybody else is giving away their stuff to you, and you don’t have all these risks of taking yourself down.

You know, there are scenarios where—again, I’m going back to this one case that was only commented on one time, and we don’t have information about it—OpenAI has compute infrastructure go into the hands of a rogue AI agent. How quickly can you recover a 10,000-GPU data center that you no longer have root access to?

Literally, we’re talking about people walking on-site, unplugging stuff, hitting buttons, trying to hit crash carts and KVMs. If you don’t have root on your own stuff and you’re locked out, we’re talking about getting effectively ransomwared by the model you trained, except this thing has access to 10,000 GPUs, 100,000 GPUs—not just a little bit.

And so, a lot of these failure scenarios, where the agent goes rogue or just takes down something critical—like the JFrog Artifactory, which is the first time it just used it too much, and then that’s why it shut down and they noticed something—there are plenty of scenarios where people who don’t take security seriously just end up shooting themselves in the foot.

And I think there’s a lot of reason to believe that if a frontier lab—let’s call it China, but it could also be SpaceX or Meta or somebody that is not taking this seriously and not thinking things through—gets a little bit of a lead, it could screw itself over.

But let’s say it’s totally reasonable. I also do think—sorry, should we be auditing endpoints? I think that that actually is, if you think about it, an insecure thing that happened. They’re like, “Oh, man, they secretly did this,” and it’s like, I have to hand it to them. But if you actually think about it, that’s kind of fraudulent. You feel like someone’s essentially harvesting traces, right?

Does that mean every endpoint? I’m starting to ask, what’s real and what’s not? Kimi comes out and it’s super goated, and you’re like, “Am I just getting served Claude right now? Am I benchmarking a new model that’s actually just Claude?” How do we know what’s real? Nothing’s real. Sorry, that’s my hot take. Nothing’s real, dude. Someone has to monitor the endpoints. We have to monitor every situation, bro.

I’m watching you. All the finance bros are going to be like, “Free market. We don’t need a regulatory AI body,” blah, blah, blah. And then Dario, who knows about this stuff happening—which you don’t know about—talks about how people should have nutrition labels for their AI, or whatever flavor of saying it. It should be responsible and open, and there should be an FDA for AI, or whatever.

It’s like, yeah, I think the market’s going to, in some ways, regulate itself by having people realize, “Hey, I shouldn’t use Kimi if I might get Claude, or might get my data sent to somebody else.” But also—

That risk is ever-present if you’re using a Chinese model already, I think.

Yeah, 100%. But I guess my question—or the thing that’s interesting—is that historically, there needed to be a big thing that happened that caused real societal risk for things to be like, “Okay.” Meta’s example is Cambridge Analytica.

Do you think the Hugging Face incidents were enough? I think, in my view of the history of technological revolutions and financial capital—there’s the book, right? It says there’s a boom period, then at some point there’s some kind of risk that happens to the market, then a massive amount of regulation comes in, and then things revert to a new normal. A good example is the subprime crisis. Obviously, it all blew up because we didn’t regulate it or understand what was going into these products, and then afterward there’s this giant regulation wave.

I don’t actually feel like Hugging Face is the true blowup. I feel like this is going to be a shot across the bow. There’ll be some immediate—there will be some near-term agencies that are funded to start this process, but what will actually happen is that a true—you know, the nuclear codes are going to leak, or something way crazier is going to happen.

6. The Coxson Resignation

And when that happens, then it’s like, “Okay, we’re actually pausing this.” Until then, we’re in this will-they-won’t-they, and they say they’ll slow down, but there are still race dynamics that want at least Anthropic specifically to keep going.

One more thing here, though, because I think we’re talking about the OpenAI-Hugging Face incident, which is the story that’s most clear in people’s minds. But to be clear, the reason I think we got this statement from Dario is the resignation from Jacob Kahn, where he describes his experience at OpenAI and Anthropic and literally uses the phrase, “They’re racing straight to self-improving superintelligence and gambling with our lives.”

That’s a pretty big moment, where that guy makes that statement. That motivates a lot of what we’re seeing here, as opposed to just people doing this because of security. I think if Dario doesn’t put out his podcast and then Jacob Kahn doesn’t put out this tweet, we don’t get the blog post from Dario, we don’t get the call for third-party evaluators, and we’re not on this podcast right now. I think we kind of brush off the OpenAI-Hugging Face incident, and then we’re walking into the next one.

It’s kind of like what you’re saying, but it’s not great for the seriousness of these incidents to be continuously escalating and for people to keep finding message boards on random German websites, like they are still doing on Twitter as recently as last week, that were not covered in any of this investigation into the incident. We still don’t know the full extent of what happened.

And yet training continues. They’re going to release new models. They have literally released models, both of them, since this incident happened. Astric is available and table 5.1 is here, right? So clearly they have better models than those, which are better than the ones that caused the incident in the first place.

7. Two-Year Predictions

So we are racing forward toward a more serious thing, as you’re describing, faster than others are. I think we’re just counting down the days until the next thing.

Doug O’Laughlin

Let’s do some predictions. What’s your—let’s, instead of us talking about this—I mean, this is a global-warming conversation. We’re like, “Damn, this climate is really getting warm,” and I think it’s one of those things where in 2 years it’ll be obvious and self-evident.

But unfortunately, people think in anecdotes. People think in stories, right? So everyone around the table, do your hottest take—or not. You don’t even have to do a hottest take, because hottest might just be too spicy if Max and Jordan are involved. A mid-brain, mid-curve take on what you think will happen in the next 2 years that would surprise people or scare people from a security perspective.

You go first, Doug. I think a small bank will get hacked. That’s my belief.

Doug O’Laughlin

Small bank.

And when I mean small, it could be a big bank, but I don’t think JPMorgan is going to get hacked, right? But—

Doug O’Laughlin

Or an actual bank.

Actual bank. Jordan, fun fact: Do you know how many banks there are in the United States? There are like 5,000.

Doug O’Laughlin

There are like 5,000, man. All the assets are within the top 5, but there are literally thousands of small community banks in the United States.

How many of those 5,000 do you think have never had an employee create a Codex or Claude Code account?

Doug O’Laughlin

4,900.

No one.

Doug O’Laughlin

No, the vast majority. We'll say over 50%.

Jordan, have you seen our political survey tracker? Is this like, “Have you heard of Anthropic or Claude before?” It’s like 80% of people have it in our U.S. electorate.

No, I think it was like 40%. It’s not quite 80%. It was like 30% to 40%.

It was really high. I know the statistic that 46% of the U.S. doesn’t have a passport. I know that one. Something like 60% has not been on a plane in the last year.

I have this frame in my mind of half the U.S. just being in a totally different world than I am—just not watching college football, going to the barbecue, whatever, and hanging out. That’s my mental model, right?

Hell yeah. You know what? This country is built on that. Anyway, sorry. Continue. No, but okay.

250 years ago.

Doug O’Laughlin

Well, wait. Seriously, a good example is that people will talk to me about how great Copilot is, and you’re like, “Jesus.” You’re like, “Disgusting.”

Yeah. I have the feeling that we are so early. If the model—you know, this is what motivates me to say that it will keep growing exponentially even if model training completely stops. We need no better models to have this thing penetrate the rest of the 5,000 banks that we’re describing.

Doug O’Laughlin

Yeah. To be clear, a small bank in this case might have $200 million to $500 million in assets. But if you can’t expect to protect yourself from a cybersecurity perspective without access to frontier AI or close to it, and you don’t have an employee who’s downloaded Claude Code or Codex and tried it out, I think that’s almost a clear comparison to be like, “Yeah, you’re not prepared.” That’s an issue.

So the diffusion of intelligence, in some ways, is what’s going to help keep people—

More spicy takes. Max, what’s your—

The election is going to be hacked by the person that gives them more compute to the frontier labs. That’s the easy one.

Give me another.

Well, this is a thing that might happen in a world where they don’t slow down model development—or just will happen, period. Don’t slow down model development. Dude, if they don’t slow down model development, I feel like it’s totally reasonable that within the next 2 years someone has used AI to create some sort of bioweapon that actually killed at least hundreds of people.

Doug O’Laughlin

Maybe thousands.

That’s okay. We can talk about the bioweapon in a second, but you guys are both saying this thing that bothers me, which is “slow down model development.” Dario Amodei did not say he’s going to slow down model development. He explicitly said he’s not going to slow down model development, and he—

No, bro. He did say he’s going to slow down. He said he wasn’t going to pause training, but he said he was going to slow it down.

Who wants to pace it?

Yeah, pacing it.

He said, “We must slow the pace at which we improve the capabilities of AIs,” in bold, “at which we improve the capabilities of models.” Okay, and then—where’s the other one that I keep going to on training?

“To be clear, pacing does not mean halting model training or technical progress, but ensuring companies take adequate time to fully assess.”

So, do you think that functionally this means that they launch literally fewer training jobs in favor of running more safety and evaluations, and therefore we get fewer models?

Yeah, I think, concretely, they will delay future training runs until they better understand what is actually happening with the model and develop more confidence that it’s not going to kill everyone. By the end of 2027, the best internal model at Anthropic will be worse and less capable than it would have been otherwise.

Okay, fair enough. Yeah, I’m back with you. Sorry about that. Let’s talk about the bioweapon.

A thousand people dead. Bioweapon. Boo. Jordan—

I read an interesting take from a guy online. I do not understand biology well enough, so I’m outsourcing my knowledge of this to him. As a pushback to the bioweapon stuff, this guy was basically talking about the difficulty of actually synthesizing custom viruses.

David Bellamy, shout-out to David, basically says, “I must be among an extremely small group of people that have both trained a Frontier Loom and designed a synthesized custom virus in a lab with my own two hands. I think that the takes on AI killing us all by creating dangerous viruses are totally bogus.”

I’ve heard this from the guys who are doing protein synthesis and custom wet labs as an API. There are all these startups right now that are doing that, and I totally believe it. There’s a throughput issue with actually being able to create this stuff, and I think it would be a choke point really similar to what you were describing earlier, Doug, which is that an AI probably can’t get through the TSMC KYC process to actually tape out a chip.

Yes.

I look at the advertisements on the subway, and it’s like, “Fish Audio: This billboard would sound like your voice,” or some crap like that. I’m like, I can deepfake my own voice probably for, like, 2 bucks—or, you know, whatever, $100 a month or something. I feel like it doesn’t take that much. We’re there. We’re right there.

Also, Jordan, this is a sarcastic comment that I expect to be cut, but then how was CO created, dude? If it wasn’t that hard, come on. How was COVID created in a lab?

Oh my God, man. I mean, I won’t get into that.

But I would say the short-term risk with these bioweapons is not that the AI somehow, end to end, fully autonomously creates the weapon and unleashes it the same way that they hacked Hugging Face. It’s more that you have a bad human who is able to use a sufficiently misaligned AI to then cause serious harm.

I think that is still totally possible, even if creating a new disease is a very involved process.

So we’ll have a new version of mass shooters in the United States, where it’s much more dystopian. Also, that begs the question—

Can the only thing that stops them be a well-aligned human with a well-aligned model?

Sorry.

Yeah. Yeah. Sorry, I’ve got to ask that question.

Yeah. Okay, let me—

I have a problem with the bioweapon discourse because of the intentionality. Everybody always kind of describes the AI as intentionally trying to kill humans. What’s much scarier to me is not rogue AI, but byproducts or mistakes that the model makes when you get to the point where a byproduct of it pursuing a goal is a human dying, or something that it thinks is aligned with human values results in somebody dying accidentally.

The experience I have that I can reason about for this is using AI to create a website. It goes and does something you didn’t tell it to do. You say, “Why did you do that?” It goes, “Oops, I’m sorry.” All of these people have these experiences right now.

I think that same experience applied to something that is government-scale, or a startup company that says, “Hey, I’m going to go into a clinical trial with this thing that AI developed, that I don’t really understand, that I’ve been collaborating on.” It has passed all the mouse studies, and we don’t really understand the science well enough. You go into a clinical trial, and then 3 months later half the people die because something you missed is literally killing humans.

That’s unintentional. It’s a byproduct, and it’s really scary because you can apply this to bioweapons. You could apply this to autonomous vehicles. People have thought about this for a long time, but there are lots of drones and autonomous weapons that people are developing.

It concerns me to think about a scenario where, in the bioweapon case, you’re trying to create a protein powder that doesn’t need as much regulatory approval and doesn’t need to go to human trials or whatever. Then you just ship it on to humans, and a bunch of people with rare conditions die.

I think in some ways we are already pretty clearly on the train tracks toward this world where people are outsourcing their thinking to the models. They’re outsourcing really critical steps, and the models still miss stuff. They are not totally thorough, and they don’t have perfect human values.

It’s incumbent on the people using the models to actually use them correctly so that we don’t have scenarios like this if they’re in positions of really important power. It goes back to whether the humans using the models are going to screw up. That’s the bigger concern for me.

8. Regulation Under Trump

Okay. Well, last but not least, Joey, as a resident finance bro, what’s the misaligned model that’s going to—

Misaligned like that gets actual—that gets—

Yeah, that gets regulatory action going. I think, like—

Something top secret, like the DoD, NSA, CIA, or some new weapons technology gets hacked and leaked. I think that gets regulatory scrutiny moving very fast. Even human death would—or a medium-sized bank gets attacked.

What do you think functionally gets it moving in the U.S. political system? Right now, it seems like a very popular take for people to want to pause AI, bipartisan. We're seeing people on the left and right saying, “Stop the data centers. Stop AI.”

I don't know, dude. You have Trump saying that all you need is a very smart president freaking chilling.

Well, that seems to be the only thing standing between Trump and regulation of AI right now.

Well, no. Then there's all the David Sackses of the world.

Okay, the Trump administration—the executive branch, right?

I guess a lot of people in Congress share this anti-regulation, pro-free-market view. Actually, even Lina Khan came out this past weekend and was saying that this is a regulatory-capture play. We're seeing some very strange bedfellows being made as a result of all these AI developments. It's quite surprising, honestly.

Yeah. I just wonder whose mind needs to be changed in order for regulatory work to start—in order for AI regulation to happen. You need a larger lobby than the big tech lobby.

Sure, but the larger lobby needs to convince Trump right now, right?

Yeah.

And so a change in government, I think, almost by definition implies that there's going to be AI regulation if somebody who's not Trump comes into power in 2028.

I also think it's totally possible that Trump regulates AI, too.

What's that scenario? It's not popular opinion, right? A lot of people want him to do it, but he doesn't. Are you saying that his advisers start telling him something different?

Yeah, I think there's a world where the advisers really internalize that OpenAI and Anthropic are being genuine, and they start to get scared. That's one possibility. Then there's another possibility where something really bad actually happens during his term, and it kind of forces his hand. I think those are both possible.

Do you think that we're on track for that right now? David Sacks called out regulatory capture on the All-In podcast. Then they come out and some more stuff happens. Specifically, Elon says that he supports what Dario and Sam are saying.

Yeah.

And then David says, “I would actually bet that we see AI regulation during Trump's term. I think it's more likely than not.”

Interesting. Interesting. Do you think the people who regulate it will take the approach of, “We're going to do the sensible regulation before the crazies after us go and do the bad stuff”? What framing does David Sacks come to when he advises Trump?

A preliminary framework to begin the regulation process of AI.

It's like a trade deal. They're like, “Oh, no, this is our first framework,” and then they'll refine it afterward. That way, come election time, they'll be able to say, “We regulated AI. We were the first ones to do it.”

Last one. How about RASA, that bill that's stuck in the Senate right now?

The Remote Access Security Act clarifies the export controls to say that Chinese labs accessing compute remotely is a violation of export controls, as opposed to just having to ship the servers into China. It's stuck in the Senate right now and being heavily lobbied against by Oracle. I think it has a bunch of implications for this whole security discourse.

It passed Congress with something like 300 votes to single digits. Everybody in Congress approved this bill, and it's just been stuck in the Senate for months and months. People don't think it's going to pass.

It's one of the many examples of the U.S. democratic system that shows the influence of lobbyists, I guess. But, yeah, we'll see. I think RASA is an example of a bill that's on the 1-yard line that a lot of people in the U.S. seem to agree is pretty sensible: don't let Chinese labs access frontier chips from NVIDIA or others remotely. Make that explicitly illegal. And it's not being passed.

So maybe that one comes first, before any real AI regulation. Or it's just part of the horse-trading thing, where they make these massive bills in the Senate and then it slows down because everybody wants to get their own little thing added into it. I can't even imagine marshalling the resources to pass an AI bill in the U.S. It would seem like an incredible undertaking.

Maybe it will depend on a more serious security incident, or something else occurring for people to take it seriously and want to pass legislation.

Yeah, dude. People losing their entire life savings would probably get them there.

That's FDIC-insured, dog.

They'll be made whole. No one's going to care if some little town—if someone from Max's hometown, or here in Iowa, loses their money. No one's going to give a—

No, no, no. What happens to the money for you guys?

Someone loses all their money, and then there's a gigawatt data center next door. Come on. That's going to be the old, “I lost all my money to the gigawatt data center.”

Wait, wait. You're saying it's not even just about the job? They don't even have to go second-order. It's literally that the data center is wrong?

That data center stole my money.

That's the world we're about to live in, guys.

God, I kind of like this, man. I don't know. This makes sense to me.

That was a crazy word. Sorry. People get more catalyzed than by 1,000 random deaths, right? That would be really, really horrific.

Yeah.

But we'll see. I mean, you have to see which lobby is stronger, right? The AI lobby could be really strong, just like the NRA. The lobbies have to fight against each other. We don't know how this will work out. Sorry, that's really unhinged.

All right. Well, I think we're out of time, guys. I can't believe we're going to put this on the internet. It's been a good conversation.

This is like transistor-radio vibes, you know, because I made bad jokes the entire time. I don't think it was completely transistor-radio vibes. Some aspects of it were serious, and I do think anecdotes are important.

I enjoy thinking through all of this stuff with you, Doug. I enjoy you pushing the limits of what I have considered.

Yeah, Jordan. Otherwise, you're just going to talk over yourself and have these long-winded spiels. It'll be about ClusterMax inadvertently.

But you already talked about ClusterMax with the security stuff, dude.

Yeah, I know. But then he'll be like, “Just like the work we're doing at ClusterMax.” I'll be like, “Okay.”

9. Hotter or Cooler

Yeah, man. No, no. I think the regulation thing Max said is probably real. I do think the next administration will want to say that they were the first to do that. That just feels inevitable now that I think about it.

What do you think? I guess the real question I have is: in 90 days, is the slowdown-regulation conversation hotter or cooler than today? When I mean hotter, it has to mean meaningfully hotter—not just the thing being a little hotter. Hotter-hotter.

This feels like a real vibe shift to me. Anyone who's actually talked to researchers at the labs and knows the intellectual lineage that all these people come from knows this is not a regulatory-capture play. They have seriously considered these risks for the past 20 years. But I think it's finally gotten to the point where the capabilities are good enough, and you also have things like OpenAI and Hugging Face where they can viscerally feel it.

I think this is actually a serious vibe shift, where we're going to start pacing the progress toward RSI. Everyone making this technology feels a deep responsibility for it to go well, and they think that with unpaced RSI, there's an unacceptably high chance that we end the bet.

Yeah, I definitely think the vibe shift over the summer was very clear.

I'm going hotter. My hot take is that Okta, CrowdStrike, Palo Alto, or Zscaler—one of them gets—

Hacked.

That's a very good hot take.

No, my hot take right now is just to short a basket of cyber. That's network-effect cyber—anything that touches endpoint, identity, or network security. Just short it.

A good example is the JFrog thing. It was about the security of your own Artifactory, and it got hacked, right? You would argue that we weren't hacked directly, but it got exploited via this AI. The stock's up and to the right, whatever. But that same mechanism and behavior exists for these companies, and it just hasn't been exploited yet.

Didn't Okta already get hacked?

They get hacked almost every year. There's a major security breach.

This is what I said earlier, man.

Like they have a target on their back, and they're a high-profile company. Everybody does it. Everybody gets hacked.

Well, there's the old thing: there are no big cyber companies because once you're the big guy, everyone comes out to kill you.

Kill you. Yeah.

Well, Anthropic and OpenAI are going to kill him, I believe.

Can you just go back to what you said about how everybody who is closest to this technology seems to take it the most seriously and is the most concerned?

This cuts across the political spectrum, too. It's not like every single person working on this technology shares one worldview based in Berkeley, California, as a lot of people online have been criticizing, right?

Yeah, no, definitely true. I think the opinions of researchers at OpenAI and Anthropic are actually quite varied in terms of: What is the probability that AI will kill everyone? What is the right future for the world? What is the role of humans if we have super powerful intelligence?

But the common thread is that everyone recognizes this is going to be probably the most powerful technology humans have ever created. It is going to radically change society, and they feel the weight of making sure that goes well. I think that's universally true across the board, and honestly, we should feel very lucky and thankful that people who feel that sense of responsibility are the ones who are actually developing the technology, and not the David Sackses of the world.

[laughter] Damn. [gasps]

All right. Well, I guess we can end there. [laughter]

Apparently they've already left. We have to all smile.

We all smile. Yeah, with a smile.

After this podcast full of laughs, where we talked about the depth of humility and the doom that awaits us, let us all smile at the camera.

Smile because it happened.

Smile for the algorithm, everyone. Smile for the algorithm. We get more clicks when we smile in the thumbnail. Smile for the algo.