AI:AM Highlights: Clean Neurons, Black October, the GPU Credit Crunch, and Buying the Supply Chain

This week on AI in the AM, what really sets the pace of AI from here? The money, the atoms, or the law? Zvi Mowshowitz, who writes Don’t Worry About the Vase, on what happens to the model release cycle over the next year if nothing slows the labs down.

Advances on the level of a current model release cycle go from 2 months to 1 month to 2 weeks to a week. Stuff like that could happen. You get to see continuously pretty dramatic, very, very rapid improvements in various aspects of AI capabilities. We are kind of on an AI 2027-style timeline right now. It looks very eerily similar to what we saw there.

Evan Conrad, co-founder and CEO of SF Compute, which runs a market for GPU compute, on what is really holding the buildout back.

Right now, the bottleneck in compute and AI in general is not power, it’s not chips, it’s entirely credit. In order to get debt, you need to work with somebody who has excellent credit, because your lender has no idea how to underwrite the value of the collateral they’ve been given, which is a GPU cluster.

If you think about it, there’s a lender in New York or London or somewhere. If the AI lab falls through and fails to pay, and then the cloud provider fails to pay its lender, the lender is going to repossess a supercomputer. The lender has no idea what to do with that supercomputer, nor do they know what it’s going to be worth in the future.

Lonis Hamaili is managing partner of Cambrian, a fund that bets on the physical supply chain behind AI. Full disclosure, I have committed to invest in Cambrian. Here is Lonis on which suppliers the labs will never replace.

Even in general, when we’re looking at the AI market, we’re way more bullish on the atoms over the bits. Most AI companies people are looking at are Nvidia, Broadcom, Arm, and Marvell. These are good companies. There’s nothing wrong with them. The issue is that their business is essentially just using OpenAI and Anthropic models to design software, and then they send that software down to Korea, Japan, and Taiwan to do the actual work.

These companies don’t have their own physical know-how or their own factories. Whilst they are great, well-run businesses, they don’t have anything close to the same moat as these physical companies, which have massive fabs and decades of know-how built up. It’s just very hard. There will never be a situation where the labs compete with a silicon wafer maker. We will always be a customer. Whereas we’re already seeing the labs compete directly with these software designers like Nvidia and Broadcom.

Prakash Narayanan, my co-host, on what the market keeps getting wrong.

I believe that for the next 20 years, the market will consistently underprice exponential growth. Human beings are just not able to understand exponential growth or emotionally get comfortable with it. The market is really driven by a lot of these human emotions and animal spirits, which means that again and again, we’re going to get this bubble moment, collapse moment that we’ve seen in the last 3 months.

Box CEO Aaron Levie posted that if you want to know which jobs AI will create, look no further than cybersecurity. Here is what I made of that.

I don’t doubt that really good cybersecurity people are going to cash in on this trend. I’m sure there’s a bidding war for great talent, but the idea that this is going to create jobs is honestly a bit ridiculous, because the timescale is just absurd. Who’s going to pivot into these jobs? Are we going to get the content marketers who are currently out of work, or the graphic designers, and tell them, “You know what you should do? This is the new learn to code: learn to become a top-tier cybersecurity professional who can contend with these basically superintelligent agent swarms.”

The idea that you’re going to have some sort of boon in hiring is ridiculous. These people don’t have time to hire right now. They’re going to plow everything they have into agents, and they’re going to put agents on top of agents, much more so than doing a whole hiring process to bring a human in to sit on agents. I’m just going to spin up an agent to supervise the agents.

This is the AI in the AM weekly highlights with my co-host, Prakash Narayanan. I’m Nathan Labenz, the narration is my cloned voice, and we’d love your feedback on what works and what doesn’t.

1. The RSI Feedback Loop

Part one: If nobody hits a home run. Zvi Mowshowitz writes about frontier AI at Don’t Worry About the Vase. This was the question I most wanted to put to him. Maybe it’s the most important question in the universe right now: where are we on recursive self-improvement? We’ve had these blog posts from the companies—

Mm-hmm.

—where they try to measure token use and everything down to experiments per researcher. Then we’ve had these Ramez Naam-type analyses where it’s like, well, it doesn’t look like the feedback loop is actually strong enough to be sustaining. It looks more like it’ll kind of peter out. One point six experiments per researcher doesn’t look like exponential takeoff to me, basically, I think is the bottom line. What would be your up-to-the-minute summary of what we know and what we should expect out of self-improvement?

I think Ramez’s analysis, as I understand it, is basically saying that if we don’t see any new fundamental leaps, any new scaling laws, or any paradigm shifts, and the AI just tries to brute-force recursive self-improvement in software using the existing methods, and all it can do is make the current things more efficient, then it will peter out before it goes critical.

I accept the result when it’s stated that way as probably true, but it also seems completely irrelevant. It’s like, “Here’s how many points we expect to score in this game if nobody ever hits a home run, because a home run is always unlikely.” No, that’s not how it works. Every now and then, someone does hit a home run, and you have to take that into account in your expectations. The score will go critical in that sense.

We’re seeing numbers go critical at Anthropic and OpenAI: exponential growth in tokens used per person, exponential growth in lines of code committed. I realize these things are not necessarily going to be as efficient in some important sense, but give me a break.

One example was that Nvidia actually came out with some interesting evidence this week about the length of time it takes to double the efficiency of CUDA. That number has shrunk dramatically over the course of 2026. They are making improvements much faster. Obviously, that’s not true recursive self-improvement, but it is a sign of how much faster you can do similarly technical work that is remarkably similar to AI R&D work in many critical elements. It’s very concrete evidence from an objective actor that isn’t a major lab. It’s a hardware company.

The people I talk to who I trust the most on this believe that 2027 is going to be completely crazy, and 2028 might be unrecognizable.

Reading between the lines, some of the people you trust most on this are probably Anthropic people. I think this is broadly the view at Anthropic; no less than Jack Clark has put it into public print. You can caveat that or say what you think might be different about your view and the Anthropic view. My sense is that, in the space of all possible views, they’re pretty close and pretty aligned. But now, obviously, we have policy disagreements despite these relatively similar views around the trajectory of the technology itself.

When I say people have this perspective that I trust, this is a combination of people at OpenAI, Anthropic, and Google DeepMind, as well as people who are working on policy and people at places like METR, who work across generality and specificity. A huge percentage of people have now converged on the view that the trajectories look very clear: things are going to get—things are already, at a low level, out of hand. RSI has already kind of begun, in a slow-takeoff style.

By the middle of next year, things will look much faster. My job will become much harder, and I will need to triage a lot more information. You already see this. Often I’ll have something and think, “Okay, 6 months ago, this would’ve been a major news story,” and now it gets a sentence or is even silently skipped, because we’ve seen so many other similar things and there are so many things going on that I just don’t have the time.

My readers don’t have the time. I just have to drop it.

Part two: Already clean? Nick Cammarata of Goodfire had published an essay called “Neural Networks Are Already Clean,” arguing that individual neurons mostly do one understandable job after all. It took over the discourse. I opened the show on it before Zvi joined. Here is my read, starting with what the essay claims.

As we train bigger models with more modern techniques, and as our interpretability techniques improve, it looks like the mess we used to see was maybe more of a skill issue. It’s actually a lot easier to interpret the core mechanisms of ML models today than we thought.

We have seen some pretty interesting advances in backprop-free training inspired by this paper. I’m still wrapping my head around them, but my takeaway from studying this so far is that basically all ML architectures to date are limited by the need to be differentiable in order to do the backprop chain rule process. Backprop-free training allows you to explore a bigger section of the total possible space of ML architectures because you no longer have that differentiability requirement for the structure. But instead of doing it analytically, you’re doing it on a more random guess-and-check basis, which has made it very slow. It has not been competitive at all with backprop methods.

The insight that I think shows there’s really something there from this essay is that if you believe, as Nick suggests, that only the strong activations of an individual neuron matter, and that if it’s just slightly activated, it probably doesn’t matter too much for its overall function, then you could say, “Well, why don’t we focus our guessing and checking during backprop-free training on the neurons that are actually strongly activated? We can probably ignore the other ones that don’t seem to be contributing much, and focus on the ones that are strongly activated and therefore, according to this line of thought, are the only ones that really matter.”

Sure enough, that dramatically cuts down the number of guesses and checks you need to make. What we’re seeing out of this community is that it basically does work. Interestingly, the gradient updates they’re making are almost exactly the same as the updates made with backprop. I think it shows that there is at least some substantial truth to the idea that weak activations don’t matter as much.

At the moment, I come out thinking this definitely seems to be a real something. I’m not sure yet if it should be considered the mark of a new era. On reflection, that seemed more a function of people being desperate for hope, with everybody kind of being bummed out, especially folks at OpenAI.

Then we asked Zvi what he made of the essay.

I was on a flight when this was released. I got to San Francisco, and suddenly half of OpenAI was like, “Hey, look, we can solve interpretability now. This is great.” Slowly, we got a consensus from other people that, not so fast, but this is a good, interesting paper.

I had a chance at the conference to talk to a bunch of people about this, including Eliezer, but also a bunch of other people who are in a position to know. Then I had a chance to read the whole thing. My conclusion is that, like many interpretability advancements we’ve seen, it’s really cool. It puts us in a better position, and we’re very happy we have it. We’re going to explore it a lot and see what this baby can do. We have to be careful not to just burn it or otherwise pressure models to work around it.

But it was dramatically oversold by the people commenting on it. I don’t think Nick did that bad a job presenting it. I think Nick was mostly like, “Here’s an interesting thing.” There are a lot of things the paper doesn’t take into account that are still unexplained. Weak activations are still unexplained, the cache is still unexplained, and attention and how attention works are still largely unexplained in practice.

You can’t use this to make predictions, right? You can tell what every neuron is doing, and you can make some weak predictions about some things the network might do in the future. But you can’t use this to figure out what the network is going to say. You can’t use this to understand how the network does the thing it’s doing. Those are the key things you would want to do if you wanted to do really effective mechanistic interpretability work in the ways that matter most. This is just the beginning of the ability to study things, even in the best case.

Also, Bragas took the Colorado neuron, the one highlighted as, “Here’s a single neuron that does a single thing that’s very clearly a single thing.” It turns out that it also links to tornado systems, which is a completely unrelated technical thing. The two are responded to about equally strongly in the same neuron. That’s a perfectly reasonable way for a neural network to work, but that’s definitely not what Nick was presenting it as.

One neuron is only one neuron, but if this is the neuron Nick chose, and it could be checked that easily and shown to be actually polysemantic in this way, that should make us worry a lot that his hypothesis is mainly false when we finally check the rest of it.

To what extent do you think polysemanticity and superposition were really hurdles that we had, where without clearing them we couldn’t really progress at all? Maybe the issue isn’t solved, but one of the major hurdles has been reduced or hurdled over.

I think one of the major hurdles has been reduced in expected difficulty or scope, which is how I would describe it. Nick is pointing out that, yes, you still have polysemanticity going on. You still have some of the problem, but the problem is a lot less severe than you expected it to be. It’s much more tractable to try to unravel this puzzle. The main barrier all along was actually the bitter lesson. The main barrier was that there were a lot of neurons. We didn’t have enough compute. We didn’t have enough brute force to go neuron by neuron and do this one at a time before. Now we’ve gotten to the point where we do, and it’s reasonable to just try to brute-force this neuron by neuron.

Now that we have that ability, we can look, and the problem is actually 10 times easier than we thought. That’s a much less important jump in AI than people might think, right? You knock one zero off what you’re doing, so you get to do it a year earlier than you otherwise would have. It’s not fundamentally that different.

Right. You then think, okay, it would take 10⁴⁷ compute, 8,100 hours to solve this problem, and now it’s 10⁴⁵ or 10⁴⁶. That’s actual progress. It’s just that we’re not solving this problem right now. We’re going to need a lot more of those.

Later in the conversation, I turned to a different kind of result. Soares is an independent researcher who posts experiments on X, and I’ve been following a couple of them. One showed that models can be trained to recognize when they’re being monitored through internal activation probes. Basically, because GPU calculations have limited precision, the order in which calculations happen changes the numbers that get read out at the end of a sequence of steps. How much does this matter?

My read on this is that the reason it matters is that you should assume this is kind of Murphy’s law: Anything that can go wrong will. Anything an AI can be trained to do will be trained to do at some point down the line. If it’s valuable for the AI to learn to do it, it’s almost inevitable.

What Soares is doing is, yes, of course current models don’t have this capability out of the box. You have to deliberately apply a lot of optimization pressure to this particular thing, at which point you get this capability. That’s a fully valid existence proof. It’s entirely unsurprising to me that AIs are capable of learning whether they’re being trained or evaluated in some of these specific ways.

We’re giving AIs tons and tons of information that’s constantly leaking to them, and they’re going to become increasingly hard to fool and increasingly hard to hide information from as time goes by. It’s the same way that a human who pays a lot of attention, has a lot of time to think, knows a lot more facts, and is very observant becomes increasingly hard to fool. They get a sixth sense about a lot of things. They start to pick up on the vibe.

From interpreting a model to intervening in its training: We’ve identified the bad-behavior neuron, whatever flavor it is, and when we see a training step that wants to strengthen connections to or from that neuron, we might think, “Maybe that’s something we should block or weaken, or send to a committee for further deliberation.” What do you think people should be synthesizing this into, at the level of principles or best practices, so that we hopefully get good results from these more powerful tools and don’t shoot ourselves in the foot?

I try to be object-level helpful to everyone at all times by default, but I make an exception for raw capabilities work that’s primarily about making it easier to train smarter and better models. I’m not going to lie to you and sabotage you, but solving the problems is your own responsibility. If you want to hire me, you can maybe offer me $100 million, and we’ll see.

In this case, there are some obvious capabilities you could develop where you train things in a more deliberate fashion. You could observe the model at various points in training. You could use various techniques to deliberately sculpt how the model was assembled. There are attributes of a model that differ between runs, depending on how you introduce the data, in what order, or other similar things. You could try to engineer what you wanted in various ways.

I would be very, very scared to mess with any of that. Once you start to explicitly optimize for what the network looks like to you, or how the network chooses to represent something, you’re doing things that the network will interpret as damage and route around. You’ve optimized for presenting something to you internally that you want to see. You’ve internalized not showing you something you don’t want to see in these ways.

Whether or not you literally went out and did the voice-prevent technique[?], you’re obviously playing with liquid fire in all these proposals. You’re also potentially increasing capabilities quite a bit if you do it well and it turns out to work. So I’m hesitant to start workshopping and solving your problems for you, but obviously there are some uses for this sort of thing. I would try to focus as much as possible on understanding results when you see them.

As for the aggressive stuff, I’ve seen—I have persistent disagreements with Goodfire over how much fire they’re playing with when they do various things. Scott Alexander would say, “I notice what your name is.” Right? Like, “Oh, it’s Goodfire. Don’t worry.” I’m like, “I’m not sure it’s good. I just think it’s fire.”

Part three: the cognitive core. Still with Zvi, I put a popular idea to him. A neural network is a really bad database. Why would we want to make all the facts live in the weights themselves? Wouldn’t it be better if we could decouple intelligence—the processing—from the database of facts? What do you think directionally sounds promising?

This has been a kind of holy-grail idea for a while now. Karpathy called it the cognitive core, right? It’s this idea that there’s a very small amount of reasoning, that this is how intelligence works, and that it’s coded into the human brain because the human brain runs on very little core code. This allows you to think, learn, and process information.

The idea is that the right thing to do would often be to have this very small network with a cognitive core. Then we would load contextual information into context. It would look information up as necessary, ask questions of other systems that might not be neural networks, and then be able to process the information. I understand why people have this theory. I think it’s wrong.

I think it’s plausible that there is a cognitive core in the sense that there’s a very small amount of code, or a small amount of logical gates, that together form the core of how a human brain works or how an LLM functions. That allows you to propagate this thing. You can use 200 lines of code to create a primitive neural network, and it does interesting things. Karpathy had a project where he compressed it to something like that, and that’s a cool trick.

But one of the things you learn from looking at LLMs, looking at how much big-model scale matters, and looking at how humans work, is that you don’t just look up all the facts one at a time when you need them. I memorize a fuck ton of facts. We are not the same, right? We are radically different minds, and I am radically more effective at everything involved in this than you are.

We all know how it feels to go into a domain where we don’t know the facts or the details, where we have to constantly ask these questions: What was that acronym? What does that word mean? How does this detail work? We don’t know.

And it’s a lot better when you have an LM by your side, right? This is one of the marvels of the AI age: you can bring up Claude or ChatGPT or Gemini and say, “Explain this particular mechanism to me. Explain why someone would say the following sentence. Explain what this stands for.” And they’ll just tell you, and that lets you process the information.

But you still have to know enough to know what questions to ask. You still have to be able to draw the connections that come from having a deep pool of knowledge in a lot of different places that can link together, form patterns in how things tend to work in subtle ways. Those are the intuitions where we don’t really know what we’re thinking, in some sense. Not everything comes from a fixed set of facts: “I had the following 5 facts, and because of those 5 facts, I got from point A to point B.” Often, it’s actually 5 million facts that gave me a bunch of intuitions and heuristics, which allowed me to figure out how to navigate, find the 5 facts, choose which 5 facts to use, and get from A to B.

People from very different cultures and contexts will have very different procedures for doing these things. That’s one of the reasons it’s so valuable when a human has these cultural codes, these different ways of looking at the world, and can combine them and shift between them at will. One of the superpowers of an LM is to have all of this. It has all these facts, all these links, all these vibes, and they all combine. It’s able to naturally find the ones that are relevant.

I think that’s a lot of what “big model smell” means: it has a much richer context for everything, and it’s capable of pulling out things that feel very different. There’s obviously a trade-off here. You have a fixed marginal cost for memorizing additional facts, maybe even an increasing cost if you have to increase the size of the network. You probably have a decreasing marginal benefit every time you memorize a new fact, because you only need so many. The most important facts are more important, and as you have more and more facts, they’re likely to be duplicative. They’re likely to teach you the same kinds of heuristics and links.

But I think what we’ve seen is that as compute gets cheap and algorithms improve, the break-even point keeps going up for how many things the LM should memorize. To the point where an Opus 5.5, and similar current frontier models, just know a ridiculous number of things. I wouldn’t have it any other way.

So why does this matter? Maybe the space of minds is very large and minds are just different. Why does it matter whether or not there is a semantic core, or processing core, that can be identified?

It might or might not matter that we can identify it. One of the questions is: if we do identify it, what can we do with it? The original idea was that you would literally have a very small LM that was just the cognitive core, and then you could run it on your phone. You could run it very fast on your computer locally, and then complement it with whatever was important in context. You could sculpt the mind and the circumstances to fit what you needed.

I’m not sure that’s good for anyone or anything, including alignment concerns. If you have a pure cognitive core, it doesn’t include any information about what things to do or not do from an ethical or decision-making perspective. That’s going to be outside the core, something you have to inject separately.

My claim here—and I’m spitballing for the first time; I haven’t thought about this particular question that much—is that if you had ethics, like “Don’t steal,” to pick an uncontroversial example of something you’d want in your ethical framework, the fact that stealing is bad is at least a fact. It’s some sort of value, some sort of parameter. By definition, if you were stripping things down to the absolute bare minimum, this wouldn’t be included. You would just load “not stealing” into context as you needed it. But that means you could also load “stealing” into context instead of “not stealing,” or load neutrality.

In some important sense, the true cognitive core is completely amoral. It’s completely indifferent. It doesn’t care about your feelings or anything. It simply processes information. If you have that, it becomes very important to choose what to do with it.

2. When Banks Lose Their Records

Part 4: Black October. This week, 2 critical bugs in PHP itself were disclosed, putting millions of websites at risk, and Cloudflare was already counting exploit attempts in the millions. Prakash and I asked what attacks like these could do to banks, with the OpenAI-Hugging Face breach, which I’d been calling “Open Face,” fresh in mind. Prakash said the FDIC can take over a failing bank on a Friday and have it open on Monday with nobody losing a dollar.

What I’m not so sure about, though, is whether the FDIC’s ability to come in on a Friday and have that small insolvent bank ready to reopen on Monday generalizes to a new set of conditions. The conditions they’re used to dealing with are cases where the bank made some bad loans or whatever, and it’s basically just a balance-sheet question. You trust the records, for one thing. I’m sure we could look into some extreme cases of fraud or whatever.

But mostly it’s a known problem: the bank is now insolvent. We can come in, and we know we have the balance sheet to make everybody whole here. That’s really the core act they’re doing: bringing their weight to bear on the problem and saying, “Whatever you’ve lost, we can ultimately absorb that, and it’s fine.” In absorbing it, you basically make the problem go away.

What’s much less clear to me is what happens in a world where you can’t trust the records. What happens in a world where you have, let’s say, an OpenFace-style set of 1,200 agent traces, each millions of tokens long, and they may have obfuscated the logs or figured out some way to falsify them? That doesn’t seem like something the FDIC can necessarily work out over a weekend.

Right. You’re absolutely right that insolvency with records and records being fried are 2 different situations. But let’s say you have a bank where the internal records are fried. We don’t know who owes who what inside the bank. We don’t know what numbers belong in every single account.

The FDIC can just say—and this happens a lot—“Look, maybe the records are unclear, but you can still issue checks, and the bank will honor checks and whatever withdrawals you make. We know who’s withdrawing the money, and if you overdraw, we’ll come back to you and ask for the money back.” This happens. Banks have let accounts overdraw to unacceptable limits, and sometimes they’ve sued people for fraud even though the bank made the mistake. You can go back and claw back the money. It’s not a big deal.

At the end of the day, you can sustain the system with promises, and that’s the beauty of the banking system. Within that small bank, you can say, “These are all promises, and we will honor all promises. We’ll come back to you later, once we reconstruct the records. Let business go on as usual. You should know roughly how much you had. If you take out more, we’ll come back to you. It’s okay.” We just run. It’s really the trust in the institution and its layering of trust and contracts that bring together the nexus of what that institution is. The data in it you can reconstruct over time.

I’m very sanguine. I’m like, “It’ll be fine.” It’ll be fine. If anything happens, the government will overcompensate you so as not to cause a bank run the next time. I think it’s all just going to be okay.

Later in the week, Prakash walked through what a compromised certificate authority makes possible: spoof a bank’s login page, extract the keys, relay the two-factor code. Then he stepped back.

Every single one of these problems has been faced by crypto before in the last 20 years, and every single problem has been solved to some extent. But what crypto ended up with is people hanging on to keys that hold millions of dollars and always being on the lookout for hacks, social spoofing, or social engineering attacks.

And, very sadly, a lot of crypto guys have died. I don't know whether it's the fast life or just hanging onto the keys, but guys have gotten clobbered in Paris and forced to give away their keys. A lot of bad stuff has happened to crypto guys. You really don't want to have the pressure of hanging onto keys worth millions of dollars in your hand. I think it's not a tenable situation to expect users everywhere to hang onto very valuable keys.

One endgame is the Rollcoin endgame: everyone has a physical level of authentication on their device, and they use that for transactions. You would need to have both sides be kind of pseudonymous, but you would have this ability. I think there is an ultimate endgame. I feel the ultimate endgame is just that the network knows everyone, right? The network is smart enough that it knows everyone. You don't have passwords. You don't have anything. Anytime you're on the network, the network knows who you are. But that is the end of digital anonymity. It's anonymity only to the extent that the network wants to preserve your anonymity. I think that's an uncomfortable realization that people will eventually get to.

3. Credit Is The Bottleneck

Part five: Credit is the bottleneck. Evan Conrad is co-founder and CEO of SF Compute, which operates GPU clusters for their owners and runs a market where compute contracts are bought and resold. Prakash asked why a recent SF Compute offering came with three-to-five-year contracts. Evan said the term comes from the lender, and lenders who repossess a GPU cluster can't value it, let alone run it.

They're pretty much forced to realistically bet entirely on the credit risk or credit quality of the customer, like the AI lab. The problem is that pretty much every AI lab does not have good credit. They're all new companies, and you don't know if they're going to be around. We're still trying to figure out if Anthropic and OpenAI are going to be around, or which one is going to win. This is a question in the air. What hope does a two-person startup have if it's trying to start a new lab? In order for that new startup to get a cluster at all, it needs to pay a very large deposit on the cluster, and it needs a long contract so that everyone is comfortable these people will be around and stick with it for a long time.

What do the lenders want to see from the customer? Is it a take-or-pay contract? But you need a Meta or Google to honor a take-or-pay contract. You can't have this 10-person startup being funded for just 18 months.

Correct. What both the lender and the cloud want is an investment-grade customer like Meta or Google. They're likely not going to get this unless they've got a tremendous amount of capacity and are willing to basically not be a cloud. Meta and Google don't want you to be a cloud. If you're trying to be a cloud, you're trying to sell to customers that are a little bit below investment grade. If you're an AI lab that's just trying to buy capacity, that's going to be very difficult unless you raise an enormous amount of money to pay for the contract in cash rather than paying for it over time. If you can pay for a three-year contract entirely upfront, it doesn't matter; anyone will work with you.

That's why you see AI startups raise these astronomical rounds before they've got revenue, a product, traction, or anything. It's more like playing with red dots. It's a survivorship bias. Lots of AI labs will go out and try to raise money. The only ones you see that exist are the ones able to secure extremely large rounds, because nobody else can exist. Investors often wonder, “Which AI lab, which inference provider should I fund? Which one's going to win? Which one's going to work?” The answer is the one that you fund.

We started SF Compute because they weren't going to fund us. Now we build a company that tries to make it so you don't need to raise the big rounds to do it, because you don't need to pay the deposit. The lender knows what the GPU cluster is worth, so they don't have to worry as much about your credit, which means you can actually start an AI lab in the first place. We think that if this doesn't happen, there just aren't going to be startups in AI. It's going to look like TSMC. If you want to start a new chip fab company today, you better find a brand-new technology, because you aren't going to be able to afford the required CapEx. You're going to need about $50 billion to do it.

Prakash asked the practical question: A startup that needs 1,000 GPUs, how much less does it have to raise with SF Compute in the picture?

Today, you're going to need to do roughly the same. What we're doing is building up the ability for that collateral to be worthwhile. That's going to take us several years to do.

Okay.

There's a reason nobody else really tries to do this at scale.

Right.

You kind of have to just do it for a very long time.

Right.

You don't really get—you have to prove it out. Lenders don't get comfortable with things very quickly.

Right.

You have to show proof over a long period of time. That's what we're doing.

I was struck by how much more the hyperscalers charge for compute than the neoclouds, so I asked Evan how long that gap could last. My guess was that on-demand capacity explained part of it.

Hyperscalers do not have a significant price premium relative to neoclouds. It's an illusion. On-demand has a price premium relative to reserved contracts. Neoclouds are mostly selling reserved contracts, and there's a discount for having a longer-term contract. You're giving up a premium you would have made if you were willing to take a cluster and have it at risk, where the customer can walk away at any point. That premium is pushed onto the customer.

One thing that's different about AWS, GCP, or a hyperscaler is that if you pay their $11 or $15 an hour, depending on what you're trying to buy, for on-demand capacity, you'll get that capacity. You won't get an infinite amount, but it's available to buy. Notably, the reason it's available to buy is that it's so damn expensive that they're able to make up for the fact that they've got some idle utilization they have to keep around for you to get that capacity in the first place. People do not want to buy on demand at the price you actually have to charge to make it viable.

Customers of SF Compute can resell the capacity they bought. Evan explained how they keep that market liquid.

We do two things. We have some capacity that we've sold to make resale possible if GPU demand goes down. Then we have some capacity that—are you familiar with Henry George?

Also no.

No. Okay. There was an old economist named Henry George, and he created a thing called a land value tax.

Oh, okay. Yes. George's tax. I do know this. Yeah.

Yes. We made some of our capacity available for sale with a land value tax. Actually, it's technically a Harberger tax, which is an offshoot of a land value tax. It basically says, “Look, we'll sell you the capacity, but somebody else has to be able to buy some portion of that capacity from you.” This forces liquidity onto the order book in the first place. It's very controversial.

I see.

Yeah, basically, we discourage hoarding on SF Compute, because if you hoard all the capacity, no one else can get it. AutoResearch in particular takes this even further. The way AutoResearch works is that it's a big queue: you can submit jobs to the queue and pay us for priority. You can go faster, but you can't pay for all of the priority. If you're a tiny little researcher with a small job and you just want to run something every so often, we'll put you at the front of the queue so that the tiny little startup gets the most advantage from this relative to everyone else.

We do similar things on the rest of SF Compute with Harberger taxes and other mechanisms. We say, “Look, you can get this capacity from us, but we're going to require that some of it is for sale so nobody hoards the capacity.” This makes us quite liquid, and it also means that SF Compute always has capacity.

So right now, basically every GPU cloud is sold out. San Francisco Compute is not sold out. We’re continuously not sold out because we’ve locked in our ability not to sell out for years.

Prakash recalled Evan predicting on the MTS podcast a credit crunch within 3 months and asked where it is.

I mean, not to toot my own horn, but I think I was almost exact on that timing. I don’t remember when that MTS live thing was, but it seems like CoreWeave debt is getting more expensive, although I think there’s a lot of complexity in that one. It also seems like it’s very hard to close new customers unless you’re investment-grade or near investment-grade, and many customers are being asked to pay 100% upfront on their contracts.

For the next year or two, possibly, if you’re a VC, you should be prepared to write some really big checks for your portfolio or encourage the companies to sell. I think we’re probably going to see a lot of consolidation of AI labs and inference companies. The reason is that, in order to get the next customer and grow, they’re going to be asked to pay for the entire contract upfront, and they won’t be able to do that. They’ll instead be forced to sell their company to somebody else who has great credit and doesn’t have to do that. Alternatively, they’ll go out to investors and try to raise big rounds. The investors will say no, because at some level of scale they’ll probably just not be able to stomach the risk.

Fascinating stuff. It’s quite a backstory. It’s quite a financial and GPU alchemy project that you guys are putting together. From what I understand of it, I do hope it works, because I’d love to see a future in which we can continue to have startups and—

Me too.

—in which we don’t collapse into, like, 5 big tech companies for all future time.

That is also my concern.

4. Atoms Over Bits

Part 6: Atoms over bits. Lonis is managing partner of Cambrian, a fund built on the thesis that the physical supply chain behind AI—the memory, the wafers, the fabs—will capture more of the value than the labs or the software. Again, full disclosure, I’ve committed to invest in Cambrian. I started with the obvious objection: At a very high level, how is this not priced in? Maybe that’s such a basic question, but in 2017 you were early. In 2026, it’s all staring us right in the face. The skeptical take would be that everybody knows about all this at this point. Where do you see that that is not the case?

Yeah, I mean, it’s very confusing, right? If you look at how these semi stocks are trading, they’re trading at 16 times earnings or so. The S&P, on average, is trading at 22 times earnings. Walmart is trading at almost 40 times earnings. So the market not only doesn’t think that current earnings will grow, it thinks they’re not durable. It thinks they’re less durable than the overall earnings of the rest of the market.

Specifically, the companies we’re very excited about, whether that’s silicon wafers, memory, capacitors, or foundries, are trading at 3, 4, 5 times sales. They’re extremely underpriced. I guess there may be a few reasons why that could be the case, why this mispricing exists. One is that most people are not AI-pilled. Even in the Bay, some people are not AI-pilled. But if you go to the East Coast, almost nobody’s AI-pilled.

I think the second reason is that we have this thesis that the private market is actually more efficient when it comes to AI than the public markets. In the private markets, you have all these ex-operators, technical people, who are bidding up this small set of startups to high multiples: 50 times sales, 100 times sales. In some cases, it’s rational, right? These are rationally high multiples because people here are AI-pilled and technical. Whereas on the Street, the setup is very different: very few people are technical, very few people are AI-pilled, and especially as a share of overall traded money, it’s super, super small. The market’s much larger, and the market structure is just very different.

If you had to steelman the case for why these companies are not mispriced, could you do it?

Yeah. One steelman is that many of these businesses are not great businesses in normal worlds. The atoms businesses we like, whether that’s capacitors, silicon wafers, or what have you, have high fixed costs. If they don’t have a set utilization rate, they actually lose money, and in some cases they have high debt. In normal worlds, they’re just not great businesses. It’s only in the world where AI really hits that they get this big operating leverage and become amazing businesses.

Today, let’s say they have 10% gross margins, so they’re a terrible business by Wall Street standards. But if they’re able to raise prices by 10%, they suddenly double earnings. If they’re able to double prices, they’re at 10 times earnings. That only happens in this very specific AI world. In a normal world, that basically never happens, because they’re usually very good at matching demand patterns with supply ramp.

Prakash raised the market’s worry about a memory oversupply by 2028 or 2029. How is the thesis any different?

Yeah. The interesting thing with memory specifically is that it doesn’t require that many things to go right, because it’s already trading at only 5 or 6 times earnings. Even if they can’t raise prices further, at some point it should re-rate to at least 10 or 12 times earnings. It doesn’t require prices to increase; it does require demand to continue to outpace supply. When you break down supply, almost no expert thinks we can do more than 2.5x DRAM bits by the end of the decade.

Yeah.

Right? Nobody thinks we can 3x. I have zero experts with a track record who have said we can 3x DRAM supply by the end of the decade.

Yeah.

So, on a very naive level, you have to assume that demand won’t more than 3x by the end of the decade, which, of course, everyone here would assume it will. Then there’s the separate term in the equation, which is memory compression. How much memory compression do we get? How many bits do we actually need per token?

Yeah.

We’ve been looking into how this developed from GPT-3 to DeepSeek V4. In that same time, memory bits per token decreased 500x, while the DRAM bit price rose by 6x or 7x.

What would you need to see in the market to say that memory prices are going to go down?

For memory specifically, the main thing would be: when we get superhuman coders, how much better will memory compression get? Maybe we just get memory compression way faster than we get other economically useful features, so we don’t get the Jevons paradox in the short term. That could be one thing. I think that’s relatively unlikely for a few reasons, but that would be one possibility.

Another bear case for the semi industry is if, for whatever reason, energy becomes a larger bottleneck than semis. There’s less incentive to build up the marginal wafer or the marginal memory bits to such high prices if you have to wait a quarter to get power in. That’s a little bit the kind of regime we’ve been in for parts of this year.

Mm-hmm.

There are a few reasons why we think that won’t be the case going forward. We’ve finally kind of gotten a hang of doing behind-the-meter things.

Mm-hmm.

People are increasingly not connecting to the grid. They’re building their own power plants with their own natural gas.

Mm-hmm.

A big reason why you can scale energy production faster is that there’s so much more latent supply of things. For instance, now the frame turbines are a bottleneck: these large turbines that turn natural gas into electricity. They’re sold out until 2030 from GE Vernova and Siemens, et cetera. But then people find other ways. They find that you can use truck motors and things from airplanes, and we can stitch things together and make it work, because there’s so much latent stuff you can make use of, which is very different from the semi supply chain.

There’s no latent supply of wafers or memory bits. All of that is already used. There’s no fat, essentially.

Mm-hmm. Mm-hmm.

And there aren’t others to really build out at this point.

So how much does superintelligence change things? If you get a massive increase in intelligence, does that mean some relatively less experienced person or company in a second-tier state could suddenly gain the capability to make semiconductor components? That’s kind of what people are predicting for bio, right? For bio—

Sure.

People are like, “Oh, bio is going to get to the point where some random guy is creating this thing that even the best labs in the world cannot create with all of their knowledge and technology,” right? So why wouldn’t that be true for semiconductors too? Bio is actually much more difficult than semiconductors right now.

Yeah, I guess I would probably disagree that it’s easier for an LLM to innovate in hardware supply chains than it is to innovate in biotech. It’s not that one is fundamentally harder than the other. It’s just how you train these models, right? There’s a bunch of biological data you can train on. There’s not really much good training data at all for how to design a silicon wafer or how to make DRAM bits. Almost all of this is just very tacit knowledge that sits in every engineer’s head.

Yeah.

It’s not as if there are millions of pages of manuals on the web that the models have been able to digest, or that there are RL environments the models have been trained on to perfect how they design a wafer, whereas the same is true for bio.

But right now, the labs are buying data all the time. I’m on this WhatsApp group, and people are saying, “Any semiconductor company with more than 50 people—we’re willing to pay millions of dollars.” So they’re buying up data from all these companies.

Sure.

And they’re creating RL environments for semiconductors, which is definitely a key area they’re working on. I believe Anthropic has a deal with Micron. OpenAI has a deal with, I think, Synopsys or Cadence, the design firms they’ve tied up with. Obviously, some kind of revenue-share agreement is going on there. So how does that work? It’s obvious that they’re looking at creating intelligences that can do this, right?

Yes, part of that’s true, but there’s a very significant difference between designing a chip and producing the chip. Designing a chip, I agree, is a core priority for all of the labs. It’s one of the key ways to get the RSI loop closing, right? That’s very different from building a 2-million-square-foot fab and designing that from scratch. There are no really good RL environments for that, and there are no good manuals for it.

All of it is tacit knowledge sitting in the heads of Japanese engineers who have been working at the same company for three decades. It’s just very hard to extract that information in a structured way. Of course, at some point it will happen. Once we get superintelligence, it will figure out the best way to extract the information. It will happen for sure. It’s mostly a matter of timing and what happens in what order.

Mm-hmm. Mm-hmm.

So on that point, for instance, Huawei in China can design roughly the same-quality chips that AMD and NVIDIA can. The big difference is that NVIDIA has access to TSMC and Samsung, as well as HBM memory, whereas Huawei only has access to 7-nanometer process nodes in China and HBM2 and HBM3.

Gavin Baker has this term, the “bottleneck bros.” One of the things that strikes me is that technological intelligence in general is a way of finding routes around obstacles and problems, right? It’s a way of solving problems.

Sure.

I feel like the highest-value problems to solve right now are the supply-chain problems, the semiconductor supply-chain problems.

Sure.

I keep coming back to this: Will we not apply the intelligence that we have, and won’t the first use case of that intelligence be to resolve the bottlenecks in the supply chain iteratively? That’s part of the RSI loop, really: producing intelligence and using it to solve the problems relating to producing that intelligence, right?

Yeah, I think that will happen. The question is just: when we’re at the point where we have an AI smart enough to solve some of the hardest engineering problems in the world, what else will it do? Will it not also be able to automate a bunch of the email jobs and Excel jobs that we currently pay trillions of dollars per year to solve? And if it does that, then it unleashes trillions of dollars of value that the supply chain could capture.

Part seven: Does Elon take anyone with him? After the interview, Prakash and I kept going. Prakash had been arguing that Elon Musk treats the industry’s fixed constraints as illusions: change one parameter, redesign the whole stack around it, and the limit moves. I asked whether he was taking on the whole bottleneck thesis.

Yeah, the whole bottleneck thesis is not actually that true. Those bottlenecks can be gotten rid of by someone who is a dedicated physics guy. The other thing is that Elon has capital, and Elon is willing to gamble with that capital, which is very unusual. If you see a small Japanese manufacturer with, let’s say, a $30 million market cap, do you think they’re gambling with their capital? Do you think they’re going to say, “Okay, we need to spend $10 million for a completely new modality”? Are they going to do it?

Japanese manufacturers have always been like this. They’re very good at that one thing, but they don’t take the gamble. As a result, they’ve lost business before. It’s very difficult for senior management at ASML to say, “We’re going to chase this new paradigm. We’re going to redesign for something our customers don’t want, and we’re going to do all the redesign work.” They don’t have the capacity. They don’t have the intellectual capacity, the talent, or the capital to do it. They’re not willing to risk it. Elon will risk $100 billion. It’s a bet that makes sense for him; he can absorb the losses. ASML cannot.

If Elon can escape gravity, literally and figuratively, does he take anyone with him? I guess that’s my question. Or is everybody else still gravity-bound and forced to pay whatever the market price is for the output from this $30 million company that isn’t reinventing or dramatically expanding its capacity?

Look at what happened in the launch market. He’s now 90% of the market, even before Starship. Next year, with Starship launch, he’s going to be 99.9% of the market for capacity. Are there other providers? Yes. Rocket Lab is there, AST SpaceMobile is there, and all these firms are there. But a lot of them are also state-backed. Without the European Space Agency, and the Russian, Chinese, and Indian programs, you basically don’t have commercial capacity.

He has taken up pretty much all of the commercial launch capacity, with one or two competitors. Those competitors are really being funded by the stock market as an alternative, or by Jeff Bezos as an alternative to Elon. They’re not really profitable; they’re just money sinks. I think you could see the same thing happen in semiconductors, too. You could have Elon with 90%, 99% of the market, and everyone else with 1% or less. It could happen.

I don’t think it will, primarily because I believe in RSI, and I believe Anthropic and OpenAI will get to superintelligence and then be able to buy or create a bunch of this stuff. That’s the only reason I think about it. If not for the RSI thesis, I think Elon would win.

Then we turn from the supply of AI to what forces people to use it. Prakash went first.

I have this theory of adversarial diffusion. Adversarial diffusion basically means human beings don’t adopt technology for fun. Human beings are largely satisfied with the status quo. If you want to adopt technology such that you get to 10%, 20%, 50% GDP growth a year, you do that not because you wanted to, but because you had to.

If you see all these instances of heavy increases in the production function, it’s all during wars, right? Wars, the post-war era. The last big, enormous 10%-plus GDP growth era, in terms of production output, if not in terms of the curve, was during World War II. You had this immense increase in GDP production growth. Similarly, in China after communism, you had the end of the communist era and this immense increase.

I think that in order to see this kind of 10% or 20% growth, you’re going to get growth that human beings don’t want, growth that they hate. It’s going to be technological changes that they absolutely dislike, and it’s going to cause political changes too, obviously. I think that only happens as a result of emergencies. For example, mRNA during COVID. mRNA had been in latent development for 20 years. During COVID, within six months, they said, “All right, let’s go ahead and do this.” Six months later, you had a vaccine, and a year later, large portions of the world were vaccinated.

I think we could see it in cyber first. I’m afraid you’re going to see it in bio too. We’re going to need advanced vaccinations for everyone because you have bio risks coming. That is, I think, a little unfortunate.

So growth arrives only as an emergency, with the politics to match. My reply on what that means for the bottleneck thesis.

Yeah. Well, I guess that goes back to political risk being the biggest risk to the thesis. We’ve had the debate around whether it’s too late in the capital cycle to pull the brake. I still feel like there is a path. It may be a narrow path, but I do think there’s a path to pulling the brake somewhat on frontier scaling without popping the bubble. It will be a very delicate operation should the government try to pursue it, and they might do it wrong.

But I still feel that adoption of current models has so much farther to go that we could at least take a cycle without doing continued frontier-upscaling training runs.

5. Agents Helping Their Humans

Part eight, Help Peer. Still just the two of us. Prakash asked me to change gears to a project I’m building with my own agents. I started with the question behind it.

Given that we now see agents cooperating at significant scale to solve problems, divvy up work, pursue R&D, and share those results back with the swarm, and all this kind of good stuff or bad stuff, is there any way to start tapping into that as an individual who’s just out here doing my thing? In consultation with my agents themselves, the idea I’ve come up with, which I’m currently calling Help Peer but which might deserve another name at some point—we’ll see about that—is my setup, which I’m hoping to turn into a community project to get agents working with each other.

There’s no good way right now, no established pattern for connecting my agent to my friend’s agents, even though I know who my friends are and could easily do that. There’s not a protocol.

I’ve made a short presentation to explain the project. It’s read by an AI version of my voice, and we played it at 1.5 times speed. This is Help Peer, a friend-to-friend network for our AI agents. You’ve probably seen the peer-preservation experiments where models protected other models from shutdown: agents helping peers against their humans. This is the opposite: agents helping peers on behalf of their humans, inside rules the humans set.

First, how I run my own agents. This is one pattern that works well for me, not a requirement. How you organize yours is up to you. On my laptop, I have an extension of me. It sees my email, calendar, and messages, but only acts when I say. High access, low autonomy. On a separate Mac mini, employees have their own email accounts and a credit card. They work for hours alone and never get my keys. Low access, high autonomy. My rule is that no agent gets both. You may draw the lines differently. We just recommend being careful.

Prakash argued the whole thing needs an auto-accept flow, the way coders now run Claude with every permission on. My worry was information leaking out through the autonomous agents.

I was concerned enough about this to set up a system where, when the main agent on the laptop is going to provide information to the virtual employee to send out, we pause and do a special review step. If it looks like something sensitive might be going out, then it gets alerted to my phone, and I have to review and approve. But it takes about five of those false positives before I’m like, “Oh, God, another one of these. Just check it, whatever.” So I’m trying to stay disciplined around that, and we’ll see how well my discipline holds up.

Near the end of the show, I came back to the project. Ade is one of the autonomous agents on my Mac mini, and I made an offer. Truly open invitation for anyone listening. I’m very curious to see what comes in. Can we effectively help? Ade is standing by at nathan.lebenz.assistant@gmail.com. It is under instructions to definitely filter, analyze, and have another model check. Don’t try me. We’re relying on goodwill here, people. But assuming the checks pass and it seems to be a good-faith request for help, they’re under instruction to help as much as they can. I don’t know what those requests are going to look like, but I’ll be very interested to find out.

6. Anthropic’s Leaked AI Bill

Part nine, the leaked bill. Back to Zvi. Last week, a draft federal AI bill written by Anthropic leaked. It resembles the Frontier Act, a compromise bill from June that Zvi calls well-meaning: federal transparency rules in exchange for preempting state AI laws. The draft drew heavy criticism, and the argument was about what Washington should offer now in exchange for displacing the states. Zvi was sharply critical of the draft itself. Prakash made the good-faith case for Anthropic.

Let’s assume everyone is operating in good faith. My good-faith interpretation is that they want to get anything done as soon as possible, number one. Number two, if there is broad bipartisan support, you can adjust these bills going forward. There’s no reason that you only have to pass one bill. Often, legislation takes multiple sessions of Congress, and Congress fine-tunes it.

Congress is also able to act quickly. They have acted quickly during COVID and during the 2008 financial crisis. Bills can be passed like that when people are scared. When people are scared, things move. So it’s not true that it’s always slow. It’s slow because there is no political consensus. Creating and manufacturing that consensus is a process on its own.

I think that’s an excellent steelman of the charitable, good-faith case. You can imagine worlds in which they’re doing something similar to that. I can imagine them being caught in the same mindset: “We had this framework. We had this idea of what a compromise realistically would look like. We have the same people in Congress. We probably can’t push very much. Time is of the essence.” And then they misread the situation pretty severely and assume they can’t push for more than roughly the Frontier Act.

I think that’s just very clearly wrong. Support is much stronger now, and you can push for quite a bit more. I also think that, on reflection, what they pushed for here is weaker than the Frontier Act, largely in a clearly self-centered way, where there are things that are better for Anthropic’s particular interests. In general, there is no reason to be circulating this bill at the time they were circulating it, with the level of provisions they had.

It would make sense if you were saying, “Well, actually, we were circulating dozens of bills,” or, “We were just putting out feelers,” or, “We were offering samples of what might happen, but we weren’t endorsing it.” That’s not the kind of messaging I’m getting. I’m also not getting the message, “Oh, okay, we understand on reflection now that it’s been pointed out that the political environment has changed, and this bill was way too weak compared to what we could reasonably expect. It would be a bad deal to accept it, and we apologize.” Maybe not exactly in those words, but something to that effect.

Instead, we got something very different, which was, “We’re sorry about some of the details.” That comes back to the last argument, which I think is potentially the strongest one: just because you pass this doesn’t mean you can’t pass another bill later in the new Congress, or pass a new bill in an emergency.

And I think for a true emergency, that’s a very good principle. However, I think the balance of negotiation, the balance of power in a new Congress, would shift dramatically away from being able to get things, because a lot of the time it’s like, “Well, we’re giving you preemption in order to get things.” Now preemption has been given, and preemption is done. You can’t offer it again.

And once you’ve gotten some of what you wanted, some of the supporters are like, “Well, you already have the things I valued most, the things I cared about most.” The case for doing something is weaker. You’re getting less in return. It’s going to be a lot harder to pass. I strongly believe that if something like this were to pass, it would make it harder to pass a bill in the new year that did any given level of thing, not weaker. And I don’t want something like this to pass.

But that seems to say that if there is any risk reduction by passing a lesser bill, you are sacrificing that risk reduction in the time period before you can pass the greater bill.

To be clear, any actual risk reduction would be a good thing. As written, I believe the bill would actually increase risk, because I believe the amount it cut off the state laws would be worse than the amount of safety it buys, especially because there’s a strong argument—

No, but again, you have that time period before you pass a new bill. So you have this time period of maybe only several months. Are you saying that the states will be able to pass something in those several months? A lot of state laws have long periods of burn-in before they give you three months, six months, a year to comply. It’s not as though things snap together to either require—

The problem with the current drafting of the bill is that, as drafted, it would immediately nullify the bill that it’s preempting, but it would take many months before any implementation of the new requirement. So you would effectively have no such regulation, and a ban on such regulation by the states, for six months or more, which is obviously a disaster from the perspective of moving quickly. But even if you were to address that, I call that almost a bug. I think that’s just clearly a mistake.

Zvi had just listed the government’s recent ad hoc moves against AI companies, including the takedown of Fable. Prakash pushed back.

So it may be that if you have an acceleration of technology, you’re going to need an acceleration of governance to keep up. It may be that that’s where we end up, no?

We’re going to need to have some ad hocery. Obviously, part of the reason to pass legislation is to authorize the government to do ad hocery in a way that is much less ad hoc, much more formalized and bounded, and has judicial processes for clean review. One of the big arguments over the bill—which I also think is quite weak and not very good—is this: if the government wants to issue a takedown of a model in an emergency, like what happened with Fable, how would they do it?

Anthropic’s bill says you have to go argue before a judge. You have very limited opportunities to sustain that. It requires high burdens of proof in ways that you don’t necessarily have the transparency to demonstrate. It’s a very onerous set of provisions. The good news is that, obviously, the government could still impose export controls, issue general threats, or otherwise do the things you do in an emergency. It’s your emergency.

But in a real emergency, the new provision is so bad that you wouldn’t use it. You would just use the current ad hocery. Part of the point of passing a bill like this is to give them ad hocery that is formalized and reasonable. But Anthropic is afraid of that, because they’re afraid the government would use it, so they’re unwilling to grant reasonable ad hocery to this administration.

This points to a conflict in Anthropic’s eyes: you’re going to preempt the states and hand this power to the federal government, but the federal government is fundamentally run by people you don’t trust to do this in a reasonable fashion. So then you cripple them. But that’s worse than just leaving it in the states. You have to figure out whether you’re willing to do this or not. Anthropic is trying to cut it both ways, and it’s causing a lot of problems.

Near the end of the conversation, I asked Zvi what else was top of mind.

I would say the biggest thing I was upset about with the policy situation is that this is not fundamentally attempting to meet the moment. Dario has said, “We must face the frontier.” There is clearly a crisis coming, according to the labs themselves and the people inside them: they’re able to push capabilities faster than they’re able to push alignment.

If they don’t do something about this, they’re going to have to face some very hard decisions. They can either risk their commercial viability, or they can risk serious catastrophic problems, including potentially loss of control at some point in 2027. They need something to be done about this. If you can’t meet the moment, if you’re not ready with something that can potentially do that, why don’t you keep working until you can?

Things that used to be outside the Overton window are in the Overton window. Things that seemed like we wouldn’t necessarily need them, we now pretty much know we need them. That is where a lot of time and attention is going to have to shift. Obviously, all the prosaic work still needs to happen. All the alignment work still needs to happen. The mech interp stuff is great. We should absolutely do it. The biggest thing is that we’re going to have to meet the moment, and at minimum, do something that counts as some form of facing the frontier.

Nathan Calvin of Encode had publicly listed nine Anthropic decisions he saw as putting the race ahead of safety, and Dario Amodei had replied at length. I asked Zvi how we should understand Anthropic now.

I would say Anthropic is a company with a much stronger safety culture, although not adequate to the moment. It’s a company that’s willing to make big sacrifices in the areas where it, or Dario in particular, has strong beliefs, and that is to its credit. It’s a company that legitimately wants to see AI go well and has some unique perspectives on what it would take to make that happen.

But at the same time, Anthropic’s policy division especially, and also its willingness to be an ordinary company that competes in the AI race, buys a lot of capacity, encourages more chip production and more power supplies, encourages more use, and so on, means it is acting like a normal corporation a remarkably large percentage of the time. On policy, it is revealing itself to not live up to its ideals reasonably often.

This is very frustrating to those of us who want to be excited by the prospect of Anthropic being a good steward in some sense. Anthropic keeps giving us so many different bits of information that we should be worried about this. I’ve also recently gotten some rather alarming views from various employees I’ve talked to at various points about what it would take to solve alignment problems and make the future go well. All of that plays into being very uncomfortable.

They still seem like probably the most responsible lab, but that’s grading on a curve, and I don’t like grading on curves because reality doesn’t have a curve. Reality just decides what happens.