20VC：“反数据中心是中国的心理战”｜规划中的数据中心究竟有多少能建成？｜能源是AI最大的瓶颈吗？——Positron联合创始人 Thomas Sohmers

I would say I'm overall opposed to the pace of the frontier direction it's going in. The scariest thing to me on the political spectrum, and in the way all of this is being treated, is that it's now become an almost unifying issue on the left and right: being anti-data centers. I think that is almost entirely a Chinese psyop. A single In-N-Out uses more water than the largest data centers in the United States.

We're probably at one of the most important times in history for technology. We have the biggest model providers saying we need to pace the frontier. But what does that actually mean in reality? How possible is it? What does it mean for the threat from China? What does it mean for the infrastructure layer moving forward?

We have a true expert in the space on the show today in the form of Thomas Sohmers. He's the co-founder and chairman of Positron AI. They just raised an $875 million Series C at a $5 billion valuation. They've got some of the best investors in the business, including the one and only Gavin Baker at Atreides, and many more great names.

Thomas did not hold back in this episode. It's a beautiful combination of incredible education on the infrastructure that powers this economy for AI, and then also—I don't know how to say it, but “analytical gossip” would be a more intellectual way of saying it—incredible discussion about what we can expect in the next few months from the biggest players in this space.

Thomas, I am so excited for this, dude. I said to you just before this that I think there are some big questions that the world doesn't know, or thinks that it knows, that I think we're going to correct today. So thank you so much for joining me.

Yeah, great to be here. Thanks, Harry.

Can we just have a brief description of what Positron is, and where does it sit in the stack?

1. Positron Builds Inference Hardware

Yeah. Positron's a fabulous semiconductor startup that's building hardware. So really, everything from the chips and the software directly running on top of that, all the way up to the full systems and rack-scale deployments to power generative AI inference.

Effectively, it's everything in the hardware and low-level, directly talking-to-hardware software stack that powers all of the applications everyone in the world is excited about right now. So, everything from the likes of ChatGPT, Claude, et cetera.

How does the infrastructure stack required for inference, what you're working on, change compared to training?

So training, I would say, from the underlying compute level, fundamentally is a compute-bound problem. It's a workload where the more FLOPs you have, the better. Regulatory and export-control frameworks are heavily focused on the FLOPs required—how many floating-point operations per second can be done.

More or less, the amazing thing the scaling laws of the past decade have shown is that the more parameters you add to a network and the more FLOPs you dedicate to it during training, the better that model is going to become.

2. Inference Hits The Memory Wall

The big difference with inference—the deployment of those models—is that the actual math and steps you're doing amount to about half of what you're doing during training in terms of the steps. That shouldn't be thought of as the actual compute involved, though. What it turns out to be is that the forward pass, that inference portion of it, is heavily, heavily memory-bound.

For every single token that's generated, every little bit of output, you have to go through the weights and the parameters. You could think of it, from a biological perspective, as the neurons: you have to read the values of those for every single individual token.

I don't know if it says anything about the value, or if it's actually saying that inference is somehow—I don't want to say more important, because of course you have to train—but fundamentally, when you're training, you already have the corpus. You already have all the training data. With all that data, you can massively parallelize the token inputs—all of the sequences of words, sentences, paragraphs, et cetera, that are going into it. So that's something you can just crush through with a bunch of compute.

But when you're inferring, because that's actually generative, you don't know what the token is 5 words down the line. You have to generate each and every one autoregressively, or in order, without foresight. So that becomes a hugely memory-bound problem that can't just be massively parallelized like training.

So I totally get that in terms of the shift from compute-bound to memory-bound. Is that what people mean when they talk about the memory wall with regard to what you're doing?

Partially. The memory wall, as a phrase, has been around for a long time, before all the hype around AI. Really, what it comes down to is that if you look at the past 50 or 60 years of computing, we've been able to have Moore's law give us more transistors per square millimeter of silicon consistently.

While that has resulted in greater raw compute—FLOPs, et cetera—the improvement of memory technology has not kept up at the same rate. Roughly speaking, between 2014, the very early innings of the new AI era, and 2024, you had about a 120× improvement in the FLOPs of GPUs. A single NVIDIA GPU had about a 120-fold improvement in FLOPs, and that's what enabled a whole lot of the improvements over that decade.

The improvement in memory bandwidth was only 17×. I would say the real embodiment of this is that we had massive improvements on a per-device basis in FLOPs, and then a whole bunch of elements on the periphery improving connectivity, et cetera. But the ratio of compute to memory bandwidth had this divergence.

So you had cases where, if a problem was memory-bound and you couldn't just scale the compute linearly with it, you were getting more and more memory-bound as the decade progressed.

Why was there such a misalignment in the progression between the two? One's 100× and one's 17×. Why is that the case?

Yeah. Well, it comes down to a lot of technical implementation details, like the fact that if you look at the lowest level, the type of memory used on the silicon itself is called SRAM, or static RAM. SRAM is made out of 6 transistors, with a bit line and word line and some other control logic around it.

But that SRAM cell has not scaled in terms of its sizing with Moore's law over the past 15 years. So they have grown—or shrunk, I should say—much more slowly than just a group of transistors that you'll use for other purposes.

I would say that there have been a lot more architectural advancements on the compute side, while a 6T SRAM more or less has not changed in 30 or 40 years from an architectural primitive perspective. That is on the input side: raw technical capabilities in fabrication, et cetera, have not been able to improve. But I would also say that there were not the right motivations for most of that decade.

With convolutional neural networks, the things that powered AlexNet, which really launched the deep learning revolution in 2012, and then ResNet and all of the advancements during the 2010s, we were in the realm of machine learning models that were fundamentally compute-bound. You could just throw more and more FLOPs at CNNs and get better results, and you did not really need all that much memory capacity or memory bandwidth.

But with the transformer—even though the Attention Is All You Need paper came out in 2017—I would say it did not really get the attention, pun intended—

Sure.

—it deserved until 2020. GPT-1 and GPT-2 came out prior to that, in 2018 and 2019, but it was really GPT-3 showing that you could go from roughly 1 billion parameters up to 175 billion parameters and actually get this massive improvement in capability. That is really where I would say the transformer revolution started, and most people did not catch on to that until the end of 2022, when ChatGPT came out.

When you look at token economics and token efficiency today, what does no one know or talk about that you think should be much more front and center?

3. Cached Tokens Drive Huge Margins

Compared to a year or two ago, there are now different prices listed for cached versus uncached tokens. But I do not think people realize how many providers that charge the same amount, even with a lot of people's cached prices, have margins that high. It is insane. You make all of your money on selling cached input and output tokens.

Why is that? Sorry, just so I understand.

When we discussed earlier that processing a cached token is essentially free, it is one-thousandth of the cost, on the order of magnitude, of actually having to recompute and generate that token. There is so much you can juice out of selling those cached tokens.

Basically, all the providers charge you to cache a token. They charge a higher rate than the normal processing fee for an input token, and then they charge you a lower rate when you read from that. It is great when you are paying that lower rate, but they are making an obscene margin on that cache read.

There is a reason why Anthropic is reported to have 80 points of gross margin right now on its API business.

Were you surprised by those 80 points?

Not really. I am impressed by 80 points of margin in basically any industry. It is difficult to get that margin, and the great thing about capitalism is that those margins will compress with competition. I am confident and happy for that, even though those people are theoretically my customers and my margin is sort of based on their margin. But I care more about a healthy ecosystem in the long term.

It is more surprising to me how many people still today think that these are horribly unprofitable businesses and that the whole market is going to zero. It is absurd to me that the meme is that OpenAI, Anthropic, et cetera, are just burning cash and eventually they will run out of cash to burn. If they stopped training, they would be massively profitable overnight, and there are a ton of other levers that they have without pacing the frontier, as Dario just said in his essay.

I jokingly think that a little bit of the pacing-the-frontier discussion is, “Oh, this is a great way to reduce costs ahead of an IPO.” But I do not think Anthropic or anyone needs to do that. I think they are amazingly profitable businesses with their scaling rates.

And it would reduce costs, just so I understand, because they would spend less on training if they slowed down the speed?

Yeah.

Right?

Yeah. I do not think that is actually the intention or anything, but yes, that is the bulk of their costs.

How did you analyze “Pacing the Frontier”? You brought it up. How did you analyze it?

4. Pacing The Frontier Risks Centralization

I have mixed feelings on the safety topic. I am a human who would like to live to old age, and more than that, I would like humanity to continue to the stars and beyond. But I believe much more in the ability for this technology to revolutionize every part of humanity in a positive way.

I do worry that pacing, in a lot of the ways it is being talked about—not necessarily how it will be implemented—has 2 big risks. One is a major pause and that playing into, for lack of a better word, the Luddite sentiment that exists. By pushing for a pause, it is actually giving ammunition and a better basis for those who just want to stop the technology altogether. I see that as a major risk for humanity.

The second piece is my actual P(doom). The thing I am most worried about with any AI outcomes is technology and capability being concentrated among relatively few people. Having a lot of what is being discussed from a regulatory framework, and limitations on AI, et cetera, I think is the modern road to serfdom.

It is the concentration of technological capability. Making it illegal to do matrix multiplications is the thing that will set us back to pre—not just the Industrial Revolution, but pre-Enlightenment capabilities. That is the biggest attack on classical liberal freedom concepts that I can think of.

While I do think that the vast majority of tokens are going to be produced by the big players, if the technology itself is restricted to just those players, then they are going to be the new lords and kings, and everyone else is back to being serfs.

With the greatest respect, is it not just lip service? Great, we will stick Meta in the corner. They can do their compliance, and then we can IPO. Sam can have a reason not to IPO because his numbers are not as good as Anthropic's. It plays into both our desires.

And Elon wants time to catch up as well.

So it plays into everyone's—

I completely agree, and I think that is the biggest internal reason for everyone other than Dario. Dario and, I would say, the vast majority of people at Anthropic are true believers, both in all of the promise and capabilities of the technology and in the risks. If I were in their shoes, I would also be taking on the massive amount of responsibility for that.

But I will say that there are a lot of strategic reasons for saying, “Okay, by having these auditors, et cetera, that removes some potential responsibility and culpability from a legal perspective.” The risk that I think this is a problem with a lot of very smart people, especially when they have amassed large wealth and power, is that they think they are going to be able to keep it.

The scariest thing, and part of my point about the centralization of technology, is that if it gets concentrated with companies, governments, et cetera, you have people who think they are the smartest people in the room, not realizing that they are not going to be the ones to actually control it when they put these measures in place.

Dario is basically, on the one hand, verbally begging governments to take over Anthropic. He is partly saying that because he does not think it will actually happen. I would love to see his reaction if and when that actually happens, and he realizes, “Oh, shit. I thought that if I was begging for regulations, they would then make me the regulator.”

When that does not happen and it just becomes a bureaucracy that halts all progress, and the capabilities that currently exist basically get squandered by select bureaucrats, that is the worst outcome I can imagine.

When you consider the advancements China is making, especially with its open ecosystem, it has an incredibly talented ecosystem moving forward. If we pace and they do not, what happens then?

I guess when I said that the worst possible outcome, I was not counting the Terminator outcome. I think, of course, everyone can agree that the Terminator or something similar is very bad, but I think it is extremely low probability. I am not a believer in that doom scenario.

For the vast majority of people, it would result in the same level of serfdom that I worry about in the scenario I described, but it would be significantly worse for some number of people in a Chinese Communist Party-controlled superintelligent AI scenario.

On the one hand, China's strategic angle right now is to have technology proliferate through open source, et cetera. I think as soon as it gets into pole position, the ladder gets pulled up with it in some way. I do not think it actually wants the technology to be easily accessible to everyone.

I do not know if China will decide that it is okay if the rest of the world has some access to the technology, but it definitely will not let the billion people who are not Communist Party members benefit equally from the technology.

So, just so I understand, do you agree with it? To me, I just didn’t get it. You can’t pace the frontier unless the global AI community paces the frontier, and I don’t see Putin signing up.

Agreed. I think this is a little bit the same naivety that I described among these company leaders and, in general, people in the Western world, thinking, “Oh, we’re so great. We’re so advanced, so far ahead that we can’t get caught up to.”

On paper, is the US the greatest military force in the world? Yes. If we suddenly had a drone incursion at the same level of what’s happening in Ukraine, with Russia coming up from Mexico—and if Mexico had developed very naive drone technology at the level of what’s happening in Russia, Ukraine, and Iran—how would we respond to that as a country if it were coming across our border?

It doesn’t matter how amazing our military might is. We built our military to fight the last war, and I think geopolitically, our thinking is, “Oh, we’re still the big dog.” When it comes to AI technology, there’s not an acceptance that export controls and all of the other elements that theoretically would allow us to pace and have people keep pace behind us just aren’t good long-term solutions.

Do you think we should have export controls?

I am a strong believer in free trade and free exchange of ideas. The exception to that is China, which has been a free rider on all of the benefits of a liberal free-trade order for the rest of the world, while getting to keep everything closed off.

I am very happy, and I think that any governments, societies, or people that want to embrace the free exchange of ideas and trade should have a very vibrant economy and ecosystem. But totalitarian regimes should not be able to participate in that, especially in cases where they get all of the benefits of it and get to export things that make them better able to maintain that totalitarian system.

Can I ask—we mentioned pacing the frontier and the different people who supported it. You had Zuck and Jensen say nothing. Well, Zuck actually came out in opposition to it, saying that we should continue as planned. What should we take from their seeming silence in opposing it?

Based on my overall beliefs right now, as evidenced by the conversation so far, I would say I’m overall opposed to the direction that pacing the frontier is going in. So I appreciate anyone who is adding to the discussion and being realistic about the benefits and risks, but you always have to take that with a grain of salt and ask what the motives are of anyone discussing it.

I would say I probably appreciate Zuck’s or Dario’s comments infinitely more than those of a random politician—not just random, but the “leading politicians” who don’t actually understand the technology. The scariest thing to me on the political spectrum, and in the way all of this is being treated, is that it’s now become an almost unifying issue on the left and right to be anti-data-center. I think that is almost entirely a Chinese psyop.

Can I ask why being anti-data-center is a Chinese psyop? It does increase—I’m totally with you on the benefits of them, and Gavin Baker said it: They’re the greatest economic needle-mover for large parts of the country. I guess people see increased electricity prices, increased water prices, and ugly data centers in their backyard. Why is it a Chinese psyop? What am I not seeing?

5. The Data Center Backlash

Just on the ugliness and all that, I totally support beautification campaigns. We need to turn them into centerpieces of our society. I think thousands of years from now, future historians should look at these massive data centers—the really massive, impressive ones—as being like the great pyramids or other wonders. We need to dress them up to be the world wonders that they are technologically.

The thing about water usage and the amount of power they consume is that so much of the early information that went out, from unsophisticated writers, was simply patently false. A single In-N-Out uses more water than the largest data centers in the United States, and golf courses use orders of magnitude more. These are closed-loop, liquid-cooled systems, and in many cases, you don’t even want to use water.

From the Chinese psyop perspective, they’re not, to your point, pacing the frontier. They’re adding gigawatts of new electricity-generation capacity, most of it dirty. They’re building massive new data centers and horribly displacing people. It just irks me so much that we have the freedom in the Western world to criticize companies and governments based on false information, and I love the freedom elements of that. But it is a strategic disadvantage when China can just say, “Yeah, we’re going to bulldoze all these people’s homes and do rolling blackouts wherever we need to in order to serve the greater good of new training capacity.”

I mean this with the greatest respect, but I don’t understand how anyone thinks the US or Europe can beat China when they have no regulatory or policy restrictions. In the UK, you can’t put up a paper airplane without getting a permit. We’re fucked.

But you are getting there. You’re becoming a European state in terms of the regulation and policy requirements. Am I wrong? Am I being overly negative? I don’t get it.

I hear you. You’re right. I think the greatest advantage the US has in that regard is that there’s still a lot of land and a lot of places that do not have the same levels of restrictions.

I don’t agree with a lot of things about most administrations of my lifetime. But the current administration gets attacked for supposedly destroying our environments and the national parks. The vast, vast majority—90-plus percent, although I don’t know the exact numbers—of federal land is just open, empty desert in the West that is not part of a national park or anything.

The fact that there are so many restrictions on using BLM land to build data centers when it’s literally hundreds of miles from any populated area is completely absurd to me. My great state of Nevada has plentiful geothermal and solar energy, along with all these green-energy technologies. We could build nuclear and other facilities in the middle of the desert, where they wouldn’t impact anyone, and there are restrictions on doing that.

There has been some political will and push to solve these things. But in just the past year, you’ve had Republican governors and other politicians who at least had part of their platform focused on being pro-growth and supporting all of these things backing away because they see their own political base becoming anti-data-center based on completely false premises.

One of the points I want to go back to that you brought up was the idea that people would have higher electricity costs. This is the most basic supply and demand. If we increase generation capacity—and no one is saying we want to take energy that’s reserved for people’s homes—then prices should not increase because of that.

One of the regulatory problems I see is that power companies have to offer energy availability that is baked into the costs and capabilities for everyone. There are absolutely zero cases where a data center could potentially pull power from anything that’s already been allocated. That’s simply impossible.

All these data centers being built right now are coming with generation capacity that covers their own use and beyond. We’re just not allowing them to hook up to the grid, where they could actually be lowering prices for everyone.

Then you have people on the power-company side lobbying against new generation capacity because market forces—more capacity—will decrease prices, which would be good for consumers. So it’s a very wonky market.

What percentage of the data centers that are planned will be completed, do you think?

From the major providers, I think the capacity they have planned may simply be in different locations. You’ve had some local communities successfully stop facilities from going in there, but those data centers just move.

I don’t think, a year ago, the major data-center builders and operators were thinking that the political problems were as bad as they were. So there’s a lot more effort being put into education in those communities now, which I think will turn the tide a bit. It’s also just going to mean that those data centers move to locations that aren’t going to have those problems.

As I said, we’ve got large tracts of land that can support them. So I’m not too worried that this is going to be an existential threat to capacity build-out. And, of course, there’s space if Elon is successful.

Do you believe that space is truly a viable alternative, or is it conference talk and lip service to justify a market cap?

I think something can start as one thing and turn into something else.

Right.

I would never, ever bet against Elon. I primarily bet for Elon. If you asked me a year ago, I just would not have thought that there would be a good reason for it in the near term because it’s going to be cheaper, easier, et cetera, to build on land.

I also think there are great alternative technologies. A company we’re partnered with, and whose CEO I’m good friends with, is a company called Panthelassa that’s building ocean-based data centers. Basically, it’s a very interesting pumped-hydro solution in the middle of the ocean. So there are alternatives that don’t require going to space, I think, long term. Part of the reason I’m a long-term, big believer in space data centers is I just think we’re going to need to have a space economy for humanity to live up to its long-term potential.

Love that. Totally agree on never betting against Elon. If we think about the cost of intelligence being tied to the cost of energy, how should we think about energy as a bottleneck moving forward? To what extent is energy a bottleneck moving forward, or is it less of a bottleneck than people consider?

6. Energy Gates All Progress

I think there are 2 pieces to it. One, Positron is trying to deliver more compute and more capabilities per watt, per megawatt. On our base case, we can turn what you would have spent 500 megawatts on with NVIDIA equipment into 100 megawatts. I don’t think that’s actually going to mean that you’re only going to build a 100-megawatt facility. You’re still going to build the maximum amount of compute that you can. You’re just getting more tokens and more intelligence per joule.

If I go back to the long-term thinking, I think that, assuming humanity continues for hundreds or thousands of years, everything turns into an energy problem. You can go back thousands of years and just look at the progression of mankind. Fundamentally, that is a perfect track of our ability to produce and use energy, from the discovery of fire up to nuclear power plants. The simple, tongue-in-cheek answer to your question is: all progress is gated by energy.

Even if there’s energy available, it may not be economical, and so it won’t be done. Actually, I would say the bigger limiter than just saying our ability to build and produce energy is that we’ve got plenty of technologies and capability to do it. I would say we have way more economic limitations. How much debt is the world willing to take on to build out everything over the next couple of years? That ties into energy, it ties into the infrastructure itself, et cetera. So I think economics is a much easier scapegoat to pick.

People are already very concerned by the levels of debt being taken out and the debt cycle. Do you think their concerns are justified, and then do you share them?

I think we’ve got a major sovereign debt problem that masks a huge amount of second- and third-order elements in the financial system. There are the inflationary consequences of governments that can print infinite amounts of their own currency. We are already seeing, with Treasuries and the greater bond markets, that there is greater and greater perceived risk in the most “risk-free asset.” I think that will trickle down to all elements of the financial system.

When people worry about Oracle’s debt and credit rating, I’m like, I believe in Oracle’s business model and ability to execute and do everything a whole lot more than I believe in the United States government. It’s just that the United States government can issue its own currency and also has guns and nukes to take tax revenue.

My biggest economic concern is that there will be a more acute, specific crisis that arises out of the compounding of national debt, leading to devaluation of the currency and all of the downstream consequences. I’m really not worried about any of the companies in the AI debt stream not hitting their revenue targets. If the past 3 or 4 years have shown anything, we’re accelerating every aspect of these businesses in terms of revenue, profits, and how they are improving productivity and value downstream.

I’m jumping around, but fuck it. When I was doing the research, I was reading about KV caching and compression as part of this, and I was honestly getting lost, but I was intrigued and digging deeper and deeper. I was like, “Why did I not know this before?” I don’t think many people will know this. What should we know about KV caching? Why is it important? Can you explain it to me a little bit?

There’s always a give-and-take relationship with innovation. One element I’ll have to explain to make all this clear is the concept of a sequence. I already talked about a token. A token, effectively, as part of the training process when any of these big model labs are developing a new model, is based on a vocabulary that they define.

They take their big, giant corpus, and they do some statistical work to figure out what the best encoding method is to take all of the text in it and break it into chunks that get reused frequently to make things more efficient. What you end up doing is, if you took an English dictionary, you’ll find that there are common prefixes and suffixes and groupings of words. If you just try to think of how you would best compress this, you might have symbols for these prefixes, suffixes, et cetera, and compress this into a thing.

Basically, what ends up happening is, if you’re using ChatGPT and you see text streaming out, if that’s going particularly slowly or you have a keen eye, you’ll see that portions of words come out at a time. Sometimes it’s a full word, sometimes a small fraction of a word, and each of those little flashes that you see is a token. Roughly speaking, a token is equivalent to half to 75% of a word on average in large English corpora.

A sequence, or the thing that builds up to be the context in a model, is the grouping of all those tokens in order. When you’re running an inference, you’re given a prompt. You have “What is the capital of France?” as your input prompt. That is tokenized—4, 5, or 6 tokens that go into the model. When you do inference, it’s going to say, “The capital of France is Paris,” and “the City of Lights,” or some other thing after that.

When you have that entire sequence, when Transformers originally came out, for every token that was generated, you were doing the computation for generating all of those tokens, including the ones that you’d already processed. Clever, and I would say kind of obvious based on all the developments in computer history, but it wasn’t done initially, was the realization that you don’t actually have to redo the computation of the things that you’ve already had as input and already generated in this turn.

That’s how the KV cache was born. Within the model, there are 2 matrices called K and V, and those matrices are fully based on the prompt and whatever is generated during a turn. By storing those 2 matrices, you can avoid having to redo computation, at the cost of now having to store this thing in memory. That’s the simple example: very, very small, hundreds of kilobytes of data. But the thing is that these grow with the sequence length.

The interesting thing is that, for the attention mechanism, the compute per token grows quadratically with the sequence length. You’re having to spend more and more compute quadratically as sequence length grows. But when you store that as just your K and V, that’s only linear growth.

What KV caching does is mean that you don’t have to do that compute, which gets very expensive very quickly, at the expense of needing to store these things. Storing that is a complexity in itself because there’s a unique KV cache for every single user that you’re serving, and it raises questions of how long you want to keep that for and how you manage all of that in a large system.

What does it mean, then, when we hear about compression and decompression of KV caching and potential entropy within the system?

There are 2 different forms of compression— a couple more than that, but the 2 main ones. One is quantization. If you’ve got each of your values, be it your weights, your KV caches, or your activations, stored in a particular data type, before the machine learning revolution, most of the world’s computation was done in FP32. You have 32 bits to represent a floating-point number, and that’s broken into a mantissa and exponent.

It was pretty quickly realized that having 32 bits of precision was superfluous for the things you’re wanting to represent, and it costs more from both a storage and computational perspective than lower precision. So we went to FP16. Google developed BF16, a little rejiggering of those bits. Then we went to FP8, and now we’re at FP4 in popular systems.

We’ve been reducing the precision quite quickly, but that does lead to, for lack of a better word, some brain loss when these models run, just because you’re now trying to encode the same information into fewer bits. There have been a lot of interesting schemes that say, “Okay, I’m going to take this group of FP16 or BF16 values and quantize those. I’m going to truncate and round that down to, let’s say, INT4 values.” Now you’ve actually saved 75% of the total size of that group of values.

You shrunk that down from 16 bits to 4 bits. But just doing that naively will mean that on a lot of benchmark scores, you'll have them get 20% or 30% worse. So you get that 75% savings in space, but you kind of lobotomize the model.

Advanced quantization techniques actually say, “Okay, these 16 values, I'm able to have a shared bias and a multiplier for them.” Let's say for those 16 now-int4 or FP4 values, you store one new FP16 value that gets applied to all of those at compute time. So you get a 75% compression on all those values at the cost of adding that one new FP16 value.

Basically, the state of the art here is that you're able to get things compressed from FP16—16 bits per value—down to around 4.5 bits per value. That can be applied to weights, the actual parameters in the model. That could be applied to the KV caches. But there's no such thing as a free lunch. You'd still have some lobotomy, but thankfully, it's kept within around 1% of an unquantized model.

Is KV caching the hardest element of building that inference infrastructure, or is it latency SLOs, load spikes, or anything else that we could come up with? Is that the hardest? What do we not see that we should see?

You can run a service and do something without having KV caching at all. The economics, performance, and everything else are going to be much worse. The dark arts and magic with it are that the workloads the industry has found most valuable so far happen to be very, very highly cacheable.

SemiAnalysis has its Agent X benchmark and suite of test data based on taking a whole lot of Claude Code sessions and having dozens to hundreds of turns in those Claude sessions, with sub-agents and everything else. What they found is that over these massive numbers of interactions in these real-traced code-generation agentic-coding sessions, about 96% of all the tokens that go through these entire sessions are cached.

If you know your workload is going to have this extremely high caching rate, where you're going to be reusing the same tokens again and again, that drastically shifts the importance of how you can retrieve those caches, because these things get to be very, very large. We've gone into trillions of parameters.

GPT-4 was leaked as a 1.8 trillion-parameter model. Assuming that is int4-quantized and rounding down a little bit, that's 900 gigabytes of data size for the model weights. If we take the high estimates for something like the Claude Fable, that's a 10 trillion-parameter model, so around 5 terabytes of model weights.

The crazy thing is that at these long context lengths for models of this size, the individual user sessions can be in the 100-gigabyte range. With just 50 users on your service, those individual user sessions end up being greater than the model weights that you're trying to store.

Claude and OpenAI have a whole lot more than 50 users. So it becomes a really interesting trade-off: How much of the accelerator memory do you want to dedicate to weights, which you need to process every single generated token? You want that to be as fast as possible, because that sets your SLO and your token latency.

But if you don't have their KV caches persisted, you're actually losing a huge amount of efficiency, because that was work that you didn't have to repeat. So it saves you, as an operator, money more than anything. At some level, having users' KV caches be persistent will provide some speed improvements that the user perceives, but it's mostly an economics thing for the service provider. If you can return to those caches and use those tokens again and again, that saves you a massive amount of money as an operator.

I totally get that it saves us money because we don't have to use as much compute, but then it's harder from a memory-challenge perspective. How do you think about the right logical next step, then? If you appreciate the importance of saving on compute but the challenge of memory with KV caching, what's the answer? Do we just have bigger and bigger memory stacks on-chip? What does that look like?

The most common deployed solution—and the vast majority of inference out there is taking place on GPUs, followed by TPUs and a couple of other devices—is that you've got your GPU accelerator memory, which is primarily responsible for holding the weights, and you'll keep some number of user sessions on there, the ones that you're actively processing.

But the larger group of users has a tiered hierarchy. You'll have users who were around, say, in the past couple of seconds but haven't returned and don't have an active request. That's residing in host memory. Let's say that's on the order of 4 to 10 times more memory on the host than in the accelerators, so you'll be able to store more of those there.

If someone hasn't been around in a couple of minutes or a couple of hours, that's going to be stored in even more distant memory. That could be in NVMe, so flash storage: a lot slower, but with a lot more capacity on that host. It could be in flash storage on a network-attached drive. Eventually, I bet the ChatGPT sessions that I had 6 months ago are residing somewhere on a slow SSD or somewhere in a data center.

It would be dumb for them to use expensive memory to store that. So that tiering is the norm, but it introduces a huge amount of complexity in deciding when and where you're going to store something for your massive number of users.

I would say our solution, and how we're trying to go about it, comes from our expectation that model sizes are going to drastically increase, the number of users for all these things is going to drastically increase, and the contexts themselves are going to increase. Two or 3 years ago, typical context lengths were on the order of 8,000 to 64,000 tokens. Then they got up to 128,000 and 256,000. Million-token context lengths are the norm now in terms of what the models support.

But million-token context lengths can only hold a portion of some of our company's largest code repositories. It would be a fraction of that. So if you really want an agent that can take over the capabilities of a whole team of programmers, I think the main limiter today isn't the model capabilities themselves or scaling the model size. It's how much context that model can have from all of the data it needs to make smart decisions.

I just want to break some of the things you said out there. You said that you think model sizes will increase. I thought we were all moving toward owning our own intelligence, with every enterprise having its own smaller model with proprietary data. Does that go against what you think in terms of model sizes increasing? Can you help me understand?

7. Frontier Models Keep Scaling

I think you can break it into 2 tiers again. There are going to be the frontier models and capabilities being developed at the forefront by OpenAI, Anthropic, maybe Google, SpaceX AI, et cetera. I still think there's a long road to go in terms of pushing the frontier of model capabilities, and those will continue to grow and get better.

There are a lot of workloads where, speaking for how Positron uses LLMs, I don't today care that much about the cost. If I can get 10 times the output value from a model today, I'm very glad to pay 10 times more per token. I really want the frontier to push that.

Some of it is cost-saving, and some of it is truly owning your proprietary data. There is a push from enterprises to have inference on-site, and it's a lot more difficult to provide that for the largest models. Most companies, if they're adapting from open source or developing their own model, don't have the resources to push the frontier. So that's what has led to smaller models.

And do you not buy that reality?

I would say that right now, somewhere around 80% or 85% of all tokens consumed and produced are handled by just the top 4 model companies, and I would say the next 5% or 10% is handled by the 3 or 4 companies after them. So I can totally believe that 5% of all tokens consumed will be handled by things running on-premises, locked into the big companies.

But both from Positron's business perspective and in terms of how I see the world evolving, I'm going to care more about the high-volume set of things. That being said, I actually think the small-model stuff is much more interesting for everything happening on your phone.

The amazing thing about that is that I think there's a misconception that if questions could be answered by your phone, or if prompts could be handled locally, that means fewer tokens will be used by the big companies in the cloud. I think it's the opposite.

The reality is that if I have an LLM running on my laptop or phone, or in my enterprise's secure on-premises cloud, whatever it is, that is going to be consuming data at such a fast rate from everything coming into it. It's going to be generating analysis based on that, and it will decide what it's actually going to return to the user locally and what actually requires more intelligence from a better model that isn't self-hosted.

For any of the tokens that are being “saved” by running locally, I think that's actually going to generate more things.

Because, in some ways, for a simple, naive use case of a personal user of LLMs, they're only going to prompt ChatGPT or Claude every so often. They're sort of limited by their thoughts of when to actually ask an LLM something. But if they have a local LLM that's constantly checking their email, calendar, messages, et cetera, and deciding to do these lookups to cloud-hosted models frequently, that's now, on a per-person basis, a massive increase in the number of tokens being consumed and generated by the cloud models, even though the naive view was that there's a shift to an on-device LLM.

So can I just understand? I completely hear you in terms of maybe 5% of them will be in this smaller-model, enterprise-owned kind of model landscape. Why are you so bullish, then, on much larger models and the sizes increasing?

I'll break it into portions. There's the increase of the model sizes, which I think is something that would be shared by a lot of people in the AI space. It's kind of a gut feel based on the fact that we have seen these scaling laws. We call them scaling laws because we've observed the fact that going from 100 million to 1 billion to 10 billion, 100 billion, and 1 trillion parameter models, we've seen this amazing increase in capabilities.

We still see that to this day, going from 1 trillion to 5 to 10 trillion at the largest end right now. We call it a law because we've observed it, but there's no actual mathematical proof that this will continue. So it's sort of on vibes that this continued scaling—there's no sign of it slowing down. Is that going to continue to 50 trillion, 100 trillion, and beyond? I don't see any indication that it's going to stop, so I'll be bullish on that.

What do scaling laws look like at 3 times what it is now? If AGI has been declared now by Jensen— forgive me—but what is 3 times this?

It's a good question. GPT-6 Astra—my first 24 hours with it were basically as magical as my first experience with GPT-3.5 in November of 2022. I was at the ChatGPT launch at NeurIPS in 2022, and it was so funny because Sam and Ilya were there. It was a party in New Orleans for the NeurIPS conference, and basically, at the end, they just said, “Hey, we launched this little, fun experiment called ChatGPT. Go check it out.” Zero fanfare. It was really just a side mention, and I don't think anyone really gave it a thought at the event.

When I went back to the hotel, I loaded it up. I got back at 10:00 or 11:00 p.m. or whatever, and I was up for 4 or 5 hours straight, just giving it random prompts. This was the most magical experience that I've ever had with a computer. I would say I got very close when Sora 2 came out. I had a similar experience over a short amount of time, but I was just mind-blown by the quality of the videos, especially the weekend that Sora 2 launched, when there were no restrictions on what you could generate.

But, yeah, GPT-6 Astra, I do think, is AGI. To your question of what that means going forward, I think my guess is as good as basically anyone's.

Why was GPT-6 Astra so good for you? Why was it comparably such a breakthrough? Because I have it and it's great, but honestly, it's kind of the same as before.

In terms of the things that I've found LLMs to fail at the most in the past, I'll give a case where it is more linear improvement. In terms of general coding capabilities, performance, analyzing problems, et cetera, it is a step-function improvement, but not mind-bogglingly so. There are a bunch of things that other models have not been able to fix or have gone in circles on and found inelegant solutions for, where it's still a human software architect who was able to come up with a better solution.

With Astra, I initially gave it a couple of really hard problems that I had not been able to solve with other LLMs, and it was able to do them in one shot. I had it go through a codebase and find performance improvements, bugs, et cetera. It was basically discovering new spaces that I didn't know existed in a bunch of portions of our codebase. So that's one element: a step-function improvement, but not mind-boggling.

The second case that was mind-boggling, just from a “wow” perspective, was the computer-use abilities with a set of generic tools. It could do Blender animations. There are a bunch of memes online of it recreating different videos, et cetera, but just the fidelity of that, when it was basically impossible with GPT 5.6 Soul, was a massive increase in capability. I had it do the interior design of my house based on a couple of pictures, and I was just amazed. I did not think that what's fundamentally a text model could do that.

Finally, the biggest thing for Positron was that I've been trying with every single new model release to have these models actually take a relatively simple logic-design problem—in this case, implementing an encryption block—and take that through the full RTL-to-GDS flow. That means going from the specification—“Do this encryption function”—to implementing the Verilog, the hardware description language for that, writing that code, and then taking that code all the way through until you've got a chip design that theoretically you could tape out.

LLMs could do different portions of that. They could write the scripts and fail at a lot of different midpoints along the way. But a big problem with the electronic design automation tools—the EDA tools for doing chip design—is that they were designed in the 1990s and early 2000s. They're really unintuitive. None of the documentation exists on the public web, so these models don't have a really good innate view of them.

But GPT-6 Astra, with a combination of computer use and an ungodly amazing scripting ability, has been able to take this Keccac block, implement it with the TSMC N3 PDKs, take that all the way to GDS, and do that in a little over 50-something hours while meeting timing at over a gigahertz.

If I were giving this to someone similarly new to the field, getting the flow mostly working would take on the order of a week. Getting it optimized to the point that Astra is at with that design would maybe take 1 or 2 additional weeks, depending on the person. Compressing those 2 to 3 weeks down to 2 days and change is still mind-boggling. It shouldn't be this good at this, as I would naively think about its training sets. But obviously, with OpenAI's own in-house chip development, they've added these capabilities to the models they're releasing to the public. I'm glad they're not just keeping them inside.

We see Jalapeño—terrible name, I think, personally—but their own chip development. Anthropic are developing their own chips, and DeepSeek is supposedly developing its own chips. We see the commoditization of the chip layer, with everyone building their own chips. How should we think about that?

As a consumer of all these things, if I take my Positron hat and shirt off, I would say that that's a great thing for the industry. Fundamentally, that's going to bring costs down, capabilities up, and bring it to more people. I think it's such an interesting world where, when I got started in the semiconductor space 13 years ago, silicon was a dirty word in Silicon Valley. Now you have all the biggest companies in the world being somehow connected to the semiconductor industry, with the most interesting, exciting applications and the companies building them vertically integrating down to the silicon layer.

The interesting thing with all the ones that you mentioned and the broader set is that companies have the same macro goals. The implementation details are all unique, though, and, just as an engineer and technologist, that's exciting to me. There are a lot of different ways to skin a cat, and people can have their own architectural view, go about implementing it, and get different results.

Hot Chips, the biggest conference for this design space, was where OpenAI unveiled Jalapeño last month. I'm happy that the industry is still pretty open and willing to share—not as many details as people would've shared 5 or 6 years ago, but still a good amount of open discussion of things.

How that applies to Positron is that we have our particular architectural views and the way we've decided to do things, and that will evolve in the future, as will everyone else's. There's still plenty of space to make bets and go in different directions. The great thing about the market is that the market gets to decide what is valuable, and those that create value will receive a reward for that.

We spoke about context-window length earlier and the expansion of it. How much does that expand? Is there infinite expansion capability for context-window length, and what does that mean we can do that we can't do today? I'm just fascinated.

I think with traditional linear or quadratic attention, there are going to be limits of scale when it comes to what the hardware could provide. Now, one of the big things for Positron is that we're trying to massively increase the memory capacity per device.

With our upcoming generation, we're going to have 8 times more memory capacity than NVIDIA's highest-memory SKU. NVIDIA is actually decreasing the amount of memory per device based on market memory conditions. But I frankly think that taking context length from around 1 million tokens today to 10 million or even much more is really, really hard with that quadratic expansion of memory cost.

The algorithmic advancements over the past year have been very interesting in terms of further reducing the amount of storage and compute necessary for that context, with linear and sparse attention mechanisms. Those have really been innovated by the Chinese model labs, and this is a great example of what happens when you have constraints: we had export controls on chips with the highest memory capacity and FLOPs, so they innovated around not needing that.

DeepSeek, beginning in 2025 with DeepSeek-V3, made a lot of waves because it was able to massively decrease KV-cache size with multi-head latent attention. You were actually spending more FLOPs to have a smaller KV cache, and that's advanced a lot over the past year and a half. Probably the most interesting—or my personal favorite right now—is Gated DeltaNet and its derivative versions, where you can have a 75% decrease in the total time you're spending on the attention portion with this mechanism.

How important, then, is new hardware if DeepSeek, without it, just through architecture innovation alone, can cut costs by 80%?

I would say that there's no such thing as a free lunch. When they have that MLA compression, it does come at a cost of model capabilities in some form, and there's a reason why the Chinese labs have heavily embraced MLA while none of the U.S. labs have. I should say that's based on rumors, but I also have very good information and belief that none of the major U.S. companies are using MLA or some of its brethren. I think that will evolve and change in the future.

The short version is that it's not just a pure savings on that side. But the reality is—and to go back to your previous question—everyone does want greater context length. If I had a 10-million-token context length, I think that would be enough to hold multiple of our largest codebases and really have that cross-pollination happen between them for an agentic coding model.

One thing I should have explained is that it's not just about having the full context. There were some early models that advertised a 1-million-token context length, but as soon as you went above 64,000 tokens, their recall ability went to garbage. That was 2 years ago. Saying that something has a maximum context length is one thing; whether it can actually use that context length effectively is an entirely different thing.

That's actually going back to the Astra thing. The amazing thing about it is that there are a couple of different benchmarks measuring long-context performance. One of them is called RULER, and there are also these “find a needle in a haystack” tests. You flood the context window with a bunch of junk—passages from books and all this—and put, somewhere randomly in the middle of it, a hash or some value that looks out of place. Then you prompt the model: “What's the secret value?”

A lot of models have done really poorly on this. GPT-5.6, which is only 6 or 7 weeks old, could do this only about 70% of the time. GPT-6 Astra does it correctly more than 95% of the time. There's a lot of room for improvement in these things.

Speaking of room for improvement, can I ask you: when I was doing the research for the show, I saw the Silicon data token price index drop below $1 per million tokens this month, and 5 years ago it was $60 per million. From $60 to $1, what does a million tokens cost in 2028, 2 years from now, do you think?

I care much more about the fact that a $60 token from 5 years ago is something no one would pay a cent for today. That was a complete garbage token, relatively speaking, 5 years ago, and the level of quality for a token that you pay $1 per million tokens for now is astronomically more valuable.

And so that's because of token efficiency and what can be done?

No, I'm saying just in model capabilities. If you say, “Okay, so it's 2026,” the best model in the world in 2021 was GPT-3. It's crazy, at the rate models get released today, that GPT-3 was basically the best in the world from 2020—I think it was August 2020, when it was released—all the way until ChatGPT in November 2022. They didn't have a new release for 2 or 2.5 years between model releases.

Really, GPT-3.5 was just doing reinforcement learning with human feedback on the same base model. If you remember how bad GPT-3.5 was, what was the economic productivity value of GPT-3.5 versus GPT-6 today? Or pick whatever comparison points you want. The value per token, in terms of what it can improve in a person's life or a company's business practices, is orders of magnitude greater—I'd say 100- or 1,000-fold.

I think there are actually 2 points to your comparison of going from $60 to $1. Yes, that's a decrease in cost, but that token today is, let's just say conservatively, 100 times more valuable. I would say there needs to be some multiplier there as well. The value per unit of intelligence is probably closer to 1,000-fold, not just the 60-fold you're talking about.

What does that mean if we extrapolate that out to 2028? Does the cost of a token actually matter? Is that the primary unit that we should measure? Everyone talks about the cost of a token. Is there actually a different metric that we should measure?

It is interesting that, with the GPT-6 launch, Greg Brockman said that he doesn't think they're going to be pricing things in tokens much longer and that they want to move toward cost per useful result. I don't think that's where it'll end up, because that's really difficult to price and, you know, qualia, et cetera.

But I think the price per token is really great because you can easily calculate the cost to generate a token. Determining a margin on that and pricing it in bulk to generic customers is really easy, and I think that's going to stick around in a large way because it's so easy.

We'll see how the largest providers of tokens potentially evolve their business models. If you have a GPT-7 or GPT-8 that is superhuman and can fully function as an employee in an amazing capacity, and OpenAI calculates, through whatever method, that running at full tilt is only going to cost them however many hundreds of thousands of dollars to produce tokens continuously, they may decide that it's easier—and that they'll get more adoption—if they just charge $1 million a year, using a random number, for full, unlimited usage of that virtual-agent worker. That may be how things evolve.

Can I ask you something? I'm always very careful about being the young, naïve one. I'm not that young anymore, but being the naïve one who's not seen cycles. Gavin Baker says it well when he says, “I can't speak to a company that doesn't have numbers that are parabolically up and to the right, and just everything is better than it's ever been.”

What would be the first signs of a crack in the dam? Would it be a shift from frontier models to open-weight models? Anthropic and OpenAI not continuing at the same level—not quite the growth rate, because it's impossible to say for sure, but the level of growth—missing numbers next year, and then the bubble getting burst a little bit? Would it be whether the 2 core leaders are having some form of strife?

8. What Could Break The Thesis

I agree that that's a possibility. The reason I don't think it's likely is that I think the development of open-source models and things happening locally will actually drive greater token volumes for the big guys.

Sorry, how does that work? I thought they were competitive.

I think the smarter and more capable Siri is on my phone, the more it will do a whole bunch of background tasks and things that remove me from having to initiate requests and have data processed by even smarter models. I really do think that, in a lot of AI applications right now, the bottleneck is actually a human making some sort of decision. Different tasks have different levels of autonomy that will result in things getting sent to be processed by a model, by OpenAI or Anthropic.

But I think the next really big increase—possibly a couple of orders of magnitude—in token volumes is going to come when we humans trust a local LLM that has access to all of our data all the time to decide to do things on its own that it is not smart enough to do. Right now, I trust Astra a lot more than myself on a whole lot of different things, but I still prompt it to do things, and maybe it will run autonomously for 12 hours or 3 days.

I think the next big leap is when I trust a model running on my laptop or on my phone to prompt the smarter models to do an even wider set of tasks.

When we look at the data economy that powers the larger models which you believe in, we see Merco, we see Surge hitting $3 billion in revenue. How big do these companies become? Because Anthropic and OpenAI being $4 to $5 trillion businesses, say, is feasible. It's wholly feasible, isn't it, that Mercor and Surge AI are $200 billion businesses which serve both frontier labs and some of the world's biggest enterprises building their own models?

I think my only skepticism there is on there being vertical integration by the frontier labs. I think the reason that hasn't happened is because Anthropic and OpenAI have better uses of their capital and mental power, et cetera, than doing all of the Scale AI, Mercor, et cetera, work. But I think that won't always exist.

Let's say that GPT-7 or 8 could be an effective replacement for Sam Altman in terms of being able to manage a large business. Why wouldn't they just have agents taking over those tasks?

Before we leave, I'd like to finish on a note of optimism. What are you most excited about today that you think the world does not spend enough time on, that we should spend time on?

I'm very sympathetic to the problems that I think the smarter set of the AI alignment and AI safety community is thinking about when it comes to how to align incentives. A key part of that is human alignment. It's like, how do we, as a society, as the human race, align ourselves to have a good outcome that I think will be empowered by artificial intelligence?

We discussed a bunch of the different problems that we're facing geopolitically and socially and how these different things are handled. I think a lot of smart people are doing good work on the AI alignment problem and thinking through how we solve that, but they may be gated in what can be done there if we don't get better human alignment, be it on regulatory frameworks, energy production, where we're going to put the data centers, et cetera. I think framing it as a similar sort of technical problem that smart people can work on and reason through will hopefully get more people thinking about it that way.

I think a core element that gets discounted by a lot of people in that sphere is that there are a lot of economic factors, and I think it's the economic factors that actually will drive real decision-making and actions. If we don't look at it from the perspective of rational, self-interested actors and all these different things, you're just not going to make progress.

Thomas, this has been the most varied discussion ever—from education and unbelievable infrastructure evolution to Dario and Sam. You are a star. Thank you so much for joining me today.

Thank you so much, Harry.