20VC: Taking on NVIDIA: Fractile CEO on the Future of Chips | What Everyone Gets Wrong About Capex Spending | The Winners and Losers in Neo Clouds with Walter Goodwin

We’re so traumatized by the dot-com bubble that we’ve priced in exuberance right from the outset. We will be spending far more on running models in 10 years’ time than we are today. It feels like we need thousands of times more compute than we have today. Frontier intelligence is not yet commoditizing. I think we’ll have a couple of multihundred-billion-dollar companies built in that space. One competent neocloud is much like another competent neocloud.

This is 20VC with me, Harry Stebbings. Now, joining me in the hot seat today is one of the leading minds when it comes to frontier chips, Walter Goodwin, founder and CEO of Fractal. The business has just raised a $720 million Series C co-led by Thrive, Founders Fund, Lightspeed, and Redpoint, valuing the company at a $7 billion price. As I said, Walter is one of the leading minds in the future of chips, and we get into that and so much more in the discussion today.

Walter, I am so excited for this, dude. It’s lovely to be able to do it in person, and I’m so thrilled that you could join me in the studio, man.

Very good to be here with you, Harry.

Now, I want to start with 60 seconds for those who don’t know: what is Fractile? What do we need to know?

We’re a frontier AI company of sorts, but we make chips. We make these big rack-scale systems that are fully focused on advancing the frontier of AI capabilities. From a team composition point of view, this means we have ML researchers, process experts who fly out to Taiwan every few weeks, and people who look at the thermals and mechanicals of putting together one of these rack-scale systems.

I was listening to you on No Priors. We were just talking about Sarah, and you were saying unequivocally that you were a chip company. I was listening to this thinking, “Great.” But then I’m looking at it and thinking: everyone’s moving full stack now. We’re seeing OpenAI have Jalapeno. We’re seeing DeepSeek have their own. We’re seeing Anthropic build their own. Obviously Meta and Mark have their own. My question is: is the future won by vertical integration, and can you be a chip company alone these days?

It’s a very good question. I think there’s a verticalization as all of these different components come together, especially as you look into the future and work out what these models are going to look like, how the workloads evolve, and how the silicon evolves. There’s an absolute necessity to have a vision for every part of that.

Fractile is a chip company, but in a sense, we’re also a frontier AI company. I think what that means for us, and this will continue to be the case, is that we monetize by selling chips. We monetize by selling racks of compute to our customers. But it’s completely essential that we have a bunch of folks internally who are ML researchers, looking at how model architectures might evolve, and maybe even running smaller-scale experiments than the ones you would see in the leading labs.

I think there is a distinction between the expertise that’s needed internally to win and the nature of the business—what the business model is. We never intend to compete with our customers. We will never monetize internal models, because that’s how we’ve chosen to build this company. We’ve chosen to compete in the chip layer.

But you can’t step back from challenging yourself to have expertise in all these other places, because in our case, that would mean you’d place the wrong bets. You have to synthesize everything you’re learning across the ecosystem, and then bring that back to be obsessive about the part of this space you’re building. If you do this right, you’ve ideally found yourself an enormous piece of green space where you’re incredibly helpful to your partners, your customers, and your upstream suppliers, and you’ve gotten to know parts of their business as well. But of course, you never seek to conflict with them.

Would you argue that everyone’s moving to vertical integration because everyone’s fighting against the commoditization of one layer? If you think about what OpenAI did with Jalapeno in nine months, it suggests some form of commoditization of the layer. Do you think everyone is desperately expanding to fight commoditization?

I think there’s definitely a fight to keep your margins expanded. If you’re serving frontier models, it’s important that you’re not squeezed all the way down by very high margins on incumbent chip players. That’s where it’s useful to have, for example, a first-party play or another third-party play that can be a counterpoint to that.

I think there’s a confluence of forces that means if you’re a frontier lab or a hyperscaler, it’s very important to have your own retort to various parts of this. I also think there’s still a pretty crucial distinction today between what’s perhaps becoming commoditized and what we still see at the frontier.

What is commoditized, then? Help me. I’m an investor. I look for value in amazing entrepreneurs and businesses. What is commoditized that most people think isn’t, and what isn’t that maybe people think is?

That’s a great question. There’s sometimes concern, sometimes excitement, about the idea of intelligence saturation: as models get smarter and smarter, there will become a kind of waterline that they exceed, beyond which we don’t really need greater intelligence for a whole set of tasks.

I think it is true that there are certain tasks for which we see this kind of intelligence saturation. This is where you then see an open model is good enough—

Mm-hmm.

—and these models become commoditized at that layer. I think where that becomes more nuanced, though, and where models even at that level still fight against commoditization, is that there’s always economic pressure to reduce the cost of intelligence.

Perhaps counterintuitively, this is why I think the frontier labs and frontier efforts remain so vital. Frontier intelligence is not yet commoditizing. We see large margins there. We see big deltas against everybody else. Also, these learnings from the frontier trickle down into building better, smaller models.

If you look at that fight across the entire gamut of frontier LLM capability, as well as behind-the-frontier LLM capability, there’s still a ferocious battle. I would characterize that as not commoditized, insofar as if you’re a market leader in something, you can still extract high margins.

A good example of this is that DeepSeek released their entire inference stack, their entire software stack, a few days ago, and it had really remarkable…

People very quickly characterized it and realized that DeepSeek had done so well building an inference stack to run its models on GPUs that it was getting much higher throughput, better interactivity, and better speeds than everybody else who had built a DeepSeek inference server.

Something you might have thought would have been commoditized by this point—the serving of an open model—wasn’t. By going full stack, DeepSeek had built a more efficient model and an inference server for that model, and carved out the right to earn really healthy margins. Even in the open-source space, we don’t yet see a complete race to the bottom, at least for the folks who try to own a lot of vertical understanding of the space.

What is commoditizing, and I don’t think anybody has ever argued it won’t be, is the whole space of cloud compute. One competent neocloud is much like another competent neocloud. The reason it’s so exciting today is that we don’t yet know which ones are competent. There’s still a lot of space for huge winners and maybe huge losers as well.

I think that’s why you see all the neoclouds desperately trying to vertically integrate, too. They’re very aware of the commoditization of their own layer. We had the founder of Nebius on the show, who talked about moving into model routing and inference as a core business and strategic priority. I think they’re aware of that.

I think that’s true, and you definitely see every neocloud trying to build an inference server. A lot of the people who built an advantage in inference serving, which is ultimately a software advantage, are also thinking about how they can have more control of their destiny on the compute side. Obviously, I’m biased, but I think this is where the neochip story and differentiated silicon can become a core differentiator for those players as well.

Can you unpack that? What is that?

We’re still in the very early innings of rolling out silicon that isn’t the mainstream conventional GPUs, TPUs, and other XPUs. It’s still much bolder, especially if you look at how these build-outs get financed, to lean into a new chip platform like Fractile, Cerebras, and so on.

And we need these new chip platforms because workloads are becoming more and more complex and intense, and the specialization of these chips will allow those workloads to be executed much more efficiently.

These new platforms, obviously, we’re one of them, have a belief that in certain workloads you can get an advantage of more than 10X, whether on cost or, in our case, on both cost and speed. In a sense, they’re still contrarian bets. We still don’t see many gigawatts of new silicon being built out, but these platforms have the potential to fundamentally alter the economics.

Imagine you’re a neocloud and you’re the first to work out how to break free from a particular GPU vendor. You could reap an enormous benefit. Fluidstack is a great example. In some ways, they were the first neocloud to decide they were going to step away from Nvidia. They deployed a bunch of Google TPUs, and they’ve had an extraordinary year as they crossed that Rubicon and earned the right to start deploying other chips. As you go more vertical, there are opportunities to take some of these slightly higher-risk bets and reap huge benefits.

What does that do to Nvidia’s business? If we think about Nvidia as a horizontal provider and you as a specialized provider, is the future of chips specialized according to workloads? How do we think about the market composition of horizontal versus specialist?

The way I think about this is that Nvidia is an extraordinary business. By many metrics, they build the best AI chips in the world, as well as the best systems, and they do extraordinary work on data-center-scale inference, for example. In the inference space, they’ve done brilliant work as well.

What’s different about this moment in time, and core to Fractile’s identity, is the idea of an evolving frontier. The frontier calls for slightly different bets and a slightly different way of thinking, because you’re setting up a business that can look around corners, make bold predictions, and steer the entire tech product toward those predictions.

Nvidia excels at delivering high-quality products at extraordinary volumes that are, in a sense, unopinionated. These are powerful general-purpose processors that are very good at various parts of the AI compute world. Fractile’s bet on inference, for example, is to build chips that are orders of magnitude faster than GPUs. That focus has led us architecturally to find paths to huge memory bandwidth, and has potentially taken us away from some of that complete generality.

What’s interesting today, and why we’re so excited about this space, is that the frontier is also an overwhelming portion of the total market. Over the last few years, we’ve seen an enormous build-out of inference compute specifically to serve frontier LLMs. What is in some sense specialist is by no means niche.

That’s the real transformation. Looking back 10 years, or even 5 years, it would have been slightly crazy to build a specialist chip. We didn’t have any single workload that made up more than, let’s say, 10% of the total compute burden. Suddenly we do, and I think that calls for slightly different company structures, with predictive power built in, that try to orchestrate the entire organization around what it means to chase after that.

You mentioned memory and HBM there. Why is the memory wall still such a prominent problem today?

It’s one of those interesting facts about chip physics. If you look at the two things—

All great days start with that line.

I know. If you look across the last 20 years in the silicon world, there are two things that really make up the top-level specs, the top trumps of your chip. One of them, certainly for AI chips, is the flop count: how many floating-point operations you can do per second. That’s a multiply or an add, and it’s the mathematical power of the chip.

The other piece is the properties of the memory on the chip: its capacity and, in particular, its bandwidth. That’s how quickly you can shuffle data back and forth. If you’re looking at a GPU, you have silicon in the middle that runs the math and has a high flop count. Then there are other pieces of silicon around it, the HBM, coming in on a highway that brings data in.

We’ve been extraordinarily good, with Moore’s Law, at scaling flop count. We’ve scaled it about a millionfold in the last 25 years on these chips. Bandwidth, by contrast, is much more intrinsically physics-limited. We’ve scaled it about 40-fold in the same time, so it’s lagged behind flop scaling by about 25,000 times. As a result, we’ve naturally found ourselves more and more bandwidth-limited in many of the operations we do.

For something like LLM inference, this has become a critical bottleneck because of the interplay with text generation. We all have a sense that it’s autoregressive, a sequential process: one word after another is generated by these models. Every time I generate one of those words, I load all the weights of my model from memory over that limited bandwidth, and today I also load all the context I’m operating on—all the past states of things I’ve seen. That’s what becomes my bottleneck on generation speed.

The reason a very large frontier model might generate at 80 tokens per second, or 80 words per second, isn’t the chip’s limited math capability. It’s that there isn’t nearly enough bandwidth to the memory. I can’t load the model’s weights from memory into the processor more than, let’s say, 80 times per second.

The reason I started with the sexy line about physics is that, if you zoom in at the interface, there are essentially copper wires going from this DRAM memory into the processor. The world has been pretty good at making them smaller and smaller, but there’s only so much you can scale that.

And so you have, let’s say, a few thousand wires—

And is that why—sorry, you have SRAM. Am I right in saying that?

Mm.

Moving memory closer to it.

So SRAM, yeah. DRAM is often in a different building, basically. You’re carrying these boxes down the stairs and then up the stairs again into the processor building, where the maths is done. And unfortunately, you have to carry them back again because there isn’t enough space in that building to store all the boxes. So maybe it’s a warehouse, and then you’ve got the office where you’re doing the work. SRAM is like having a bunch of boxes permanently in the processor. It’s a small amount of memory that you cram onto the same silicon where you’re doing your processing.

But that’s the snag as well. Over the last 30 years, we’ve externalized memory into this thing called DRAM and put it on different silicon. They’re built in completely different foundries and specialized for very different things. DRAM is completely focused on density. The DRAM bit cell, as it’s called, is an incredibly simple structure, and you can cram many billions of these into a very small area of silicon.

SRAM, by contrast, is built on the same process nodes. It’s built on the silicon you’re using to build your processor, by TSMC, for example, but it isn’t optimized for density. So you might have less than one-tenth as much SRAM capacity per unit area as you could get in DRAM—and, by extension, per dollar as well, because your silicon wafer costs a certain amount.

When you look at inference and the future of inference, which is better suited? Is it DRAM or SRAM? Latency is important.

That’s right. Latency and bandwidth as well: how much can I cram through? This is one of the things we’ve been working on at Fractile over the past couple of years: rejecting this trade-off. SRAM is the only memory today that can be built on the same processor. It’s the only one that can be incredibly close and have incredibly high bandwidth. DRAM is much cheaper, denser, and higher capacity, but fundamentally limited in how we access it.

If you look at the physics, the fact that it’s coming in over wires is because we’ve got it in the wrong place. We’ve got the DRAM sitting in plane with the chip, and we’ve got this limited beachfront. It’s literally the limited real estate on the edge of the chip. What we’ve been working on at Fractile over the past couple of years is stacking that memory on top of the logic. Instead of having this limited beachfront, where everybody gets a seaside property, the entire memory layer sits on top of the logic die.

How is that possible?

3D is a dimension that people in the chip world have been scaling for a long time, but it comes with all of these complexities. If you look at HBM, or high-bandwidth memory, it’s a stack of DRAM chips. But it has certain properties that don’t make it amenable to just being taken and plonked on top of your logic die. So we don’t do that today. But there are these process technologies, and Fractile’s team has a long history with 3D stacking.

So you’re building condos on the beach instead of beach houses to allow for more capacity?

I guess we’re building one of those new-build archipelagos where everybody has a bit of beachfront. It’s strips of land; everybody’s right next to a piece of coast. When you put this on top, suddenly you have this whole two-dimensional area where those copper wires aren’t going laterally. You have millions of them coming straight down.

When you look at differentiation between other providers, is that your differentiating factor? And how do you analyze the specialized providers that also compete with you in this realm, like Etched? What is the differentiation?

I think there are a few ways to look at this. One is philosophical, and the other is technology today. Philosophically, I do think Fractile is different. We have a mission: to advance frontier AI capability through building better hardware. Today, that’s taken us over the past couple of years to this particular solution. The team has done extraordinary work unlocking a future where you can have the bandwidth of SRAM with the capacity and scalability of DRAM. It’s a world-changing innovation, and it’s going to have longevity as well. We’re very enthused about taking this forward over multiple generations and compounding a technology advantage.

At the same time, with a mission, if the workload changes, you also want to be able to change your response. One thing we’re cautious not to become is the company that cracked 3D DRAM. We want to be challenged to continue to innovate on whatever comes next.

Mechanistically, though, over the next few years, there’s a challenge in the chip space. On the one hand, you have all of these HBM-based chips. At some level, it’s very hard for them to diversify, whether they’re GPUs, TPUs, the Jalapeño chip from OpenAI, Etched, or MatX. They often have brilliant architectures on the chip that let them do things slightly more efficiently than one another. But they’re fundamentally boxed in by the Top Trumps specs: you can get only so many FLOPS and only so much bandwidth. The HBM that sits off to the side is exactly the same piece of silicon—the same module from the same vendors—for each of those chips.

On the other side, you’ve got SRAM-based chips with high memory bandwidth but a huge capacity constraint. If you look at Groq or Cerebras, what they can do today, which is extraordinary, is let you run powerful models much, much faster.

One thing I’ve always been excited about with Fractile, and what I set the company up to do four years ago, is build chips that are better at LLM inference. In particular, we want to build chips that let us run inference much, much faster, because we always believed we’d get to this point where we had what we actually have today: what’s called test-time compute scaling. Reasoning models and long-running agents run longer and longer to get better and better at what they’re doing, and give you a better answer. This is the profile of all the frontier workloads that are changing the world today.

If you look at what happens when you turn a frontier LLM to the problem of solving a Millennium Prize problem, it will go off and generate many millions of tokens sequentially. You scale up this swarm of models as well, so there’s a parallel element to it. But generating many millions of tokens sequentially means you’re waiting days and days to get one of these responses. It’s a space that no silicon or chips we have today have really cracked, because there’s a tantalizing mismatch between the fast inference chips—the SRAM-based chips that can access memory and generate tokens quickly—and the needs of this new workload. A long-running agent needs a huge amount of capacity because really long contexts are emerging. The world urgently needs this up-and-to-the-right combination: high bandwidth and high speed, but also high capacity, so you can scale to longer contexts and larger models.

Is this one of those things that always takes longer than you think, or do you see dramatic inflection points in the acceleration of technology?

What’s exciting about this is that, as is often true of technological breakthroughs, you’re always standing on the shoulders of giants. Some of the technology we’re using for 3D integration, for instance, has actually been around for quite a long time. It just hasn’t been applied to this exact problem: the memory-stacking challenge.

If you’re willing to do the architectural work and redesign your chips to use this, and you’re also willing to do the manufacturing operations legwork and persuade your partners that you’re not crazy along the way, there’s actually the possibility of overnight transformations.

Can I be blunt? I was talking to the founder of a brilliant chip company last night, and he said there’s a huge amount of hot air and BS in the industry. There’s a lot of marketing fluff, but not much actually in production. When you look at companies like Etched or Fractile, are we actually in production? How do we think about what it means to be in production on the ground, and is he right?

It’s a great sentiment.

There are 2 parts to this. Yes, Fractile has silicon that’s gone through the foundries and come back to us. We’re running further trial runs, and we’ll be doing full product tape-outs in a few months as well, which is quite exciting.

I think the scale of production is something very much worth challenging. There’s a real challenge, especially for new chip entrants at the frontier, in accessing TSMC capacity and HBM capacity. In particular, for some of these efforts, there’s a race for exactly the same resource. A lot of these chips today are built with the same back-end partner, for example. Whether it’s the TPU, Jalapeño, MTIA from Meta, MatX, or Etched, all of those are rumored to be using Broadcom, for instance. They use the same or very similar TSMC process nodes and the same HBM technology, so you can see why there might be a real gold rush for that particular finite resource. That means it can be challenging to get to really meaningful scales in a short space of time.

One of the things we’ve always been quite focused on at Fractile is that we’ve had this almost unfair reach into the stack. The point of a back-end partner is that they do the last mile: the physical design, and interacting with the foundries and the manufacturing side of the business. Fractile’s always had that in-house, and we have this extraordinary team that’s always worked that way, even before Fractile, for decades, delivering their chips. It’s become a core strategic advantage for us because we’ve always had direct communication with our manufacturing partners. We’ve had to do that legwork from the outset.

And just so I understand, the alternative is you go through Broadcom, who then have the manufacturing partner relationships.

That’s right.

Got you. So Broadcom is the intermediary layer that most people work with, which you don’t.

And that we don’t.

What does that enable you to do, then?

One thing is to build completely different building blocks for these chips. I was at a conference this summer where Charlie Kawwas, one of the leaders there, held up a LEGO model of the next-generation platform. He said, “This is going to be in production with 6 major customers in 2028.” It’s a strange thing to say, because what he means is that something that looks identical to this is going to be made with 6 of the major players and given different names. It’ll be the TPU, MTIA, Jalapeño, you know—

Dorito 2.

Exactly. It reveals something about the structure of this market. Each of those chips will have a different name. They’ll also be different. There are incredible chip teams inside all of those companies that come up with insanely clever architectural optimizations, but in the end it’s going to be exactly that same platform. It’s going to be the same stack of HBMs, the same number of chips. A lot of the fundamental building blocks are dictated by the back-end partner because it’s a platform technology play. You get to change the color of the paint, basically.

For us, what it’s allowed us to do is maneuver and build these new building blocks. The memory-stacking work, for instance, is incredibly complex and painful, and requires convening multiple, much larger supply chain partners to get them to do something highly custom. But once you crack through that, as we have, you then have an asset that nobody else has. This is a slightly long-winded way of getting to your question about volume production: that also more likely than not means you’ve found a way to use resources that others haven’t. In our case, that might mean a different process node, or a slightly different DRAM node or DRAM generation than what’s going into frontier HBM. So you find the opportunity to build gigawatts of capacity down the back of the sofa, using things that other people in this set of frontier platforms have already moved away from.

Can I ask you about the production line, so to speak? A lot of people told me that I need to ask about tape-out. What is tape-out, what do we need to know, and how do you think about it?

Tape-out is the moment when the final file from your chip design process—which is basically an enormous image—is sent out. We call it a GDSII file, and the image dictates where all of the copper traces are, where the transistors are, and where the logic is. That gets sent to TSMC or your logic foundry partner, and they commence the process of building the mask sets, the stencils that will ultimately produce the chip. Then you start doing your first trial runs, and potentially your first production runs as well.

It’s a seminal handoff for a chip designer, because at the end of this long process of architecture, front-end design, verification, and physical design layout, you send the file. If there’s a bug in it at this point, it’s too late. The mask is getting printed.

Oh.

That’s why it’s a major date on the calendar. It’s also just step 1 in quite a long process, because that silicon then goes through the fab, and you wait months for it to come back. You have to be poised and ready to receive it and run your bring-up process. The packaged silicon returns to you in the lab, and then you sprint to get a workload running on a single chip on a tray, and eventually on a rack as well.

That’s part 2, I guess. At Fractile, we track progress. There are tape-out dates in the middle of it, but the thing we’re continually shooting for, which we call task zero, is the day that a particular reference workload runs on an entire rack of Fractile chips.

What does no one see about this process that’s most prominent to you? If we think about tape-out as a seminal handoff, where is the “God, no one quite understands the difficulty, the momentousness of the post-tape-out date”?

I think that’s it, actually. It’s what comes next, because there’s an awful lot you can fail to prepare yourself for if you’re too fixated on that one thing, the handoff of the first chip to the fab. For us, it’s about continually tracking what we’re actually trying to build. We’re trying to build systems that yield well, so the underlying economics are great, because we’ve designed silicon that has the right levels of repair and redundancy. It’s highly manufacturable, makes a lot of sense as a product, and then fits into a system that also yields well and is reliable.

The goal, in some ways, is to build a product that’s manufacturable in volume, where you can then find a path to ramping as quickly as possible. I think tape-out has, over the last few years, become an accidental shared obsession between chip startup founders and the venture capitalists backing them. It’s such a tangible moment, such a tangible date. But a lot of the work we do at Fractile—all the work we do on building software, all the work we do on systems, and a lot of the silicon team’s work—is focused on the parts of the chip that are, in a sense, intangible, ineffable, but are about getting this stuff up and running, reliable, and to volume as quickly as possible.

When we were downstairs, you said that silicon may no longer be the bottleneck in the same way it was before, and maybe data centers, or data center capacity, are. Can you talk to me about that relationship and the transition from shortage to not?

Yeah, this is why the last 4 years have been so exhilarating: it feels like every quarter, the thing that most constrains our ability to run more frontier models at scale in the data center has shifted. There are underlying properties that we urgently need, like the ability to run inference much, much faster and the ability to run at longer context with better economics. But if you think back 3 years ago, there was no question of other chips. There was an NVIDIA GPU shortage.

Then I think we found ourselves in an underlying silicon and supply chain shortage. There are acute memory shortages today for HBM and so on, which stretch years into the future. There are logic silicon shortages with TSMC as well. If you look at the space today, land and power are definitely another limitation. We’ve had all of the low-hanging fruit for industrial sites that can be retrofitted to be data centers: places where, in the US, you can get permitting to bring gas turbines close by and spin these things up in 90 days. These kinds of often one-time tricks have all been exhausted, and so we’re now exploring the longer tail of harder ways to find land and power to install these chips.

And so, on the one hand, it is data center capacity that is our fundamental bottleneck on how much compute we can build out. If you want to be completely agnostic and invest in the long-term AI trade, buy energy, because that is obviously the most fundamental. But at the same time, it’s precisely that constraint that makes it vital to have the best possible chips for any given job.

If you look at frontier labs and hyperscalers today that have a fast mode on their models, if you’re running that on an HBM-based chip, you’re doing so at an extraordinarily inefficient point on the trade-off curve. On these chips, you can be slow and efficient in generating tokens, or fast but deeply inefficient, with far fewer tokens per second per megawatt of power. So what happens if I want to take a multi-trillion-parameter model and run it at thousands of tokens per second, but I have a 500-megawatt data center and not enough aggregate throughput? I can’t serve my user base on those chips.

There’s real pressure to install chips that retain really good aggregate throughput even as you squeeze them to much higher speeds. Fractile would be an example of that. So although land and power may be the most acute bottleneck, even ahead of supply chain constraints, that’s exactly what incentivizes chips that are much more fit for purpose.

How much of an energy deficit do we have? I know it’s an incredibly macro and bold question. When you look at energy requirements versus energy supply, what do you think the deficit looks like?

Mm.

Going back to what we said earlier about DeepSeek and their incredible innovation on that inference stack, when you look at China’s policy and regulatory environment towards energy, it’s “green light anything.” It’s “green light anything,” including moving millions of people out of their homes because they need the sites for energy provisioning. In the US, you have NIMBYism, and in the UK you have bats and butterflies preventing sites of scientific or natural beauty. Are we not inherently disadvantaged in a world of energy requirements versus China?

I think this is one of the places where China has somewhat stolen the march on the West, particularly with these mega-projects, like huge hydro projects. They’re ideal for AI compute build-outs because they have this particular signature: it might once have been inconvenient to have 15 gigawatts of power all arriving at a single site that then needs to be distributed. An AI data center doesn’t really care where it’s located. Why not put 15 gigawatts of AI compute right next to a hydro plant in the middle of nowhere?

In the US, and particularly in the UK and Europe, there’s a lot we can do to look a little more like that. The good news—and this is something I think governments here and in Europe should do more to highlight—is that a lot of the power sources that could be brought to bear on this are renewables. Renewables can be a huge part of this story. You can locate a data center in Scotland next to an enormous wind farm, where previously there might not have been a reason to have energy generation there.

Do you know what’s really interesting to me? We often talk about this—we do a lot of Instagram now and short-form content.

Mm.

The minute we talk about data centers and them not happening, we did it the other day with Microsoft pulling out of the Nscale project—

Mm.

—and everyone goes—

Good.

“Good riddance.”

Yeah.

“Good riddance.”

Yeah.

“You tech bros, good riddance.”

Yeah.

And it was astonishing to see the universal backlash—

Mm.

—from, bluntly, moronic civilians.

Mm.

Can I ask you: you saw inference as a market before so many other people. I wish I’d listened years ago. The thing I know now that I wish I’d known 12 months ago is just the infinite demand for inference. What do you know now about the market for inference that you think isn’t obvious, based on what you hear other people say?

I think what remains is, in some ways, an even more pronounced version of my original bet. Four years ago, the insight was twofold. One is that as you start to get these foundation models that have been trained on everything and then generalized to anything you might turn them to, you have this hard logic: training becomes a fixed cost, and deployment—inference—is just the use of these models to do useful things. It’s the only part that matters. If you were spending more on training in the long run, why are we even building these things? You really could pack up and go home.

I always describe it to my mother like the Olympics and Usain Bolt. He trains in the gym, great, and then decides, “Ah, I’m not going to actually do the race.” What’s the point, dude?

Right.

This is why we’re here.

Exactly.

Yeah.

Or imagine you spent far more money building a factory than you ever actually recoup from generating something out of it. The hard logic of building something with a high fixed cost is that you must believe you’re going to do enough deployment to justify that.

Yeah.

I think that was part one. Then there was this bet that we would have what we call test-time compute scaling. Reasoning models didn’t arrive until two years later, for which I’m very grateful to Noam Brown and the team at OpenAI, who were the first to publish one of these.

I think what’s happening today is that we continue to see this vector of test-time compute scaling laws. It’s a stack of different ways that we pursue greater capability. One is increasing by 10X, and then 10X again, the amount of compute we use to train these models. But the other is how we pour more compute into deploying these models, on the inference side. That’s a huge part of how you get to greater capability.

If you look at the example of solving a Millennium Prize problem, that Navier–Stokes solution was, I think, 80 hours of execution from 10,000 agents in a swarm. You have these two axes that trace out an enormous volume of token generation: I’ve got 10,000 parallel agents all generating their own chains of thought, and then they’re running for 80 hours.

That’s so different from the intuition we might have about an LLM, where we think we ask a question and the single-threaded thing generates an answer almost immediately. When we bring these models to bear on the hardest problems—and I think the hardest problems are generally also the ones with by far the most positive societal impact, which is a bit of good news—they’re solving a science problem or something in medicine that we maybe haven’t been able to solve before.

To do something incredibly beneficial economically and societally, beyond the reach of just applying the model and asking it one question, “What do you think? What’s the answer to this?”, inference compute is the axis that we have. The Navier–Stokes example is evidence of that. I think the thing that is—

What do you mean inference compute is the axis that we have? It’s the axis that we have to—

It’s the knob that we have to turn to get greater capability. Imagine that at any moment in time we have the best frontier model in the world, from one lab or another. It’s extraordinarily capable. You talk to it and it’s smarter than any you’ve spoken to before.

But when you ask it, “Hi, I’m working on this challenge. We’re really trying to find a cure for this rare disease,” we’re not at a place—and of course we’re not at a place—where it’s just going to say, “Oh, here it is. I’ve thought of this. It’s in my neurons; this has fired out immediately.”

But we also discover that if you throw enough compute at it, if you say, “Hey, how about rather than just trying to give me an answer, you propose many hundreds of candidate potential solutions that you could go and explore.”

And you create multiple copies of yourself, traverse the internet, read the literature, debate with copies of yourself, exchange information, and come back to me in basically a week’s time, once you’ve generated maybe billions of tokens in pursuit of finding an answer. A little bit like if we gave a bunch of smart people a lot of time to solve a problem, you have a chance of coming back with a solution. We really see this. This is the key. It’s what every single application of a frontier model to solve a hard problem is going to look like.

My question to you is: I’m a venture capitalist, I’m an investor, and I see the crunch, the debt cycles, the credit, the problems that we’re hitting now and are about to hit. There isn’t enough money in the world for the requirements of this cycle and everything you just said, which I agree with. Then I look and go, “And we haven’t addressed the physical world, physical data—”

Right, right.

“—and robotics and everything—”

Right.

“—in between that’s required there.” So I’m holding these two seemingly opposing thoughts in my mind, going, “What the fuck happens now?”

Right. Well, this is why we need continuous innovation in every part of the stack that leads to serving intelligence. The good news is that, for all that we’re obviously focused over here on chips, if you look at the algorithmic side, like model efficiency, the cost of serving a given level of intelligence, assuming no hardware change, is coming down precipitously quickly.

Deep, algorithmic-style improvements in efficiency with no hardware change.

Yeah. As we see longer context becoming a challenge, we’re seeing incredible innovations in how you can compress context, do sparse attention, and handle this in other ways. So there’s sustained innovation in what you might call the compute efficiency of the models. With the same chip at ISO compute, how much more intelligence, how many more solutions can I arrive at? And of course, we have the same thing on the hardware side, and we’re working very hard to make sure we have that hardware effect as well.

What’s really thrilling, and the reason I think this build-out is entirely justified and we won’t see a collapse in this market, is that, far from the fiber build-out of 2000 and 2001, we have complete, full utilization of the GPUs that are powered on in data centers. We don’t have dark GPU fleets. As you’ve seen, we have an enormous number of problems as a species that we still have to solve. We now have tools we can throw at those problems, and we have many orders of magnitude more thinking we could be doing at inference time to solve the hardest challenges if we had the compute.

So I agree with you. On the one hand, it feels like we need thousands of times more compute than we have today. We’re not going to get thousands of times more data centers, and I think that’s probably a good thing for the world. What we will be able to do is take roughly the land, power, and set of data center sites that exist today and, over the next few years, bring model and hardware innovations together to squeeze dramatically more intelligence per megawatt and per rack in those data centers.

I had Mike Cannon-Brookes on the show, from Atlassian, and he said to me, “I think the cost of inference is going to come down dramatically over the next three to five years.” That’s why I’m—

He’s pretty good, actually.

That’s why I’m on that. You should—

Yeah, you can do me on the next one.

Well, I mean, you on the next one is relatively close to just me. You want me doing Russian. That really is—

Mm.

—a good one. But with the Mike Cannon-Brookes theme in mind, what happens to the cost of inference?

1. Where Inference Value Accrues

If you look at the cost, what will a token cost in 10 years’ time? It’s one of those questions that begs another question: What is the model that we’re running? It’s probably orders of magnitude larger than the model we have today, so that token is also delivering substantially more capability in some ways.

I think inference margins will fluctuate, and they’ll occur in different places. We will be spending far more on running models in 10 years’ time than we are today. That part is clear. But who’s going to win in this space? Today, you’ve got chips, clouds, frontier labs building frontier models, and folks building and serving open models.

One reason I’m very long on chips in terms of where that value accrues—or where one big part of that value accrues—is the nature of the challenge of placing the right chip bet. It has an intrinsic lag: the time to design the chip, get it out, have it built, and then make sure it has a useful lifetime in a data center. That means you don’t have quite the same fully efficient race, potentially a race to the bottom, in the chip space as you do elsewhere. If you have a better idea, if you can build some sustained advantage, I think you’ll see a lot of the value accrue there.

I think a lot of it will also accrue at the frontier: the construction of frontier models that have a sustained advantage over some of the others. It’s a costly process. It requires incredibly talented teams working with huge amounts of compute. But it delivers something people want: the greatest intelligence on Earth. Of course, people will continue to pay for that. So I think the overall prize pool expands. There will be relative winners and losers within that, but inference is a positive-sum game.

We just had Satya on the show the other day, and he said we’d see a future where every company has its own specialized model with its own unique and proprietary data. A cool value segment could be evaluations and benchmarks for each company and its own objectives. Do you agree with that world of many specialized models on a per-company basis? Do you think it’s relatively techno-optimistic and wrong? How do you think about that?

I think there’s absolutely space for that today. If you look at the technology, there’s a positive impact from taking a model and fine-tuning it for your use cases. But it’s a transient thing, right? Generally, the advantage of having something internally fine-tuned will disappear as soon as the next release or the next open model arrives.

I think that can still become a bit of a “yes, and” situation. As long as you have a model where this construction stacks on top of the best frontier capability, I can absolutely believe there’s an advantage to having an internal, specialized variant of something that’s otherwise constructed at the frontier.

But I do think this is one of the hardest questions to answer, because over time it may well be that you can only do this with open models, for example. If that’s the case, what does it mean for the trade-off? Is it better for me to specialize an open model, or just go all in on the thing released at the frontier, which is still pursuing better compute efficiencies and might, by extension, be very cost-effective for my challenge?

When you look at that value stack, you said chips are obviously the most valuable. Where is the least valuable segment?

I think there’s a danger for anybody working purely in the software layer. I’ve seen a few plays recently in orchestration. There’s a coming wave of heterogeneity in the chips we’re going to have in the world: GPUs, other XPUs, these new chip efforts. Some clouds and software providers have set themselves up to be the orchestration layer over this coming wave.

I think there’s a world in which that kind of heterogeneity is slightly transient. Today, for example, we have a world where Nvidia, having acquired Groq, will deploy two chips alongside one another: Rubin GPUs and Groq chips. They’ll do what’s called prefill-decode disaggregation and attention-MoE disaggregation, where these pieces of silicon are doing different parts of the LLM inference workload. So there’s sophisticated orchestration between the two.

At the same time, it’s kind of like model routing, but chip routing.

Yeah, exactly. Chip routing. I think there’s a technologist’s view that maybe, hey, why stop there? Why not have seven different pieces of silicon? Maybe they’re from different vendors, and they each specialize in different parts of this. I think there’s a belief that this could be a huge wave of value creation for specialists in orchestrating across them.

I think a lot of that could collapse back down over time. There isn’t a fundamental existential reason why we desperately need 7 different platforms to run a workload, and there isn’t really a strong economic case for why we would be able to sustain that either.

If you imagine even just the NVIDIA-Groq data center deployment, how many NVIDIA GPU chips do I put in there versus Groq chips? The bet today would be this attention-feed-forward disaggregation. If you’ll give me a minute on the tech here—

Oh, I will.

—it’s that I put the weights of my model on the Groq chip because I access them really fast, with that high bandwidth. Every user I’m cramming through that data center wants the same model weights, right? So I can amortize and share the cost of that much more expensive memory. I can recoup it and make it up on volume.

Attention, which is the part that operates over the context, isn’t shared across users. My context is different from your context. For that piece, I’m incredibly exposed to the cost of my memory, and we talked earlier about how SRAM is far too expensive. DRAM is much cheaper, so I need to fall back to the GPU and the GPU’s memory to run that piece. That’s the logic of having an NVIDIA data center with these 2 chips today.

But as I spec that data center, how many Groq chips do I put in? What if my context lengths get longer over time? Suddenly, I probably want to have more NVIDIA GPUs than I did last year because the context lengths have got longer. How much longer will they get? Maybe the model gets larger, and I wish I had more Groq chips. You’re very likely to end up in a place where some portion of that capacity in your data center is stranded. You’re just not filling it because you’ve got the wrong mix.

That’s unacceptable.

And it’s completely unacceptable, right?

Yeah.

Because you have a certain number of megawatts, and you’re trying to juice as much revenue as you possibly can out of them. This idea that you’re placing a fixed blend, and you’re not going to be able to change that data center for the next few years—that’s fixed. It’s CapEx. It’s being amortized over time.

Heterogeneity is actually a really challenging thing, and I think it’s a transient response to the fact that today you don’t have a chip that can do it all, right? These SRAM-based chips solve the fast inference challenge a bit, but they don’t have the memory capacity to do attention, so we still need to bring that GPU in. They also don’t have the FLOPs density to run the compute-intensive part of LLM inference, the prefill portion, the prompt processing. You’ve got one piece of silicon that cost you a lot, and you’ve had to share the area you might put maths in with the area you put memory in. You definitely haven’t got as much maths as your counterparty that used HBM, who could use 100% of that chip area for the tensor cores that run matrix multipliers.

If you look at what Fractile is doing, you stack the SRAM upstairs, basically, and have a DRAM layer stacked on top. All of that compute die on the base is now free to put tensor cores down. You can have as many FLOPs as anybody. You can be a top trumps maxer on that. But you also get the SRAM-level bandwidth from the 3D access, and you get the capacity from the DRAM.

Suddenly, you have this holy trinity where you’ve got all of these things in one, and it becomes clear that you don’t need heterogeneity, all this complex orchestration, and the risk of stranded capacity.

Was NVIDIA’s acquisition of Groq an insurance buy?

There are so many fun rumors about this, right? I think the Groq team is incredibly smart, and Jensen is a smart guy who wants smart people on his team. They also built a platform that I think answers—or answered—a big challenge NVIDIA faced at the time: “How do we move into the fast inference space?”

I think Jensen is quite positive-sum in this space. He said some nice things about Fractile last week when he was in London, and is in some ways a bit of an ecosystem player. The way NVIDIA captures the bulk of this market is that they will sometimes be deployed next to other platforms that augment parts of their capability.

There’s another rumor that Cerebras got a deal with OpenAI. I think that can be an alarming moment if you’re running a multi-trillion-dollar GPU company and one of your largest customers suddenly signs on for something you perhaps hadn’t previously thought was existentially necessary: a fast inference platform. What’s your response? I think both deals were signed on the 24th of December, which must have been a fun time to be a lawyer as well.

And so you think it’s a knee-jerk reaction to that diversification from OpenAI?

I think it’s an acknowledgement that some parts of this space at the time required a different solution, and this SRAM play from Groq was the right solution at that moment to run LLMs faster.

Today, though, I think we’ve already seen ourselves move on from this. You cannot solve LLM inference with these SRAM-based chips. There isn’t enough capacity. The cost to achieve speed is too high. The constraint on the context lengths you can operate on is very stark.

I’d characterize that solution as giving you snappier chatbots, but not necessarily this ability to take overwhelming frontier intelligence—these agents that run for days on end—and make it orders of magnitude faster. I think that really is where the huge prize is, and today that’s not something that provides an answer to.

So the different players in the inference space that people know are Modal, Fireworks, Baseten, and Together. If you bet on one, which one would you bet on?

I’m friends with founders of multiple of those companies, so I’m bullish on them all.

But it’s like children: you always have a favorite.

Yeah.

Yeah.

You know what I like about all of these guys? They still have the capacity to dodge and weave, right? They haven’t yet locked up huge amounts of their own capacity. They’re not a NeoCloud. They have the potential to bet on a new chip as it comes up, and to carve out new portions of the market.

I think there’s a world in which this piece doesn’t fully commoditize, right? You can specialize in serving multimodal models, text models, or fast inference. So I won’t place a specific bet. I will say that I’d rather be an inference provider today—a specialist in running these models and getting them up—than a NeoCloud, actually. I think there’s more mileage in being able to create the business flywheel of signing up these enterprise customers that consume these tokens.

Totally get you. Will they be multi-hundred-billion-dollar companies? And is it an Uber-Lyft market, where the winner is significantly more valuable than second and third place, or is it a GCP, Azure, AWS market?

Even that’s only 3 players, I guess. Yeah, I think you’re not going to have 10, and today you probably do have a long tail that takes you toward that. If you add the NeoClouds into that, you definitely have tens of players, all of whom are maneuvering around billions of dollars.

Are they multi-hundred-billion-dollar companies?

I think we’ll have a couple of multi-hundred-billion-dollar companies built in that space.

Can you talk to me about the new funding round? How much did we raise?

It was a $720 million Series C.

Wow. That’s quite a lot of money. How was the process? Was it fun? Because now you’re the belle of the ball.

If you go back 2 years, we were still teaching people the word “inference.” It makes for a trickier investor pitch, right? Because—

When you educate an investor, it’s kind of like, “Leave then,” and—

You’ve got 30 minutes, right?

Yeah.

And you have to spend 20 minutes. Four years ago, you had to spend 10 minutes explaining what an LLM was, and then another 10 explaining why you felt that training these things wasn’t going to be the be-all and end-all, and that you’d deploy them. I’d then spend 5 minutes saying, “And here’s why I think speed is going to matter.” We hadn’t yet seen reasoning models and so on.

Compared to then, it’s been very fun to be able to go straight in and talk about how we have a completely unique and extraordinary team that can do things differently from everybody else, how we have a very unique position in the supply chain and can maneuver differently, and how we have a different set of philosophies on how you build to win in this space. That’s what I get to spend my time talking about over coffee. I think there’s a real advantage to now being in the eye of the storm, in a space where there’s an appreciation of why this is such a crucial problem to solve.

$720 million. Talk to me about whether that’s enough. Suddenly, you have that in your account. What changes? What do you do that you couldn’t do before?

Suddenly, there’s a weight of responsibility, right? A number of folks have staked a lot of capital on us, and I think one of our responsibilities is to ensure that we are always putting multiple shots on goal, as it were.

One thing that capital allows us to do is distribute ourselves to look at multiple things at the same time. We’re now able to start looking more generations ahead. We’re able to explore more parts of the stack than we maybe previously would have. More innovation at the systems layer and the networking layer, and building this entire roadmap of future systems, is one thing that capital allows us to do.

This is an expensive space, but I think the underlying capital needed to get that chip out and get those first systems back has been a fixed cost for the last few years. In some sense, that’s not what most of this capital is used for. I think a lot of this is now important for our ability to maneuver faster in the rest of the world, in the supply chain: placing long-term agreements for certain components that we need years into the future.

Mm.

And the ability to build more of that muscle of predictive power as well. We can now start to think about how we run more of our own experiments and work out what might be important in the future. As I said, Fractile will always be a chip company, but we do need strong insights so that we can be more helpful to our customers about what scales well, where model architectures might head next, how you can handle very long contexts, and what it looks like to run an agent swarm versus one very long serial agent.

All of these are almost ML research challenges that we also get to tackle now. We’re also expanding geographically. We’ve got folks in Taiwan and a team in the States. I’m very excited that just a few weeks ago we were joined by Dan Rabinovich from Meta, who worked on a lot of their AI compute systems, to drive our systems buildout.

That’s everything from the chip upwards: rack-scale systems, networking, and integration with the data center. It’s a very long stack, and we get to build parts of the business where it makes sense to, and pull in the world’s best talent as we go after that.

Can I ask you, speaking of investors, if you could invest in any one of your suppliers, who would it be and why?

TSMC is an extraordinary business. If you look at everything that’s happening today, everything traces back to them. This is not a scoop for your podcast; I think it’s pretty well known that TSMC underpins all of the world’s chip production. It is cheap at the price. I would buy that stock every day of the week.

The majority of the world’s chips at the frontier are built there. They’re an extraordinary business with an extraordinary team, so I think we’d go in there.

If you could invest in any of your customers, who would it be and why?

I really like some of these scrappy neo-labs. Again, I’m thinking like an investor here.

Sure. Yeah, that’s what I want.

They’re a little cheaper, and I think we’re going to see some—

Which one?

Friends of mine have started Inherent in London. I don’t know if you’ve come across them. They’re looking at both the RSI challenge and, in particular, what it takes to build systems that automate science and research.

I think there’s a lot of mileage here. It’s not quite the same as pursuing the best frontier model that goes and tackles everything, but there are so many interesting questions in what it looks like to orchestrate and deploy models, and build systems that are better at solving these challenges.

If you were forced to invest in one of your competitors, final one before we do a quick fire, who would it be and why?

I’m a big believer in 3D DRAM, and another company called d-Matrix is pursuing a different architectural direction in the same space. I think both companies are going to be very successful off the back of that.

What was their last price?

I don’t know.

Are you ready for a quick fire?

Mm.

What have you changed your mind on in the last 12 months?

Maybe I’m not quite ready for a quick fire. Do you want to read me the quick-fire questions, and I’ll just—Is that not an honest quick fire?

Please, can we keep that in? No one’s ever ready for the quick fire. So, quick fire—which we’re definitely keeping that last version in—what have you changed your mind on in the last 12 months?

2. The Next Ten Years

I think the underlying pace of capability advance in models has been extraordinary. The maths results we saw this summer, in terms of the timelines people talk about truncating, have been significant. I think what has become clear is that anything verifiable—maths and coding tend to fall into this category—will be solved in the next 12 months. We will be able to tackle any of these hard, verifiable challenges with frontier models.

Who do you not have on your board that you would love to have on your board?

This is an easy one. I think Demis Hassabis is both an academic hero of mine and a bit of a British hero, and I think he should come aboard.

Have you asked him?

Not yet.

Well, this is a good way to do it.

Demis.

What’s the biggest misnomer about the chip industry today?

Misnomers about technology businesses in general are that they’re very technological. It’s a very, very, very human space. So much of it is about relationships. So much of it, in our case, is about having this extraordinary team that can collaborate across the entire stack. It’s powered by humans.

Will Nvidia be a $10 trillion business in five years?

Yes.

You said that pretty confidently.

It’s a very positive-sum game. Nvidia trades at a lower forward multiple today than Cisco did after the crash, after the dot-com bubble burst. This is a sound bite, so it may be unsourced.

Why is that?

We talked about this bubble before it even started booming, right? We’re so traumatized by the dot-com bubble that we’ve priced in exuberance right from the outset. I think we’re not yet appreciating how sustained this buildout will be.

You don’t have to have incremental data centers added necessarily. We talked about this advance in compute efficiency: if I have a finite number of megawatts, I need to replace my chips every few years with better chips. That’s a huge advantage for Fractile as a market entrant, because every chip you see in a data center today will definitely be out of those data centers in six or seven years, tops. Many of them won’t last more than a couple of years.

That’s an enormous market opportunity for the market leader. They have extraordinary supply chain relationships, and they’re going to do very well. There’s then this enormous, and I would say new, opportunity in being the chip vendor that serves the needs of the frontier. I’m very long on Fractile’s chances over that horizon as well.

You are in charge of the UK. Congratulations. What do you change to make the UK a superpower in technology?

Sadly, so much of what unlocks growth is unlocking energy. It’s doing everything we can to bring more renewables, more nuclear, and more sources of power online, so we can build out the industries that matter here.

Yeah, I love Alan Chang at Fuse. I don’t know if you know Alan, but—

I love Alan.

Yeah, phenomenal. Final one: what are you most excited for about the next 10 years?

Again, we’re using these models now. What are we using them for? We’re using them to solve problems we couldn’t solve before. I think we’re only just starting to see that pick up and take off in terms of what it means for the real world.

It’s going to be all the ways in which life gets a bit better. We’ll find cures for diseases we couldn’t previously tackle. We’ll have high-quality education for everybody. These things are unequivocally going to come. We have the technology to make it happen now.

I totally agree with you. I think it’s so awesome. Look at class sizes traditionally: 40 children in a class. With AI, you have personalized education for some who like multiple choice, some who like visual learning, at their own speed.

I was shit at maths, probably unlike you. I was always too scared to put my hand up and say I didn’t understand because I’d look stupid. Having personalized learning to that point is one tiny example.

Yeah.

I so agree with you. Listen, Walter, I love doing this show because I really get to learn from amazing people like you. Thank you for doing it with me. This has been so much fun.

Thanks very much, Harry. It’s been very fun.