Ep. 030 - Long Live the Short King: Why 4-hi HBM Wins (Memory)

We’re going to talk about why 4-high HBM wins in this episode. I already have Myron laughing. We put out an article just yesterday—I think it was 2 days ago by the time this episode comes out—that called out a few pretty big changes in the industry, everything from the HBM suppliers all the way to the NVIDIA GPUs that this is going to impact, and potentially other accelerators. We’ll discuss and dig into some of the motivations for this change and why people are revisiting the balance between capacity and bandwidth. Myron, how are you doing, man?

I’m doing very well. This was a fun article to write, and I think the thesis has challenged a lot of conventional assumptions about HBM content per accelerator. To give listeners some background, the trend over the last few years has been that every major chip—or AI chip—generation packs in more HBM content per chip. They add more stacks of HBM, and one of the big changes is that they also make the stack taller, with more layers.

Back in the Hopper or Ampere era, 8-high was common, and then with Blackwell Ultra they moved to 12-high. That means 12 layers of DRAM dies in an HBM stack, so you increase your capacity by 50% over 8-high. The presumption from there was that you would keep going taller. The industry was contemplating 16-high for HBM4E—16 DRAM layers in a cube.

When NVIDIA originally previewed Rubin Ultra to people at GTC last year, they said that Rubin Ultra would have 1 TB of HBM per package, per chip. To get there, it would have 4 compute dies and 16 stacks of HBM4E. Each HBM4E DRAM die has 32 gigabits, or 4 GB, of capacity, and they make it 16-high. That’s how you get to 1 TB.

1. Rubin Ultra Cuts HBM

But if you fast-forward a year and a half, what we understand now is that Rubin Ultra is going to have just 192 GB of HBM4. So how do you get from 1 TB to 192 GB? First of all, it’s no longer going to have 4 compute dies; it’s just going to have 2 compute dies per package. It’s also no longer going to be HBM4E. There might be an HBM4E version, but it’s going to be HBM4 at the start. HBM4 is only 3 GB per die, and it’s going to be 8-high.

You’ll also notice that this is even lower HBM content than Blackwell Ultra and conventional, or what we call vanilla, Rubin, which were both at 288 GB because they use 12-high HBM. So why is this happening? The big reason is that it’s primarily motivated by supply.

As many readers are aware, there’s a very big shortage of memory supply in DRAM and NAND. Let’s talk about DRAM. There’s very little wafer capacity being added, and as the number of accelerator shipments goes up, the amount of DRAM and HBM demand keeps going up. It’s very hard to keep up with that demand.

What the suppliers and customers like NVIDIA, Google, and Broadcom have realized is that we have a certain amount of HBM secured, and we have this much logic secured at TSMC. If we take the HBM supply we have and ship it in 12-high cubes, it isn’t enough supply to ship all that logic. So let’s ration our HBM supply and ship it in 8-high cubes. You get more cubes from the implied wafers, and that helps balance the equation. That’s what NVIDIA has realized.

Of course, does downgrading from 12-high HBM to 8-high HBM have performance impacts? How does that affect performance and TCO? In terms of how important HBM capacity is for inference specifically, there’s a threshold that you need. You need enough HBM to hold the model weights and enough capacity to support KV cache for a lot of users so that you can batch effectively.

But once you get beyond that threshold, the returns diminish. Because HBM is getting much more expensive next year, as the shortage is reflected in prices, the cost of additional HBM that you might not need is also much higher. The penalty for having too much HBM capacity becomes much worse.

2. Capacity Gives Way To Bandwidth

Okay, let’s take a step back and talk about capacity versus bandwidth and the trade-offs in a little more detail. It seems to me that, going into the Rubin generation, from a design perspective and when the announcements happened, people were just starting to use Blackwell. There was a lot of talk about 10-trillion-plus-parameter models, going to 20 trillion, and people were talking about NVL72, NVL144, NVL576, and all sorts of other rack configurations, which really seemed to be about the models getting bigger.

To me, if models get bigger, that means you need more capacity. But that assumption may not necessarily be true today, because we can get a lot more performance out of existing models with the same total parameter counts by using a lot of different strategies in terms of reasoning, looped transformers, and other techniques in both training and inference. Can you talk through the trade-offs between capacity and bandwidth, and why NVIDIA may have previously designed Rubin to be the biggest, maxed-out version of everything, but we think they’re going in the direction of making the exact right tool for the job?

Yeah, exactly. If we take a step back into what models look like compared with what systems looked like a few years ago, in the Hopper era, if you have a Hopper HGX node with 8 GPUs and 80 GB of HBM per GPU, that’s around 640 GB in the server. At the time, Llama 3.1 405B was probably the best open-source model. That would take 405 GB if you quantized it to FP8, so it would take up around 60% of the server’s available HBM, assuming you kept the deployment at that one scale-up node.

But now what’s happened is that parameter counts have scaled, but probably not as aggressively as we thought. The biggest model today is Kimi K3 at 2.88 trillion parameters, which is almost 6 times—actually, a bit more than 6 times—the size of Llama 3.1. But we can quantize that to MXFP4, which reduces the capacity needs by half.

If you compare it with the biggest system right now, the NVL72 GB200, which is 72 GPUs with 288 GB each, one set of weights for Kimi K3 stored on one scale-up system is around 8% of the available HBM capacity. So parameter counts have scaled, but not as much as we expected. Meanwhile, the deployment sizes—the size of the system on which you want to serve a model—have grown a lot, so the capacity pressure is much lower.

If you go back to Llama and the Hopper HGX system, the upgrade to the H200, which used HBM3E with higher die density, gave a lot more valuable breathing room to serve Llama on one HGX system. That was why it was more important back then to increase HBM capacity.

Now you have a lot more breathing room, and whether you really need to use all of it isn’t as clear-cut. Going back to the decision NVIDIA made with Rubin Ultra, they’re also going to scale the system to NVL576. That’s another 8-times increase in the scale-up size. So even though they go from 12-high HBM to 8-high HBM, they increase the scale-up size by 8 times. That’s a lot of aggregate HBM capacity within that domain.

Yeah. To harp on that point about capacity versus bandwidth, it seems like today a lot of people are focused on increased performance from existing models at existing sizes. They want faster versions of Kimi K3, and they want faster versions of 5.6 six soul[?]. To be clear, just stacking more capacity on the existing bus does not address the problem of making models go faster.

Yeah. Speeding up tokens is all about bandwidth, basically. Capacity doesn’t really add anything. To your point about why model parameter sizes haven’t scaled as much, there are all these other techniques that can get performance from models rather than just pure size scaling. There’s more reasoning and a lot more post-training and reinforcement learning.

The other point is that post-training is very inference-like, so it also tilts toward more bandwidth. We can see that the amount of compute that’s more inference-sensitive—in the blue and white bars, post-training and standard inference—has quickly dominated. Meanwhile, classic pretraining, which does require more capacity, has become a much lower share of the total compute pie in terms of where it goes for frontier labs.

Yeah. And our Tokenomics model is tracking this in great detail, sourced from a lot of work from the data center model that the guys have for tracking sites. But it's pretty clear to tell if a given site is targeted for pre-training or not because of the implications for the network and how big the individual building needs to be. If you're going to have more than 100,000 GPUs in an individual building, or an interconnected campus of multiple buildings, that's a very different data center design compared with 2,000, 4,000, or 8,000 GPUs spread across 4 data halls and then connected all around the world.

It's clear that as the labs bring on more compute in a capacity-constrained world right now, the marginal data center that they bring into their fleet is not going toward pre-training. It's going toward—

Research, post-training, inference, and some collection of other—

Yeah, not—not pre-training runs for that 20-trillion-parameter model that we were thinking about 2 or 3 years ago, right?

Yeah.

Just 2 or 3 years ago, right?

So—

Exactly. So why don't we talk about maybe the specifics on the relative price? I think this is something you mentioned right at the beginning. If you could just explain it once again in a different way: The idea is that going from 12-high to 8-high to 4-high gives you a different ratio of bandwidth to capacity. If they have a constraint on the total amount of bits of memory that they can put into a given system, with, let's say, an unlimited number of packages, then marginally you're not going to want to stack more capacity or have more bandwidth, and therefore the individual stack is smaller. To finish the thought experiment, physically we can't go lower than 4-high, right? We can't go to 1-high or 0-high. So what's the limit here?

Yeah. HBM bandwidth is pretty much fixed within each stack, or cube, regardless of the stack height. Put another way, you need at least 4 layers of DRAM to get the full bandwidth out of a cube because the bandwidth is driven by the interface between the HBM and the SoC. For HBM4 and 4-high, there are 2,048 I/Os—data wires between the cube and the compute. Depending on whether it's 4-high, 8-high, or 12-high, these I/Os are divided across the stack.

Each die can support up to 512 of these signal I/Os, so you need 4-high to maximize utilization of the whole 2,048. Of course, suppliers charge you for capacity, so customers effectively pay on a dollar-per-gigabyte basis for HBM. The bandwidth is the same, so if you're paying for more capacity with 8-high or 12-high, you're paying much more—almost double or triple what you pay for the same 4-high HBM cube.

If you're really bandwidth-focused, you're getting the same bandwidth from each of these cubes regardless. The dollar-per-bandwidth proposition is much better at 4-high. Paying for 4-high only, when all you really need is the bandwidth and 4-high capacity gives you enough, is almost a free lunch.

So, can you talk a little bit about the different hardware architectures that may be taking advantage of this? Obviously, the headline for this entire podcast is the fact that we're making it very clear to everybody with a nice big chart up front that Rubin Ultra was going to be 1 terabyte, and it's now being revised to 192 GB. That's less than the 288 GB that's shipping today that people are using in GB300, NVL72, and what's coming in Rubin this year. It's the first time in the history of me keeping track of Nvidia that the next frontier GPU—their flagship GPU—is going to have less capacity than the previous one.

That's just a little mind-blowing. Are you expecting to see this? You didn't call any of this in the article, right? But at a high level, are you expecting to see other accelerator vendors follow suit if Nvidia is doing this?

3. Other Accelerators Follow Suit

In short, yes. I think the capacity challenges, or the supply chain challenges, that Nvidia faces are the same for everyone. A lot of road maps have been reset to factor this in. They want to maximize the number of accelerators that they can ship, and if they can make the compromise on HBM capacity per accelerator to make this happen, I think it's the most logical path going forward.

Especially given that it's really bandwidth that's important, and HBM has always been expensive and is going to get more expensive next year as well. In summary, a lot of customers' road maps have been reset to favor lower-height HBM instead of 12-high.

Everybody is obviously looking at how this is going to impact both the accelerator vendors and the memory vendors. I know that we've kept some of that behind the paywall on the memory-supplier side, so we won't get into that in detail. But you made a couple of things public that are not behind the paywall, specifically saying that this is going to have an impact on DRAM supply in the typical server supply chain because effectively the amount of HBM cubes that are harvestable compared with 8-high could more than double.

That's the call. I think doubling makes sense, but why would there be more than double the amount of HBM cubes that are harvestable at 4-high? And maybe you can talk a little bit about the downstream impact on the rest of the supply chain at a high level, like logic wafers, substrates, PCBs, and things like that.

HBM4—basically, the taller the stack, the harder it is to manufacture that finished cube. As you stack each die in the module, each additional layer has some yield loss. It's not a 100% yield for the process of building this module. If you compound that by 8 or 12 times, even with 99% yield for each layer, that 1% yield loss compounded 8 or 12 times is much higher than the loss if you compound it only 4 times, which is what you get from 4-high.

There are also other factors. In terms of being able to send power up that stack, it's much easier with a 4-DRAM-die stack than with 8 or 12. Overall, you get more than double because the yields are much better than with 8-high or 12-high. At a high level, 4-high is a product that has a longer manufacturing history and is simpler to manufacture. They're going back to—

It's simpler to manufacture. There are fewer steps to making the stack.

Fair enough. So, moving the bottleneck away: You have a bunch more HBM cubes, but the bottleneck is now no longer wafers, and there may be other bottlenecks in logic wafers, substrates, or PCBs that you might focus on.

Let's say everything goes to 4-high, and then we can get double the cubes compared with 8-high, or more than double the cubes compared with 12-high. I think the bottleneck moves away from HBM into probably logic—leading-edge logic—in terms of whether TSMC can support enough logic wafers to co-package alongside all this HBM.

Right, double the HBM—

Yeah, you can double the amount of logic capacity required—

You know, that's probably not quite—

And then there's also other parts of the supply chain. One of the tightest areas is substrates. Again, if you can double the amount of substrates from the baseline, that's going to be a big ask. Then there's also, of course, power, which we've spoken about a lot as being a big constraint.

It doesn't matter if we can't maximize the use of all this HBM. It's not only HBM that's tight. Conventional DRAM is very tight, and we're seeing a lot of de-spec'ing, especially in servers—just taking down the amount of DRAM per socket. Again, that's because there isn't enough DRAM to go around, and a lot of that is driven by DRAM wafers being cannibalized for HBM production.

If we reduce the HBM wafer demand and free up more capacity for server DRAM, that's also great for the whole industry. When it comes to GenAI, it's not just AI accelerators that we need; we need a lot of CPUs just to perform all the tool calls, and so on. A lot of the time, we're also bottlenecked by waiting for a CPU task, right? That's why CPU demand is so high, but the CPUs are probably going to be poorly utilized because they don't have the right amount of DRAM to support them.

So that's also another benefit of relaxing HBM constraints: You can use those wafers for commodity DRAM. But another way to look at this is that we're in a DRAM-constrained world. In terms of maximizing tokens per HBM wafer, 4-high is how you best optimize for that, in the same way we're talking about tokens per watt and tokens per dollar, because watts and dollars are valuable and scarce resources.

HBM wafers are valuable and scarce resources. DRAM wafers are valuable and scarce resources, too. If you want to deliver the most tokens in aggregate, 4-high is how you do that as well. So maybe we can end on a burning question that I always get here, which is related to how people actually expand the supply of memory.

It's in the news a lot, right? Whenever people say we are constrained by memory, not by logic from TSMC—not always from data center space, although there are local constraints—it's clear that industry-wide, all accelerator vendors would be able to produce more than they're producing today if they could get their hands on more HBM, assuming they have HBM in the package—generally speaking, HBM or DRAM for the server.

However, you have to juxtapose this with the fact that we're also seeing stories about SK hynix printing massive profits, people going to the local Lamborghini dealerships the day after paychecks are cut and buying out all the cars and driving around, and Samsung's doing well, and Micron's doing well. So when people say, “Okay, do you have a clear answer for why these memory vendors, which have raised prices into this massive demand shock and made a bunch of money, can't just produce more? Why can't they increase supply and respond to the demand signal faster?”

4. The Memory Crunch Persists

I mean, on that, I think it all comes down to semiconductor manufacturing having fairly long lead times. There are so many things that you need to do before you can just—you can't just click your fingers and add wafer capacity. If your lines are fully utilized, you need, number 1, clean rooms, which has probably been the number 1 reason why we haven't had short-term capacity being added.

You need to build these clean rooms, and then you need to fill them up with equipment. That equipment is also very limited because, at the very least, you need EUV tools from ASML, and they can only manufacture so many of them a year because they have their own supply chain that's very long. To add more tools, you have to bring up suppliers who do a very specialized thing for them, like making very smooth mirrors, and that takes a while as well.

So unfortunately, it's just these physical constraints that prevent capacity from coming on demand as we need it.

So what's the high-level forecast? Everybody keeps saying, “When's the HBM or memory crunch going to ease off?” What's the current timeline you're telling people is the earliest time at which we could start to produce a lot more memory and have everybody—

I mean, we don't. I think we subscribe to the memory model to find more, but I'd say the TL;DR is not—not within this decade.

Not within this decade. 2030, here we go. All right, Myron, anything you think we missed talking through in this article?

5. When Four High Falls Short

I think the counterargument is: At what point is 4-high not enough? I think that basically comes back to model sizes and how many parameters they have. If model sizes explode a lot, then the economics start to really favor higher stacks. It's less favorable to go 4-high; that sort of additional capacity that 8-high gets you is much more valuable.

The reason it is more valuable comes down to batching economics. With batching, for every user, you only need one read of the weights, and the larger the weights are, the more efficiency gain extra concurrency gives you. That's when additional capacity is more valuable.

We also did the same analysis for Kimi K3. We modeled something like 3× Kimi K3. If you have a model that's 3× the size of Kimi K3, then the throughput gain from 8-high and 12-high is much better and is probably worth the cost.

The counterargument would be: If everyone goes to 4-high, do they shoot themselves in the foot and have a suboptimal system in a world where these model sizes grow to be huge? Again, this goes back to model sizes. What we're observing is that parameter sizes aren't really increasing that much.

There are so many techniques that are trying to keep that footprint small. Looped transformers is a relatively recent one, and it's basically confirmed that GPT-6 Astra uses looped transformers. Instead of adding parameter count, the input goes through the layers more than once, so you add compute depth without increasing the size of the model.

The other point is that we're seeing the loudest cries for 4-high coming from the labs. The labs are probably the people who are best positioned to say which way models are scaling. I think that's also a tell that they're not seeing parameter sizes scaling as aggressively in their road maps, in what they find in their research.

Yeah, look, at a simple level, I've made this point previously, but it would be very embarrassing for OpenAI and Anthropic if Kimi K3 was this close to their performance.

And it's sitting there at 2.8 trillion parameters, and they're at—

10. Exactly.

Right. Yeah. It's kind of implied that the models are of similar size to Kimi.

Yeah, yeah, today. Now, let me push on this a little bit further. When you think about the future of these accelerators, do you think it's possible that we reenter this world where there's multiple SKUs for an accelerator? It starts at a certain amount of memory capacity, and then maybe a revision comes later that's bigger.

I'm thinking back to V100 launches at 16 GB and then going to 32 GB. A100 launches at 40 GB and goes to 80 GB. H100 starts at 80 GB, and H200 is basically a revision of it to 144 GB. There's this trend of having these minor revisions to accelerators, but they're producing so many of them and the customers are so big. Can't there just be multiple SKUs? Can't somebody have a 4-high SKU and then somebody else buy an 8-high SKU of the same base logic die?

You've got the Meta flavor of the Rubin Ultra, and then you've got the OpenAI flavor of the Rubin Ultra that have different—

Yeah, absolutely. We're already seeing that, right? I know that, for instance, Meta has had a lot of custom or semi-custom SKUs where they'd have different memory from the mainstream configuration.

We're seeing, for instance, for MI450, there's a custom Meta version that has 8-high HBM instead of 12-high. So absolutely, I think more SKU segmentation in a world where supply-chain capacity is really tight makes sense. It becomes much more expensive or costly to overprovision certain resources if customers don't need them. So, yeah, I think more SKU segmentation is absolutely on the cards.

Makes sense, man. Well, we're on a trend. Everybody wants to understand more about the accelerators and the models and how everything all works, so it's going to be a fun couple of years as this stuff gets produced.

I'm actually fascinated that it's only happening in the Rubin Ultra generation. Everybody kind of got away with these 288 GB Rubins, and nobody is pushing that hard for the current generation to change at all. It's just the stuff that's coming this time next year or into 2028 that we're actually going to see all this, which maybe is an important thing we didn't make clear up front.

Rubin Ultra is the, you know, if there was an R200, it would be the R300.

So it's the next version after the stuff that's getting put in right now.

Okay, man. Good job.

Okay, cool. Thanks.