Ep. 003 - Deep Dive on NVIDIA Vera Rubin VR NVL72 (AI Supply Chain) | Jordan Nanos, Myron Xie, Copper Wei (Wega), Howie

Hello, everyone. Welcome back to episode 3 of the SemiAnalysis weekly podcast. I'm Jordan Nanos. I'm here with Copper Wei (Wega), Myron Xie, and Howie, the undisclosed Howie from an undisclosed location. Copper's also got his glasses on; he's ready to play poker. Myron's back.

This week, we're going to talk about the article that we put out on Vera Rubin, “Extreme Co-Design,” and walk through a lot of the updates that we're expecting this year. We'll talk about sparsity, peak marketed FLOPs, the changes from Grace to Vera, the system server side, and how everything's going cableless. I've got 3 experts here to talk about it, all of whom are working on our accelerator and supply chain team for AI. Welcome to the show, guys.

Good to be on.

Thank you, Jordan.

Thanks, Jordan.

All right, let's start by talking about Vera Rubin and the concept of Extreme Co-Design. Obviously, Rubin is the new GPU that's going to replace Blackwell. There have been a lot of improvements. Can you give me a high-level overview of what's new with it and how you would introduce these changes?

1. Rubin Changes The Compute Equation

Howie, we have a Rubin floor plan. If you bring that up, we can see a lot of interesting things. What's interesting with Rubin is that, compared to Blackwell, it has 2 big compute dies and 8 stacks of HBM, with HBM moving to HBM4. We also have the I/O chiplets being disaggregated into separate chiplets on the sides for NVLink C2C and NVLink 6.

This comes with a big improvement in memory bandwidth and peak marketed FLOPs. For FP4, it goes to 35 petaflops dense versus 15 for Blackwell Ultra. They also have 50 petaflops sparse, which is interesting because this is a different type of sparsity. Howie, can you talk a bit about the adaptive compression engine that we see in Rubin and how it differs from the sparsity that NVIDIA has marketed previously, in the Blackwell and Hopper generations?

Sure. In this example, NVIDIA is marketing up to a 5-times increase in sparse FP4 performance compared to the Blackwell GB200. What they're actually comparing is sparse on Rubin versus dense on Blackwell, and they're able to claim that because of this new hardware compression in the Transformer Engine.

Instead of doing structured sparsity, which is what has been done before, where you force every other data point in your tensor and matrix into zeros, the Transformer Engine now directly looks at the data stream and dynamically compresses it. That allows you to deliver up to 50 petaflops of effective compute while the actual processing is done at 35 petaflops.

It's almost like a hybrid approach where you don't have the accuracy losses of forcing the data to be zero, while also achieving some level of speedup. We believe this is important because structured sparsity wasn't really used at all, especially at the lower precisions down to FP4. The models wouldn't converge, and there were a lot of accuracy issues. With this new adaptive compression, we believe you can effectively reclaim your sparse performance.

Exactly.

Okay, cool. Let's take it back a step just to do some history. We've been dealing with Jensen math for 4 or 5 generations now, since sparsity was introduced as a concept. People would double the peak theoretical FLOPs for a given data type: FP32, then FP16 with the bfloat16 data type, and then FP8. Now we're at FP4, with different variations, and NVIDIA obviously pushing NVFP4.

The peak marketed FLOPs of 50 petaflops is a huge generation-on-generation increase that you might think is attributable to this Jensen math with sparsity. To be clear, the difference with this one is that we expect this sparsity to work, or to be useful, in a way that it previously wasn't.

Exactly.

Okay, cool. What else? We have the picture on screen for those who are just listening. The point about the 50 petaflops and the sparse NVFP4 points to the SMs within the GPC on the GPU, but there is plenty of other stuff that makes up this GPU. We have PCIe interfaces, NVLink C2C, NVLink 6.0, and the HBM controllers with 288 GB of HBM4.

Let's go through that one by one, starting with HBM. The 2 fundamental pieces here are FLOPs and memory bandwidth. We don't actually have a firm specification for the HBM4, depending on the SKUs, but it's still a significant increase over the Blackwell generation.

2. HBM4 Creates A Supply Bottleneck

What's interesting with this generation is that we've moved from HBM3e to HBM4. The density, in terms of capacity, of HBM3e and HBM4 is the same as 3e, but there's a huge increase in memory bandwidth. The reason is that you effectively double the number of I/Os per stack to 2,048 versus 1,024, which was the case for HBM3e and all the HBM generations before that. You have double the wires.

What's happening is that NVIDIA is pushing suppliers to offer HBM4 with the same pin speed. That effectively gives you approximately double the bandwidth per stack. In this case, it's more than double. NVIDIA wants to hit 22 TB per second of memory bandwidth for Rubin, compared to 8 TB per second per chip for Blackwell. That's about 2.5 times the memory bandwidth.

The interesting thing is that the original specification for HBM4 from JEDEC—the committee that sets all the standards for memory—was much lower in terms of speed than what NVIDIA is pushing for. As a result, suppliers—Samsung, SK hynix, and Micron—have different levels of preparedness for these speeds.

We've seen some specific choices in the architecture of HBM4 from Micron and, to an extent, SK hynix that may not have been correct for hitting these speeds. Notably, one of the big differences is in the base die of an HBM4 stack, which is where the PHY that interfaces with the host XPU is located.

Micron has chosen to stay on a DRAM process for the base die, which has relatively poor-quality transistors. SK hynix chose to adopt an N12 logic process from TSMC, while Samsung is going all out and has chosen to use its own Samsung 4-nanometer process.

As a consequence of these choices, we're seeing some of these memory suppliers, particularly Micron, having trouble hitting the speeds that NVIDIA wants to reach 22 TB per second. I think it's quite likely that a lot of Rubin will ship with lower-than-advertised memory bandwidth.

They're going to bin the Rubin GPUs effectively based on which HBM supplier they get capacity from for it.

Yeah.

Super interesting. What do you think about the design decision to stick with 288 GB of density compared with the Blackwell Ultra, such as GB300?

There are a few ways you can expand HBM capacity. You can add more stacks, which is probably not possible given the packaging and die-outline constraints here. The other way is to go to a higher layer count in the HBM.

The thing is, we haven't really seen 16-high stacks, which would be the next higher tier of stack counts. Twelve is probably the default now, so we haven't seen any of the suppliers able to ship 16-high stacks in volume. As we increase those stack counts within an HBM cube, it's becoming a greater packaging challenge and getting more and more difficult.

With HBM4 and 3e, the DRAM density within each layer is the same. In some generations, you get a density increase. For example, with HBM3e, each die had higher density than HBM3. But between HBM3 and HBM4, it's the same.

Part of that is because you need to dedicate more area to the TSVs, which goes back to the wiring point I mentioned earlier: doubling the I/Os has an area implication. Therefore, with HBM4, to keep die sizes in check, the density isn't really increasing. That's why, for Rubin versus Blackwell Ultra, the 288 GB of HBM per chip is static.

Now, obviously, we're talking strictly about the R200 in these cases, right? There is a—

Yeah. Although—

Rubin Trail—

Although the R200 terminology is not what we used.

It's just the Rubin GPU.

Okay, my fault.

Drop the 200.

Fair, fair enough. Yeah.

They don't like the 200.

Yeah.

Oh, yeah. Okay. But there will be a Rubin Ultra in addition, which—

Yes.

Yeah, it takes us all the way up to 1 terabyte of memory—1024.

1 terabyte. So, when you think about the Rubin Ultra, how they get there is they double the stacks to 16 from 8. So that's 2× scaling. They want to use 16-high HBM4E, which brings the capacity of each stack to—is it 48 gigs? Sorry, no, 54 gigs—versus, say, only 32 for a 12-high stack of HBM4.

So that's how you get a quadrupling of memory capacity from Rubin to Rubin Ultra. But I think the question is, DRAM—HBM and conventional DDR—is very tight. I think XPU designers are probably thinking about supply-chain realities and, to the extent that they can secure memory, this may mean there's more rationing or skewing of different XPUs, with perhaps lower capacity than initially planned.

Cool. So, what do you guys want to talk about next? Do you want to go to NVLink-6 and talk a bit about packaging? Wega, we can bring you in there. We can also talk about—

We can talk about NVLink-6, and what's interesting is the bidirectional SerDes.

Yeah, yeah, let's do that. Because I think there are basically 3 ways in which these GPUs change significantly from generation to generation. One is the tensor core, measured in FLOPS, roughly. One is the memory bandwidth or memory capacity—HBM—measured in gigabytes or terabytes now, or per second for the bandwidth. And then the last one would be the interface—

Yeah, so—

And then the last one would be the interface—

3. NVLink Changes The Scale Up Design

So, on the NVLink side, the big upgrade from Rubin to Blackwell was the doubling of the bandwidth of the NVLink domain. But the NVSwitches are not upgraded—the bandwidth per switch isn't upgraded. That means they have to double the number of switches per system, per rack. So, the number of NVSwitches goes up from 18 per rack to 36 per rack. Then we have 9 NVSwitch trays, with 4 per NVSwitch tray.

However, there's another major upgrade that they made here, which is the bidirectional SerDes. This essentially allows you to double your bandwidth without doubling the wiring—the physical connections between these switches and GPUs. They've kept the physical connections, which is the backplane, with the same sort of hardware as Blackwell. They've achieved that with the bidirectional SerDes.

How they've done that is, NVIDIA claims it has 400G SerDes, but instead of sending, let's say, 200G SerDes on a physical wire in 1 direction, each physical wire will have 200G SerDes going in both directions. So, essentially, with that, you double the bandwidth with the same amount of physical wiring.

Yeah. And this was another source of Jensen math in the past, where people would state the theoretical bandwidth as including bidirectionality, when that's not really how anybody uses it. So, this is another case of the hardware catching up to what some of the marketing specs have said in the past, or being moved again to increase the theoretical amount of bandwidth that you can get out of a given switch ASIC and the connections on the GPU itself.

Right. And because of the bidirectional bandwidth, it's marketed as doubling 200G SerDes, right? But in reality, the real performance because of this special setup should be lower, because there might be some real-life—

You're never going to get 400 gig—

Yeah, yeah.

—unidirectional on a link.

That is the best-case scenario, but it's unlikely they're going to hit that.

Sure.

The other interesting thing about the NVSwitch is they've doubled the number of switch chips. To double the bandwidth, instead of doubling the bandwidth per switch, they've just doubled the number of switches. So now, as you can see here, there are 4 switch packages per switch tray, and that means that each link between a Rubin GPU and one of these packages is just a single 400G link, as compared to multiple ports in the previous generation.

I think this was done mainly to keep the switch die as a monolithic, single piece of silicon, because the previous generation was already at the reticle limit—over 800 square millimeters. So, for them to double the bandwidth again while keeping it in the same package would be really challenging. They might have to go to a chiplet design and increase the complexity too much. So, this is the design decision that they went with.

Makes sense. Now, obviously, the thing that anybody who's worked with a GB200 or GB300 system would be thinking about is that backplane and cabling. But that's a big change as well.

4. Cableless Design Simplifies Assembly

There isn't much change on the backplane side. It's the same backplane from Rubin to Blackwell. But the changes that you're mentioning here are mostly on, as Jensen said, the cableless design. That is mostly within the compute tray. Howie, could you pull up a picture of the compute tray?

If you're trying to run twice as many cables to the NVLink switch because you've doubled the number of switches per rack, you would think that there's double the amount of cables. But in fact, there are 0 cables in a compute tray for the NVLink scale-up domain at this point.

There are still cables in the backplane, but within the compute tray and within the switch tray, there are no cables.

Yeah.

But the cables that are removed are actually not the NVLink protocol ones. They're the Ethernet and PCIe ones.

Oh, okay.

As you can see, this is the compute tray of the Vera Rubin. It's a very seamlessly integrated design with a very modular approach. You have 2—we call them the Strada modules—which are the superchips that house 2 Rubin and 1 Vera. Then you have 2 of them at the back of the chassis, and you have the networking modules at the front of the chassis.

With this design, all the signals within the compute tray run on the PCB. There are no more cables. The reason for that is because the initial challenge with GB200 manufacturing and assembly was the very low yield with the cabling. You have workers routing these cables in very tight, dense spaces. You have fans in there, too, and 10 or 20 cables within a very tightly compact footprint.

Oftentimes, these cables are scratched where the cable connects to the connector. That's the most vulnerable part of the cable, and they're scratched during the assembly process as people are rushing assembly. It's also very difficult to route these cables. Whenever they're testing the system, they'll have to go back and figure out which cable is causing connection problems. They have to go back and debug.

With this approach, the cables are removed. It takes human error out of the assembly process. As Jensen said at CES—sorry, not at GTC, at CES—he mentioned that there is an automation process for this assembly, which can reduce the assembly of the compute tray from 2 hours to 5 minutes. That's an incredible step up.

As we understand it, there are only 3 vendors that can do the automation or are working with NVIDIA on the automation. But even without automation, this approach should be much easier, given that you're just putting things together. It's kind of like building Legos, right? You're just putting things together instead of putting them in place and then routing cables.

This all goes in. There will be blind-mate guiding mechanisms, and everything should go into the place where it's designed to be. So, yeah, the big one here is PCB content, of course, because all the signals run on the PCB now.

Given the distance the signal has to travel on the PCB, the material needs to be upgraded. We have 2 drivers of PCB content: material increases and PCB-area growth. As you can see here, the Strada board is slightly larger than the Bianca board of the Grace Blackwell generation.

You also have a midplane between the Strada and the front networking modules that acts as the bridge between the modules. They’re connected via that midplane. At the front of the chassis, there are 4 Orchid modules that house the CX-9. The PCBs don’t just cover the compute tray at the front; they’re stacked on top of each other.

The PCB area in Vera Rubin compared to Blackwell, I believe, is 2 times or more. We disclosed that in our BOM model estimates. It’s a significant step up in PCB area, and the material upgrades are incredible too, so the signal can perform over that distance, which was previously covered by cables. Cable has better signal performance, but now everything has to run on the PCB.

This cableless design is pretty clever in terms of maximizing manufacturability. Is NVIDIA the first vendor to do this?

Actually, Trainium is the first hardware system that adopts this cableless philosophy. They’ve advocated this at re:Invent for 2 years now. Trainium is the first design, and they don’t have this modular approach. They have just 1 massive baseboard that spans the entire chassis, with the signals running through it.

Amazon claims that it was a time-to-market decision, and NVIDIA later chose a modular approach with the same PCB and cableless design philosophy.

I guess the trade-off there may be in terms of serviceability or reliability. You have issues with the heat?

I think the serviceability is better here. The trade-off is probably cost, as PCBs are getting expensive with these material upgrades, higher layer counts, and worse yield. As you get to higher layers, these are state-of-the-art PCB processes.

The midplane uses a material called quartz fiberglass. It’s a state-of-the-art material for PCBs and PCB layers, and it’s very challenging to process because quartz glass is so hard. That will lead to worse yields when drilling the PCB.

There’s a debate right now about whether they will de-spec from quartz glass, and whether the next, lower-generation material—which is just the higher-end glass-fiber cloth—can have the same signal performance. That is currently under debate.

Cool. Awesome. Does that wrap up the system stuff, or was there other stuff you wanted to talk about on the packaging?

Myron’s on mute here, but yeah.

Yeah, packaging, not really. I would say the packaging is broadly similar to Blackwell, except you have a few more chip I/O chiplets on the sides. That increases the interface size and the substrate size.

Technology-wise, there’s nothing too new. I’d say NVIDIA is more conservative on the packaging side, whereas AMD is pushing the technology on the packaging side, given that they have implemented 3D bump-less stacking, or hybrid bonding, which is SoIC according to TSMC’s trade name. They’ve been doing that since MI300X.

I think we’ll start to see more and more implementation of hybrid bonding as we go, as those technologies advance and the cost or economics of it make more sense. We’ll see more and more adoption from both the merchant guys and the ASIC guys.

Makes sense. I guess there’s maybe more to talk about in terms of the rest of the system. Obviously, there’s the change to the Vera CPU. For the scale-out network, we’ve got the CX-9 NICs. For the front-end network, we’ve got the BlueField-4. At the switch, we’ve got new silicon for Spectrum-X: Spectrum-6. Anything else you guys want to cover going through?

5. Thermal And Power Limits Rise

The other thing is that we see this with every generation. One way to get more performance is to increase the power. Rubin is no exception. The maximum TDP per Rubin GPU is going to be 2.3 kilowatts. That compares to Grace Blackwell Ultra at only 1.4 kilowatts.

More power means more heat that you need to dissipate. Wega, maybe you can talk a bit about what’s happening with innovations in cooling or thermal management.

There are several innovations on the thermal-management side of Vera Rubin compared to Blackwell. First, there are innovations on the packaging side. There is a technology called a microchannel lid.

Initially, NVIDIA wanted to push for adoption of this technology, which is essentially having the cold plate packaged together instead of having a heat-spreader lid. You replace the heat-spreader lid with a cold plate integrated into the package.

But there are packaging, logistics, and testing challenges with that design. It is not available in time for market, and it’s not a viable solution for NVIDIA’s first deployment of Rubin. It might be deployed later or in the next generation, but for Rubin, the technology isn’t mature enough in terms of logistics and testing.

However, there are still innovations on the packaging side. As you can see here, the package is the piece below. There’s a lid, an integrated heat spreader, but it’s actually 2 pieces instead of 1 giant piece. Sorry, guys, it’s still Chinese New Year’s time. Someone just set off firecrackers.

What was that?

I have no idea.

The firecrackers.

The firecrackers, yeah. Watching a rocket ship take off.

Yeah. Coming back to this, the heat spreader is broken up into 2 pieces. The reason for that is that, given the increased packaging size, if the heat spreader is 1 piece, it will have warpage itself. Even if the heat spreader is so big, it will have warpage, and that warpage will be transferred to the package.

Breaking it up avoids that, so it is a 2-piece solution. It will be combined into a module. There is also the introduction of a stiffener, which brings mechanical support to the package and helps with the warpage of the package.

On top of the heat spreader, there is a layer of electroplated gold. The reason for that is that the liquid-metal hybrid TIM between the cold plate and the heat spreader, which is the package, is corrosive to copper or whatever metal the heat spreader is made of.

As you can see, the cold plate also has a layer of electroplated gold, which helps prevent corrosion from the liquid-metal TIM.

There’s also innovation on the cold-plate side. It’s still a module, similar to Bianca. It will be a big module for the Strata board. However, as you can see here, the channels—which provide the surface area for heat exchange—have a reduced pitch. That increases the surface area for heat exchange, allowing you to remove more heat from the package.

So these are the innovations on the packaging side and the cold plate side for them to achieve that 2,300-watt power design that they’re aiming for.

Can you talk about the inlet water temperature that everybody was talking about when this announcement happened—how it’s going to be 45°C, but it’s not really a big change?

Yeah. That innovation is actually more impactful for the infrastructure design of the data center. The chiller-less design means that you don’t really have to cool the loop that goes into the system to room temperature or below room temperature.

How they achieve that is by increasing the flow rate. When you’re looking at the difference between the inlet and outlet temperature, with a 45°C inlet and a 65°C outlet, which is similar to what Blackwell had, you’ve sort of increased the minimum—sorry, the maximum you have. Basically, you’re minimizing the delta between the inlet and the outlet. By doing that, you have to increase the flow rate to remove the same amount of heat from the system.

Yeah. I mean, I think that’s a pretty incredible trade-off if you can make it work: a 2,300-watt GPU.

Yeah. On the data center design side, we’re seeing some data centers doing this chiller-less design, but not everyone is doing it, because people have different considerations when they’re designing a data center. Not everything is optimized for Rubin. There might be other hardware in the data center, or there could be roadmap changes. Things change very quickly all the time.

We see this trend happening in the market, but the chiller-less design is not a one-for-all solution for Rubin deployment. It’s an option that helps with your TCO.

Yeah. At least the reference design doesn’t mean everybody has to follow it, but NVIDIA’s reference design is suggesting that it would have double the thermal performance compared with Blackwell just by making these changes. Pretty impressive stuff.

Maybe we could wrap by talking about power delivery and then any final thoughts on Rubin. It’s like some major changes, but in some ways it’s quite similar to the GB200 and GB300 design. It’s not as significant a change as what we saw going from Hopper to Blackwell, considering that the full-system design is obviously just increasing specs and increasing the rack design, right?

Yeah. Let’s start with the power delivery. On the power delivery side, there actually are more changes at the rack-level power delivery. Within the IT rack, you’re still receiving 50-volt power. The busbar is a liquid-cooled busbar now because it has a much higher current rating. There are no fans in the system, so you have to use liquid to cool that busbar.

Besides that, another innovation is the 50 volts going directly to the Strata board. I think that is interesting because it increases efficiency. When you do the step-down of the voltage closer to the package, the efficiency is much higher.

Given that the Strata module is taking in 5 kilowatts of power, it is the right design choice to move that 50-volt-to-12-volt conversion closer to the GPU, which previously for GB200 was done on the power delivery board. Previously for GB200, the power delivery board would step down 50-volt power to 12 volts and then deliver that to all the modules in the chassis.

Here, the 50-volt step-down is being done in 2 different places: 1 at the power delivery board for the networking modules at the front of the chassis, and then, for the Strata modules, the 50-volt step-down conversion will happen on the board. That is a power-efficient design.

At the rack level, the power shelf is much higher power. The reference is that the power shelf has much higher power. There aren’t many changes from previous generations, except for higher-power power shelves with much higher density.

But we’re seeing more adoption of the power rack design. That is not a necessity for Rubin, but it’s an option for hyperscalers or customers who are looking at more efficient power delivery at the infrastructure level. We’re looking at high-voltage direct current in these power racks. The timeline for that is being pulled forward. We could see some deployment for Rubin, but it still depends on customers’ choices. That is interesting.

Besides that, the overall power delivery infrastructure is similar to Grace Blackwell. To wrap this up, from the system design perspective—from the hardware perspective—the major difference is on the PCB side. The architecture is very similar to the previous generation. It’s still an Albaron architecture, but they’re taking the lessons they’ve learned from the very incredible ramp of Grace Blackwell.

Of course, there are many challenges with that, but they’ve implemented the lessons and experience learned from that ramp and integrated that into the Rubin design. So, it’s a very interesting evolution from Grace Blackwell to Rubin. Most of the improvement comes from the semiconductor side, I would say. Myron, could you comment more on that?

Yeah. I think, as we mentioned, there’s a massive improvement in compute. One of the big things, again, is the huge improvement in memory bandwidth. That’s going to be great for inference, which is mostly memory-bandwidth-bound.

Then there’s the whole suite of silicon—not just the Rubin GPU, but also Vera, ConnectX-9, the new NVLink switch, and new networking switches. NVIDIA really is the only company that has end-to-end silicon for pretty much the entirety of the system. It’s pretty impressive, and I’m pretty excited for Vera Rubin. I’m sure Jordan is really excited to test these things when they’re available.

6. NVIDIA Locks Up DRAM

I want to add something. We forgot to talk about the memory situation here. SOCAMM was previously—there’s a step up.

So, what is SOCAMM? What’s that?

Yeah. SOCAMM is NVIDIA’s specification for a socketed LPDDR4, LPDDR5 module. It’s customized for NVIDIA, and the specification is set by NVIDIA. It replaces the LPDDR that was soldered onto the board for the Grace Blackwell generation.

The benefit is that customers can choose what sort of memory SKUs they want by choosing a SOCAMM with different capacities available. But we understand that previously, SOCAMM would have been procured by the customers themselves.

Given the memory tightness in the market currently, NVIDIA has taken the procurement responsibility back themselves. That means they will bear the cost of the SOCAMM and charge a margin on it. But that provides several benefits.

First, given that they’re probably procuring for the entire Vera Rubin generation or part of it, they’ve contracted maybe half of the deployment availability for SOCAMM currently. So they can have better pricing negotiations with memory vendors. Jensen went to Korea for fried chicken—that is definitely for something. There’s a purpose there, right?

Yeah.

Another thing is that it also ensures there is no awkward situation where customers have secured the Strata module—the Vera Rubin rack—but haven’t secured the SOCAMM modules because of memory tightness.

Yeah.

Just to be clear, we're talking about the memory that complements the Vera CPU, which is the traditional—

Yes.

DRAM, so tight in the market, right?

Yes. It is the traditional DRAM, not the HBM. That is on the Strata module. That is accessed via the Vera CPU. Correct.

Yeah. What you're saying is that, in general, there's a lot of discussion in the market right now about memory prices being passed through the supply chain to people who are going to be buying servers. I think what you're saying is that you think NVIDIA is less exposed to increases because they've locked up all of the supply that they'll need for these SOCAMM modules on the Vera Rubin CPUs.

Yes, yes. That ensures pricing stability, at least for what they've contracted. Maybe they haven't contracted the entire amount for the whole Rubin generation, but they have contracted—

How much do you think you can contract with 1 fried chicken visit? Does that get you 6 months' worth of supply? How many billions are we talking about here?

That is a question for Ray.

We'll find out. But NVIDIA probably saw this coming, and they realized, “Okay, we need to lock up as much DRAM as we can.” So, as a consequence, they're going to pay a lot less than what we see in the latest contract pricing or spot pricing.

To be fair, the end customers will still pay for this memory at a price similar to this contract price, right? Or it will be much higher than—

Yeah.

what NVIDIA contracted, right? Because NVIDIA is passing—

And they're going to charge the standard NVIDIA margin of 4 times.

Yes, yes. They should, right? Well, maybe they can't, but they'll try to bring down costs on the other components to make up for the margins they cannot charge.

And, roughly speaking, in terms of the content of the total system, SOCAMM modules attached to the Vera CPU are relatively small compared to the HBM on the GPUs or just all of the rest of the components in the system. We're not talking about SOCAMM being 50% of the BOM or even 25% of the BOM here.

Actually, based on our calculations, it's a significant chunk. It's in the double digits. We've disclosed that in our model, the BOM model, but it is a significant chunk because of the margin that NVIDIA is passing to it. But if there's no margin on it, it'll be in the single digits, right?

Hmm.

But the point here is that the pricing for this is stable, and they can have a secure supply for their customers. That might be challenging if other GPU system vendors did not procure that ahead of time or didn't have any—

Yeah.

—

It's a huge impact on somebody's purchasing decision, right? They always have to make an adjusted performance-per-dollar decision. We can look at the specs and see what the performance is, but the other thing is how much they're going to have to pay for the system, which is the denominator there.

If their input costs on their GPU systems are stable, then they have a lot more control over how much they're going to charge their customers, and they can make deals that are on the margin work if they're competing against GPU, Trainium, or AMD, which may be more exposed to changes in the DRAM pricing market. Or HBM, too. I mean, everything's constrained, right?

Yeah. There could be a high price hike coming for HBM, right?

Cool. Okay. Well, good job so far, guys. Anything else that you think we should cover here before we wrap up?

I think I've covered most of what's in the article and the important part of this Vera Rubin evolution. It's been fun, taking a lot of time, but—

7. Rubin Deployment Takes Time

What's the timeline? When are we going to see this stuff ship, and when can I use one for the first time, do you think? Time to first code.

Well, there—

The delivery systems ship in the second half.

Second half of the year.

Yeah.

For time to first-token deployment, with Blackwell, it took a long time to get things stable even after systems shipped, right? It's probably the same for Rubin. Out-of-the-box performance straight away isn't always—it's, I guess, never really super-optimized.

Yeah.

The design seems much more streamlined and integrated, right? So that should help, compared to GB200, where there were lots of challenges with manufacturing at the beginning.

The supply chain is also much more ready, right? The rack system assemblers have ramped up the capacity for rack-level manufacturing. These things need time for a capacity ramp, right? Because the assembly of these systems is very power-hungry, too. You're testing these systems at your assembly site, so you've got to have power secured.

That was also a challenge at the beginning of the GB200 ramp. The infrastructure for the ramp is ready, and the supply chain is ready, so it should be much smoother compared to GB200. But it's still a very ambitious, state-of-the-art rack, so there are always challenges during the ramp of these new systems.

With the speed that NVIDIA is pushing for things, there could always be hiccups. But, as Myron said, we're expecting second-half delivery of the rack system. That's delivery to the customers, and they have to set it up with the networking, bring in the on-site setup, and everything. There could be some availability in the second half, but mass deployment could be in the first half of '27.

Going back to Blackwell, we saw pictures from CoreWeave with their first rack in November 2024, but people really started using it probably in February or March, and at scale around May or June 2025.

Yeah.

So do you think it's a 2-year rhythm, where Blackwell Ultra with GB300 followed that same timeline in 2025, and then this one will be—

Actually, GB300 was much smoother. The transition from 200 to 300 was much smoother, right? There wasn't much of a hiccup between that transition. As you know, in the first half of '25, people had been working very hard to ramp Grace Blackwell deployment.

Going forward, with the supply chain and the data center side—the location and operations people being more familiar with the system—this process should be much more streamlined and familiar.

Yeah. Okay. So let's stick to your—

The next big challenge will be Kyber, right? That's the next system.

Right. So whatever milestone you want to pick—first engineering sample versus initial deployments versus at-scale deployments—do you think it's going to be a little less than 2 years between the first GB200 NVL72 and the first Vera Rubin NVL72?

You mean from samples to mass deployment?

I'm saying that if you compare a given milestone—engineering sample to engineering sample or mass deployment to mass deployment—between GB200 and Vera Rubin, it's a little less than 2 years. It's not more than 2 years from what we saw with them, right?

I think it'll probably be close to 2 years. It could come in under—

Maybe it'll be due to the GB200 delay, right?

But, yeah, it'll be near 2 years.

It won't be as fast.

Yeah. It depends on when you say Hopper was released, but they claim a 1-year rhythm, which would imply H100 to H200, B200 to B300, and now Vera Rubin to Vera Rubin Ultra, or anything in between, right? Each of those steps would be 1 year, and therefore the major change from Blackwell—the new name, Rubin—would be a 2-year difference.

It makes sense, right? We're on track for that. GB200 was, you know…

Well, there was some delivery in the single-digit racks in the second half of 2024.

And a big announcement at GTC 2024, right?

Yeah, the announcement was made at GTC 2024, but the supply chain knew it wasn't going to be ready for the second half of 2024. That was impossible.

Right. This Vera Rubin announcement came at CES 2 months earlier, in January, right? So that bumps up the timeline for everybody by 2 months. If Jensen makes the announcement 2 months earlier, now it's going to be a year and 10 months.

Yeah, but I think the supply chain is much more ready than when he announced GB200 at GTC 2024. There is a big difference in terms of supply chain readiness.

Okay.

Yeah. But it is on track for that, right? The second half of 2025 is GB300, the second half of 2026 is Vera Rubin, and the second half of 2027 is the timeline scheduled for Rubin Ultra.

Well, it'll be interesting to see what happens next, but yeah, very exciting.