Ep. 006 - The AI Silicon Shortage Explained (AI Supply Chain & Fabs) | Sravan Kundojjala, Ivan Chiam, Jordan Nanos

We've put out 2 articles since the last episode: one called “The Great AI Silicon Shortage” and the other, “NVIDIA: The Inference Kingdom Expands.” We're going to start with the former. Ivan, can you give me a high-level overview of the article? How is demand growing, and why? What's the backstory there?

Yeah, thanks for having me on. Maybe to start, we can rewind a bit. At the very start, in late 2022, at the dawn of ChatGPT, what we saw was that this AI buildout really cycled through very distinct bottlenecks.

1. The Silicon Supply Bottleneck

Maybe the first phase, in 2023, was more about CoWoS packaging: there wasn't enough advanced packaging to assemble chips. After that, from 2023 to 2025, there was also the issue of data center power—not enough electricity or physical space. Right now, we're in this silicon shortage era, which is what this article is about, where we're arguing that there's not enough wafer-fab capacity. Power is no longer the biggest constraint; right now, it's really a lack of front-end capacity.

Starting with demand, I think it's worth discussing where all this demand is coming from. A huge part of it is that, if you look at token demand, it's skyrocketing, mainly from 2 drivers. The first is better models driving higher adoption, and the second is agentic workflows such as Claude Code and multistep agents.

Our firm, SemiAnalysis, burns through tokens like crazy—I think a few thousand dollars a day. We know firsthand how many tokens we're consuming. If you look at Anthropic, they added $6 billion in annual recurring revenue in a single month, February alone, mainly driven by Claude Code. If you look at Hopper pricing as well, you're seeing on-demand pricing really shoot up. Hopper is a chip that is nearly 2 generations old. Maybe someone can add more about this.

Clearly, there's a silicon shortage in just the ability to produce chips. We've seen people talk about lots of other shortages impacting the industry in the past, with memory being a big one and people talking about drives as well. Clearly, this one is at the fab, and it impacts more than just AI accelerators; it's downstream, impacting consumer electronics as well. Sravan, do you have a high-level description of what's happening to all TSMC customers at this point?

2. TSMC Rations Leading Edge Capacity

Thanks for having me. Basically, what's happening with TSMC is that, all these years, from the first iPhone through 2022, before the A100 arrived, smartphones were the key demand driver for TSMC. Almost every year, TSMC was first to market with the leading-edge node.

Now there's a big change in the demand profile. Earlier, Apple, Qualcomm, and all these companies used to drive demand for TSMC. Now they have another demand driver that's much bigger than what they've seen before. For example, NVIDIA recently overtook Apple as TSMC's largest customer in 2025. That's the biggest one, I think. HPC is now outgrowing smartphones by a very big margin.

All of these factors are driving demand for TSMC's leading-edge nodes. Unfortunately, there isn't enough capacity to meet all this demand, and TSMC is severely constrained. They didn't invest much in 2024 and 2025. Even though they were at almost $30 billion in CapEx, this year they increased it to almost $54–55 billion. I think $52–54 billion is the range they gave, but we expect it to be more than that, to be honest. In 2027, they'll probably spend close to $70 billion.

Will that be enough? I don't think so. Right now, if I look at demand coming from 3-nanometer, for example, TSMC exited 2025 with 120,000 wafer starts per month of 3-nanometer capacity. Of that, Apple, Qualcomm, Intel, and all this consumer demand—smartphones and PCs—together accounted for almost 70,000–80,000. Almost two-thirds of that demand came from smartphones and PCs alone.

So what's going to happen this year and next year is that the new chips coming from NVIDIA and AMD are going to take over. They're going to ramp this year and next year, so a majority of N3 capacity will go to accelerators. Obviously, this year isn't a good year for smartphones because of memory constraints. DRAM is severely constrained. As a result, we'll probably see a 10–15% decline in smartphone units and probably a 10–15% decline in PC units as well. Companies like Apple are different because they have the procurement power and purchasing power, and they have leverage over suppliers. They can get DRAM and gain share. But companies like Qualcomm, MediaTek, and their Chinese low-end customers are really struggling to get—

Yeah.

DRAM. Sorry.

Can you talk a little more about the role TSMC is playing? Obviously, Apple and NVIDIA—big companies—want capacity for both wafers and memory. But in some cases, TSMC itself is playing the role of kingmaker and choosing who gets allocation, right?

That is true. TSMC, as far as I know, is pretty disciplined when it comes to capital allocation and choosing customers. They always look at the customer profile: who can give me proper, stable demand rather than a very volatile profile.

For example, if I go back to 2018, they burned their hands with crypto-asset customers. At that time, the CEO was saying there was a lot of demand coming from crypto and all. But within 2–3 quarters, it all disappeared—the crypto disappeared. I think they learned a lesson, and they know which customers can give me solid demand rather than a volatile profile.

So they're looking at Apple, for example. Even though Apple isn't going to be as important as it was before, it still gives a very predictable demand profile compared to any other company. As of today, Apple accounts for almost 10% of TSMC's total wafer shipments. Apple is a very big customer for leading-edge demand. Apple is 25–30% of TSMC's leading-edge demand.

TSMC is very closely looking at all these customers and studying whether, if it gives a company a certain allocation, that company will be able to use it and whether it has enough clout to drive demand for the next few years, not just the next 1 or 2 quarters. They're looking at all these factors. Pricing-wise, they're not so opportunistic, to be honest. If I look at TSMC's wafer pricing over the last few years, it's usually all driven by customer demand and tightness, and they call it value capture. They're not as opportunistic as people think, actually.

3. AI Crowds Out Consumer Chips

So there are other shortages as well, right? We hear about memory and CoWoS. When TSMC is playing the role of kingmaker and deciding who gets access to what, it's forcing some people to bring their own memory allocation to prove that they can use the wafers at this point. What does that mean? This seems obviously good for the big companies and bad for the small companies, right?

Definitely. Yeah, go ahead.

Sorry. Maybe I can move back a bit first to share more about how bad the N3 shortage is. To share a more specific data point, based on our modeling, AI as a percentage of N3 output in 2026 is 60%, and this goes to 90%, or 85%, in 2027. This really squeezes out smartphones and consumer electronics. If you look at the major AI accelerators, almost all of them are moving to N3 at the same time in 2026.

NVIDIA's Blackwell is moving from 4NP to Rubin, which uses N3. AMD, which uses N3 for its compute dies, is also using N3 for its AIDs in MI400. TPU v7 is already using N3, and it has been shipping in 2025. TPU v8 is going to use N3 as well, and the same goes for Trainium and so on, which are all ramping in the second half of 2026 and continuing to ramp in 2027. That's why you see this huge squeeze.

Adding on to the memory point, this is what we are seeing: you have to secure HBM as well as your foundry capacity and all of that. It really goes to show how important having a good procurement strategy is, and this is a new competitive moat that we are seeing. Maybe Sravan has more to add on.

I got muted, I think.

On the consumer memory side, if you look at the typical memory bill of materials, it used to be 17% to 20%. Now it has shot up to 25% to 30%, so it's a big chunk. This is all compressing the operating margins of low-end companies. For example, a lot of Chinese companies—Xiaomi, OPPO, and Vivo—are all cutting orders on the low end, right? They're cutting orders by up to 30% in some cases.

Flagship devices are safe. I would say iPhone-class devices are safe, but the mid-range is declining a little bit. The low end is very much taking a hit because of this memory. Mostly, the people who don't have volume and who don't have much leverage on the supply chain are the ones taking the hit at this point.

I want to put this chart on screen that you guys had in the article. I think it's pretty striking, just to make the point that Ivan brought up earlier: as we look at N3 wafer demand—3 nanometers—in 2025, AI was 9%. This year, it's going to be 60%, and next year, it's going to be 90% of all 3-nanometer demand. What is the release valve? How are people actually going to produce the chips that consumer electronics needs, or even AI needs, beyond 3 nanometers?

Unfortunately, there is—

Yeah—

Yeah, sorry.

Yeah, maybe Sravan, you can go first.

I'll be very brief. Unfortunately, there is no near-term solution to this whole thing, right? It usually takes—I mean, if TSMC invests in capacity this year, it's going to take a minimum of 12 to 24 months for that capacity to come online. So there is no short-term relief.

Luckily for accelerators, memory constraints are actually compressing wafer demand for consumers. That's a positive thing. Otherwise, the situation would have been even worse. The other thing is that, unfortunately, there is no alternative to TSMC in the immediate term. Samsung is there, and Intel is there, but Samsung's capacity on 3 nanometers is still struggling with yields.

The near-term solution is some triaging that people can do. For example, smartphone companies such as Apple and Qualcomm can migrate their premium tier to 2 nanometers. There is some capacity at 2 nanometers, so instead of fighting for allocation with NVIDIA and AMD, they can go to 2 nanometers. Even though it's expensive, I think TSMC is trying to accommodate them with reasonable pricing because they understand that there's no capacity. Why don't you go to 2 nanometers, take that capacity, and release some capacity to NVIDIA and other people? Mid-range and other devices can keep using 4 nanometers.

To be honest, we have heard of things like some game consoles getting pushed because there's no memory and not much capacity available. All these things are happening.

Maybe just adding on to what Sravan said, we generally feel that smartphones as the release valve aren't sufficient. We did some modeling there, but before we get there, I think what we shared earlier was how an increase in memory prices will lead to an increase in the price of the handset. These higher consumer prices would lead to demand weakness, and generally we are seeing low-double-digit year-on-year smartphone unit declines.

There are also some node migrations for smartphones from N3 to N2 that will help alleviate the N3 shortage a little. But we did some modeling there, and it shows that it's insufficient. Let's say we reallocate 5% of 2026 N3 smartphone wafer demand. This will only allow you to produce slightly more than 100,000 Rubin GPUs or slightly over 300,000 TPU v7s, which is not huge.

In an extreme scenario, let's say we reallocate 25% of this smartphone N3 wafer demand. It would lead to only 700,000 Rubin GPUs and 1.5 million TPU v7s, which is good, but really isn't enough to move the needle significantly.

The big question that a lot of people have been asking is when real relief will come across the whole supply chain. People are asking, how long is this memory shortage going to last? It can't possibly go on for years, with people paying 2 or 3 times more than they used to for memory. What does your modeling say, exactly?

Maybe I can speak a bit about HBM. Our house view is that memory is still very tight and continues to be tight because we only see meaningful capacity being added in the second half of 2027. Demand is so strong because of a couple of factors.

If we look at the HBM that we have right now, it currently consumes 3 times more wafer capacity per bit than commodity DRAM. This really makes the whole market very tight. If you go to HBM4 and HBM4E as well, this ratio goes up to 4. The HBM content per chip is also really increasing generation on generation.

Another interesting dynamic that we have seen recently is how NVIDIA requests very high pin speeds, such as 11 Gbps, and many memory vendors fail to hit that requirement. This whole dynamic—where NVIDIA requests very high pin speeds but memory vendors struggle to hit those speeds—keeps the HBM environment very tight. We do see things remaining tight for HBM, and we see this as a structural trend.

Makes sense. Sravan, any alternative opinions there? Do you think you can find some HBM or DRAM hiding under the bed that can cover this whole shortage?

No, I think it's going to take time. Samsung, SK hynix, and Micron will start bringing new-fab capacity online sometime in 2027. Even that is not going to be enough because smartphone demand is going to be compressed this year, so all of that demand will come back at some point.

Again, a perfect storm is brewing now. I think this is going to take some time unless we see some huge CapEx cuts by hyperscalers, which is unlikely. This is not going to get any relief anytime soon.

What were you hearing at OFC last week? Are you expecting huge CapEx cuts from hyperscalers?

4. GPU Demand Keeps Prices High

Dan Nishball

Definitely not. I think it's full steam ahead. They're thinking about what they're going to do with Vera Rubin. If you think about the whole stack, the issue is that it's ROI-driven. We've written about this a few times.

For instance, we'll do tasks on Claude Code, like updating earnings or updating models, where it will cost us anywhere from $5 to $6 or $7. That's a task that would have taken 3 or 4 hours of analyst time, right? So the ROI is probably there, and I think everyone else is seeing it. Otherwise, why would Claude Code commits and engagement revenue be going up so much?

That's percolated through to what Jordan, you and I have been seeing, which is a pretty sharp increase in rental prices. I don't know if you have the chart, if you want to put it up. We expected the GPU rental price for H100s to fall by 30% in 2026, and that's obviously because, as GB300s ramp up, the cost per compute gets lower.

If you measure compute by tokens per second or by FLOPS, the cost gets lower, which should push down the dollars per hour per GPU for legacy GPUs. But that actually didn't happen. Things actually inflected in October of last year. You'll see it bottomed at 170 in the 1-year, and then increased a little bit to about 180.

Dan Nishball

This is dollars per hour per GPU for H100. We actually see it rocket up by around 15% to 20% just in the first couple of months alone. And Jordan, I think it's another probably 10% in March. The irony is we're actually having trouble even deciding what the price is—the ranges have gotten wide, right? Because this is a very low-liquidity, low-volume market.

Jordan, what was it like trying to source compute at GTC? And I think even now we're trying to source compute. How has it been going? What's your experience been like? What kind of anecdotes can you share?

Yeah, I'm—it's a fascinating market. It feels like trying to find airplane tickets on the last flight out because the price is ramping up so quickly for the existing capacity of Hopper that we've been able to find, and then when it's gone, it's gone. People just have these contracts, and they respond, "Nope, I can't find any H100s. We don't have any coming off contract," which is a unique experience. I've not had that when working in the NeoCloud world up until this point.

The idea that not only are a bulk of the GPUs that were in the rental market already sold, but there's no capacity coming online that's not already spoken for until roughly August or September, based on what we're hearing from NeoClouds, is really strange. But it also means that I'm not sure the relief is coming in August or September, because what's to say demand is going to slow down? Our use of Claude is increasing. They keep adding more ARR every week whenever they make some disclosure or announcement. So, if anything, this modeling that I hear the guys are doing when it comes to NVIDIA supply or Google supply or really Broadcom or something at TSMC—what's the eventuality where that gets worse, not better? Are we modeling a worst case for semis that's even worse than the worst case right now? That's the trend, it seems.

Dan Nishball

Yeah, let me try. I don't know if you're able to share the 1-year chart I sent you, or I can share it, but I think I have to share it. I'm not even sure how to share.

Yeah, you can share here.

Dan Nishball

Oh, there. I can share a window. Okay, great. Give me a second; I need to go find a window.

I'll stay up on this one while you figure it out. But there were different forecasts for the 1-year that we were putting in, and it kind of looks like those forecasts for the amount of solar power generation—

Yeah.

—being used—

I got it.

Every forecast was wrong. All right, nice. Yeah, cool.

Dan Nishball

This is another way to look at it, right? We'll have to see if we put our forecasts, but anyway. It's a very common question: how much is the memory factoring into these price increases, right?

Mm-hmm.

What we've sort of figured out is, if you look at an ISO IRR curve, what we say is, "Okay, how much did the memory cost increase the cost of the server?" It probably increased the cost of an AI server anywhere from 5% to 10%, depending on the model. But what we've actually seen is that the increases are well beyond that, right? So it implies it's very clearly demand-oriented.

I think there's an interesting way of looking at this market. You can think of it as a 3-tiered market. At the very long end, there's 4- to 5-year offtake contracts, which you see the large AI labs, OpenAI and Anthropic, sign. Those have typically been GB300s, and they'll be hundreds of megawatts, right?

In the middle, you'll see the contract market, which is anywhere from 1 year up to 3 or 4 years, and you'll see a slightly different set of folks, mostly AI natives. But you will see a bit of the AI labs playing in that, because it's one of the places they can find capacity, right? And then, of course, the other part is the on-demand market, which is probably the smallest part.

What you've seen is that a lot of the AI labs have to take capacity in that middle part. Also, a lot of the capacity that people thought would be released in the H100 market—for example, CoreWeave recently renewed a lot of H100s, I think it was late last year. They would have renewed it with AI labs, so that keeps supply out of the market.

What will be really interesting to see is when the GB300s, which are mostly ramping up this year, provide a tremendous amount of inference capacity. Just look at InferenceX. Will that satisfy the demand? Will it alleviate the demand? Or will the AI labs still be behind the 8-ball and pushing up prices?

It's created a very interesting dynamic. We think it's really created a lot of opportunity for NeoCloud. It's dispelled a lot of the fear about the economic useful life of a GPU, and we see a lot of H100s and L40s still very active for all sorts of models. You and I both know that just from our challenges finding capacity in the market.

Yeah, I heard today that one of the NeoClouds had a long-term contract coming due for an H100, and they renewed it for a 4-year term. So they're signing GPUs through to 2030. That's an 8-year life that somebody wants to commit to paying for. If it dies and there are no spares, I guess they stop paying for it. But for the ones that don't die, it seems like they're committed to paying for it for 8 years at this point.

That changes a lot of the financial modeling of the stuff that's coming online right now, and I think will juice hyperscaler CapEx even higher, unfortunately, Sravan. That's not really the option of asking everybody to chill out on spending right now.

Dan Nishball

It'll probably benefit hyperscalers as well, Jordan, because they all have terms that are coming off from the very first batch of H100s. They were the largest investors in GPUs, and they monetized them at a pretty good rate initially, so—

I agree that will be a benefit, but I'm not sure what percentage of their total rental fleet is going to be H100s coming off term this year compared with the new—

Yeah.

amount of Blackwell. NVIDIA just had a $500 billion year, right? They're forecasting another $500 billion year next year. That's so much bigger than whatever they were doing in 2022 and 2023, when H100s were coming online. Just imagine all the GPUs from then are getting renewed right now. It's fascinating.

Dan Nishball

They'll probably monetize at a better rate than they thought they would, right? And I think the terminal value has always been the concern, right? What is the economic useful life? We've been asked that pretty much every week, and I think it's a significantly better outlook now than it was even 2 months ago or 3 months ago.

5. The AI Cycle's Warning Signs

What's your favorite answer when people ask you the question for the 15th time: what are the signs of the top?

Dan Nishball

Well, I think it's when you hear the 4 words, "This time is different." It's usually the classic sign of the top.

The way we model is, we always say that supply is easy to observe, and it's relatively easy for us to price. We observe price. We spend a lot of time with NeoClouds and hyperscalers understanding where price is going, and we talk to buyers and sellers.

Demand is always the hardest to observe, right? We have data points. We've got Claude Code commits, we've got ARR, all sorts of things, and we'll get better at measuring that over time. That's what this tokenomics model is all about.

But at the end of the day, it's going to really be determined by how price evolves versus what it should do based on the increasing compute capabilities, right? That's sort of what happened: when we saw it not falling as much, we knew demand was strong.

Similarly, if you see the price evolve in a much weaker fashion, or we see the tension come off, then I think that'll be an indication. The key test point is, for all this GB300 capacity coming online, does it alleviate the supply shortage? If you actually see that 1- to 4-year area loosening up as the GB300s ramp up, then it's a little more balanced. But if that doesn't let up, then I think there's more to come. It's going to get worse.

And I think the other thing, Jordan, is ROI. If we see the ROI become hard to make, if it becomes hard to make sense, that's when we'll see it.

Dan Nishball

And I think the final thing is, for all of us in our circles, it seems like everyone has a team of agents working for them, right? But I think the thing to bear in mind is that we’re really the tip of the spear. So many people don’t even have IT clearance to use this. There are so many Fortune 500 enterprises that haven’t even gotten started, right? It was only, I want to say, 2 months ago that we all got prodded to start learning this, right? So I really think that the penetration is extremely low in corporate America and everywhere.

Yeah.

Dan Nishball

Those are my thoughts.

That’s a pretty bullish case for AI demand going forward. Ivan, same original question to you. I’m going to go around the horn and try to get some takes. When you get asked about calling the top, do you have a signal that you think you would look for the most?

I think I would monitor end demand: how fast Anthropic and OpenAI are growing in terms of adding users, growing revenue, and ROI. I think that’s one aspect. The next aspect is maybe looking at data center leases. Sometime in 2025, Microsoft was a bit worried about this whole AI buildout, and then they started pulling out of these non-binding data center contracts. I think those are early indicators, I would say. But I still think we’re early in the cycle and still very bullish on AI.

You still think we’re early. Okay. Sravan, how about you?

Yeah, usually this industry is pretty tricky to predict. In the semiconductor industry, inventory cycles are pretty brutal. There’s a lot of double ordering in this industry, which is very hard to track and pinpoint because even at the moment, as we speak, a lot of double ordering is going on in memory, as well as in N3 capacity that we discussed earlier.

The cycles come every 3 or 4 years, no matter what. As Dan said, “This time is different.” People keep saying that, but every time, at the end of the day, it’s a cyclical industry. Booms and busts will happen. When people try to challenge the thesis that memory is getting more secular and is no longer a cyclical industry, no, that’s not going to be the case. At some point, demand will meet supply, and things will start to wind down. That should happen.

So, can you just dig in on what you mean by double ordering, just to define it for someone who isn’t familiar?

Usually what happens, for example, if I am TSMC, is that an end customer might order through a channel, and sometimes from another source. What you see at the end of the day is the same amount of demand coming from multiple orders from a single customer, which might be canceled.

At the end of the day, they might cancel if they over-order or if they think they’re getting that demand, because these customers get panicked and order through multiple channels. For example, I’ll order through this distributor, I’ll order through that distributor, and they end up doing that. Once they meet their demand, once they get a certain amount of demand, they’ll cancel all the others. So the real customer might be missing this demand altogether.

Who’s the customer ordering what product? Is this a customer ordering a server from multiple OEMs, and then those servers are forecast to the chip vendor, which then needs to go get wafer starts?

I’m talking about the semiconductor industry. Generally, there are 2 types of channels. One is that you can order directly from the vendor; the other is that you can order through channel distribution companies.

For example, if you are Xiaomi or Samsung, a handset company, you might order memory chips directly from the vendor and through a channel company, a distributor. Once you get panicked, you feel that you probably won’t get your demand served from this particular source, so you order from multiple sources. Once you get the demand, you cancel all the others, and that will end up as inventory for those guys.

This happened before, this happens, and this will keep happening. There’s no way this industry is learning its lessons. If you look at CEO statements, they keep saying that this industry never learns this lesson. This keeps happening. There’s no way to end this. Unfortunately.

Okay. Can you then talk a little bit about your experience using Claude Code recently? Is this time different for you personally?

Yeah, definitely. I would say 2026 is probably the best year for me in terms of getting my productivity up. I think it has gone up a lot—almost 10×. It’s like I always have 2 or 3 people working for me with Claude Code. I can accomplish things much faster than I used to because some of my main tasks are sifting through all these earnings call transcripts, filings, and presentation slides. There’s just so much industry information and connecting the dots: what this company is saying, what that company is saying.

For example, if I want to track TSMC’s wafer demand, I have to look at TSMC. I have to look at what its associated fabless companies and customers are saying—for example, fabless companies like Qualcomm, MediaTek, NVIDIA, and AMD. Claude Code is pretty helpful for me in organizing information and pulling data from multiple sources. It’s especially good at working with hundreds and hundreds of PDF files. You can just run through that. SemiAnalysis has multiple APIs, and those are pretty helpful. I can just ping Walter or Royce or somebody. I don’t need to download the transcript and do some analysis.

These transcripts are like 5,000- to 10,000-word articles. Claude Code can run through them, pick the keyword, and bring the relevant information to me. That makes my job easier. I would say it’s highly, highly productive. The dashboards have now become child’s play. Basically, anyone can now design dashboards.

Dashboards are pretty good too, because our Excel files have 40 or 50 tabs. If I put that same information in dashboard format, it gets quite a bit easier. I can just look through trends: what is growing, what is declining, what is inflecting, and what is incremental. I can observe all these trends straight up. So it’s been pretty useful and productive, I would say. Actually, 2026 is probably the best year so far.

Awesome.

Yeah.

Yeah, yeah, yeah. Well, maybe this time is different, Sravan.

I hope so.

All right. Do you guys mind if we shift gears a little bit and talk a bit about OFC? Grill Dan on some explainers on CPO, co-packaged optics. I think this will be a lot of TLAs needing definitions for the general audience. TLA means three-letter acronym, as you know, of course.

All right, let’s jump in. How was OFC?

6. CPO Reshapes Optical Networking

Dan Nishball

Well, OFC stands for Optical Fiber Conference. So let’s start with that. It was good. I think this year, the conference circuit sort of tortured everyone by having GTC and OFC in the same week. So we split the team up. Jordan was up at GTC, and I was down at OFC. But we were really almost covering 2 conferences because GTC had some of the most important CPO updates. CPO just stands for co-packaged optics. We wrote a long 25,000-word book, we call it, published at the beginning of the year.

Co-packaged optics is when you have a switch, a GPU, or any kind of processor where, normally, in order for it to communicate using optics, it will actually have an electrical trace or an electrical channel that goes to the faceplate. Then you’d have a pluggable optic, which converts the electrical signal to the optical signal. What CPO does, and why it’s called co-packaged optics, is that you’re taking the optical engine and putting it right next to the chip and co-packaging it into the substrate, in many cases.

What this does is shorten the length that the electrical signal has to travel, thereby allowing you to avoid using DSP, which is just a digital signal processor, to condition that signal and do the proper modulation. This has been a huge theme.

There are 2 forms of CPO: scale-up and scale-out. Scale-out, of course, is just connecting different GPUs across many racks. It's for connecting tens of thousands or even hundreds of thousands of GPUs. Scale-up is about how you connect GPUs in the same domain. It could be a rack, or it could be a set of racks at a much higher speed—about 10 times the speed.

Why is it important? It's important in general for scale-out because you get to save power, and eventually you get to save cost, if you can integrate this optical engine tightly with the switch chip or the GPU. For scale-up, it becomes important because scale-up currently uses copper, and the reach of copper is only about 2 meters at 224 gigs.

You can really only connect within a rack for a scale-up network with a lot of bandwidth. If you need to go further, you need to use optics. Pluggables have been 1 option. Google uses pluggables to connect the TPUs.

But it really becomes difficult because, again, if we're talking about escaping something like 7.2 terabits or 14.4 terabits of bandwidth, that gets untenable. That could be 6 to 12 different 1.6T pluggables. It's just not enough space.

What CPO allows you to do, by actually taking the optics directly to the chip or the switch chip, is skip all that pluggable faceplate and all the extra energy on DSPs. Eventually, it's going to allow more bandwidth to escape from the chip. That's sort of what CPO is.

Last week, we had a few announcements, and I can sort of—let's talk a bit about scale-out CPO. NVIDIA is definitely the vanguard there. We actually think they're mainly going to market it to neoclouds because it'll simplify deployment.

Neoclouds are already avid users of NVIDIA's transceivers. They like to have a turnkey solution, sort of nuts and bolts, end-to-end, from NVIDIA's switches to their transceivers. This is a very natural transition because you're going from an all-NVIDIA ecosystem, where they have the transceivers and the switches, to something that's also a very end-to-end NVIDIA solution.

Some of these switches actually have a very high aggregate bandwidth capacity. A typical switch will have maybe 100 terabits in the Tomahawk 6 generation, but these CPO switches can allow you to go to multiplane switches, which can get you 409 terabits. It's a little bit of an involved discussion, but that can simplify deployments for neoclouds.

That's scale-out. Scale-out will be the vanguard, and that's actually been a very interesting change in the market in the last week. A lot of folks thought scale-up would actually be the vanguard. A lot of people who follow supply chains see a lot of optical engines being built, and they assumed it was scale-up.

There was a big expectation that NVIDIA's Kyber rack would actually be leading it. I'll show you a quick roadmap table here. Let me just share a screen.

If you look at this roadmap, it summarizes where NVIDIA went in the scale-up domain. Kyber is a 600-kilowatt rack, right? They went through all this trouble to get 144 logical GPUs into 1 rack within the range of copper, and a lot of people expect them to connect with CPO.

It's also a bit of an oxymoron. Why spend all this effort to compress it into 600 kilowatts in this rack, get everything within copper range, and then not use copper? It makes much more sense to use optics to connect different racks to each other and expand that world size beyond 144 GPUs.

That's actually what we saw announced at GTC. We saw the NVL576, which is a Rubin system. It connects 8 racks using scale-up CPO. The other thing we saw announced for CPO was Feynman, the NVL1152, where you're connecting 8 Kyber racks with CPO.

Those are the 2 most important things on CPO. I can talk about a couple of other themes. There are a couple of MSAs that are worth getting into, but I think those were the main events in the optical world.

Everybody's hanging on NVIDIA's keynote at GTC while attending the OFC conference.

Dan Nishball

OFC, actually, Jordan.

Yeah.

Dan Nishball

Everyone at OFC is like, “Oh, I think Jensen—he's going to come on in 10 minutes. He's going to…” It stole the show on the first day of OFC, I'll tell you that.

Awesome. Do you think the sense was that these were investors, engineers, or both who were waiting on this announcement from Jensen?

Dan Nishball

Yeah, I think everyone wants to see the direction they're going to take. With scale-up CPO, investors are keenly weighing the TAM. The variation in total addressable market—the TAM—is really wide.

I didn't stop you for acronym definitions there. I caught DSP and MSA. I need a few of them, maybe—

Dan Nishball

Oh gosh. What does GTC stand for, actually?

GPU Technology Conference, man.

Oh, really? I didn't know that.

You don't like acronyms that stand for other acronyms, where they shorten GPU to G in GPU Technology Conference? Yeah.

Dan Nishball

Oh. Anyway, what were you asking again, Jordan?

Oh, I don't know. You're still sharing on screen there. But yeah, I think it seems to me—

Dan Nishball

No, no. That's—

—that investors are hanging on every word—

Dan Nishball

I guess—

—because they're all trying to bet—

Yeah.

—on CPO and it being the next big thing.

Dan Nishball

Yeah.

Meanwhile, the pragmatic engineers who work at some of these companies—they like it, they don't, they know it's coming. But whether it happens in 2027 or 2028 with Rubin Ultra or Feynman isn't making or breaking their week of work while they're attending the conference.

My perception is that investors really care about these announcements, and the engineers are kind of like, “It's going to happen. It's inevitable. But when it happens, we kind of have no idea, and we'll just be pragmatic about it.”

Dan Nishball

Yeah. I think that's definitely true. It was TAM where I lost my train of thought. The reason TAM is so important is because the big debate a lot of people are having is whether this thing here—the Feynman NVL1152—is going to use CPO for everything, between the GPUs and the NVLink switches, the scale-up switches, or whether it's only going to be between racks.

Again, is that CPO within the rack as well as between the racks, or just between the racks? There are interesting arguments for either. If you take the assumption that CPO will be everywhere, that basically triples your TAM.

The ranges are really wide, and it's very difficult to gauge the ramp in production. Lumentum has to ramp to something like 20 or 30 times the production of ultra-high-power lasers, but no one knows what the base is. Very few people actually know what the base is, so it's very hard to estimate where this could go.

We've always said it's going to be large—10 times the scale-out bandwidth. From our perspective, it's always been large enough. But a lot of these companies have moved up a lot in share price, so a lot has been priced in.

Yeah, yeah. They need to deliver on something, and that all kind of depends on engineering decisions from NVIDIA. Interesting, man. Okay, I think we're—

7. Dueling Optical Standards Emerge

I think the last thing on the MSA—let's talk about that. I think we want to talk about the dueling MSAs, if you will. You've got the CPX MSA, and you've got the OCI MSA.

The OCI MSA came out the week before OFC, and what that's all about is that it's signaling a huge pivot from what's called DR optics, which is kind of gray optics. It's 1 lambda per fiber to DWDM, where you can carry multiple optical signals over 1 fiber.

NVIDIA showcased that. It allows you to simplify the modulation, fit more traffic onto the same number of fibers, and there's a big shift underway.

It'll eventually allow much more bandwidth scaling and a lot more flexibility in the future. I think that's the other important thing that came out of OFC.

Yeah. Can you define that a little bit? You got caught up with some acronyms there, maybe for the general audience. Can you define an MSA in this case and why there are battling or dueling MSAs for sourcing? Basically, the industry that's going to produce these components is going to need to standardize on some approach, right?

I'm looking it up: multi-source agreement.

Yeah.

Multi-source agreement.

The industry that's going to produce these components is going to need to standardize on some approach, and that's—

Exactly.

Yeah.

Yep. So let me share my screen on the OCI MSA. Let me see. Okay, so here it is. It's got all the right logos, right? NVIDIA is here, Broadcom is here, AMD, Meta, Microsoft, OpenAI. And the specification—this is the specification here—is quite detailed. It has all the specs here, but this is probably the important part, right?

This tells you, at its heart, that it's 4 wavelengths for transmit and 4 for receive. Sorry, 4 transmit—well, these all look like TX groups. Anyway, the point is, it's going to be 4 wavelengths for transmit and 4 wavelengths of receive. They're going to go bidirectionally, right? They'll have different wavelengths, so they're not going to interfere with each other. It'll be 50G NRZ.

This is the vision, and all these companies have agreed, “Hey, this is the architecture. Let's get all the optical engine suppliers and all the laser providers to center around this shared set of specs,” right? The idea is to create a common ecosystem, streamline procurement, avoid duplication, and focus resources. This is the roadmap going forward for scale-up, so it's a very, very important pivot.

Okay. So is there drama here? What's happening with XPO and CPX, the other 2 that were dueling with this one? They just didn't get sign-off from any chip vendor because they're all on OCI? Is there no drama? I need some semiconductor industry drama here, man.

Yeah. So the CPX MSA—whereas this specifies how you do the channel, using DWDM and 50G NRZ—is meant to be silent on how you actually do the physical connection, right? What is the form factor? The CPX MSA is about how you actually do that physical connection. It aims to be silent on this kind of stuff: Are you going to use DWDM? Are you going to use NRZ or PAM4? What kind of modulation? It's meant to be silent and leave that to everyone else. It's meant to be a connector, right?

Okay.

But—

The—

Dan Nishball

The 1 interesting thing is that because it is pluggable, even if it will be CPC, which is co-packaged copper, it may have the connectors coming off the substrate. Eventually, you will still need a serialized link—some kind of serialized link—to go from the chip to that connector.

Versus with the OCI MSA, the final boss, as I put it, for the OCI MSA is that you can do a die-to-die connection at NRZ. Keep it slow and wide. But we don't think we're quite there yet, right? And so this has a different view of what the final boss is, which I think is the interesting disconnect.

Yeah. So sorry. The logos—like NVIDIA, Broadcom, AMD—they can all sign up for this consortium and then still be opinionated on other sides, which can cause a bunch of issues.

Dan Nishball

Well, I think it's very implicit, right? Because they seem to stay out of each other's lanes, right? But there are 2 things that are actually implicitly defined, if you will.

First of all, a lot of people have said, “Well, hey, how come this is supposed not to specify the implementation, but this seems to presume that you're going to use ring modulators?” There are different types of modulators. There's a Mach-Zehnder, there are EAMs, and you could even use VCSELs to modulate. So that's 1 seeming—I don't want to call it a contradiction, but a curiosity with the Open CPX MSA.

But the other interesting thing is that because it is pluggable, even if it will be CPC, which is co-packaged copper, and may have the connectors coming off the substrate, eventually you will still need a serialized link—some kind of serialized link—to go from the chip to that connector.

Versus with the OCI MSA, the final boss, as I put it, for the OCI MSA is that you can do a die-to-die connection at NRZ. Keep it slow and wide. But we don't think we're quite there yet, right? And so this has a different view of what the final boss is, which I think is the interesting disconnect.

And that's why I call it dueling. I think everyone is in different camps, right? You have folks who want to go for CPO, but they want to do it with pluggables, which puts them in the Open CPO camp. Open CPO is how you do near-package optics—co-packaged optics, but pluggable—which gives all these hyperscalers flexibility.

The XPO MSA is an Arista-led thing, and all it's trying to do is extend pluggables for longer. I think they've done a really great job with native cooling. They've created a lot of density, and they've made it very flexible, whether you're doing coherent optics.

Because XPO also envisions co-packaged copper coming off the substrate and going to flyover cables into the XPO module, it actually makes it quite possible that you're going to have linear pluggable optics, or LPO, by the time it comes around. But it's a few years out—this Tomahawk 7.

Makes sense, Ben. Awesome. I don't have anything more in the notes that we have to get through. I think this was an hour well spent. I learned a lot about the fabs, the AI silicon shortage, and everything that happened with the dueling MSAs. Sravan, I think you've been playing maybe a little bit too many video games recently in all your spare time, ha-ha.

Dan Nishball

I don't have any spare time, Jordan.

No, that's the joke, but you're calling back to RuneScape or something out here with the references to dueling—

Well—

—and the final boss.

Dan Nishball

Do you remember our Trainium article? Do you remember “Challenger approaches”?

Yeah. Yeah.

So Super Smash Bros. fans will appreciate that. But we always have gaming references in mind.

Yeah, I should have gone with Mario. I should have gone—yeah.

Dan Nishball

But they'll all coexist. All 3 will coexist. They all have different approaches, and they're all great approaches, but they're just different approaches, right? They're different ways to play the game.

Makes sense. Awesome. Thanks, guys, for coming on. Another good episode. Well done. Good job.

Dan Nishball

Thanks, Jordan.

Thank you. See you guys.

Dan Nishball

Bye.