Intel 前 CEO：为什么现在是打造硬件的最佳时机

In the AI-driven digital age, energy capacity is equivalent to economic capacity. If we can’t supply power, what’s the point of building a new data center and buying 1 million GPUs? Due to energy shortages, defaults will become increasingly common in many data center projects. If there’s a technology that makes something easy, the bottleneck will shift to another location.

It’s no longer a chip. It’s a rack. The design took 3 months, but it took 9 months before it was actually put into use. How many major new memory technologies have there been in the past 30 years? Zero. A memory revolution is imminent—the first in 30 years. I declared the end of copper about 25 years ago. Eventually, things will turn out just as I said.

For us hardware engineers, this is like a Renaissance right before our eyes. AI inference accelerator chips—I don’t know everything about them. Why are you funding so many chips? I believe you all contributed your fair share of the funds. Historically, no industry has ever had as many as 100 competing processor vendors. Is this just a temporary thing? Eventually, it will likely narrow down to just a few companies.

1. From tech school to Intel at 18

I’m doing this with Pat Gelsinger, a respected guest, dear friend, and former boss. Welcome, Pat. Pat is currently a general partner at Playground Global. However, you’re very well known in the industry as a leader at Intel—as Intel’s CTO—and, of course, as a leader at VMware, among many other things. Welcome.

Hey, thanks, Raghu. I’m so happy to be with you and Guido. For Raghu and me, this place feels like we’re back in our old home, right? It’s almost like a year’s worth of changes have been superimposed, right? But you all look the same. I’m full of energy, so let’s get started right away.

This is not how we intended to start. You’re not aging anymore, so that’s not news. But let’s start with a career at Intel. We’ll be discussing Harvard Academy, which was recently established by Harvard University and other institutions. The goal is to identify talented individuals aged 16 to 22 and place them in a modern educational environment, ensuring they’re fully prepared to become independent, pursue their own goals, or join today’s companies. You had a similar experience, didn’t you?

I joined Intel after attending a regular vocational training school.

How old were you again? Were you 18 or 19 years old? Was that how it felt?

It was truly a magical time. When I was 16, I happened to receive a scholarship and attended a technical school. I skipped a year and a half during my final year of high school, and Intel came to recruit me when I was 18. I had never even been on an airplane before.

I was already fascinated by computers at technical school, and the interviewers were too. One of them was Ron Smith. He wrote this on his page: “That day, I was the 12th person I interviewed. When you interview 12 people in a row, you start to lose track of whether they’re male or female.” By the end, I was the 12th. He said, “He’s smart, proactive, and arrogant. He’s perfect.” So I was invited to join Intel, and that’s how my truly amazing career began.

I initially joined as an engineer, moved to the design team toward the end of the 286, became the fourth engineer for the 386, and then became an architect and design manager for the 486. I did all of this during my bachelor’s, master’s, and doctoral studies. I thought it was incredible. It was the best career I could have hoped for.

You were studying during the day and putting it into practice at night.

That’s right.

And you were creating a chip for an entirely new era. I think there was a big gap between what you learned and what you created.

Yes, because you were forging new horizons. In many ways, it was truly an incredibly wonderful time for the industry.

I remember that, in a Stanford professor’s class, there was a professor who proposed a new carry-lookahead adder design. I said I was working on a carry-lookahead adder for the 386 and asked if I could try it out. Two days later, I went back to the professor and told him it wasn’t working.

I started a discussion with a professor from Stanford. The next 2 classes consisted solely of the professor and me discussing why his design wasn’t working. He was planning to publish a paper containing that design. It was very annoying.

For him, it was a matter of great time urgency.

As you know, we started doing built-in self-test. Ed McCluskey of Stanford was my professor, and he was like a father figure to me. We had a great discussion about why most of his ideas were bad and impractical for actual chips. However, I was running built-in self-tests on a 386. John Hennessy was my thesis advisor, and he clearly did well. I helped him with his career, he became president, and he’s been doing well ever since.

Yes, exactly. We’ll talk about the Cisco-versus-Sun Microsystems debate later, but I wanted to talk about the 486. Sorry, go ahead.

I had the same experience when I was teaching at Stanford. When I brought up what I believed to be the truth, students who were actually working in the industry would come along and say that it wasn’t actually true. So, in the next lecture, I’d say, “I looked into it, and you were right.”

That’s part of the basic student experience. I think that interaction is one of the elements that makes Silicon Valley so unique. In other words, there are so many companies, and the entrepreneurial spirit is like water flowing through the ocean.

If you’re going to launch a new idea, challenge a professor, and become a professor at a place like Stanford, you can’t help but expect challenges and enjoy them. Many people come and go from that ivory tower.

2. The 486 and the birth of modern EDA

The 486 was a great achievement for you all. Clearly, many people were involved, and you opened up some new horizons in chip design, increasing the capabilities that a chip could perform.

3. A VMware for agents

Yes, that was also a time before what we now call the EDA industry, so it was quite a magical era.

That’s amazing. Toward the end of the 386 era, the 486 was the first chip to implement what is now called EDA technology. You did high-level design, right? Regarding RTL descriptions, Verilog wasn’t available.

So, we developed an HDL, right? This was Intel’s hardware description language. In other words, I wrote my own language.

You needed to create a compiler for that language, right?

So I created a compiler for that language. There was no automated placement-and-routing system, so we needed to develop one. Working with Alberto Sangiovanni-Vincentelli and his students at Berkeley, we achieved the initial placement and routing, the first automated timing analysis, and more. To some extent, EDA was born.

That’s correct. That was truly one of its distinguishing features. The 486 was not only a compatible pipelined microprocessor, but it also laid the foundation for much of the modern EDA industry. It truly was a magical time for the industry.

4. How AI changes chip design

Today, do people look back at Blackwell and say that this was the last generation before the microprocessor design process was replaced by AI? Are we currently in a similar transitional period?

I think there is certainly that aspect to it. When we look at things like AlphaChip today, we can say that it’s a first-principles use of AI in the chip design process.

You have to get rid of a lot of things. As you know, there are still some very difficult parts of the design process. In that sense, many analog circuits, such as 3D structures [?], cannot yet be handled by AI. We need a lot of silicon data. In other words, we either have to make the analog design extremely conservative to accommodate AI, or we have no choice but to do it the difficult way.

However, nowadays, many logic functions can be implemented with surprising ease using AI tools and technologies. We have an ample transistor budget, and our design tools are becoming more sophisticated. There are a small number of experts who can provide guidance on how to use and apply the tools.

In that respect, I think it will be something like the 486 series. Looking back, we’ll probably say, “That was the dawn of a new era in chip design.”

When a technology is developed that makes something easy to achieve, the bottleneck shifts to another location. At this point, can you predict what will become a bottleneck in the future?

There are some things that AI seems to be very good at. Many of the capabilities that allow you to specify goals at a high level—such as creating software layers and writing kernels—are translated into Verilog. So, what is the new frontier? What will become more difficult in the future?

5. Why silicon still takes nine months

Let’s say today, for example, that Guido and Pat are working together to create a fantastic new chip. Because we understand AI workloads, understand how to configure them, and also have visibility into the appropriate multiply-add and register structures, we intend to create chips that represent a significant leap forward.

Yes, that’s right. It’s that kind of thing. For example, let’s say you give someone free access to AI tools and they create a fantastic design within 3 months.

Yes, that’s why we were a little conservative when it came to analog components. Even now, there are still several really big bottlenecks. One issue is that silicon processing will still take another 9 months.

In other words, the design can be completed in 3 months, but it takes 9 months to actually implement it in a large-scale silicon fab. That’s terrible.

Yes. In order to realize this innovation, we need to improve that bottleneck. Even during the verification and validation stages, or beyond, you can’t get anything out of the fab.

That’s assuming we have an ultra-fast design flow, of course. So how can we compress it? Furthermore, because advanced packaging and 3D packaging are becoming more complex, it takes another 3 months. Then it needs to be integrated into a rack-scale solution, because it’s no longer a chip; it’s a rack. In other words, it will take 9 months.

The design took 3 months, but it took 9 months before we actually started using it. So how do we begin to compress these aspects of the design? I believe a new form of lithography is needed to make that possible. We need a design flow that doesn’t require a $50 million mask cost to create a prototype. For me, the question is how to compress that into 1 or 2 months. I can now complete the design in 3 months, so how can I finish it in 1 or 2 months?

If it takes a year and a half to actually scale up and deploy the software, my understanding of AI workloads is no longer applicable to the chip I designed. As I said, this can be seen almost entirely from history. Graphcore wasn’t a bad design, but the world had moved on. In that respect, the fact that AI has made the design part easier is somewhat negated by all those other aspects, and the manufacturing-scale issues become a major bottleneck for me.

Another bottleneck is memory. I noticed the memory bandwidth. As I explained, HBM is terrible memory. That’s the best memory we have. It’s terrible memory: It suffers from poor bit density, limited bandwidth, and power and heat issues. DRAM does not like high temperatures. It’s a terrible idea to concentrate everything in one place and give AI a memory-compute workload.

Therefore, development and implementation need to be scaled to the actual size. The memory needs to be corrected. Related to that, this is another important bottleneck that we need to address. Then there’s electricity.

Power, heat, and most aspects of today’s power simulations are bad. There aren’t any good 3D modeling tools for things like power and power-loss hotspots. Regarding voltage, it’s almost a full-power envelope surrounded by a guard band. In other words, we’re losing 40% of the power in the guard band, so we need to improve our power modeling. Furthermore, the method used to account for power loss is also inadequate. We completely remodeled and changed everything.

In short, all of these are the next major bottlenecks for me. How can we improve each of these aspects? Once that’s achieved, an exciting new era will begin.

Thank you for this wonderful AI tool. However, even if these are fully implemented, significant challenges remain. As a result, the number of chips and chip architectures has increased, and we’ve seen a kind of resurgence. That’s thanks to these workloads. With something like 100 AI inference accelerator startups, I don’t think they can all win. So why are you all funding so many chips? I imagine you all invested a considerable amount of money as well.

They all have different design assumptions about the components you’ve talked about and the parts of the Pareto curve they’re targeting.

Do you think there’s a place for all of these? Or are we in a world where, thanks to the AI tools we’re talking about, we can build a wider variety of chips?

Yes, that’s right. The amount of time is enormous.

6. Will 100 AI chips converge to a few?

Either way, and to put it even more provocatively, historically speaking, no industry has ever had 100 competing processor vendors, right? Is this a more diverse future? Will we see it? Is this just a temporary thing? Will it converge to a small number?

I see this as more of a temporary thing, and I think it will subside. That’s my prediction. There are probably 3 reasons why I think that.

One example is this new diversity in computing environments. There are now tools specifically designed for prefill and decode, and people are saying that we need to specialize not only in prefill, but also in decode. In other words, we now have prefill and decode. This means that, as OpenAI suggests, inference and verification are different things.

My computing fleet suddenly became increasingly fragmented across the entire workload, but historically, I don’t think such a state is sustainable. Then, when people saw it, they started saying, “As we move on to inference models, we want something like a CPU again.” Suddenly, I didn’t want to be at that level. In other words, it doesn’t account for a large portion of my computing power. We needed more scale in areas such as models and model workflows.

Therefore, I’m not a big fan of large-scale specialization, because I believe the workload will continue to evolve significantly and shift in the future. First, today’s models are also on the verge of an evolutionary breakthrough. As you know, LLMs are reaching their limits, and people are searching for ways to truly model in 3D.

When dealing with molecules, chemicals, imaging, and so on, a planar LLM doesn’t work well. Therefore, I believe we will see limitations that will cause various changes in the algorithmic domain. I also think some of the most interesting work will start with bringing back things like HPC to AI. Suddenly, 64-bit processing and precision become important again.

To give an example, imagine my AI model derives the 5 most interesting areas in chemical analysis. These chemical algorithms perform with high accuracy when they represent the dominant part of the workload. You will have 5 algorithms that need to be executed, and run these 5 algorithms in various chemical and biological systems. Therefore, as we optimize AI systems, I think we will end up returning to some of the more traditional high-performance computing methods. We’re also looking at that aspect of the workload.

In short, all of this, for me and computer architects, represents extreme specialization. I don’t really like it because I don’t know what the workload will be like in 2, 3, or 4 years. Therefore, this is one reason why I don’t necessarily think that AI chips specializing in specific computing workloads are the right approach.

The second point is that there are currently 100 different types. There are optical chips, probabilistic chips, and so on. Not everything is a winner. Ultimately, these things need scale. To scale up, you need to win, acquire capital, and invest in the workload. Therefore, I think it’s illogical for there to be 100 of these things. We’re seeing them come back as a result of simple capital-market dynamics.

It’s about which team wins, which design wins, and which architecture wins. In other words, I think the workload is driving it. We’re also seeing natural industry consolidation taking place. I also anticipate that the winners will choose the winners. OpenAI, NVIDIA, and Anthropic would say, “This is good,” because it’s not just about the hardware, but also about how the hardware and software co-evolve.

It’s progress. I think there are 10 companies to choose from for this purpose. Leveraging those platforms requires investment in software and workload advancements. Therefore, for these 3 reasons, you will see that field narrow.

The complexity of the problem makes perfect sense, and I think most people would agree with it. Let me try to stir up a bit of controversy here. Do you want to lead the debate?

No, that’s completely wrong. I changed it, didn’t I? Obviously, there aren’t 100 companies, and they’ll merge, right? I completely agree with your assertion that, ultimately, large consumers will, to some extent, choose the winners.

Today, I believe that whether a chip startup is considered top-tier is, to some extent, determined by its track record of doing business with major corporations. That’s all. There’s also the argument that, during the transition period, various approaches will proliferate explosively, and everything will be weeded out.

However, one could also argue that, historically, handling various instruction sets was difficult, requiring the construction of multiple compilers, so it was standardized to 1 or 2 architectures. In other words, the complexity of the software meant that it could only be used on a limited number of hardware platforms. It seems that AI is starting to change that aspect a little.

Today, we’re talking about writing optimized kernels, and I think Groq is a great example. They basically said, “Humans can no longer program this, but a swarm of agents can do it overnight, and that’s good enough.” In other words, if you can build a chip with a specific function, it becomes much easier to build a software layer on top of it that makes the most of that function.

As a result, will the degree of heterogeneity increase slightly? It will become easier to program that heterogeneity in the future. I think there will be a change in where the bottleneck lies. The first bottleneck, as I’ve already mentioned, is that it will still take another year and a half to build these things.

The second bottleneck is not only that it takes a year and a half to integrate these things into racks and scale them, but also whether we can borrow $50 billion to make them practical. In other words, we’re not talking about small-scale capital investments.

We're talking about very large-scale capital investments, including data centers, electricity, and contracts. Therefore, it is necessary to build this on a large scale—and everything that comes with it. As a result, it becomes difficult to address many differences.

Now, obviously, major companies have seen this through what NVIDIA did with Groq. In other words, we end up possessing a part of that heterogeneity. Two years later, we don't know what Groq is within NVIDIA. It will simply be something special.

So, looking at the architecture hides the heterogeneity, right?

That's not a problem for me at all. In other words, that's layering, and abstraction has always been done in connection with it.

But do you think those forces are powerful enough to expose the majority of heterogeneity?

7. Why HBM is a hideous memory

No, that's right. I think some of those things will be abstracted down below. However, ultimately, building these things on a large scale remains difficult.

Pat, you said that HBM is terrible memory, right?

Yeah. This is by far the best thing we have today.

That's exactly right. That's one of the most foolish things about DRAM. I honestly can't remember when the last significant innovation in memory was. It might have been Optane. I suspect you were involved in it as well.

Yes, it's dead. We killed it.

Yes, we killed it. Do you think memory innovation is becoming more difficult, or what do you think needs to happen?

Clearly, we understand the scale issue, because it's even bigger there. Setting aside scale for a moment, let's also look at its history from a technical standpoint. I have personally been involved with at least 5 different new memory architectures. Optane is one of them, and as you know, it never saw the light of day.

Furthermore, you are likely familiar with nearly 100 events that have occurred in the industry over the past 30 years. So how many major new memory technologies have emerged in the last 30 years? It's 0.

Yeah. DRAM, SRAM, flash memory. What else is there?

DRAM, SRAM, flash memory.

Yes, so I think there will be a little bit of talk about flash—barely.

Yeah. But anyway, it means that we haven't created the next physics that will enable low-cost, high-performance, resilient, and manufacturable memory.

That's right.

I think the memory industry has been a terrible industry, both now and over the past 30 or 40 years. It generates a profit in 1 year and then incurs losses for the next 4 years. Therefore, let's conduct research and development in an industry that only generates profits 1 year out of 5.

That's right.

It was always extremely difficult because it was driven by a very intense commoditization cycle. However, that has changed a little recently.

In the past 5 years?

Yes, these are good times. As you know, the 3 major memory vendors are among the top 20 most valuable companies in the world.

Yeah. This is a huge change.

Yes, that's absolutely true. Who could have imagined something like this?

Yeah. That's right. It's related to that.

When looking at AI workloads, we always find that predicting the past is easy, but predicting the future is a little more difficult.

That's right. AI can be described as a memory workload. In other words, it's inside.

Therefore, I believe there is a combination of capital and technological needs that justifies innovation in memory.

There are 2 areas that I'm excited about. One is the idea that HBM's shoreline bandwidth limit is a fundamental problem. A company called d-Matrix is working on this, but I think other companies, like Cerebras, and their technology will follow suit.

8. How tall can chips get?

The future looks promising. We need to integrate memory and computing in some way. In multiple senses, memory and computing need to be integrated, with a fundamentally new structure.

That's right.

Given the significant funding currently being invested in the AI field for workloads, I believe some of the physical challenges can indeed be overcome. I recently funded Substrate, but I also funded a new memory company. It's still a secret, but I'm very excited about some of the innovations that will come with the new memory.

New materials for memory will emerge. Many people are focusing on materials like ferroelectrics and are looking for non-capacitive, high-density memory. We are looking for high-performance, high-density, stackable memory.

I agree that some people are focusing on low-cost flash memory, but I myself am not very enthusiastic about that idea. I believe there are issues with the speed and performance of the underlying cells in flash memory and MRAM structures.

However, I think we're about to see a revolution in memory technology for the first time in 30 years, and that will be an exciting field.

And then several companies will enter that market.

Yes, hopefully we can do some of them together. What do you think?

Yes, but I intend to find some things that need to be done here. I also think memory innovation is just around the corner for the first time in 30 years.

Yeah. I think the market capitalization of the memory industry has increased by $2.5 trillion in the last 4 years. I'm not sure if you know it or not, but it's several hundred million dollars—or was it $2 billion?

I think that's mostly correct.

Yes, we could have invested there and not done anything related to ventures.

Yes, that's exactly right. It's an easy way to make money.

How long is that? What's your opinion? Pat, just a little bit about the stacking aspect: If I had to make a prediction, how fast would the chips get higher? Will this be something that goes up to the second layer and stays there for a while, or will it be something like a 16-layer sandwich that cleverly mixes logic, memory, and SRAM? HBM is lower than V-Cache. How do you think this will evolve?

The problem with stacking is that the value of the stack increases as the dimension of the stack increases. In other words, the yield of each individual layer in the stacking process is continuously required. It cannot increase linearly in principle; it needs to increase exponentially. Furthermore, it is necessary to prove the manufacturing and defect characteristics exponentially.

When you have an 8-layer or 16-layer stack, performing calculations on it becomes extremely difficult. Related to that, the manufacturing flow must be almost perfect. You can see a design that takes costs into consideration to a certain extent, as we've seen for a long time in 2D.

Yes, of course, but remember that was a characteristic of DRAM. There were always replacements for faulty blocks and spare lines available.

I'm certainly not minimizing it, but the yield must be very good. If the die is cracked, it doesn't matter how much redundancy you incorporate into it. That's a defective die.

As a result, I'm not the kind of person who would stack up 16 or 32 units like that. I think a good stack of 3, 4, or 5 would be the sweet spot. It's mid-rise.

Right?

Part of the manufacturing process and part of the complexity associated with it is finding that sweet spot. I think most things will end up there.

For example, let's say we have an excellent computing tile. How big do those tiles need to be? Everything is subject to reticle limitations. Is it possible to increase the number of chiplets that have good yield characteristics and good thermal characteristics?

Then you'll probably want a memory stack on top of that. Is that a 2-stack memory or a 4-stack memory? I don't think it will be 8 stacks of memory. In other words, 2 or 4 stacks would be reasonable.

However, the need to integrate the power supply into that system adds another layer of complexity, and I want to incorporate that in the Z-axis direction. We also want to connect the power rail here. Furthermore, a fairly advanced RDL layer is required to distribute signals across the entire platform.

In other words, my standard configuration, when you add up all the related elements, is already about 8 layers thick. At least based on our current understanding of manufacturing methods, I believe that's the limit.

Of course, optical components will also be needed, so the material structure will likely increase to 3–5 layers. Since we also want to deeply integrate I/O into that structure, it will likely approach this complex configuration. Silicon-based components require the addition of 3–5 other materials.

How can we incorporate an optical connection into this? How do I install a power rail? It could be described as an engineering marvel and a manufacturing nightmare. We need to bring all those parts with us, but that's the direction the laws of physics are guiding us.

Therefore, throwing in a 16-layer stack is crazy. A 2- or 4-layer memory stack would likely be the sweet spot for speed.

Didn't we talk about heat dissipation?

Yeah, and what new materials are needed for that? Obviously, first and foremost, physical hotspots like those we have today are unacceptable. Therefore, we need to think more intelligently about how we use and dissipate heat.

But then the question arises: How can we supply electricity and simultaneously release it from the environment? Incidentally, if power can be supplied more efficiently at the required voltage, and power response and voltage regulation are far superior, then so little heat will be generated.

Yes. But we have to consider: How can we cool these down? New materials, such as diamonds, are being proposed as ways to improve thermal cooling, but what role do they play? And how can we implement liquid-cooling technologies, which have superior thermal properties, on top of that?

Yes, we engineers are becoming plumbers. That's exactly right. You don't need to worry about MC at the chip scale.

Yeah. I'd like to ask about optical interconnects. Some people say that if you remove the memory from its package and connect the compute unit and memory entirely with optical cables, you'll have more flexibility and more configuration options. In contrast to stacking, what are your thoughts on this?

That's another extreme example. As you know, the more OEO—optical-electrical-optical—interfaces there are, meaning interfaces that convert light to electricity and back to light, the more demands are placed on the core compute complex. That makes it more difficult, and losses always occur.

People are working hard to devise more sophisticated technologies to reduce both costs and electricity use. It's about how much power is wasted in the DAC and conversion process, right? I don't like that kind of thing. While it's good to be able to access large amounts of memory, it consumes a lot of power to do so.

The femtojoules per calculation are actually quite impressive, aren't they? The femtojoules per communication bit are a thousand times worse in comparison, right? If accessing a large amount of memory requires that much power, then I don't like that approach. I think that solution goes against the fundamental laws of physics.

Then there's the approach of, How can we directly integrate computation and memory? By shifting part of the computation into memory. That's the processing-in-memory, or PIM, approach. Yes, that's exactly right. So, what do you think about that?

I'm not a fan of those either. Considering the workload, it's necessary to cram the workload into memory. In other words, that means less memory bandwidth is required for the actual computing, and that imposes constraints on the workload.

Again, I may be wrong, especially regarding design flexibility and the intelligence that can be achieved through design. However, most PIM ideas have existed for 25 or 30 years, and I believe they'll still be relevant in that sense 25 or 30 years from now.

By providing high-density, high-performance memory bandwidth and bringing it closer to a real compute engine, we expect this to become a more versatile solution for many evolving workloads over time. In other words, chip density is increasing, and so are computing power, logic, and power consumption. However, I/O is far superior. I'm a big fan of optical communications.

About 25 years ago, I declared the end of copper. In any case, my prediction was correct. Soon, my prediction will prove correct. However, I believe that all I/O should be migrated to optical communication.

This is not the case for the core compute-memory complex. So, the first step in transitioning to optical communication is scaling up rather than scaling out.

What are your thoughts on that?

Yes, and I think things like applications and workload drivers are now appearing before our eyes. I believe the workload driver is something that's right in front of us. Today, in a scale-up environment, we're using a lot of copper, right?

Basically, copper waveguides get progressively shorter, right? Suddenly, it's more expensive to make copper work at 5 meters than to make optics work at 100 meters. I think the laws of physics are driving us there.

If that's the case, Pat, why haven't we moved there yet?

It's a complex supply chain, and it's never been tested on a large scale. Now, I think everyone is considering establishing some form of NPO, CPO, or similar solution around 2028 or 2029. Without this, it would be impossible to scale up to a computing cluster with a large radix suitable for a scale-up environment.

Therefore, NVIDIA is clearly suggesting that. I think everyone else in the industry is predicting that this is a turning point. In fact, the NVL72 should never have been built. It's an engineering marvel but a manufacturing nightmare.

It took the industry another year and a half to understand this monster. It's truly amazing that they've been able to scale it up. However, that proves that with a more mature optical supply chain, we would be far more advanced today.

Today, we should be able to create a better scalable environment—with better power, better cost, and better energy—but the optical supply chain wasn't mature enough to make that possible. Solving those problems isn't easy. There are many things that need to be done, but I think 2028 to 2029 is the year.

How cool is that? We really enjoy seeing how the light within the package functions within the network, and I think much of that could be successfully repurposed for GPUs.

Yes, I think so. However, suddenly, these things have to run at a scale of 10 million GPUs. That's amazing, isn't it? It needs to fit into a package, and there are many factors involved. Many forms of light are sensitive to heat.

So, what do you think?

As you know, heat is an unavoidable issue given the nature of this game. Therefore, we have to address thermal issues, and there are also package-integration problems. In the industry, laser capacity was insufficient.

There are many problems, but I can't find any fundamental issues. At this point, a great deal of effort is required to put all those pieces together.

Regarding network architecture, since these extremely high-speed optical networks cannot be scaled indefinitely, does this mean a switch to more circuit-switched networks?

Well, as you know, I like Nick McKeown. He's a professor at Stanford University and also a close friend of mine.

Guido knows him a little, doesn't he? Was he your academic advisor?

He was my thesis advisor. I think he explains it well. In other words, we built a network that can process packets whose destination is unknown. When you think about AI, it's almost the exact opposite, isn't it?

We're dealing with large-scale, predictable flows. In that sense, the explanation itself is saying, “Hmm, this looks more like a circuit than a packet.”

Therefore, the shift to light and the convergence of workloads with much more predictable, manageable, and large-scale flows will, to me, progress toward introducing light into compute complexes and enabling OCS, or some form of optical switching, which means more predictable flows.

For me, that's the right architecture. This then raises the question: Is it scaling up or scaling out? Once the scale is large enough, you can simply ask, “What is the processing speed in the scale-up environment, and when do you want to connect to other large clusters?”

For me, that is a very beautiful answer to the future.

However, having seen people who are far more knowledgeable in this field than I am, I've always felt that scaling up and scaling out are, to some extent, an artificial distinction. And you know the origin of that, right?

That's because they're using different protocols. One is the front-end, and the other is the back-end network, and so on. So, practically speaking, there's a big difference now.

Yeah. Stepping back and looking at it from a system-architecture perspective, we see that slightly different flavors and slightly different protocols achieve the same goal, but I believe they'll converge over time.

Yeah, especially regarding AI training workloads.

However, training and inference are actually quite similar.

Well, the inference workloads aren't all that similar, though.

Yeah. The brain never stops learning, right?

That's right. By the way, this is another characteristic of the evolution of workloads. I believe that the separation between training and inference will become less distinct in the future, rather than becoming even more so.

That's one of the counterarguments against increasingly heterogeneous architectures. I believe there's a domain of algorithms where continuous learning is essential—where continuous learning is the essence of the algorithms.

The reason we want to update the model weights in the inference environment is that it leads to evolution across the entire workload as we move to a less formalized training environment where the dataset is also smaller.

I'm not sure if that's correct, but the number of nodes required for inference is smaller in scale than for training. That's what it means.

That's true today. Yeah.

Yeah. In other words, I still think there might be a heterogeneous role to be played from another perspective: different types of AI workloads require very different underlying hardware. For example, some programs have limited computing power, while others have limited memory bandwidth.

When there is also a small-scale, dispersed workload, the optimal hardware is quite different from the top-tier 50-billion-parameter LLMs that require large-scale operations.

That's right. Therefore, if there's a factor that promotes heterogeneity, I think it is this. I think there's room for that.

Yes, but you could also say we're talking about running 1 million GPUs in a data center. For that specific distributed workload, you're saying that this optimized version should be deployed, right?

So, now we'll start placing some of them. That workload requires 2,000 of them. And for some workloads, you suddenly lose that scalability flexibility, right? That's what you'd say, right?

How much will all of these things cost, including management, upgrades, connectivity, and fault domains?

The agent will perform all upgrades.

Therefore, it’s not a very big problem.

Yes. Yes. Yes. Thank you. Everything will become easier. That’s all for now. Fleet management is complete. What did we talk about? Computing, memory, networking—next is power.

9. Energy capacity equals economic capacity

Yes. Then, regarding vertical power delivery within chip-to-chip, there is an 800 VDC power supply that is used to power the data center.

There’s an entire ecosystem involved in generating electricity, and it also has sociopolitical aspects. So where does innovation come from, and where does pure execution come from?

Well, to start with the basics, our country has done a terrible job when it comes to energy capacity.

That’s right. Basically, we’ve been adding renewable energy at the same pace as we phased out coal over a period of 10 to 15 years.

And, basically, the country’s overall energy capacity remained flat.

That’s not very common. In the AI-driven digital age, energy capacity is equivalent to economic capacity.

That’s right. So, if we believe that theory, then my economic power as a nation has remained stagnant for 15 years, right?

Well, I think that’s very much unacceptable. This massive increase has been observed over the past 5 years, and we’re going to need more capacity. Even so, I think the country’s energy capacity will only increase by 4% per year.

That’s about it. So, basically, it means going from zero to one, right?

Wow, that’s four times the amount, right? Or, hmm, it’s 4%, right?

Therefore, for me, that’s a bad situation. We need more energy capacity.

First, and obviously, there are several companies in that field.

I’ve spoken about nuclear power at Helion. You’re probably thinking it’s nuclear power, right? Baseload power supplies are fantastic, and I’d like to increase the number of them.

Unfortunately, we are heavily dependent on China for all our renewable energy, which is very disappointing for us.

The lead time for gas turbines is currently as long as 8 years. Therefore, it’s an extremely difficult environment for rapidly expanding the scale of operations. However, first and foremost, we need more capacity.

Fundamentally, energy capacity is hindering the progress of AI to some extent. Without a power supply, there’s no point in building new data centers and buying millions of GPUs. Because of energy shortages, I think we’ll see more and more defaults in many data center projects.

So, do you think this will be a major setback?

I think it will be a headwind, and I believe the first signs of this are already beginning to appear. Oracle’s comments are the very first example of this, and I think we’ll see many more in the future.

That doesn’t mean the pace will slow down, but people will start asking, “Do we really have the energy to support the capital commitments we’re making, like committing to data center racks or buying GPUs?” Therefore, I think there is an upper limit.

However, we need to increase energy capacity, incorporate more innovation in that area, and have more supply chains in the United States. Nuclear power is one of my favorite things.

When was the last nuclear reactor to become operational in the United States?

That was 20 years ago, right? As you know, we have shut down a very capable industry. Therefore, this is something that’s needed even more.

The power network needs to be supplied far more efficiently. And, as you mentioned, it’s the transition to an 800 VDC data center.

That’s absolutely right. Due to historical and standardization reasons, many conversion steps can be eliminated along the way.

Okay, this needs to be fixed. 800 VDC—Edison’s revenge is closing in on us.

Yes, that’s exactly what I was about to say. In other words, I think we’ll need better power conversion.

We own several companies in that field. I think something like vertical GaN will become a killer technology because it will be able to perform conversions from 800 to 48, 800 to 12, or 800 to 5 in a single step.

That way, you can get a lot more energy.

I mentioned solid-state transformers, and yes, they’re all solid-state transformers, switches, and so on.

This will involve rebuilding the power supply network, and I believe that will generate a lot of innovation.

It’s like electricity is getting cool again.

I agree. All of those things are also necessary for the cooling system. Therefore, many innovations will likely emerge in that field as well.

Turbines, refrigerants, and other such technologies will suddenly become exciting new technologies once again. So, for all of us involved with hardware, this is truly like a renaissance.

I had to reread about Carnot’s principle and Carnot efficiency for the first time in 25 years.

Oh, this is wonderful. How does the first law of thermodynamics work?

Physics and thermodynamics were indeed important subjects. Therefore, in that sense, it will be an exciting era. However, it’s also said that physics needs to be changed.

This is because CMOS technology has remained stagnant in terms of power consumption per terabyte and power consumption per teraflop across the past 5 generations of NVIDIA chips. This is unacceptable.

That’s where Snowcap, a company specializing in superconductivity technology, comes in, offering an opportunity to fundamentally change physics. In other words, it’s possible to improve power performance by 1,000 times. Therefore, I believe that such innovations will become a reality in the near future.

I’ve had terrible time management over the past hour. There are still many topics I haven’t covered yet, but I must address at least one before I depart.

Got it. You’re in charge now.

Yes, that’s what I thought. VMware is back up and running. As the former CEO of VMware, what are your thoughts on that?

Well, I feel like I’m being excluded from this conversation.

No, no. As a former CEO, I would like to talk about networking as well, but I don’t think we have much time.

I think that every innovation has always led to the next abstraction, right? I would say that many of these ideas are what’s new in our world today.

We will return to the fundamentals, such as data structures, algorithms, and abstractions. I believe that virtual machine abstraction, whether done at the infrastructure level or the application level, should always be a fundamental abstraction model situated within the computing hierarchy.

What do you think? Should one of us try running VMware again?

Yes, as I think I mentioned before, I think it’s time to run VMware again. Some of these abstractions—the things we did at VMware—were basically about managing every aspect of the computing hierarchy.

You manage workloads, networks, storage, and systems, but how do you abstract those things away? In the context of AI, we think about groups of agents, right? So, who will manage the agents? Who is responsible for creating the security profiles for all agents? Who is responsible for managing agent performance?

Essentially, all the fundamental elements of virtualization and management need to be rebuilt in this next computing tier. In terms of how to achieve those goals, many amazing companies will emerge that accomplish groundbreaking things.

Got it. Shall we wrap things up with a discussion about networks?

No, actually, I’d like to talk about VMs a little longer. What interests me most is building a VM not for human use, but for agent use.

The changes we’re seeing in several other contexts are significant, and form factors are really changing how we manage things. I want the agent to be able to choose whether or not to provide the VM. The degree to which you can make it complex will vary greatly, and the startup speed will also change. Humans are far more patient than agents.

So, if we had to rebuild VMware today, how would that change things?

I think VMware is for agents, not humans.

Do you think we need VMware for agents or for humans?

That’s right. Yes, I understand what you’re saying. Since we need human beings, let’s stick with the term “agent.”

Humans are far more efficient.

Yes, but I think that’s an important point. I think we need to be able to set the agent’s constitution, guardrails, and policies.

It comes down to this: Every aspect of virtual machines that we know and use today is a service provided to agents, rather than providing services to hardware or people. Therefore, I think this will be a fundamental design constraint on this side.

How can we make agents better? How can we make agents safe? How can we improve agent performance? How can we abstract the agent? How can we migrate the agents? All of these have a new manifestation.

This is vMotion for agents.

That’s right. Herein lies a new manifestation of all of these ideas.

However, we also need to consider how humans can set policies, write constitutions, and access dashboards. I think that’s what makes it interesting, because you need to do both. Neither one alone is sufficient.

The fundamental thing—the hard design work we did with virtual machines—was to efficiently abstract the hardware for applications and operating systems. The goal here is to abstract the hardware and operations for the agent. That’s the basic thing you’re providing.

I’m glad we were able to discuss this. Thank you so much for coming, Pat.

I’d love to do it again if the opportunity arises.

Well, they are 2 of my favorite people, after all.

Anytime.

It was wonderful. Thank you very much.