第027期 - OpenAI Jalapeño：优于 Nvidia Blackwell（加速器）

Today I'm joined by Bryan and Myron. We're going to talk about the new article we put out on OpenAI's Jalapeño versus Blackwell—their self-designed ASIC—which we compared with Vera Rubin. We analyzed the TCO, assessed the throughput-per-megawatt claims, and looked at more details in terms of the microarchitecture and system architecture, how they used AI to design it and write kernels, and why the speed from tape-out to a first working system with real benchmarks has been so impressive.

So, guys, excited to dig in. This is a fun one.

Is this a virtual background that you got? Is that a pink cotton-candy background or what?

1. Jalapeno Overview

Of course. I think it's the color most similar to what I think OpenAI's logo is. I don't remember off the top of my head, but they've got some nice designs for their blocks.

Jalapeño has been great—a real surprise on the last day of Hot Chips for those attending. It was one of the most surprising talks, in my opinion, and I believe its performance caught everyone by surprise. We all knew that OpenAI's chip was in the works, and they had announced deals with Broadcom, but the first look at the performance results is really quite good for OpenAI.

A brief summary for those who didn't read the article—which is somewhat unlikely—is that, on a per-megawatt basis and a performance-per-TCO basis, meaning how much it costs to actually run the chip, Jalapeño beats Vera Rubin's July results. We're comparing against the July results because we believe the software is in a similar state between those 2 points.

As shown in the picture, this is Jalapeño compared against GB300. Jalapeño blows it out of the water completely. But as we mentioned in our article, it's a bit unfair to compare it against Blackwell because Blackwell uses HBM3, while Jalapeño uses HBM4. Bryan can talk about the differences between the HBM types a bit later, but it's unfair to compare HBM4 against HBM3, so we're comparing it against Rubin instead.

The results show that it does beat Rubin on output-token throughput per utility megawatt. But again, these are July results, and Vera Rubin right now is likely much better than it was in July. Jalapeño, of course, will improve as time goes on. As we've shown in another diagram, Jalapeño has improved over 25 days.

Yeah, I'll pull that one up. Maybe the key point you were making is that, if you can explain a little bit more about the benchmarks they're running, it's DeepSeek R1 at 8K/1K. It's not really the absolute biggest model or the most demanding inference workload because it's just random data, not agentic workflows, but they got this up really quickly.

Clearly, the performance is strong, as you're implying with this new chart. Performance is improving by the week at this point, even by the day. Maybe you can also explain the denominator there—why they're choosing to divide by power consumption.

2. Tokens Per Megawatt

Jensen brought up this point at Computex 2026 during his keynote: Data centers nowadays are becoming power-limited, and power is starting to become the constraint. If you have money, you can always get more servers and more chips, but the constraint is starting to become the data center's power. Companies have tried getting around this using behind-the-meter power, but at the end of the day, power tends to be the constraint for building a data center.

3. AI-Designed Silicon

Absolutely. If you have 100 megawatts of power, you can fit only so many chips in there. It doesn't really matter if these chips are more expensive. On a per-megawatt basis, if they're producing more tokens per megawatt, you think you can make more money from the tokens they produce and justify the extra expense of the actual chips.

While it's a valid metric for some customers, it may not be valid for others. It's not the most typical way we see things measured. Most typically, we see token throughput per GPU, or per package. In this case, OpenAI doesn't have customers for Jalapeño. They just run it for themselves, so they don't really care about tokens per package. They care about tokens per megawatt going into the system.

Going off this topic, someone mentioned—I forget which one of us—that performance per chip is, at the end of the day, an imaginary thing. You can just glue 2 chips together and say you've doubled your performance per chip, so looking at performance per chip isn't really the wrong metric; it can just be easily misleading. That's what Rubin Ultra and Blackwell Ultra do, if I'm not mistaken: You glue 2 chips together, and suddenly you have double the throughput per chip.

You can do this if you're Cerebras, too, right? You can just put 3 wafers in a rack and put them all on a chart where 3 is better than 1.

Or just say your chip is the whole wafer, right?

4. Benchmark Caveats

It doesn't mean that they don't care about performance per cost, because performance per watt and performance per cost are pretty closely linked. The more power the chip consumes, the more expensive the chip probably is as well, generally.

We did our TCO analysis of what we think OpenAI would pay for a Jalapeño system, and they still pretty much win—or are very close—on performance per TCO as well.

Here's that chart on screen. This one is more interesting because the TCO calculation, in terms of how many tokens you're going to get per dollar, depends on the input for cost per dollar. We've got some variables on screen about the total cost per hour to own, for example, a GB300 or to own Jalapeño.

We're assuming that OpenAI is going to pay $2.79 an hour for a GB300. They're buying them themselves and running the data centers themselves, for example, in that case. They're not paying the current NeoCloud prices, which are around $6 an hour for GB300 right now. We put Vera Rubin at $3.61 and Jalapeño at $1.56.

Let's compare it per package. Clearly, the thesis of designing a chip in-house is that you want to pay the Broadcom margins only—not the NVIDIA margins, or the Broadcom-plus-Google-TPU margins, or some equivalent there. That's being borne out on this chart.

All right, I guess the next part of this is: How could these curves move? With Jalapeño, they're actually sandbagging these results significantly, right?

The caveat on the performance is that the tests they ran were DeepSeek R1, which is based on the V3 architecture that came out in January 2025. It's not the most current DeepSeek model, but it is a relatively large model with 600 billion total parameters. They also ran Kimi K2.5, which is 1 trillion parameters, so that's an actually large model, and GPT-OSS 12B, their own open-source model, which is relatively small.

They had solid performance on all 3. They're beating Vera Rubin on all 3 today, and they're doing it with single-token prediction and no prefill-decode disaggregation. Bryan, maybe you can explain single-token versus multi-token prediction, specifically in these performance claims when compared directly to Vera Rubin, which is using MTP. This is OpenAI fighting with one hand behind its back. MTP is a really significant optimization for interactivity.

Going back to DeepSeek R1, it reminded me of an X post I saw some time ago that said the closed labs—OpenAI and Anthropic—probably have an internal version of MLA, and they probably discovered MLA long before DeepSeek. It's quite interesting because we were told that OpenAI didn't have any internal, optimized MLA kernel implementation. I'm not talking about the open-source Triton one; they don't have any internally optimized MLA kernels, which is quite interesting to me because that means none of OpenAI's models uses MLA.

On the 8K/1K and DeepSeek R1 model choice, we did release an AgentX benchmark case, but the timing was very bad, so I don't think we got the OpenAI Jalapeño team to actually run it on AgentX. It measures things like prefix caching, which exposes a lot more areas for optimization.

In our article, we describe a lot of the load-bearing components being tested for AgentX, and it's starting to show what's breaking, what's not, and areas for improvement that the 8K/1K benchmark didn't properly show.

Going back to the MTP question, for those unaware, MTP stands for multi-token prediction. It's a form of speculative decoding where you guess future tokens and then verify them in a single forward pass. Interestingly, the language model doesn't output just the probability of the next token; it outputs the token probability for every token position before the last token. We weren't given any MTP results.

It’s my personal guess that OpenAI has some internal speculative decoding technique that’s not MTP, DFlash, or any open-source configurations. So they didn’t give us speculative decoding results because there’s no way to actually verify it through open source, which is also quite interesting in my opinion. It means DFlash isn’t the best that we can do, and DFlash actually gives quite a crazy advantage over MTP by guessing all the tokens at once, instead of doing MTP, which is just one layer of a model trying to guess future tokens one by one rather than doing them all at once.

5. The CUDA Moat

We were told that the internal speculative decoding method gives 3–5× improvement on production models, which would just knock the Jalapeño versus Vera Rubin comparisons out of the water once again. So you can imagine that graph shifting 3–5×. It’s quite insane. So, CUDA moat’s gone, Bryan.

I mean, everyone on X is saying, “Oh, it’s just an ASIC. That’s what you expected it to do.” And yeah, in some sense, that’s what we expect an ASIC to do. But the software has come to such a point that this line doesn’t really matter if your software is good enough that you can get a moat up very quickly. What’s the difference between a general-purpose GPU, or, like they say, a GPGPU, and an ASIC if software can just bridge this gap?

Yeah.

Yeah, yeah. I mean, this chip has—it’s a toss-up with Vera Rubin because we haven’t seen anything since they published that note in June or July. So maybe they’ve had an extra month to develop these chips.

But conceptually, we’ve never seen anybody else put out a chart where there’s a curve showing they’re beating NVIDIA on every point of the curve, and in a real test, right? So it’s shocking that they did this. We can talk about the timeline a little bit later, but we keep talking about the point in the curve, and I think we maybe haven’t explained this in great detail—or we’ve done it on previous podcasts and people aren’t familiar with this.

I’m going to put this chart back up on screen and try to explain a Pareto curve here for the purposes of understanding the performance claims made by Jalapeño. To do that, we need to explain that the y-axis is how many tokens you can produce per megawatt you’re putting into the system, and the x-axis is how fast the tokens appear to each individual user.

So whether you’re optimizing for the y-axis or the x-axis, Jalapeño is beating the GB300 right now on the DeepSeek model. If you were to fix the interactivity per user—so everybody sees 100 tokens per second or 50 tokens per second—and zoom in on exactly that part of the curve, we’re basically seeing that Jalapeño has double the number of tokens that it can produce per megawatt, implying 2× more revenue, 2× more profitability, whatever you want to say, from an inference endpoint-serving provider.

And then if you look at the far-right side of this curve across the x-axis, you see that at a very low batch size they can go all the way to 700 tokens per second per user. You compare that to where the others peak out at 350, and this is once again double the performance. I think the conclusion is that they’re basically winning on both sides of the curve: both fast tokens and cheap tokens.

It’s just so interesting because we’ve seen so many other companies make claims about how they’re going to beat NVIDIA, and they just pick one of those. Groq or Cerebras, or any of the other startups that are going to focus on SRAM—call it d-Matrix, which is coming up with stuff, or SambaNova, where we’ve even seen some results on the InferenceX benchmark or something similar to it. They’re saying they’re going for fast tokens, just decode speed and low batch size; they don’t worry about throughput.

Then you’ve got other guys that are worried about throughput—call it AMD as a simple example—but even TPU or Trainium could be in this bucket of accelerators that are going for throughput. They go, “Yeah, but we’re not going to be able to compete with the other guys at high interactivity.” OpenAI has a chip that can do both for them really well. At a minimum, this thing is doing really well right now and is going to serve real production tokens for them.

Yeah, it’s actually quite surprising. I was surprised that OpenAI was the first non-NVIDIA, non-AMD chip to actually appear on our public InferenceX benchmark. We were expecting SambaNova or Cerebras, or even a TPU or Trainium, to be one of the first.

Maybe this really puts into perspective how this time might be different from the rest. This is the first chip that actually poses a real threat to the CUDA moat. In open source, we welcome results from anyone. If you believe your chip is good, run the results, run the curves, run the benchmarks, show us what your chip does, and we gladly put it on our dashboard and write an article about it if it’s good.

We extended this offer to Etched very recently on X, and of course, Etched didn’t get back to us. But if your chip is good, just run the benchmark, show us the results, and we let the results talk.

Yeah, it’s—

We love to see more competition from these chips. Yeah, exactly.

So, did you guys see Jensen’s response to this? I feel like the CUDA moat has been getting slowly drained over the last couple of years. Top models in the world, like Claude and Gemini, are trained without NVIDIA GPUs—on TPU, right? Anthropic uses lots of Trainium. It’s not like you can only use GPUs, but NVIDIA is an incredibly valuable company. They’re going to keep shipping all these GPUs, and I think there’s such a thing as an NVIDIA moat, which includes everything in the supply chain: the whole developer ecosystem, all of the availability to purchase and support, and how you’re going to do the logistics of deploying these data centers and monitoring them over time.

OpenAI has now got to figure out how to turn on 100 megawatts of these chips, not just 3 test racks, which is a monumental challenge to get over—as if taping out a chip of this quality is easy. The next phase will be pretty hard for them as well. Even Cerebras is going through this themselves right now.

So I guess the question for me—well, I think Jensen kind of answered it in the style I was saying there when he was on Mad Money with Jim Cramer, everybody’s favorite. He was basically like, “I’m not bothered.” Myron, what’s your take on this? Well, okay, what’s your take on that, and just the whole timeline to go from, “Yeah, we’re tired of buying NVIDIA GPUs for everything. We’re going to go build it”—decision made at OpenAI—to actually having a chip that can run in an InferenceX benchmark?

6. How OpenAI Did It

It’s under 2 years from concept to a real chip in the lab, and under 9 months to actually get it taped out, right?

Yeah, I have several thoughts on this. I guess, starting from the CUDA moat eroding, I think a lot of how you can succeed, both in terms of the silicon design as well as bringing up the software, has been AI-assisted, right? The irony is that this was all done on NVIDIA GPUs—training these models to bring up the capability to a point where AI is able to program kernels. That’s been, I think, probably the big shift in terms of making it easier for the labs, and anyone, to adopt alternative systems for their serving stack.

I think one of the big reasons that Anthropic has decided to bring in AMD as one of their hardware providers is because AI-centric programming allows them to get around the challenges of using the AMD software stack, for instance. Somewhat ironically, it’s NVIDIA’s hardware that has enabled moving off NVIDIA’s hardware.

In terms of Jalapeño, it’s such a big surprise because we always knew the team was capable. They’ve had experience building the other main successful ASIC program in the form of TPU. A lot of the hardware team behind this is from former TPU people, as we see in a lot of other AI accelerator startups or other ASIC teams. They tend to come from former people with TPU backgrounds.

For what it’s worth—sorry, sorry to interrupt—but there are basically no chip startups I can point to where it’s a bunch of guys who are ex-NVIDIA. But there are a lot of ex-Google people out there doing stuff, which is interesting.

Yeah, I do wonder sort of why that is the case. But anyway, another topic.

Designing an AI accelerator that’s competitive with NVIDIA is not easy, right? It’s such a huge market. Of course, everyone wants to try it, but time and time again, we’ve seen a lot of entrants, and they haven’t really been able to do it.

I think the expectation was that OpenAI would deliver a decent effort with their first generation, and it turns out it was much better than decent. They came out with something pretty much competitive with, or better than, what NVIDIA has to offer.

So that's surprise number 1. I think that also says something probably about other ASIC programs, especially Meta and Microsoft. Is OpenAI really that good? Do the silicon teams at Meta and Microsoft have skill issues? It's probably a bit of both, right? I think they're the guys that look the worst from this announcement.

Going back to where next: you've designed a chip. Obviously, you're scaling up the supply chain to deliver systems at mass scale—you're talking about delivering millions of these chips, thousands of racks, and deploying them in data centers with gigawatts of power. That's not going to be easy, but other people have done it successfully. I think the hardest part is really having that system design, and OpenAI has partnered with people who have experience scaling this up. Basically, it's Broadcom and Celestica on the system side, and they've had experience doing this with TPUs.

Digging into the comment you made about the difference between an in-house silicon program that's been going for years and years, like MTIA at Meta or Maia at Microsoft, if we just literally look at the specs of Jalapeño, it doesn't look super fancy on paper. I mean, it is an HBM4 chip, so it's going to have great HBM bandwidth. It's got lots of FP4 FLOPs, but still less than half of what Rubin has on FP4, less HBM capacity, and less TDP. So it's almost 3 times less TDP per chip than Rubin, right?

When people are making the bull case for AMD, they say, “Just look at MI450.” It's going to have more FP4 FLOPs, more HBM capacity, more HBM bandwidth, and more TDP, and therefore it's going to be better. They never want to look at the benchmarks or the results of the chip, or at the MI355 when it was coming into production for the first time.

But now you've got an OpenAI chip whose specs on paper are objectively worse than everything on Rubin, comparable to GB300 on everything except HBM bandwidth, and yet it's way outperforming GB300 and outperforming Rubin so far. Clearly, this is due to something about the microarchitecture, which we can get into, and the software. What's your take on what it takes to design a chip now? Is it purely having access to the latest models and being willing to YOLO-trust them on RTL and kernels? Is that the barrier?

Yeah, that's a good question, and I don't know the answer to that. But to your point, everyone can deliver great stacks on paper. When we look at these stacks, they're all peak theoretical, and I think there's a lot of emphasis on theoretical because some of the FLOPs numbers, no matter how you try to reach them, are impossible to reach. It's somewhat determined by the chip company's marketing teams.

Basically, can you actually realize those FLOPs in an actual workload? The same is true of HBM bandwidth. One of the big tenets of OpenAI's design philosophy for I/O is that everyone can deliver raw HBM bandwidth: you just buy the best HBM and put more stacks of it. Buying HBM itself isn't easy these days, but it's basically not super hard, right? It doesn't take tremendous design skill.

What's really throttling a lot of these chips is that they can't realize anything close to the raw HBM bandwidth because there are so many other things in the microarchitecture that stop you from doing that. A lot of that comes down to complicated memory subsystems or simply the amount of data movement required. The OpenAI team has focused on a microarchitecture that reduces data movement so they can realize a lot of the massive HBM bandwidth they have. I think that's really the main skill in all this, and why it's been so challenging for the competitors: they can deliver great specs, but it's difficult to actually realize them in a real workload.

Yeah. I agree with the fact that you can't actually reach this. I know there's a saying from someone—I'm not sure who—that the figures are not numbers you can reach; they're numbers that the manufacturer can guarantee you never exceed. Those are numbers that you're guaranteed not to exceed.

In one of the previous articles, I think in the Cerebras articles, they talk about roofline models of chips compared to NVIDIA. Although NVIDIA's FLOPs are huge and crazy, especially the FP4 ones, at the end of the day, they are extremely high on the roofline in the compute-bound region, and a lot of workloads would rarely even reach those rooflines. So it doesn't matter at the end of the day: the FLOPs in the table don't really matter because workloads most of the time do not actually hit those rooflines.

7. Samsung HBM4

Oops, Jordan dipped again. Yeah, Jordan is gone. So maybe I have something curious to ask you about HBM. There's been talk about Samsung HBM being better than the other HBM, and Jalapeño was lucky—or maybe it was a decision. I'm not a hardware guy. What makes Samsung HBM better quality than the rest?

Yeah. So Samsung, for a long time, from the HBM3 and HBM3E generations, had HBM that was really quite inferior to SK hynix, which dominated—and still dominates—the HBM market share. It's the leading supplier for NVIDIA, for instance. The HBM3E generation was really bad from Samsung. Part of it was built on an inferior process, right?

Samsung realized this, and they really went all out on their HBM core technology. The DRAM dies are built on a more advanced 1c process, whereas SK hynix and Micron are using 1b-class processes, the same class of process that HBM3E is built on. There's also a logic base die in the HBM cube that has the PHY, and Samsung has built this on an advanced logic process, which is SF4—Samsung Foundry's 4-nanometer node—whereas SK hynix is using 12-nanometer TSMC, and Micron is still using its own DRAM process for this base die.

Even though the bandwidth requirements are significantly higher for HBM4, Micron has had some issues achieving the highest speeds for HBM4, and similarly, SK hynix has had some issues. They've had to redesign the base die, and this is why they've had to delay shipments of HBM4 for NVIDIA's Rubin.

Samsung appears to have the best technology for HBM4. This is why Jalapeño is HBM4 and can deliver 15.4 TB/s of HBM bandwidth, which means a 10-gigabits-per-second pin speed. That's a little bit higher than the 9.6 gigabits per second that we think NVIDIA will ship Rubin with.

I think it probably does end up being because Samsung is the supplier here. Whether it's luck or skill, we can debate, but traditionally, Broadcom's HBM has mostly come from Samsung, so I think it's partly luck. This hurt Broadcom for HBM3, but for HBM4, this has turned out pretty well for them.

Yeah. It's very interesting that SK hynix and AMD invented HBM, but now Samsung is doing better in HBM4, and NVIDIA is doing better than AMD. On the topic of HBM bandwidth, bandwidth is a problem, right, and not capacity? Is that why companies are going toward 6-high and 4-high instead of 8-high?

I'd say the primary benefit—I mean, it's in the name, high-bandwidth memory—is the bandwidth. Capacity is important, but I think the main benefit is really bandwidth, because there are cases where you pay for the additional capacity but you might not need it, whereas for memory bandwidth, you can always use that to serve tokens faster.

Whether it's an 8-high, 12-high, or 16-high stack, the bandwidth is the same. You pay for capacity, and more layers are what add to the supplier's cost.

You pay for the extra capacity, but the dollar per bandwidth gets much worse. For some companies, if they want to optimize and say, “Actually, we just want to get the best dollar per bandwidth,” then going to lower stacks is the right optimization. Of course, you want that balance between capacity and bandwidth, but I think there is a philosophy—especially now that HBM is getting much more expensive because we have very limited supply of HBM wafers. I think the trade-off is starting to look more in favor of going to lower stack heights rather than just increasing them further and further.

Yeah, it’s interesting. It makes sense to me, especially with increasingly more rack-scale architectures.

But capacity is not becoming as much of an issue anymore.

Yeah, exactly.

Yeah. This is specifically borne out in the per-watt argument as well. If you look at the raw HBM bandwidth, comparing the specs of the chips, it’s like, okay, it’s up there with the other ones. But then, if you divide by the amount of power consumed by the chip, the fact that they’re getting 15.4 terabytes per second on a chip with a TDP of 700 watts is incredible.

This bandwidth-per-watt ratio—arbitrary units here—of 22 is so much bigger than anything else. It’s literally double Rubin. Even at the Max-Q, low-power-setting option on Rubin, running it at 1,800 watts, Rubin has a little bit more HBM bandwidth—whatever that is, 25% more, 20 versus 15 terabytes per second—but it’s literally more than double the power.

That’s where your token-output-per-watt advantage comes from. If you can realize that bandwidth with that much power, that’s where the advantage comes from. I guess the big question, of course, is: B0 is in the fab right now. They should just be able to step up the power and get even more bandwidth from this, right? Or are they at some limit?

I think on the HBM bandwidth, they are probably at a limit. That’s just the constraint of the memory itself. But B0 should deliver more FLOPs at basically the same power. Depending on the workload, if there are compute constraints, then that should benefit the B0 stepping. The A0 stepping is what all the current results are.

There are many different configurations. We’re seeing so many chip startups explore the surface area of possible chip configurations right now. But if you are optimizing for HBM bandwidth per watt—a metric that seems pretty relevant in LLM inference—this is the design to go with at this point, right? There’s nothing else that compares that we’ve seen specs on, or that has public specs on, let’s say. I’m not giving away too much there.

So, maybe just to talk about realizing it a little bit more: I don’t know, Bryan, do you want to talk a little bit about the software programming model and the microarchitecture? Do you want me to talk about that?

I think you’re more knowledgeable in that aspect, right? But actually, before I let you answer your own question, another very big, interesting point is the role of AI in all of this.

Right now, you can make an argument that OpenAI’s biggest advantage is that it’s able to access its own SOTA models—the new GPT-5 models—before anyone else. That’s the biggest difference, I would say, between OpenAI and some of the other new chip companies, SambaNova, Cerebras, et cetera.

The question is, how much did GPT-5 actually contribute to this? If you look at it from a differences point of view, this is one of the only differences between Jalapeño and the other chips. Did GPT-5 really contribute to most of these performance differences, or just a bit? But that’s just a tangent on AI’s work on development.

Actually, when Kimi K3 was released, there was a part on the blog about its designing of a chip. I forgot what it was about—maybe something. I’m not sure what the architecture was about, but they did talk about Kimi K3 developing a chip. I would guess GPT-5 has similar capabilities, and I would say to a better extent or to a higher degree. Sorry, back to you, Jordan.

Well, let me comment on the AI assistance on the architecture, because I think there are 2 ways in which AI clearly assisted the design and then the bring-up of the chip—namely, design and then bring-up.

On the design side, this clearly wasn’t GPT-5, because the RTL freeze was in July of last year. From February to July of last year, when they claimed that AI assistance helped them get an 8% reduction in SIMD area and then a 10% reduction in the matrix-engine area during design, this was pre-GPT-5 that we’re talking about. Conceptually, the models have to be getting better at RTL in the meantime, but they were already good enough to rapidly accelerate the really tedious, human-driven work that is RTL before a tape-out. I think that’s maybe the biggest claim here.

A lot of people understand that you can use these models for kernels or software engineering because you put them in a Codex harness, you put them in a loop, you let them test the thing, and then you just set a goal. People have had that experience, so they can kind of understand it. But I don’t think a lot of people have had the experience of designing a chip. It’s not like traditional software programming, and this was done with an older model. So I think that’s point number 1.

On the actual bring-up, the work is iterative and it’s a verifiable domain. This is exactly what RLVR should be good at. You should be able to give a model a task of improving the performance of a kernel or getting the kernel to be functionally correct against some unit verification script, against some test cases, and then just let the model rip. Let it try.

Dylan made this point on the Dwarkesh Podcast that he was on recently: for years, the companies getting the most value out of using AI were not actually the companies providing AI. OpenAI and Anthropic were not profitable for a very long time, and now they’re just turning a profit—I mean, still running incredibly high-margin businesses—but they’re just realizing the profitability of training these advanced models. Meanwhile, Jane Street is going out there and printing $15 billion in a quarter, clearly using AI for trading or something like that, right? There are many other companies being started based on the use of AI.

This is a very clear example of OpenAI keeping the benefits of having access to a model before everybody else for themselves. They can tape out a chip and their competitors can’t. It’s a sign of what’s to come, I think. That’s the simplest way to put it. They’re going to be able to go into many domains that are tangentially related to software, where the model needs to be able to control a computer, but it’s not explicitly the thing you’re training it for. You build RL environments, spend enough tokens, spend enough time on reasoning and enough rollouts—enough attempts at the problem—and you’re going to get a good result. That seems to be the lesson here.

Yeah, exactly. I think Anthropic is also realizing the same thing. They are starting to hire silicon people—that’s in the same vein as OpenAI—and they are doing a lot of stuff in the laboratory. They got LLMs to control microscopes and whatnot recently, and they’re going quite far into this biological-sciences field.

So I really agree with you, Jordan, on the point that these frontier-model companies are realizing what this—oops, lots of interference from Jordan—but there are a lot of good downstream impacts of having a good model first, not just making money from inference revenue. There are lots of interesting developments.

What are your thoughts, dude?

8. Architecture Deep Dive

Yeah, I agree. I think the progress that AI labs are making is only getting faster, right? That’s really because they’re using their own models very effectively to drive product innovation much faster. I remember—I think it was earlier this year—Anthropic was releasing a new product every week or something. I think everything was basically on autopilot; they were just using Claude Code for everything, right? So, yeah, I agree with what you guys have said.

Yeah. I mean, the cynical view of a program like this, for both OpenAI and Anthropic, and even Meta and Microsoft, is that it’s kind of a head fake that gets them a discount on the Nvidia GPUs and therefore pays for itself. You only need to spend a few hundred million on a program to tape out a chip, and the year or 2 it takes to do it could potentially help with the negotiations. If those negotiations are measured to the tune of hundreds of billions of dollars, then it pays for itself pretty quickly.

But the non-cynical view is that this is a real thing, and they’re only going to do more of this in the future. There’s no reason that they’re going to be less vertically integrated and less interested in developing chips this time next year.

And there's no reason to say that the RTL time from initial design to freeze to tape-out can't get even shorter than 9 months. I think you just need to think about where to go from there.

Maybe the other thing that was kind of interesting here is the architecture. There's lots to say about it, but the quick, high-level take is that it looks like a TPU with much smaller systolic arrays. A systolic array is the way a TPU has its processing elements laid out.

The criticism of chips like a TPU, Trainium, or even some of the TPU-inspired ones, like Etched or MatX, is that when they use these really big systolic arrays, they can have these weird cliffs. Small batch dimensions, like the M dimension in your M-N-K for a matrix multiplication, get small, which is what happens when you have lots of experts and very few requests—low concurrency. These skinny GEMMs, or skinny matrix multiplications, can waste a lot of resources.

Even with odd numbers, if you go slightly over 256 or slightly over 128, you're spending an entire kernel launch on the device side just to run one little skinny GEMM. All of the processing elements on the systolic array aren't being used, so the chip is inefficient and you don't actually maximize the FLOPs on the chip itself.

The trade-off here, of course, is that to get more efficiency with tiling—to reduce issues with padding overhead or alignment on the matrix dimensions—you just use smaller systolic arrays, and that's what they've done here. I think that's been really smart, clearly, for efficiency across the curve. The argument the other way is that they're going to miss out on some power efficiency and data-movement efficiency because you have to have more small elements instead of one big element.

The way they've solved this, I guess, is by being really smart about how they place weights and KV caches, using synchronization between cores selectively, and saving the collective network—the NoC, or network on chip, that connects the HBM slices and the computing elements together—really, really sparingly. The results speak for themselves; you can see the performance there.

This might be both a chip that's simpler to reason about than a GPU and a chip that's a little more flexible for some of these weird, changing dimensions over time than a TPU. Clearly, the people who have experience using GPUs—which OpenAI has plenty of experience programming—combined with the people who have experience designing TPUs have resulted in a pretty well-balanced system.

Maybe the other thing is that it has an L1 cache, which is quite funny. All of these accelerators don't have L1 caches now. They rely so much on L2, namely SRAM. We hear SRAM all the time.

The reason, I guess, is that the companies designing other accelerators do want to use a software-managed scratchpad, and OpenAI is not using a scratchpad cache here. That makes the chip potentially harder to reason about when you think about barrier latencies and where you're going to move data. You have to be able to amortize the data movement by doing work on the CPU itself.

But that ties in with what I was saying at the very beginning, which is the second phase of using AI: actually programming kernels, meaning software that runs on the device side. To do this, we haven't really been able to verify it, other than scrolling through a 30,000-line file with some of the OpenAI engineers when we went on site with them.

It's literally just slop. It's not slop because it performs, but it's literally just AI-generated assembly, basically puked out in Gluon, this low-level kernel-programming language that they built on top of Triton, which uses this really interesting programming model. The people who were scrolling through this code with us clearly knew a whole bunch about hardware and the concepts in the system, but they had no idea what this MLA kernel they were showing us for DeepSeek actually did. You could go line by line and it was, “Nope, nope, nope.”

But it doesn't matter, right? The AI understands it, the AI tests it, and you see the results. It produces correct kernels that perform really well. I think this is just a sign of what's to come again: that actual code is not necessarily something a human has to reason about deeply if the AI knows how to manipulate the data movement and the processing elements on the hardware that you've given it.

Okay, we're kind of running out of time here. We've been going for a while. There are 3 things that I had in my notes that we wanted to talk about. They don't do PD disaggregation. Bryan, maybe you can rant about that, because you spend all of your time debugging PD disaggregation.

9. Doom and Wrap

The second thing is that we didn't really talk about the system architecture. We can talk a little bit about how they do the scale-up and scale-out domains. They don't call it scale-out, but whatever it is—the multi-tier scale-up stuff—is just a total mess to try to understand. We probably can't communicate it on a podcast.

The third thing is that it's a generalized inference chip. It's not co-designed with their models, like they keep saying, and the proof of that is that it runs Doom at 36 frames per second.

Yeah. We remember talking to these guys, and we were like the angry, disappointed mother who comes in. They show us this groundbreaking chip that's so fast and runs all this stuff, and we're like, “But does it run Agent X?”

“No.”

96% on the test. What 4 questions did you get wrong? Anyway, does anyone feel there's anything left unsaid about the chip? Pretty exciting release, eh?

Yeah, very. I'm really excited to see where the roadmap goes next. Bryan mentioned this earlier, but Anthropic is building one, or rather, they're hiring a team to do it. I think this really sets a pretty high benchmark for Anthropic to meet or beat, but I think there's every reason to believe that Anthropic could achieve a similar outcome. I'm very excited to see that as well.

And they're hiring the team now, right? So it's only about a month and a half until the RTL freeze, and then 4 or 5 more months for the tape-out. We should be able to get Anthropic custom chip tape-outs after 1 9-month cycle, right? One pregnancy term.

A little bit longer. Yeah.

You guys don't like that one. Okay, we're going to end on one other joke. Bryan, you like the one about converting energy usage to calories, right? So I want to finish with this one.

We converted human speech and compared some of this stuff on a calorie basis, right? A calorie is how much energy it takes to burn a cubic centimeter of water, I think, which is the equivalent of whatever 1 joule is—0.239 food calories. We'll throw this one up on screen to lead everybody off with a little joke.

If you convert the efficiency of some of these DeepSeek results that we have on Agent X, we've identified that human speech is roughly 20 to 22 times more energy-efficient than the concurrency-1 B300 configuration that we were testing it against. A human speaks at 3.3 tokens per second, but these batch-1 configurations are going at 180 tokens per second—much faster than the human brain can work, consume calories, and produce speech.

I think the caveat is that not all human-spoken tokens are very high quality.

I've got to caveat some of my interactions with Claude, too. Some of this nonsense that it's been spitting back at me recently—I haven't been too pleased with it, either. I can't see all the thinking traces, anyway, but the Claude version of English it's been giving me has not been that great.

If we're comparing the machines based on how many calories they're consuming per token and comparing them to our speech, we're really in competition with the machines at this point.

Hopefully Jevons's paradox continues, and everybody that produces chips wants to consume more tokens, produce more chips, and hire more people. Everybody gets to come have fun.

One last joke before we leave, because I just saw it: the best cover image in a while, I'd say. We'll leave that one on screen—Lisa and Jensen enjoying a nice spicy pot of Chana or Katsu, whatever the codenames were for the trays and the racks and stuff in there. That's the motivation behind that.

Vindaloo.

Yeah, right.

Sorry about that. All right, we'll sign off with that image in everybody's brain who's watching online. [laughter] Thanks for listening, guys.