Ep. 017 - DeepSeek V4 and Huawei Ascend NPU Performance (InferenceX) | Kimbo Chen, Cam Quilici, Bryan Shan, Jordan Nanos

This week, we've got Cam, Kimbo, and Bryan, and we're going to talk about everything InferenceX. The latest InferenceX article included a whole review of DeepSeek V4 and what it takes to go from day 0 performance to day 43 performance—in other words, how much these guys grind over the month. I'm really excited to dig into some topics here. We're going to talk about the model itself, the implementation, how it's running on different hardware, some of the vLLM-versus-SGLang Twitter drama that's been going on, an agentic benchmark that we're working on, and lots of fun stuff. Guys, welcome to the show. I'm excited for this one.

Super excited to get started. Let's go.

Nice to be here, yeah.

Awesome. All right, Kimbo, let's start with you. I think the burning question for the general audience here is: What's different going from DeepSeek V3 to V4? When you describe what's different, what impact does that have on running InferenceX? In other words, what does day 0 support actually mean when it comes to a new model architecture?

1. The V4 Architecture Shift

Yeah. I think the headline change, or feature, of V4 compared with V3 and R1 is the 1 million-token context length. To achieve a 1 million-token context length, DeepSeek made several very aggressive innovations in the attention mechanism. That's the first thing. The second thing is MegaMoE, which speeds up the expert FFN computations.

First, the attention mechanism: DeepSeek V4 comes with 2 variants of sparse attention—compressed sparse attention and highly compressed attention. Both are built on top of the previous version, DeepSeek Sparse Attention, which came out with 3.2. The idea is that, by sparsifying attention between the query and the key-values, you can reduce memory reads, allowing a 1 million-token context length without exploding memory requirements. In addition, both compressed sparse attention and highly compressed attention have an additional embedding compressor, which further decreases the size of each KV-cache entry.

After that, they both incorporate some sort of sliding window, which further decreases KV-cache usage. Combined with all this, I think DeepSeek is quoting around a 100x reduction in KV-cache usage compared with a standard MOA model. That's the headline feature of DeepSeek V4.

2. Day Zero Support Gets Harder

Yeah, and it's also a different architecture in terms of more total parameters and fewer active parameters compared with V3, so it's literally a different model, not just an update to the weights. Bryan, maybe I can kick this to you: What does this mean when it comes to actually getting support from the various inference runtimes, and what work is required to get performance results out of a system on day 0 when the model is released?

There were a lot of changes—not just, as you say, not just the weights, but the sizes of certain things as well, such as hidden sizes and dimensions. So it's not just adding support for new attention mechanisms or attention blocks; it might also mean changing or tuning things that the community thought were constant. In this case, for NVIDIA, I guess it was the mHC dimension. DeepSeek had previously always used an mHC dimension that was, I think, half of what Pro uses.

V4 Flash and previous iterations of DeepSeek V3 all use the same mHC dimension, but Pro used a new one, which caused a hiccup for NVIDIA on day 0. For the rest, everyone adopted it quite well on day 0.

It's worth mentioning that some companies like vLLM and SGLang had early access under NDA, but companies like NVIDIA did not, so that made it more difficult to implement on day 0.

They had a head start with implementation, which is why there weren't as many hiccups. I don't remember if you guys saw it, but there was a video circulating on Twitter of Huawei talking about the optimizations. It was in Chinese, but someone provided a translation. They had some very good optimization suggestions for day 0. If you just follow what Huawei did, I think you'll get quite good performance when optimizing from day 0. They talked about kernel fusions and index optimizations. It's a really good guide for day 0 optimization.

I've been reading a lot about how this architecture is extremely tailored to GPU and GPGPU architectures rather than things based on systolic arrays like TPUs. Is there a reason for that? Have you heard that as well? I'm curious.

I think, solely in terms of what you said about systolic arrays, no, because a systolic array is essentially a hardware architecture specialized for matrix multiplication, right? GPUs literally have tensor cores, which are also specialized for matrix multiplications. Models do matrix multiplications, so I don't see anything very special that V4 is specializing for. I guess what you might be seeing is probably related to MegaMoE. That's very tailored to GPUs because of the way the kernels launch and everything. Is that what you're seeing?

Yeah, probably.

3. MegaMoE Breaks Kernel Boundaries

Let's talk about MegaMoE. What is MegaMoE actually, Kimbo? My basic idea is just that it's a huge fused kernel, but I'm curious what it is.

Fused kernels are one thing. The whole concept of a megakernel is that you're breaking the typical kernel boundaries. Compared with kernel fusion, which is either manually written by kernel engineers who go through the math and see what can be merged together, or compiler optimization that removes kernel launches, a megakernel is more about breaking kernel boundaries in ways that aren't typical.

One concrete example is register usage. In a typical kernel, what happens is that you launch a kernel and do computation. In a von Neumann architecture, you load data into registers, perform computations in registers, and then, at some point, store it back to HBM. What a megakernel does is fuse different kernels so that it can remove the round trip of moving data between registers and HBM. When a register is free, it can automatically do the next operations from future kernels instead of waiting for an entire kernel to complete.

For MegaMoE, in addition to that, they also merge compute and communication, allowing computation and communication to overlap. It's all merged into one kernel, so they can do very aggressive overlapping.

What's the downside, then, of fusing kernels? It seems like you just fuse everything, right?

In the extreme case, as we see in Hazy Research, they literally fuse a whole model, like Llama 8B. I would say the downside—or the reason preventing people from doing that—is that it's a lot of engineering work, and in addition to the engineering work, you have to justify it with something, right?

MegaMoE is about aggressively reducing latency, and in a large-batch or training scenario, it kind of doesn't make sense, or there's a limit to where it stops making sense. If you're training with a super-large batch, like a couple hundred million tokens per batch, your computation and communication are sufficiently overlapped. Kernel launch time isn't the bottleneck, so it doesn't make sense to use MegaMoE.

Another downside, or another difficulty with megakernels, is that aggressive resource allocation on the fly creates a lot of memory pressure. I imagine that would require a lot of work managing the GPUs so that they don't overheat or run into other related issues, which would in turn affect performance.

Makes sense.

Okay, I want to throw something on screen here: a screenshot from the V4 paper explaining the open-source megakernel. It specifically says 2 things: It's been validated on NVIDIA GPUs and Huawei Ascend NPUs, and nothing else. Obviously, we've seen the open-source CUDA-based megakernel called MegaMoE, which inference-serving runtimes such as vLLM, SGLang, and TensorRT-LLM can implement for themselves. The claimed performance improvement is somewhere between 1.5x and 1.73x faster.

So clearly, they’ve realized the benefits of this engineering work by doing this. But I want to maybe share a second thing on screen, which is that the open-source community, just because the code is out there, doesn’t necessarily have the opportunity to benefit from day 1. What we’ve been tracking, really—and Bryan, maybe you can take us through this experience in detail—is how quickly these configs can come online. In other words, how quickly can a given inference-serving runtime actually support things?

4. Performance Compounds Through Optimization

I thought this little GIF video from the article was quite nice in terms of how it shows progress over time. As we count up the days, we start to see more hardware being supported, as the B200, B300, GB200, and GB300 come online, as well as the MI355 from Nvidia, and then you can start to see the performance improve. For people watching, these lines going from left to right mean either more throughput or lower latency per user or higher throughput per user in terms of interactivity. Maybe you can take me through it: day 0, there are 2 or 3 different hardware platforms supported, and then a month later, everything’s supported. But in that whole time, a lot of performance improvements happen. What are some examples of performance improvements that were just dropped overnight and then were a big win, where people can suddenly have, I don’t know, 20% more throughput—which means 20% more users or 20% more profitability for the existing users—or whatever for a given inference endpoint provider?

Wait, before I move on to that, huge shout-out to our front-end engineer, Alec, for making that video feature. It is very cool and shows you very nicely how improvements were made.

Moving on to the question, I guess one thing we can look at to understand these improvements is AMD. As the paper said earlier, a lot of the stuff was optimized really for NVIDIA GPUs, and because of how big the CUDA community is and how many people use CUDA, there were huge incentives for SGLang and vLLM, in the limited time they had with the model, to prioritize NVIDIA. So AMD catching up is actually an interesting look at real optimization, getting from not working on day 0 to working on day 0.

Maybe a first step would be getting the quantized versions to work. AMD on day 0 had FP8 only. There was no native FP4 support, which was quite unfortunate. Actually, very large gains came from getting it to work in FP4, of course, because the calculation is faster and less memory needs to be transferred. We have a very nice plot on InferenceX on the AMD improvements, if you could bring it up, Jordan. Thanks.

The biggest jump was converting a lot of the kernels from Torch fallbacks into either AITER or Triton. You can see the SGLang graph evolving the most. There’s a breakdown of the optimizations below in the image, I think. There’s a pretty messy image with orange text in Excalidraw. Sorry.

There were very big improvements just from changing kernels from Torch fallbacks into either AITER or Triton, et cetera. It’s very cool to see the impact these individual improvements have. Of course, it’s not just one humongous step; it’s always step by step. You optimize one part, and then you optimize another.

Yeah.

Yeah, very cool stuff.

I want to take a second to just talk about how cool this is and also shout out all the AMD engineers and NVIDIA engineers that we work with, because this is all them. We contribute a few things upstream, like kernel libraries and stuff, but we’re just a team of 3 people, and this is all the engineers.

I think this is something so cool about InferenceX, and this was the whole thesis of the project when we started it. A lot of things online, a lot of benchmarks, just show the end product in terms of performance. But really, it’s a lot of hard work and tiny iterations, increasing throughput by 5% at a time, and that eventually compounds to make things performant. You can see that here, right?

Whenever a new model is released, like MiniMax-M3 just released, we’re doing the same thing. Over the course of a month or 2, you’re going to see that you start out with PyTorch, just Torch-native fallback, and then you use custom kernels, and then you make the custom kernels even better. You just keep making them better and better and pushing the frontier forward. So, yeah, this is really cool to see.

I also find it fascinating when I look at these things and dig in to try to understand what these optimizations are. Every single one of them, if you tease out just that one optimization—if you look at the tuned parallelism point or the fused KV-compressed tiling point—it might seem simple in isolation and, in some cases, maybe obvious that people should have been doing it. But if you try to reason about the entire system all at once, where there are all of these improvements constantly happening, it becomes incredibly complicated.

That’s what’s so impressive about watching these engineering teams work: the ability to take all of these individual optimizations and build them all together into one thing, which is just better performance or more efficiency. Super cool.

5. The Runtime Competition Drives Progress

The next thing that this obviously leads to is the comparison between the different runtimes. You guys are big on avoiding the war between vLLM and SGLang, the Twitter drama. Can you talk about the reason for this and why, in some ways, having competition between different open-source inference engines, as well as vendor-specific libraries, is actually driving a lot of innovation and improving things instead of just waiting for somebody to be nice enough to share what they’ve learned? Many of them are forced to innovate really fast in order to stay ahead.

Yeah, we tend to stray away from comparing vLLM to SGLang on InferenceX specifically, mainly because we didn’t find it beneficial necessarily to make it a competition. We more so just wanted to showcase each one in isolation. But, on the other hand, it does kind of create some competitive spirit, which makes both sides move faster.

That can also be a problem because we only have limited compute to run things. So when you double the amount of submissions on a certain model, it’s a lot of CI time. But, yeah, we’ve seen great improvements by both vLLM and SGLang, and I think a little bit of competition is good, but—

And what about vendor-specific libraries? We can talk about some of the other chips, but maybe the most open or easy for us to run is NVIDIA and AMD because we have NVIDIA and AMD GPUs. We’ll talk about Huawei in a second.

Both of them have closed-source proprietary libraries in the form of TensorRT-LLM with Dynamo for NVIDIA, and then AITER and Mori, all the libraries from AMD that make it up. But they’re not—I guess, what’s the—they’re not—the vendor-specific libraries that are closed source are not always the best performance and don’t always allow you the most customization. So what’s the open-source versus closed-source dynamic that’s going on when you have to try these things out?

The way I think about it is, TensorRT-LLM and, you know, AITER, the optimized engine for ROCm, are hyper-specific to certain AMD and NVIDIA architectures that will run really fast, but they’re not necessarily completely portable and are not always 100% open source. Whereas vLLM and SGLang, you can fork them and go start your own codebase altogether—whatever, Fireworks. They’re very user-friendly, and they have OpenAI API specifications and such. That’s kind of how I look at it, but I don’t know. Maybe Bryan and Kimbo have different opinions.

In my opinion, it’s just more competition. On our benchmark, we also have a disclaimer saying that AITER has no customers.

But that’s the other problem: if your market just isn’t as big as NVIDIA’s, your product needs to be really, really much better. Mori is a lot better, though, than the previous fork. But AITER—the development is okay. AMD is putting support into it, which you’d like to see.

SGLang just has much more help from the open-source community and kernel development, which is really good. AITER just needs to close that gap, I guess. Who knows? Maybe it’ll take some time for that gap to close and for AITER to get more accepted by the community, I guess.

Mm-hmm.

Yeah.

Kimbo, I have a question for you. One of the things that I’ve been asked before and I’ve seen on Twitter is basically: why SGLang and vLLM? Why wouldn’t SGLang just copy all the good parts of vLLM and vice versa, to the point where they basically just converge to the same thing?

We’re talking about inference, right? We’re talking about being 1% faster or something, right? Everything is kind of being worked out in the open. So it doesn’t have to be you, Kimbo, but maybe Jordan and Bryan, why do you think it’s a natural thing that there are basically precisely 2—maybe a few other open runtimes—and they’re so head-to-head, and we’re not necessarily converging on 1 inference engine? I mean, it is kind of an interesting topic.

Based on my understanding, first of all, objectively speaking, both of them came from the same—literally the same—Berkeley lab. They are lab mates, so that’s the thing. They started with a similar idea in the same lab and started the project, and apparently something happened between them. Some of the major maintainers, the major project developers, had some problems between them, which I honestly don’t know anything about. Then they just parted ways while having very similar ideas.

In principle, they are doing the same things, and throughout the development, throughout these years, they are, like you said, copying features from each other. I’m sure you probably heard from our collaborators that the other side is copying and copy-pasting our code to support a feature. I’ve heard both sides, so there’s that.

Maybe the easiest analogy is: Why do OpenAI and Anthropic exist? They were all one company, and then Dario was working on safety stuff and something happened. Then he left, and now they really drive each other in a way that I think is benefiting the ecosystem, because they both land counterpunches. We benefit as they improve quality and drive prices lower.

Yeah. I think the TL;DR is that competition is good. It’s interesting that they’re very similar, but they have—

I just want to say one other thing, though: I don’t think it’s completely a historical artifact the way Kimbo describes it. There is obviously stuff that’s downstream of these runtimes now, whether it be customers or other libraries that depend on them.

Having 2 different providers allows you to make a decision about who’s going to prioritize your features, who’s going to merge your stuff, where you’re going to get support from, and how you’re going to work with vendors, because they’re just 1 point. There’s stuff that’s upstream of them, like vendor-specific libraries in some cases, or the hardware—literally CI testing, inference tech stuff.

Then there’s stuff downstream, which is RL libraries. You look at Slime, VeRL, or Prime RL and stuff. If there are features that they want to see from the runtime, they need to make a choice and go with the ones that are going to support them. There are many customers, like you mentioned: the inference endpoint-serving providers, as well as the labs.

We know companies like Microsoft are using these technologies. They don’t have their own stuff in-house. Poolside talked about that in their paper. GLM—the whole thing’s actually SGLang, right? A lot of people depend on this stuff right now, and I think it’s at least good to have 2 options to choose from in open source if you’re somebody making a library and you need support from somebody.

You can have multiruntime backend support if you really need to, if you’re not getting what you want from the community and you need to go somewhere else. Okay, not to take this in too far a direction, but I really, really want to hear about Huawei. This is making me think a lot about proprietary runtimes and software experience, and what this NPU chip is going to look like when there are a lot more users.

It goes from what we thought was 1 or 2 lines in a paper saying, “Yeah, we have Huawei support,” and then no real proof. But Bryan, man, there’s real proof now that DeepSeek is running on Huawei. Can you take us through what you’ve found in the analysis you did on the numbers they’ve shared?

Yeah. Actually, before that, I’m going to talk about the DeepSeek delays, which there were a lot of rumors about on X. DeepSeek was planned to release on Chinese New Year, and then it got pushed back and back and back. Every time it was, “Oh, it’s going to drop this weekend. Oh, never mind, it’s this weekend.”

There were conflicting reasons. Some people on X were talking about how they were actually trying to get a chat model working on Huawei. That’s why they were delaying it and optimizing it first before releasing it. Of course, there’s the other explanation that they just wanted better evaluation results.

Regardless, the performance on Huawei at release was real. There were benchmarks and profiles shared by the Huawei team, and looking at the profiles, they were very elegant. The kernel optimizations, as I previously mentioned, were quite sophisticated at that point. So they probably did get much longer access than vLLM or SGLang to optimize for the architecture.

In the future, we would release another follow-up article on this regarding Huawei’s results and comparing them with other chips. I think the biggest—

I don’t want to distract you too much, but I’m curious if you can compare the state of the software toolkit, CANN from Huawei, to where things sit with some of the other chips, like NVIDIA and AMD. I know you haven’t used it hands-on, but in theory, the software is open source, and it can now run a close-to-frontier open-source model, like one of the best open-source models in the world. So this chip, or this system, has come a long way, right?

Yeah. But like you mentioned, their code is all open source, same as vLLM or SGLang. If you want to look at it, you have to go to a Chinese version of GitHub. It’s called GitCode. But their documentation is very good.

The code is open source, similar to vLLM and SGLang, but not similar to TensorRT from NVIDIA or AITER from AMD.

Yeah.

In other words, you can read how they implement the kernels.

Yeah. It’s all on GitCode. It’s very interesting to see how they do things differently compared with NVIDIA or AMD. Their documentation, in my opinion, has been very good. They do frequent meetups in China, where you can talk to the CANN developers. They have weekly calls, I think—a schedule of weekly calls where you can hop on and talk to engineers.

Our takeaway from this is just that Huawei is very enthusiastic about getting this working and about open-source contributions to CANN. The direction has been very good. Some minor notes: Huawei has implemented some optimizations from papers slightly before NVIDIA.

The one that I talked about in the article was about fusing communication and computation kernels. If I’m not wrong, NCCL released it in 2024, something like that, but Huawei released it much earlier, right after the paper on it was released, actually, which is quite interesting to me. The Chinese developers are indeed 10X developers, and their velocity is actually great.

When we compound that with open source—yeah, thanks, Jordan, for mentioning MC²—their velocity will definitely be propelled by Chinese open source.

Yeah, fascinating stuff. Kimbo, let’s bring you in here. When you hear this stuff about support on Huawei and then you see something like DeepSeek V4 Pro’s 75% off discount at the start, which then becomes permanent, what does that make you think for the future of open-source model competitiveness with closed-source models if they’re just going to try to drive the price down to zero and run it on Huawei hardware or whatever GPUs they can get their hands on?

Yeah. First of all, it’s pronounced Huawei. There’s an H in it there. I’m a bit too annoyed by this.

Sorry.

No, no, no. It’s my bad. I have this issue.

Aside from that, ever since we’ve seen Z.ai with GLM able to serve its models on Huawei Ascend chips, we’ve also seen that this time DeepSeek V4 can do that too. It shows that Huawei chips are getting there and becoming a very strong option in addition to what we know from GPUs and TPUs.

The Chinese have a very different ecosystem. They built a lot of things on their own, like Bryan mentioned: HCCL, MC², and everything. The engineers have also been very aggressive in optimizing everything. I think the accumulation of all these things ends up with DeepSeek being able to offer a very low price.

I imagine one of the reasons they could probably keep the price low is that they’ve also pushed a lot more optimization since release. I would also suspect that they are trying to capture the market. Based on my understanding, in China’s chat or AI model market, ByteDance is still taking the majority, so everyone else is fighting for market share.

They’re trying very hard, at the cost of probably a negative margin or at least just zero margin, to get people to use it. So I think that’s the 2 main things—2 main implications.

Makes sense, man. Cam, do you think David versus Goliath is the right analogy for Huawei versus everybody else?

Yeah, dude, I think so. We’ll see. Maybe Huawei is actually Goliath. We wouldn’t know that.

It’s David versus Goliath, man.

Thank you, Cam.

David, yeah.

Cool. All right, guys, this has been quite the tour of everything DeepSeek V4 on InferenceX. Cam, maybe we could leave by talking a little bit about the future.

Yeah.

Some big work is coming with the agentic benchmark. Everything we've done so far has been fixed input and output sequence lengths. What's coming next?

6. The Agentic Benchmark Changes Everything

One of the things with MiniMax M3 and DeepSeek V4 is the 1-million-token context length. Right now, obviously, on InferenceX, we're not testing that. We're testing fixed sequence lengths of 8K, 1K, 1K, 1K, which nobody uses.

But I said this in the last podcast, and it's still good to have the 8K, 1K and 1K, 1K because it's testing and showcasing, in my opinion, basically pure chip performance, right? There's no prefix caching or anything like that. So we're moving it up, and we're going to showcase—I mean, inference is a systems problem, so we're moving up the stack.

We're going to showcase and run an agentic benchmark with real Claude Code traces that we've collected internally. We're going to showcase things like NVIDIA Dynamo and KV Block Manager, other sort of KV block managers, and showcase different PD disaggregation optimizations. I think it's going to be really cool, and that's going to come in the next couple of weeks, so be on the lookout for that.

That's all I have to say about it right now, but it'll be really cool when it comes out.

Excited for it, man. Cool. Okay, guys, is there anything you think is left unsaid on this stuff so far? Anything about DeepSeek, future plans or InferenceX in general that you guys are looking forward to?

Let me think. Bryan, do we have anything else upcoming that we can talk about? I'd say be on the lookout—

AgentX.

Yeah, AgentX. That's what we were just talking about. So that's coming in the next couple of weeks, but I'd say be on the lookout—

New chips.

Yeah, be on the lookout for new chips. We have quite a few in the pipeline, so that should be really, really interesting.

Awesome. And Kimbo, maybe next time we can talk about RL system performance as well. I know you've been doing a lot of work there. Some of it depends on the inference runtime, but there's much more to an RL system performing well than just having a fast runtime, right?

This is a great show. Appreciate you taking us through this. If anybody wants to find you, InferenceX on GitHub, find us on Twitter making memes. Maybe it creeps to LinkedIn and YouTube sometimes as well.

Yeah, I like YouTube short.

More memes about wide expert parallelism and related topics.

Yeah.

All right. Thanks for coming on.

Thanks, guys. Bye-bye.

Thank you, Jordan. Thank you, everyone.