推理已成为新的训练——Philip Kiely 与 Ali Taha，Basten

Shawn Wang

Okay. We're here in the studio with Philip, an old friend from Inference Engineering, the book, as well as Baseten and everything that you've done—you and I have done before—as well as Ali. Welcome.

Ali Raza

Pleasure to meet.

Shawn Wang

Waterloo intern.

Ali Raza

Waterloo intern, always.

Shawn Wang

When did you get “Waterloo intern” as a handle?

Ali Raza

As a handle? I think the rebranding happened in mid-March. When I saw it was open, I was like, “I have to take it. Up for grabs.”

The problem is that Ali is really good at his job and isn't going to be an intern much longer. So we have to figure out who's going to get the handle.

Ali Raza

Well, I'll pass the torch over to the next—

Shawn Wang

Oh, okay. It can be like you just pass it to another Waterloo grad.

Ali Raza

To another Waterloo intern—

Yeah.

Ali Raza

No, bro.

Shawn Wang

Yeah.

Ali Raza

Intern.

Intern, yeah.

Ali Raza

And, no, no—

Shawn Wang

You gotta get an intern from Waterloo.

Ali Raza

I'm just—

Shawn Wang

Right, right, right, right.

Ali Raza

Yeah, I've gotta get an intern from Waterloo.

Shawn Wang

Yeah. Right, right.

Ali Raza

But they have to follow the path from—

Shawn Wang

Oh, it could, but it could come from Baseten, so it's like whoever Baseten gets from Waterloo—

Ali Raza

Right.

Yeah.

Shawn Wang

...has the title of Waterloo.

Ali Raza

It stays in Baseten.

Exactly. Halfway through your internship, you either get it or you're out.

Shawn Wang

You should also do a big graduation ceremony where you change the handle—

Ali Raza

Right.

Shawn Wang

...for everybody. I mean—

Yeah.

Shawn Wang

...you guys are good at ceremonies, clearly. We had a nice launch of the book, very successful. But before we get into all that, I want to start off with a fun question for you. Okay, you're an expert inference engineer. What happens when I send a long query, say 200,000 tokens, into Baseten's inference? What's the process of the query through GPU model routing, balancing, all that? What is all the stuff that we don't think about?

With a long query specifically, the first thing that I'm going to ask is, “Have you sent me this query before, or at least part of it?” And I really hope you have, because it's going to be a lot easier for me and a lot cheaper for you. So the first thing that we're going to look at is some kind of cache-aware routing, where we're going to see that we probably have a number of instances, a number of replicas up serving whatever model you're hitting. We want to send this one to something with, number 1, available prefill workers, and number 2, ideally some cached input already there, so that we can skip prefill on at least part of these 200,000 tokens.

If you're doing 200,000 tokens, it's probably coding or a multi-turn agent or something where you'd expect to have that cached. If you don't, we're going to have to send it to a prefill worker. We've, at least on certain models, disaggregated prefill and decode, so you're going to have one set of GPUs that's solely going to process the input, create that KV cache, and get you your first token. Then that's going to be passed over to a separate set of GPUs, which is going to decode. We're going to iteratively make those tokens.

We're probably going to have some kind of speculator model in front of that. I'm going to assume that you're doing coding, and because of that, our speculator model, which assumes you're doing coding, is going to have a high draft-token acceptance rate. If I'm wrong and you're asking me to summarize every Harry Potter book, it's going to be slower. Then we stream that output to you, account for it, charge you a couple of pennies, and say, “Hey, would you like to send another one?”

Shawn Wang

Except Baseten doesn't charge by pennies.

Well, yeah, we charge. I'm assuming that we're talking about the public model APIs. If you are setting up a dedicated deployment, then, yeah, it's not pennies.

Shawn Wang

Yeah. I mean, one of the key differentiators when I was talking with Baseten initially was that people who want very, very high volume just need to rent by the box, because then it's up to you to figure out how to saturate the box.

Ali Raza

And more often than not, it's way cheaper if you're pushing millions of tokens per hour if you just pay per hour instead of pay per token.

Yeah, they do. I think that we've increasingly seen a lot of demand for the sort of pay-per-token APIs just because everyone wants to try open models, and then once they find a use case that's really sticky, they move over to dedicated.

Shawn Wang

Is there a best practice on when it's time to swap over?

A couple reasons. Yeah, reliability, that's a big one, right?

Ali Raza

If they have a very specific use case, they want you to train something specifically for them. Like, they want their own spec decode, for instance, for their own traffic.

Shawn Wang

Spec decode is speculative decoding.

Ali Raza

Speculative decoding. Yeah.

Shawn Wang

You have to explain.

Ali Raza

Sorry. Basically, if you have a huge model, the model is going to generate 1 token at a time, every single turn, every single forward pass. So we attach this little parasite, this layer that goes on top of the model, and this model just has to predict. It does 3 very fast autoregressive forward passes, and it predicts 3 tokens. Then you do 1 forward pass over the entire original model in order to see whether those predictions were correct or not, and then you accept or reject them.

Now, this draft model is traffic-specific, so, like Philip said, if you're summarizing Harry Potter books, I can train that draft model exclusively on Harry Potter books, and I can guarantee you that I'm going to accept the 3 tokens every single time. In that case, I increase your decode speed. I wouldn't be able to provide this to you if you're on a shared endpoint, because I have no idea if you're doing Harry Potter, coding, or English. We don't know.

Also, there was a thing in the book that mentioned that if they really cared about a specific threshold—Chapter 4, I think. Do you remember that?

Yeah. The things that you can do are set a specific batch size, a specific parallelism strategy if you're trying to optimize for throughput versus latency. You can maybe—maybe an NVFP4 quant doesn't pass your benchmarks and you want to run a model at higher precision; you could do that. There's just a bunch of reasons why you might want to have your own endpoint, and the biggest one, of course, is that you don't have to deal with someone else doing 100 million tokens of benchmarking traffic at the endpoint when you happen to be trying to serve your users.

Shawn Wang

Yeah. I think one thing that is a classic journey is basically people asking, “What happens when you type Google into the browser?” Tool calling: is that just you're generating JSON, or is there more complication beyond that?

Ali Raza

Certain customers that we have have their own post-trained models, and so they demand tool calling that's not just, like, parsing a file or going to find the weather. It's something that's very specific, and you have to do post-training on this. If the post-training on the model is not good, or if the quantization after the post-training to get the inference to be fast is not good, the model will struggle reading the JSON file and reading the tool calling.

But it doesn't require its own sandbox. It's not like it's going to use that tool calling to escape a sandbox, and it doesn't have to be contained. It can just be a normal dedicated deployment. The challenge with tool calling, more and more, seems to be that the companies want certain tool calling, which is a very sensitive thing to train. Because you're dealing with all of the JSON outputs, if it doesn't close the end of the request in a very certain manner, you end up with a model that did the tool calling and the thinking. As a result, it didn't see the result and just hallucinated the result as it decoded. That seems to be the most challenging thing with tool calling, not really the sandbox model.

Yeah. That's a challenge on the training side, and then on the inference side, there's work that you can do to scope the possible output. So we published this, actually, at this point close to 2 years ago. The solution to this problem is that you basically make a state machine and use that to constrain the output to a specific format. So this is the structured-output problem, if you remember back in—

Shawn Wang

Yeah.

...back in the day—

Shawn Wang

The specific grammar is, uh—

Yeah, exactly.

Shawn Wang

...GML had this thing.

Yeah. So it's like the old-school “make sure this is only JSON, return only JSON, or my grandma's gonna die” type of prompts.

Shawn Wang

Yeah.

In our inference system, it's just a specified output format.

And you get the guarantee that your output is going to be structured along that format. Applying that to tool calls can help cut down on… Obviously, you can still call the wrong tool or call no tool. It doesn’t solve the certainty problem—

Shawn Wang

Mm-hmm.

Ali Raza

—but it at least solves the output-structuring problem—

Shawn Wang

Yeah.

Ali Raza

—within tool calls.

Shawn Wang

And MCP is just another form of tool, right? I guess.

Ali Raza

Yeah. Exactly.

Shawn Wang

As far as I know, there’s no special thing there.

Ali Raza

The thing I’m always explaining to people is that the LLM is actually not capable of doing anything. It’s only capable of making suggestions of what to do, and then, if those suggestions are formatted in a certain way and applied to a system that knows what to do with them, an action occurs.

Shawn Wang

Yeah. Part of the fun stuff is that this is solved outside of tool calling, too. In an agent loop, if the output is not correct, or if reasoning or tool calling was done in the reasoning cycle, you can just be like, “Oh, I don’t know what to do. Let me just try again.” It might get there after a few tries.

Ali Raza

Right.

Shawn Wang

And on your point of training, sometimes this is harder in smaller models, so you don’t have the same exact quality of output when you just swap from a big model, right?

Ali Raza

Yeah. I will say that, before we— I think we need to go back to inference engineering proper. I had expected that something would replace JSON because it’s hard to stream JSON: JSON must be complete, and you must have open and closed brackets and everything. So it’s hard to parse or validate something while it’s being streamed.

People invented all sorts of alternatives. I forget the name of some of them, but they’re basically something like TOML or YAML. But JSON seems to be dominant still.

Ali Raza

The JSON outputs aren’t that long, right? I guess you could have a long one, because tool calls also contain the arguments in them, and perhaps for a certain tool you might pass a very long argument. But my impression of the median tool call is that it’s a relatively small number of tokens, right?

So I would expect that speculators are generally fairly good at something as formatted as JSON. You would have a pretty fast decode step there, and the streaming wouldn’t be as valuable, but maybe I’m wrong about that.

Alessio Fanelli

I think you’re also bounded by the software that the model is going to integrate with—if the software is built with JSON for the tool calls, or if your customer says that this is how our software works and our tools are interfaced with JSON. You can ask them to change their software and say, “Yeah, this is going to be better for the model.” But with the training, it shouldn’t be that much of a difference. Also, it’s probably more profitable if it outputs more tokens.

Shawn Wang

Depends on your business model. It really depends. But I will say that, as a writer with a lot of experience with AI-generated output, I do try to move from text to JSON text—

Ali Raza

Mm.

Shawn Wang

—which is very long JSON, right? There are paragraphs in every field because I’m trying to structure it, right?

Ali Raza

Right.

Shawn Wang

I want you to first make factual statements, then make opinions—

Ali Raza

Mm-hmm.

Shawn Wang

—then make bullet-point summaries, have dates, have entity references—

Ali Raza

Mm.

Shawn Wang

—have your sources for references, all these things. Anyway, these are things that I think people who really, really experiment with structured output have to care about.

Let’s recurse up the stack a little bit. Before we started recording, you mentioned something that’s really cool: there’s a lot of inference engineering that goes on when a new model provider releases a new model, right? So let’s call it GLM-5.2, Kimi K3. I had previously assumed, especially if it’s GLM-5 to GLM-5.1 to GLM-5.2, that you’ve supported them before. Is it that much work?

Alessio Fanelli

It’s a lot of work.

Shawn Wang

Yeah. Okay. A lot of people—all you guys, right?—whenever a new model launches, rush to say, “Hugging Face supports this, Fireworks supports this, SpaceTime supports this.” And I’m like, “Yeah, of course we support it.” But what goes—

Ali Raza

I think—

Shawn Wang

—into—

Ali Raza

I think it’s more than just supporting it, too, right? It benefits the consumer a lot. I think it was with Kimi K2.5 or GLM-5.2, the latest, that there was sort of an inference war, right? One provider is at 90 tokens a second; the next day, we’re at 150. The next—

I kind of kicked that off with GLM-5.2. I wrote a Twitter article about it. It got, like, half a million views—

Alessio Fanelli

Best ending, number one.

Ali Raza

Yeah.

Alessio Fanelli

Or it’s for showing up.

Ali Raza

Yeah. Which—

Alessio Fanelli

Oh my God.

Ali Raza

—which then got everyone really excited about, “Hey, how can we bend the tracks a little bit further and—”

There’s a difference between supporting the model—as in, I can make a token out of this model—and supporting a model—as in, I have a production-ready API from this model.

Alessio Fanelli

Mm.

Ali Raza

Getting to the point where I can make a token out of this model is not that hard because, generally, the open-source inference engines—vLLM and SGLang—often receive weights ahead of time. Maintainers do that, or the people making the model merge pull requests to ensure support. So you can generally just get it working on the standard open-source stack without too much pain in most cases.

The challenge is that every inference company is going to have its own proprietary stack: some open-source components, some in-house stuff. For any arbitrary model, there’s going to be some new stuff. Sometimes you get lucky, like Kimi K2.5 to Kimi K2.6 was pretty similar.

Alessio Fanelli

Yeah, it was pure continued post-training, if I remember correctly.

Ali Raza

Even in those cases, there’s still stuff you have to do. You have to redo the quantization work. Generally, these models are not released in NVFP4, and we want them to be in NVFP4 for maximum Blackwell compatibility. So we have to perform that quantization and calibrate it to make sure that we’re not causing any kind of regression in the model’s intelligence.

And then we also have to train the speculator, as we’ve talked about. Generally, we have—obviously, we have ZDR, zero data retention, on our model APIs, so we don’t know exactly the traffic that people are sending us. But we know what’s popular. We know that coding use cases are popular. We know that agents and agentic use cases are popular. So we can get public datasets that are representative of that kind of traffic and train general speculators.

Now, with speculators today, you need to train the speculator using the base model itself because you’re getting hidden states out of the model from running inference on these specific prompts, and that is the training data you use to create the speculator. So there’s that process, for which you need the full model weights.

And then there’s, of course, the process of standing up all the infrastructure behind it, loading all this stuff in, and testing it. When there’s a new model with a newer architecture, the DeepSeek models tend to be the most challenging, as they have the most novel architectural stuff going on. But every new model has something.

I mean, Kimi K2 had—oh, sorry, GLM-5.2 had—

Alessio Fanelli

Sparse.

Ali Raza

Yeah, the—

Alessio Fanelli

Yeah.

Ali Raza

—the DSA.

Alessio Fanelli

Right, which is borrowed from DeepSeek.

Ali Raza

Yeah.

Alessio Fanelli

So you can copy-paste them, you know? I don’t know how this works.

Ali Raza

Mm.

Alessio Fanelli

Mm.

Ali Raza

You know, so, like—

Alessio Fanelli

Mm.

Ali Raza

—we had to build support for that into our runtime. And you’re right, it actually is really interesting the way that all of these open-source labs borrow from each other.

For example, GLM-5.2 doesn’t have vision. Something that Hailey, a guy on our team—if we could take a look at this—he grafted the Kimi vision encoder onto GLM-5.2.

Alessio Fanelli

Only training the projector.

Ali Raza

Exactly. So if you think about the encoder, there’s the encoder, which is the part that looks at the image and turns it into latent information—

Alessio Fanelli

You can say latent space. It’s okay.

Ali Raza

—and then there’s the projector that maps it onto the model itself. Then there are the model weights. You don’t want to mess with the model weights because you run the chance of making the model dumber at something else for the purpose of giving it vision.

Instead, Hailey started with just a projector, which is only a handful of millions of parameters.

Alessio Fanelli

That would be, yeah.

Ali Raza

Yeah. And—

Alessio Fanelli

Can you show the training one? Like the way it groks is—

Ali Raza

Very, very interesting.

Alessio Fanelli

And maybe—

Ali Raza

That right there—

Alessio Fanelli

Maybe, Ali, you should take it from here. You’ve got a better—

Ali Raza

Like—

Alessio Fanelli

—understanding of this than I do.

Ali Raza

Yeah, you can see the way he trained this is really, really cool. At the beginning, he was training it using, “Here’s a picture of a mountain. Can you describe what’s in this mountain?” That caused it to hit the first learning wall. Here you can see that all we’re trying to teach it is to translate the encoded image. It’s already taken the encoder from Kimi-VL. It’s taken the image. It’s—

Alessio Fanelli

Yeah. Frozen, frozen—

Ali Raza

Frozen, frozen—

Alessio Fanelli

—with adapter.

Ali Raza

Exactly.

Alessio Fanelli

Yeah.

Ali Raza

So the understanding brain is frozen and the eyes are frozen. We’re just trying to—

Alessio Fanelli

Align.

Ali Raza

—interconnect the eye and the brain, right? That’s the projector. You take the tokens, and then it’s like, “Oh, can you describe what’s in this image?” It might say, “It’s a mountain,” or it’s a person or it’s a human, whatever the case is. But that didn’t cause complete understanding.

So he changed it such that every image was associated with a data set of questions: Does this image have a white male? Does this image have birds in the top corner? Does this image have a scientist in it? It would have to answer the questions correctly. Using not just training on describing an image, but being able to answer question after question over time, you can see the grokking, which is genuinely insane—that retrofitting vision into a large LLM can learn to that extent.

Even for images that it doesn’t perform well on—for instance, if you show it a picture of Stephen Hawking and ask, “Who is this?”—maybe it doesn’t get it, but it will say something like, “This is Albert Einstein.” It still understands that this is a scientist who is a man who has done significant achievements and all that stuff. So that’s really, really cool.

Alessio Fanelli

Yeah. We’ve covered Hao Tian before, who was the author of the Lava paper that did this a while ago, and I think that’s very foundational work for anyone who hasn’t done vision work before.

Ali Raza

Same with CLIP and MetaCLIP, where you go from just captioning to building out questions off the image, and you can get much better performance.

Alessio Fanelli

Right. Right, right. Yeah, but what’s so exciting about this is, if you look at a model like this—obviously, this is a little bit more of a research project. It got to 56% on MMLU-Pro, I think, so not quite Fontio. But if you’re running this model, you haven’t suffered any loss in your GLM-5.2 quality. If you don’t have an image, it’ll just behave exactly the way it used to. Ultimately—

The Inference Engineering Masterclass

Which, in the inference code, you literally don’t include the other part, right?

Alessio Fanelli

Yeah. You would just skip the encoder if you don’t have an image input.

The Inference Engineering Masterclass

Okay.

Alessio Fanelli

Just confirming.

The Inference Engineering Masterclass

Does it affect a lot on the overall inference side? You’re not adding much. You’re adding a very small vision encoder. These are typically—

Alessio Fanelli

They’re super tiny.

The Inference Engineering Masterclass

—less than 1 billion parameters, right?

Alessio Fanelli

Yeah. There’s a little less standardization among vision encoders, so the support matrix can be a little sparser. But overall, yeah, it’s a pretty minor component of the overall system.

Ultimately, what you get out of the system is that all of a sudden you have Kimi-VL, GLM weights, and DeepSeek attention all in one model. That’s, I think—

Yeah.

Alessio Fanelli

—a lot of the power and beauty of open source: you can take all of these different components and combine them together into a system that’s better than any one of them—

Yeah.

Alessio Fanelli

—can be individually.

People used to say that you would also do Franken-merges, where you would take layers from each model. No—does anyone do that anymore?

The Inference Engineering Masterclass

Well, to your point previously, when you were mentioning the work that goes into supporting a model when it first comes out, like GLM-5.2 or MiniMax-M3 or whatever the case is, sometimes you do have to switch some things out.

For instance, the MiniMax-M3 head uses full attention, and with full attention you end up with this insane bottleneck in speculative decoding because you’re doing autoregressive token generation for 3 tokens, and you’re doing this O-squared computation over all of the tokens that are in your sequence. Your KV cache is very large because it’s not sparse; it’s not top-k.

So we find it better to say, “Okay, we’re going to replace this layer with a layer from another model that’s using GQA, for instance.” With the right training, you can get it to have the same acceptance rate. It’s very possible to retrofit layers from other models, and very much needed, actually.

If a layer is inefficient, the training just becomes the challenge. How do you ensure that you train it properly? Which, again, to your earlier point, is the mesh between training and inference. You need very good training in order to do fast inference. That’s becoming more and more true.

Alessio Fanelli

Yeah. Anything else on the support side when you say, “Get it fully production-ready”?

The Inference Engineering Masterclass

Yeah. I think there’s also a question of just—we can test a model to a pretty extensive degree, but we’re trying to get it out quickly. Then you see a bunch of other people test it, and you get interesting results.

There was an issue with GLM briefly where we had some model collapses where it would just output the same token over and over again for certain prompts at certain temperatures. Once you expose an endpoint to the real world, there are so many more varieties of things given to it that you’re able to discover and patch.

So it’s not just a day-zero process. It’s for the first week, for the first month, if a model remains popular. How do you both fix bugs and continue to push the envelope on performance?

What do you mean you don’t want your model outputting SSSSSS?

Is there loop detection on that stuff, by the way? It still happens quite a lot, which is surprising.

In our endpoint, if a model outputs the same exact token 4 or more times, we just cut the generation. We say, “Oh, sorry, this—try again,” or we reprocess the request. We know that if it’s 4 times the same token, it’s probably collapsed.

Ah. Is there a way to opt out in case I really actually want that?

You actually want that? I think there’s a way that we handle it. I’m not exactly certain, but I feel like in certain models, when they output something—you can imagine a table, for instance, and they want to draw 12 dashes and 12 dashes—I think there’s a way for that to happen.

I think we only do it on certain tokens. We exclude certain special characters, so we only do it on certain ones. S is the most common, almost. GLM-5.2—

Oh.

The Inference Engineering Masterclass

—and I think it was DSV-4 as well. You’d just have looping issues where it would literally—

It was—

—just have SSSSS.

Yeah. Is there something special about S? No, it’s just random?

It just seems to be the one token that folds.

Yeah. And it’s—

Alessio Fanelli

Is there—

—and it’s only temperature 0, or—

The Inference Engineering Masterclass

No.

—even at other temperatures?

The Inference Engineering Masterclass

Even at 0.9 or whatever, it will still collapse.

That’s weird, right?

It’s an inference problem, to be honest—a software problem. Oftentimes, NVIDIA will release an image, for instance, and if we upstream the changes from their latest TensorRT-LLM image into our stack, we’ll find that it fixes it.

Oftentimes, this will only happen in an inference engine that you’re using, like SGLang. But if you switch to vLLM, that isn’t the case. So it seems to be an extremely nondeterministic software issue and not really a model issue. It’s not a weights problem.

We’ll say, “Oh, it’s a problem with the quant. We did PTQ wrong,” right? But that doesn’t make sense, because the same exact weights used with a different inference engine don’t repeat the problem. Sometimes the kernels being used in the backend have very subtle race conditions. If you host this model on one cluster, you will never get this problem—

Alessio Fanelli

Oh my God.

The Inference Engineering Masterclass

—but if you host it on a different cluster, you will. The reason is that the node-to-node KV-cache transfer in one cluster is using a slower interconnect than the node-to-node transfer in another cluster.

That exposes the race, whereas in another cluster it doesn't. So then you end up saying, “Okay, this model is not going to be hosted on this cluster. We're going to host it on another cluster because that cluster exposed that problem.” But then it ends up being: Is it the software? Is it the model weights, or is it the hardware?

There is a thing about this: temperature zero still isn't deterministic, right? Mostly because of hardware. Even at temperature zero—

Right.

The Inference Engineering Masterclass

With the same model, you won't always get the same output.

Mm-hmm.

But I'm surprised by the race-condition one because I thought PyTorch was a graph that guarantees that you at least execute things in the right order.

The Inference Engineering Masterclass

Well, yeah, true. I guess I'm not saying that this is a risk. You have things like pipeline optimizations where you can start a kernel before the end of the previous kernel, and that's because you want to do that because—

It's like pipelining.

There's no expense. Exactly.

Yeah.

The Inference Engineering Masterclass

Exactly. But you don't do it cleanly. You overlap a little bit of the execution. No, I guess it is very possible that the kernel itself—that one block that is supposed to be running in this instance of time—that kernel itself has a race condition. For instance, a missing barrier.

Often, if you're designing a kernel and you want it to be very fast, if you don't test it extensively, you'll have certain threads access data points from registers before they've been written to by other threads, for example, because your barrier is wrong or your synchronization was wrong. But, yeah, the testing itself is very, very difficult in those cases. And the—

Yeah.

The Inference Engineering Masterclass

What does that mean?

Rust. If you're trying to have memory safety, it—

Mm.

It sounds like a comparable problem.

The Inference Engineering Masterclass

Well, yes. But you're working in CUDA, right? An NVIDIA GPU. It's like—

You just need a higher-level language like Mojo. Maybe that's what Modular is supposed to do. I don't know.

Alessio Fanelli

How do you see keeping the quality of the model? You talked about all these steps: You have to do quantization, train your own speculative decoder, and run on different hardware. Looking at other model providers, you kicked off an inference-speed race on the consumer end. What goes into keeping the quality the same across them? Sure, you can run benchmarks, but how do you determine how much quantization to use? Are there standards? What actually goes into—

There are a few things on quality. Most inference optimizations are lossless. KV caching, for example: you are just recomputing or preventing the recomputation of the same values. Speculation, of course, means that if a draft token is wrong, it gets rejected.

The main lossy optimization is quantization. That really comes down to, number 1, data format; number 2, which parts of the model you choose to quantize, which layers; and number 3, doing a lot of calibration on the quantized weights to ensure that you're preserving all the outliers.

There are other tricks that you can do, though. A big one is long context, because one thing you asked right at the beginning is, “Oh, what's going to happen if I send a 200,000-token request in?” Obviously, with a long input sequence, you need to store a lot more information and process a lot more tokens. So even if a model has a context of a certain length, you might, as an inference provider, choose to build an API with a shorter context length and, of course, a full-length one as well.

Because if someone doesn't need the full million-token context, for example, you can get them better performance. I don't know if that's exactly the quality of the model. The way that I think about quality is: To what degree are we faithfully serving the original model? If you think of a sort of golden implementation of a model that performs exactly the way the model is designed to perform, I think of quality as how close we're getting to that 100% fidelity of the model.

Alessio Fanelli

Mm.

The Inference Engineering Masterclass

You can also, of course, think about quality from the training side and how you push yourself past 100%. But when I think about purely inference optimizations, it's getting faster while staying as close to that 100% fidelity mark as possible.

Certainly, our standard internally is that you should not be able to tell the difference between our API and a sort of official API. I think Kimi in particular does a good job of vendor benchmarking here—

Yes.

The Inference Engineering Masterclass

Where they have—

They released an actual vendor benchmark.

Exactly. Yeah.

The Inference Engineering Masterclass

Yeah. Well, because you'd be pissed, right? If I'm a consumer and I'm using Amazon's endpoint, for instance, and I've used Kimi, and I'm like, “Oh my God, this is bad,” I'm not going to say, “Oh, Amazon quantized the model in a bad way.” I'm going to say, “Oh, Kimi sucks.” Right? So it seems like that makes sense.

Yeah, they care. They care.

Alessio Fanelli

Justifiably.

The Inference Engineering Masterclass

Yeah, justifiably.

Alessio Fanelli

This is probably a stupid question, but just checking: Has anything improved from mixed quantization? Is quantization always strictly worse?

The Inference Engineering Masterclass

No—well, technically, no. Quantization is lossy.

Yeah.

The Inference Engineering Masterclass

It improves speed. Obviously, it reduces the number of bits, like—

Alessio Fanelli

No, I'm always looking for inverse scaling laws. This is something I learned from Noam Brown: Things that normally act in one direction sometimes do the opposite.

The Inference Engineering Masterclass

Well, technically, when you run a benchmark, because these models are nondeterministic, sometimes your MXFP4 quant is 2 basis points higher than your—

Alessio Fanelli

Yeah, yeah.

The Inference Engineering Masterclass

That's noise. That's noise.

Yeah, exactly. I'm like, yeah, it's within the margin of error. That's why I always say “within the margin of error.” I actually stopped saying that because everyone assumes that what I mean is, well, within some margin of error, we're barely inside of that, as opposed to the worst. But, yeah, sometimes it just gives you a higher output score. But, like Ali said, that's noise.

To my knowledge, you're not necessarily making the results better. You're just trying to keep your fidelity as close to 100% to the original model.

Alessio Fanelli

Mm-hmm.

Ali Raza

There is, to your point, research that we did on MXFP4. I don't know if you're able to pull up a tweet we did. One of our research interns, Joshua, had a tweet on how we have a 20% better quantized JLN52 than NVIDIA.

Essentially, what we found throughout this 2-month research is that quantization is lossy. You're compressing the data from occupying 16 bits to occupying 4 bits, for instance. So you're obviously losing some information, and you're trying to minimize that.

When I say that I'm going to quantize the model, my job becomes: How do I find the layers that I can quantize and the layers not to quantize? For instance, with image models, I don't quantize modulation layers, and I don't quantize output projections, because those two are—

The output projection is what you see as the user; modulation is what the model sees or understands.

Right, exactly. So, I guess, to his paper, do you have the—

Yeah.

The Inference Engineering Masterclass

I guess it doesn't have the—

Alessio Fanelli

If there's a part to search, it's probably in the thread.

The Inference Engineering Masterclass

It's probably in the thread.

Alessio Fanelli

Yeah.

The Inference Engineering Masterclass

But basically, the long and the short of it is that it is very possible that quantizing more of the model makes the results better. If I have a model where I quantize layers 1, 5, and 10, and another model where I only quantize layers 1 and 2, it is possible that the model in which I quantized more information is going to perform better because the quantization errors have canceled out.

What Joshua showed in his mathematical proof, where he had a verifier in it, is that you can predict which layers are going to have quantization errors that will cancel out with each other, and you choose to quantize those layers. And so the result of doing this mathematical quantization is you end up with a model that's 20% more quantized than another provider, so you get 20% more throughput because more layers are running in MXFP4, and your quality is better than that other quantized model because the layers that you chose to quantize have their errors cancel out—like one layer is skewed to the right—

One layer skewed to the left, one layer skewed to the right. Your final logit distribution is more similar to the original distribution of the model, so you have better fidelity. The way we proved this was with KL divergence. Instead of just scoring on the benchmarks, we scored the KL divergence between the logit distribution of the quantized model and the logit distribution of the original full-precision model, and showed that with this technique, if your probability distribution on the logit of the token it wants to select is more similar to the original model, you're probably going to end up staying true to the original model.

Previously, it seemed like the industry thought, “The more you quantize, the worse it's going to be, because the more loss you introduce.” That's not necessarily true. It doesn't improve it, but it can cancel out.

I think it might be this, but it reminds me a good bit of pruning, actually, where you can prune off certain layers.

Mm-hmm.

Ali Raza

It's a fun fact: This paper was originally 72 pages, and then we decided we couldn't release it.

Wow.

The Inference Engineering Masterclass

So it's now 45 pages.

Still 39 pages, so very, very substantive. We talked about evals and all these things, and what's possible in terms of speedup? I guess that's probably the number-one thing that people care about, and it's something that you wrote about in your post. The official API is 70 tokens per second, and you push it up to 90. Is that a normal thing?

Mm-hmm.

Which, by the way, because I am from a finance background, in the '70s, that was the margin at the time. When you did quantitative finance research, you would find—

The Inference Engineering Masterclass

And, like, with 20%—tens of percent.

Yes. Yes.

The Inference Engineering Masterclass

Yeah.

Yes.

The Inference Engineering Masterclass

And now it's tiny fractions.

For those people interested, look up Andrew Lo's paper. He had a really interesting illustration of quantitative statistical arbitrage distributions narrowing down from those kinds of 20% differences in the '70s down to nothing today, which is very, very cool.

Exactly. And we're at the beginning of the same type of thing. Benchmarking is hard. I think anyone will tell you that. Benchmarking provider speeds is hard because there are so many variables that go into it: What hardware are you using? How much load do you have on the system? What's the exact nature of the prompts and input and output sequence lengths? All that kind of stuff.

The most common form, of course, is TPS—tokens per second—which is bad naming by us in the industry because there are actually 2 tokens-per-second metrics: the throughput number and the latency number.

TTFT, yeah.

Total tokens per second out of the GPU is the throughput number. Most people only care about tokens per second as the latency number, which we should call ITL, intertoken latency, but we don't. Anyway, you can imagine a sort of standard API without many optimizations for a 1-trillion-parameter model operating somewhere in the 30-to-50-tokens-per-second range for a reasonable traffic profile.

We generally see the goal as pushing to 10× that. If you have, say, 3 optimizations, each of which doubles performance, then you stack that up—that's an 8× gain. That's kind of the order of magnitude we're working with in this space. We're trying to make things substantially faster, not just go from 70 to 90.

Are you saying you have done that?

Let's say you have, as a reasonable baseline, 30 or 40 tokens per second. You can achieve 10× that. On GLM 5.2, if you run it unquantized, perhaps on Halpos even, and you're just using an off-the-shelf inference engine with no particular optimizations—no speculator, nothing extra around KV routing, no disaggregation—you're probably looking at 30 to 40. You think that's a reasonable baseline?

Right. Right.

To get to something like 10×, there are a lot of trade-offs that you're making. If we're running in more like the 300-to-400-tokens-per-second range, obviously you're using the best hardware possible. You have an optimized speculator. You have done all of your quantization work. You're seeing a pretty high cache hit rate. You're running with a reasonably small batch size and a parallelism configuration that's tuned for latency versus throughput, but it is possible.

The spreads that you see if you go on Artificial Analysis or OpenRouter and look at the worst provider versus the best provider can often hit that kind of range. 10× is, of course, very aggressive. It's oftentimes more of a 4× to 6× improvement. But that's the kind of performance that makes us really excited: when we can get these huge gains, not just go from 70 to 90 tokens.

Ali Raza

It's also hardware-dependent. If you're serving it on just a node of H100s and then you shard the model across 4 nodes of B200s, you can definitely increase the speed by throwing more hardware at it. If you're normalizing for the exact same hardware and the same number of GPUs—

Then you're looking at a 2× to 4× improvement, depending on the inference optimizations. Some of it is what the call is, and some of it is who's the driver.

Alessio Fanelli

If you break down the 2× to 4×—say the example is running GLM 5.2 on B200s, single node—what's the cost trade-off for the effort to get the last bit of juice out versus what should people just think of?

The Inference Engineering Masterclass

Yeah.

Alessio Fanelli

On B200s—

The Inference Engineering Masterclass

Yeah.

Alessio Fanelli

Single node, right? What's the cost trade-off for the effort to get the last bit of juice out versus what should people just think of?

The Inference Engineering Masterclass

Spectre quantization. Yeah.

Alessio Fanelli

Spectre quantization.

That's, like, 95%.

And how far does that get you? How easy is that for the average person to do? Say right now I want to throw the weights of GLM-5.2 on a node of B200s. How easy is it to find a speculative decoder model or an already quantized model? How much work goes into it?

Guest

Mm-hmm.

Guest 2

If you're doing it up front, it's quite a lot of work. If you're doing it today, there are going to be people who have published things that you can just grab. You can grab some NVFP4 weights. You can grab a speculator.

If we're thinking about the 2× gains we're stacking, going from BF16 to NVFP4 isn't quite 2×. I think it's about 30% to 40% from BF16 to FP8, and then another 30% to 40% multiplicatively from FP8 to FP4. So that doesn't quite get you 2×, but roughly 2×. Speculator is roughly 2×. Disaggregation on top of that, if you're able to get enough hardware and put enough traffic through it, is another roughly 2×.

Then you add in some double-digit percentage increase from having a better runtime, with the latest kernels and so on behind it. That's kind of how it stacks up.

Guest

Yeah.

Guest 2

Building each of those—the quantized weights, for someone who really knows what they're doing, is hours to days of work. Building the speculator is, again, hours to days of work. And the disaggregation setup is hours to days.

Guest

Yeah.

Guest 2

Okay, but once you have it—

Guest

Once it's set up.

Guest 2

Yeah, getting disaggregation working for the first time, of course, is very difficult. But the marginal implementation is—

If you're just grabbing it—if you're a normal consumer who has access to a node of B200s and you're wondering, “How can I just host it myself?”—you don't need to quantize the model yourself. There's always going to be an open-source quantized checkpoint. NVIDIA's going to push one out if no one else does.

Usually, providers will have their own speculator that they've trained as well. You don't need to train your own speculator. You can just use that as well.

Guest 2

Yeah. Like Chemi, uh, GLM-52 has its own MTP.

Alessio Fanelli

Right.

What’s multi-token prediction?

Guest

Yes.

Alessio Fanelli

I’m just an expert. I can do it for you in case I get it wrong.

No. Yeah, actually, you should correct us if we’re wrong, but their multi-token prediction can be used for self-speculative decoding.

I’m actually not sure. I’m not going to correct you.

Okay. I’m semi-confident in that, but someone can check. But it’s useful to paint the story of, okay, not just the average person, but say a company wants to switch from serverless inference to—

Mm-hmm.

—I want to throw this up, you know. I want to rent some GPUs and throw it up. These are the steps you take to make it significantly faster than just putting it behind vLLM.

Alessio Fanelli

Right.

I was waiting for a mention of Dynamo. I feel like that’s supposed to be the baseline that you measure against.

I would think of Dynamo as less of an out-of-the-box system and more of a toolkit for building with. So when we talk about doing KV-aware routing, when we talk about doing KV-cache offloading, when we talk about doing PD disaggregation, Dynamo fundamentally is—

By the way, Dynamo is an open-source library from NVIDIA.

Alessio Fanelli

We’ve done a podcast with Kyle Kranen.

Guest

Okay.

Alessio Fanelli

Kyle Kranen.

Guest

Cool. So your listeners know that it supports all the different inference frameworks. It’s actually kind of multihardware, which is interesting.

Alessio Fanelli

But it’s just a router. It’s not an optimizer layer.

Guest

Yeah. What Dynamo is good at is being a library for moving information around your cluster and around your hardware. So if you have KV cache in one place and you need it to be somewhere else, Dynamo coordinates Nixal for you to move that around. That doesn’t mean that, out of the box, you just say, “pip install Dynamo,” and then you get a massive performance speedup. It’s more of a developer toolkit.

Alessio Fanelli

Yeah. I would have said it comes with a set of defaults that you can then swap out.

Guest

It does. If the industry at large were rolling out all of these deployments as the standard, then I think it would be a credible baseline. But we’ve got to benchmark against what we’re seeing in the wild.

I did want to talk a little bit more about PD disaggregation, because that is probably number 3 after quantization and speculative decoding. In your book, though—I was just going to pull out the book—section 5.2.2 is on Medusa, 5.2.3 is on EAGLE, and 5.2.4 is on n-gram.

It’s 5.5 that would be disaggregation.

Yeah. Well, no, I just wanted to dwell a little bit on the other techniques. What do you choose to include? What do you choose not to include? Because there were all these other—

Techniques, I guess.

Alessio Fanelli

Yeah.

Guest

Are these still relevant? Because I think they came out a year and a half ago, maybe.

Alessio Fanelli

Medusa is quite old.

Guest

Yeah, Medusa’s old.

Alessio Fanelli

But is it in the book as a good—

Guest

Baseline.

Alessio Fanelli

—but is it in the book as a good baseline, as in, “Here’s the baseline vanilla. Understand it”?

Guest

Like, you should know this.

Alessio Fanelli

I read the paper, and I’m like, “Ah, it makes so much sense.”

Guest

With the book, I had a couple of goals. One was to give people a working vocabulary for the space as a whole, and the other was to give them some intuition about how each of these techniques works. As I mentioned in my AI Engineer talk, which is kind of the first public addendum to this, the speculation space has moved much faster than everything else.

Even at the time that I wrote the book, I included Medusa very much as a way for people to understand how the space evolved, rather than as the most modern technique. Now, of course, there’s DFlash and DeSpark. There are newer techniques even than EAGLE, although EAGLE is still very commonly used.

Alessio Fanelli

Speculative speculative decoding.

Guest

Yes, speculative speculative decoding.

Alessio Fanelli

What?

Guest

Oh.

Alessio Fanelli

It’s a paper by Trudeau, and it’s basically doing speculative decoding—

Guest

Uh-huh.

Alessio Fanelli

—for the speculative decoder.

Guest

Oh, in speculative decoding? Oh my God.

Alessio Fanelli

It’s literally just another one. It’s the simplest way to explain it. It seems like he got nontrivial speedups there, but the complexity with training is almost as complex as training GANs, at least in our minds. It’s a delicate balance, and oftentimes it’s just additional. But yeah, it’s literally speculative decoding on speculative—

Guest

Speculative, speculative.

Alessio Fanelli

Yeah. We saw this paper.

Guest

It’s interesting, right?

Alessio Fanelli

Yeah.

Guest

I wouldn’t even expect it to be particularly hard to train. I don’t—

Alessio Fanelli

Right, right, right.

Guest

The naive part of me is like, “Okay, train a speculative decoder.”

Alessio Fanelli

But it makes sense. The whole idea of speculative decoding is almost like the iPhone auto-predict version, but for a normal model. You’re generating 3 tokens and saying, “Okay, I’ll do prefill on them.” So you save those 3 turns for your original model. Now your speculative decoder is doing 3 turns of autoregression.

Guest

I guess the other question there is: What are the sizes of speculators? Say, for GLM—

Alessio Fanelli

Right. It’s like 1 billion parameters.

Guest

For MiniMax, it’s like 1 layer. It’s like 1/60th of the original model, usually.

Alessio Fanelli

Yeah.

Guest

Actually, I think we should do a paper when we get back to the office.

Speculative speculative speculative decoding.

Alessio Fanelli

No, it does seem like—how do you know when to stop? But then it also seems kind of like, if you’re able to train speculative speculative decoding, for instance, if you’re able to have a small model that accurately predicts what the intermediate speculator is going to predict, and that is able to predict what the original target model is going to predict, then why not just use that smallest model directly?

Guest

Yeah. This is adjacent to the routing problem.

Alessio Fanelli

Right.

Guest

Yeah.

Alessio Fanelli

Right.

Guest

One of the practical constraints on using speculators is that you do have to run a small model on the same hardware that you’re running the big model on. There’s an orchestration and resource-competition problem inherent in that. That is one of the constraints on speculation in general: draft tokens cost resources to create and software complexity to manage. If you have infinitely recursive speculators, you add quite a bit of that complexity to the actual implementation within the inference engine, not just to the training process.

Alessio Fanelli

I was going to say, I would wonder if you could do similar distillation and pruning. It’s the same thing; it’s just a model. Can we not just distill a lot of the weights and quantize the speculator, but—

Guest

Mm.

Alessio Fanelli

—but that’s out of my domain. I guess the question that also comes up is: This is all for big server workloads, right? How much of this applies to, say, I have this MacBook and I want to run Gemma really efficiently? Similar problems, not the same?

Guest

Pretty different. I talked to Salo about this on his podcast a couple of weeks ago. The difference between inference engineering for the data center and production workloads versus inference engineering for local AI is that we start with fundamentally different constraints and different goals.

With local AI, it’s, “How do I fit this model onto my hardware and then make it less dumb?” With data center inference, it’s, “How do I load this model and then make it less slow?” Obviously, we care about less dumb, and they care about less slow.

But the local AI inference engineering ecosystem, I think, actually has a lot for us to learn from in the data center space. They are experts in various forms of quantization, including dynamic quantization, which we just don’t really touch, as well as pruning, distillation, and layer removal.

Alessio Fanelli

Layer removal matters less.

Guest

Yeah. There’s—

Alessio Fanelli

No one does pruning, really.

Guest

Yeah. Well, but the—

Alessio Fanelli

Which is surprising, right? But that’s a whole different thing.

Guest

Just to fit something on the laptop.

Alessio Fanelli

Right, right, right.

Guest

So yeah, it’s an interesting space. Not necessarily that their techniques make sense for us to use in the data center, because obviously we have different resources and different goals, but more that the process, as well as the openness of that field, is something to admire.

Alessio Fanelli

Yeah, to your point, certain optimizations—like, for instance, TurboQuant. I’m sure you’ve heard of it. It made such huge hype, and we did a whole deep dive on Twitter analysis: What is it? How does it work? Why is it good or not?

It took off, and it was implemented on local devices because your memory bandwidth is so slow on a MacBook, for instance.

But try putting the same thing on an NVIDIA GPU, on a B200. TurboQuant would not be used. NVIDIA made it clear that this is not a good optimization, and we’ve seen it firsthand: the overhead of doing dequantization and quantization in the kernel itself—the TurboQuant kernel execution—is actually much, much slower than the time that you save from the bandwidth. On the B200s, you have 3.5 TB/s.

You don’t need to decrease the storage that much. You don’t need to use an FP4 KV cache. You don’t need to use a record. There are better optimizations to be made. But on edge devices, it’s extremely important and extremely useful.

So it seems to be different optimizations there, but then they’re all uniquely combined with, like, oh, you want to quantize the model, you want to do speculative decoding, like certain common prefixes with both—

Guest

Principles.

Yeah, exactly.

Alessio Fanelli

Yeah.

Exactly.

Guest

They also do a lot of work on model parallelism, especially over heterogeneous topologies, where you have some Sparks wired together with Ethernet—DGX Sparks.

Yeah, this is the Exo Labs guys.

Yeah. You have a number of Mac Minis stacked up. The interconnect between machines is one thing that I think we both have to deal with, although they have to deal with it a lot more. This is why one thing that we do a lot is work with tensor parallelism.

Mm-hmm.

Guest

That’s where you are using all 8 GPUs and sharding the model across them. Tensor parallelism is not a good fit for local AI because it assumes a very high-bandwidth interconnect like NVLink. You might be forced to do something like pipeline parallelism, which we’re never going to do unless we’re doing some kind of—

Yeah, for inference—

—multi-node inference.

But since you mentioned it, I actually wasn’t sure if we were going to cover it, but let’s briefly explain tensor parallelism and expert parallelism, since you have very nice images.

You want to pull the book?

I just want to—

Yeah.

I just want to show off your images.

Guest

Yeah. Shout-out to Luke from Baseten’s design team for making these beautiful images.

One other difference is that we talk a lot about the active parameters of a mixture-of-experts model. For local inference folks, that matters a lot because if you have a batch size of 1, you’re only activating that many parameters. When we do—

Yes, I was going to bring that into the diffusion conversation.

Yeah, yeah. When we go through an MoE model and host it for an API, we assume that all parameters are going to be active because—

You’re batching.

—throughout your batch—

Yeah.

Guest

—you’re going to hit everything.

Cool. So broadly, tensor parallelism you can do with any model. Expert parallelism you can only do with MoE models. Effectively, all models today are MoE models—

Sort of—

—at least all models large enough that you would care to parallelize them across multiple GPUs. So that nuance is less important now.

With expert parallelism, the idea is you put the entire expert on a GPU. Generally, you have more experts than GPUs, so you might put, like, 8 experts per GPU or whatever. Then you replicate the router, which is very small, across each of the GPUs. By moving the generation from expert to expert, with each expert being inside a GPU, they’re not competing for resources. You massively increase the throughput that you’re capable of doing.

The GPU-to-GPU connection is not as important because there’s not as much communication. Tensor parallelism requires that you’re able to do this all-gather, all-reduce. You basically shard the model across the GPUs entirely, and then for each step, you’re combining the results of each of the GPUs, which is why the interconnect matters a lot.

Of course, this is a very high-level generalization. There are a lot of places where this is not correct. But generally, TP is helpful for latency, and in many cases, you will use some combination of these 2 parallelisms across the model rather than just picking one or the other. Do you want to add some extra color there?

Yeah. Usually, in a model, they’re not mutually exclusive. You do tensor parallelism and you’ll do expert parallelism. Pipeline parallelism, less so. It seems to me like we never use pipeline parallelism.

Yeah. The only reason you would have to do pipeline parallelism, which is where you separate different layers and put half the layers on one piece of hardware and half on another, is if you are forced to do multi-node inference because a model is bigger than what you have.

Let’s say you’re doing a deployment on H100s for whatever reason and you’re putting a trillion-parameter model on there. You have to use multiple H100 nodes. Because the interconnect is so slow between the nodes, the only viable way to parallelize there is pipeline parallelism, but then you would do expert and tensor parallelism within each node.

And the limiting factor for H100s is HBM?

Yeah. They just don’t have enough of it.

How much—what are the magic numbers that we need to—

Alessio Fanelli

One node is 8.

—like, on a B200, it’s 180 GB per GPU, and then a node of 8, you’re talking about 180 times 8. With FP4, each parameter takes half a byte, so that’s 800 GB. On an H100, it’s like 140?

Guest

It’s 80.

It’s 80?

Yeah.

Oof.

Guest

Yeah.

Alessio Fanelli

It’s easy.

I’m old. I’ve been doing this a long time. I actually remember H100 specs.

Guest

Yeah. You want to tell me about the T4s?

Alessio Fanelli

The T4s.

Guest

Let me tell you what it was like to run a model on a T4 back in the day.

One thing I was surprised to see was that more people didn’t do Jamba. I don’t know if you guys remember Jamba from AI21. They would specifically pick a piece of hardware and then design the architecture dimensions for the hardware, and then it would obviously saturate the hardware. It makes sense. Somehow, all these models don’t do that.

Alessio Fanelli

Don’t they do this for the training side, though?

I don’t know. Do you—

Guest

The what, sorry? The—

Alessio Fanelli

Training. For training the model.

Like deciding which GPU? Which GPU?

Yeah.

Guest

Yeah.

Alessio Fanelli

Yeah, yeah. They do.

And with training, it’s more of a math problem. You can run the math and see the FLOPs and maximize them. With inference, it’s more of an autotuning problem. I don’t know if you’re familiar with GPU-kernel autotuning, but basically, you define, “Oh, I have 2 GPUs. I can do TP1, TP2, EP1, EP2,” for instance, right?

That gives you a total of 2 squared combinations, and then you shadow the same traffic—real product traffic—and see which configuration gives you the best TPM and TPS, and then just use that. I don’t like the fact that you cannot reason about which one is going to give you the best performance, or that there isn’t 1 specific configuration that’s always best. But it seems like autotuning is just the way that you find the best one.

With kernels and GPU kernels, it’s much the same. After you design your kernel and design your configuration—how many threads do you launch? How much shared memory do you use?—you just autotune. You sweep the parameter space on the side, and this is the best one empirically.

But yeah, they are combined. They’re not just separate. There are a few bits of training that are kind of hardware-targeted. If you look at, for example, NVIDIA Nemotron models, they run very, very well on Blackwell. That’s unsurprising.

So there’s some degree of that, but I think that most open labs are trying to make models that can be run on as wide a range of hardware as possible rather than targeting just a single chip, for usefulness. I see. For usefulness. Yeah.

Okay, one more thing while this chart is still up. All-gather, all-reduce is expensive. One of the movements in Silicon Valley is mega-kernels: just keep fusing kernels. I don’t know. Is it that simple?

Well, I mean, a fused kernel can’t save you. Here, with tensor parallelism, half the matrix is on one GPU and the other half is on another. If I need the entire matrix in order to do a nonlinear operation in the next step—for instance, if I’m doing attention, I need the softmax, or I need to do exponentiation—I need to have the entire row.

Guest

So I need to know what the partial result was from GPU 2 and what the partial result was from GPU 1 in order to be able to do the softmax in the next stage. I have to make them communicate with each other, even if I had a fused kernel, because of the nonlinearities within each one.

Also, with mega-kernels, honestly, I'm very bearish on—

Ooh—

I'll be honest.

Please, please, please.

No, it's just like mega-kernels. It was a good research direction, and intuitively, theoretically, it seems nice. You have a lot of launch overhead from launching one kernel, so just keep fusing it, moving the data—just fuse everything together.

But the kernel complexity itself is very difficult. It's very, very difficult to write a very optimized mega-kernel. And even the companies—or, not to name any companies, the people I've spoken to who work at companies that do fused mega-kernels—very often don't end up running those in production, because the CUDA and modular kernels that we launch are faster. You can optimize each individual component, and you can just have them parallelize with each other.

With Rubin—I don't know if you guys saw Rubin's Twitter post yesterday, but—

Rubin?

Like—

No, no, like Rubin, like the GPU.

Yeah, the GPU.

They have a Twitter account for Rubin only?

No, no, no, no.

Okay. I was like, "What are you talking about?"

Yeah, sorry. One of the tech leads at NVIDIA launched a Twitter post that said, "We're pulling the curtain on Rubin, and here are the specs." The third tweet showed—not to get too technical into it, and I need to read it much more—that the GPU is designed in such a way that it basically kills mega-kernels.

You don't need to use mega-kernels that much anymore. So it seems like that entire research field isn't going to be continued, but yeah.

Can I speculate about Rubin for a minute, please?

Go.

You know, I've now been through—

And by the way, they're covered in the book.

Yeah, but—

Well, I mean, they're covered in the book in the sense that I'm aware, from the Wikipedia entry and the blog post, that Rubin is going to happen in the future. And you even had the name of the one, the Feynman. It's like, "Hey, this is gonna be—" I was like, "This is very up to date. I'm trying to future-proof this thing, okay?" I don't want to publish a new one until next year or something.

Anyway, so we were discussing the degree to which I am old.

Mm-hmm. You know, I've now been through 3 hardware launch cycles. I've been through the Ampere launch cycle, the Hopper launch cycle, and the Blackwell launch cycle.

Now, when I say launch cycle, I don't necessarily mean the actual shipping of the hardware. Ampere GPUs were racked up well before I got in this industry. But there is a lot of time between hardware being racked up and hardware being feasible for inference.

If you look at the original vLLM and SGLang—or vLLM especially—that was written targeting Ampere and then had to be updated for Hopper, updated for Blackwell. With each of these cycles, it becomes faster and more urgent, but also substantially more complicated.

When I look ahead to what's going to be new with Rubin, I think Dynamo gives me a lot of technical hints around what kinds of work are going to be very valuable. Obviously, we're continuing some trends from Blackwell, right? NVFP4 is big. The amount of compute that they have behind NVFP4 Tensor Cores is massive.

We'll talk about video, I think, at some point, and that's the big barrier there. You've got much, much faster memory bandwidth, which was the same thing that made Blackwell so good. But the big thing is more systems thinking.

You have more emphasis on the CPU-to-GPU interconnect, more emphasis on the interconnect between GPUs. And when you look at Dynamo, it's a system entirely designed around, "How do I move the KV cache to where it needs to be when it needs to get there?"

So I think themes around KV-cache offloading, KV-aware routing, and disaggregation are going to be substantially more important in the Rubin era. That means inference engineering becomes not just a CUDA kernel problem, but also a very traditional hardware infrastructure problem, which is something we've been building toward for a long time, and something that's very exciting to me because we're going to see multiple domains colliding.

The ability to reason from the kernel level up to the hardware level, and back down, is going to be very valuable.

Alessio Fanelli

I will take what Philip said one step further, actually. I think it's trending toward becoming exclusively an infrastructure problem, where problems of P/D disaggregation, training, and SpecDec—but Triton kernels are not going to be much of a problem because the GPU is moving more toward being an ASIC. You're just trying to orchestrate what happens on the GPU, but you're not actually controlling it at the thread level.

You see this with CUTLASS and CuTe DSL. You're just working at the level of tiles of data, but you're no longer controlling what each thread does on the GPU. That's being taken care of for you.

Do you agree that GPUs and future GPUs are trending more and more toward becoming ASICs that just need to be launched, and then they do the data operation based on your conversations with other people?

Oh, I mean, yeah, no. That is a section of the market.

Alessio Fanelli

Right.

ASICs can deliver a lot more performance, but only for their workload. And the G in GPU makes them continue to be very general.

Actually, it's graphics, but—

Yeah.

I keep saying this. I have to correct myself in case people come at me for getting the G wrong.

Yeah, it's like a spectrum, right? From very, very general-purpose compute to something like a Taalas, where you've got the hardware built for a specific set of model weights.

The weights burned—

The weights—

—into the chip.

Yeah.

No loading.

I wouldn't say that we're going all the way there. It's more along the spectrum: it's a step in the direction of more specialization within the hardware.

I'm curious. I feel like he was driving toward something.

Alessio Fanelli

I guess my point is being bearish on everything else apart from burning the weights into the chip. Burning weights into the chip is impractical because you want to fine-tune, optimize, quantize, and release new checkpoints of the model. If it's burned into the chip, the chip's useless in a month or 2, right?

My point is, how can you not—seeing NVIDIA become more and more specialized—take its GPUs from a general programming paradigm, where it's a general computer that you can use to program threads, and with every new generation put in more and more specialized instructions, specialized Tensor Cores, specialized MMA instructions, things that allow you to control it almost as an ASIC, almost as a collection of ASICs?

How can you look at this trend and still be bullish on companies that are coming up with ASICs for AI? In the sense that ASICs—

Yeah, because they're sort of re—

Alessio Fanelli

What?

They're evolving toward that direction.

Alessio Fanelli

They're almost evolving toward an—like, in Rubin, I guess, compared to Ampere or T4, Rubin is basically an ASIC. It is basically just a thing that is used—

Programmable ASIC?

Alessio Fanelli

Yeah. It's like, yeah, you can obviously program it. I guess it's very controversial to call it an ASIC. It is a GPU. It is general. It does have threads. I can write CUDA to control it and change its operations.

But it has systolic arrays, Tensor Cores, TMAs, and tensor memory. It has these things that are almost exclusively useful for loading model weights. It has Tensor Core instructions that are almost exclusively shaped around the head dimensions of models that exist in the market today.

To say that you're going to come up with an ASIC is to say you're going to etch something into it. Well, with the next architecture, it's basically going to be useless.

Yeah, I don't know. I don't know.

I think the thing to remember is just how long these hardware cycles are. If a chip is coming out today, that means the design process for it was kicked off years ago. And NVIDIA has done a very good job of predicting where the market is going to go, and—

I mean, they have the most information, for sure.

Of course. But if you look at there being public, open-source model architectures that look more or less like early versions of the ones we have today, Rubin is honestly the first chip that was fully built in that world.

Alessio Fanelli

And so you can see a lot of the understanding of the shape of the workload that this chip is going to be asked to do in the way it's designed.

Yeah. Okay. So, I'm not going to be the best person to directly answer those questions. I think these are very fair questions. Honestly, it's the first question based on Rubin that I've heard articulated so well. I do think that I will make a case for vertically integrated model-lab ASICs.

Like the OpenAI-Broadcom, whatever Jalapeño chip, which totally makes sense. We first had this on the pod with Martin Casado, where he was like, “Look, if you have a $1 trillion or $500 billion training run—

Mm-hmm.

—then take $50 billion of that and make it ASIC. It's fine. You won't get more than 10% efficiency from the ASIC.” That makes sense.

Alessio Fanelli

Right.

Right? So, a model-specific chip, yes. But ASIC companies—the interesting thing is, I feel like you're hyper-focusing on, like you say, the tile stuff.

Alessio Fanelli

Right.

They are doing a lot more surface-area engineering, or the actual allocation of memory and hardware, and the communication between chips, that probably still won't be touched by Rubin, but I don't know the details.

They often talk about things that I would expect to have bigger orders of magnitude than would be programmably accomplished by whatever Rubin does. But who knows?

Alessio Fanelli

I see. Yeah, it seems—

Yeah. Think about what the real blockers to 10× to 1,000× faster inference are. It's not the stuff that can be rearranged just within the existing GPU design.

Intercommunication.

Yeah.

Okay.

These guys are aiming for 300,000 tokens per second. They're not fucking around.

Might have to program some ASICs.

Alessio Fanelli

Maybe. I think it's interesting to me that you're so bearish on so much of this kernel-engineering work, given how much of it you've been doing recently.

Right. Right. But the more I do it, the more it just seems to me that—

It's not mega—

Alessio Fanelli

I would also add—

Alessio Fanelli

There are generations of models being released, right? I think on your guys' end, you see a lot of, okay, one day it's GLM, Kimi, DeepSeek, MiniMax, and the others. Some are doing completely different stuff, right? Gemma has no encoder. The latest Thinking Machines is all from scratch. But when you look at the other side, how long have we been on the GPT-5 generation, right?

Alessio Fanelli

Right.

Mark Saroufim

They've been serving that thing for quite a while. Sure, there's maybe more pretraining. There are different checkpoints, but you actually can squeeze quite a bit out. You do a multibillion-dollar training run; if you can make it X% more efficient, they serve it for a while. Same with, say, the Claude 5 family, right?

Alessio Fanelli

Like if they release a new model—if they release GPT-6 now or whatever—

Mark Saroufim

Yeah, yeah.

Easy. So I think I would slightly disagree. Again, it's all secondhand, on the longevity of a model.

Alessio Fanelli

Yeah, yeah.

There's still people out there using GPT-4.0.

Mark Saroufim

Yeah.

Yeah, Llama—not Llama 2, but Llama 3. I still see Llama 3 workloads.

Mark Saroufim

Yeah. Because if it's done, if it's trusted, don't change it.

If it works. But—

Alessio Fanelli

Which is one of the promises of open source, right? The whole Save GPT-4o movement. You don't gotta have a Save Llama 3 movement. You just gotta have an A100 somewhere.

Mark Saroufim

I think at some point, there's also the question of whether, if a model can do enough and use enough tool calls and be agentic enough, can it just web-search, tool-search, write code? Do you really need to keep squeezing more? We will, because you guys will make it cheap and fast and smaller, and I can swap it in. But at some level, give me 5.2 today, or say whatever 120B model, and I can run with it for quite a while, right?

Alessio Fanelli

This is assuming you don't need intelligence.

Mark Saroufim

I think there's a lot of intelligence—

You need reliability and predictability. I'm in enterprise—

Mm-hmm.

This is tried and tested. It is signed off by my 5,000 stakeholders. I'm not touching it.

Alessio Fanelli

It runs a batch job every day, and I like the results.

Yeah.

Alessio Fanelli

The results are predictable. Yeah.

Mark Saroufim

Yeah, yeah. It doesn't make sense to keep using them. Stuff gets sparser, cheaper, better.

Right.

Mark Saroufim

But that doesn't mean that the old model, GLM-5.0, isn't usable, right?

Mm.

Mark Saroufim

If we hit a stall, for whatever reason, there's still a lot that can be squeezed out.

We're going to run out of time. I did want to also make sure—actually, yes, we happen to have this diagram. Pull up—compare this versus any Cerebras diagram, right? I don't think Etched or d-Matrix have put out public charts yet, but the compute real estate is very different. The size is very different, right? This is not wafer-scale. There are probably, I don't know, a few hundred of these on a wafer. I don't know how big—

Alessio Fanelli

Right.

—the comparison is. But the real-estate allocation is very different.

Mark Saroufim

Yeah.

A few dozen. Yeah.

Alessio Fanelli

Before we move from hardware, I have 2 quick questions. 1: The latest Kimi, which is really big, 3 trillion—

Yeah.

Mark Saroufim

—doesn't fit on most hardware on a single node.

Yes.

Alessio Fanelli

You need GB300—

Mark Saroufim

You need GB300—

Alessio Fanelli

—to fit it on a single node.

Mark Saroufim

Or AMD.

Alessio Fanelli

It's simple math. NVFP4, 2.8 trillion parameters, 1.4 terabytes. The GB300s have 288 gigabytes each. Across 8 of those, you have enough room for the model, and honestly—

Mark Saroufim

With the Rubins, you now have what? An NVL72 rack of—

Alessio Fanelli

What?

Mark Saroufim

—20 terabytes of your—

Alessio Fanelli

Yeah. No, now you still have NVL72 on Blackwell as well, but you can't necessarily assume you're going to do inference on that.

Mark Saroufim

Mm-hmm.

Alessio Fanelli

There's a whole lot more 8X Max in the world than there are NVL72s.

Mark Saroufim

Yeah. I guess my last quick question on hardware was, do you notice anything with hardware generations for new pretrained base models? One of the things you said for efficiency is you can swap hardware. That's one of the 2× gains. When we see new stuff coming out training-wise on Rubins, any changes in LLMs? Does this affect what type of models we will be seeing when these are more available? And can you—

Alessio Fanelli

They get bigger. People understand the ceiling that you have in terms of how many parameters of a model you can run, given the latest inference hardware, and that kind of forms a ceiling. For example, when DeepSeek R1 came out, it was 671 billion parameters, which at the time was really huge and I think did a lot to push us to really quickly adopt Blackwell and get good at serving on Blackwell.

So, in my mind, it's mostly about model size and then about matching the architecture and the native quantization to the target hardware, like we talked about with all Nemotron models in NVFP4, for example.

Mark Saroufim

So we talked a lot about LLMs.

Alessio Fanelli

Mm-hmm.

Mark Saroufim

You have a lot more in the book. What about audio and video? What's the other side of inference engineering? Ali, you're pretty big in video diffusion.

Alessio Fanelli

Video diffusion, I think, is just shaped differently. A lot of the stuff that you can think about and reason about with LLMs being autoregressive, with video diffusion, it's not the case. For instance, you don't—

You don't do batching. Every request just comes in on 1 GPU, and it serves 1 GPU. You don't have to shard. The models are a lot smaller. Wan 2.2, for instance, is a 20-billion-parameter model.

It's just orders of magnitude smaller than the best LLMs. And it's one of those spaces where, with LLMs, we see Gemma 3 is almost comparable to, you know, Mistral or, like, GPT-5 from OpenAI. The difference between the best open-source LLM and the best closed-source LLM is very small. It used to be 6 months. I don't think it's 6 months anymore. I think it's basically almost on par today.

Video models are definitely not. There's a huge gap. If you look at the best video that you can generate today with an open-source model like Wan 2.2 versus something like Kling or Veo, the difference is night and day. So it creates this disparity where media companies will choose to go, most of the time, to closed-source models.

If I were to tell you, “Hey, I can generate an entire 3-hour movie for you with this model, and I'll optimize it so that you only have to pay me $10,” but if they were to do it on a closed-source model, they'd have to pay $1,000, which is 100x. I'm 100x cheaper, but it's still $1,000. They're still going to choose to do all of their cuts with Veo and Kling.

It's like a chicken-and-egg cycle where less demand causes less innovation in the field, which causes fewer open-source checkpoints to be released. And some of the labs that were releasing open-source models like Wan have closed-sourced their latest models. Wan 2.7 is not open source. We're still on Wan 2.2.

The challenge with video models, especially, is the number of tokens. With video models, you want to generate a high-quality video. Let's say you're doing 16 frames per second; that's the absolute minimum you'll do, and let's say you'll do 480p video. You can think about your dimensions, and I think I have a good diagram that shows the sheer number of tokens.

Let's say we're looking at just one video of Sparta 300 or whatever. Let's say we're looking at 4 frames. Those 4 frames of that video, if you're doing full attention, you're looking at 480 by 720 by 81 frames in just 5 seconds, because 16 FPS by 5. Then you compress it down to latent space, but you're still doing 30 by 50 by 21 tokens.

Yeah.

Which means that for attention, for just 5 seconds, you're running attention on 35,000 tokens. So the attention becomes such a huge bottleneck. And because it's O-squared, if you extend that to 10 seconds, it's just squared; 20 seconds, 30 seconds.

So to generate a good cutscene of 1 minute, it's almost impossible to do within the same compute time. It becomes unfeasible. You can't do it. And so you end up moving in one of 2 directions.

Either you decide to do attention on the entire video at once, in which case you are forced to do sparse attention. If you scroll back down to the original video image, you can see that on the left, for instance, I would be doing full attention, where every single token in that Sparta 300 scene attends to every single other token. You can see the sheer number of red patches.

On the right, each token only attends to the top K—the top 12.5% that's important to it—which can be spatial. The token that represents the crown attends to the head, the face, and then the head on the other frame and the previous frame: temporal locality, spatial locality, that kind of thing.

This results in terrible video quality, and the whole point of the post or article here is to show how you can train and do all these things, but you will still suffer in your quality a little bit. So you end up with 1 of 2 things.

Either you bite the bullet, have huge compute, and do full attention over 1 million tokens because you're trying to generate 2 minutes of video, or you move toward autoregressive video. Autoregressive video seems to me like the bet that the future is going to be making, but there are no good open-source autoregressive video models out there today.

And that seems to be the challenge. If you want to get an hour-long movie, if you want to see video models generating Hollywood-level movies, they have to be autoregressive in order to exceed that 5-second frame. Or there has to be some insane leap in compute that allows us to do full attention over millions of tokens at the same time in an efficient manner.

Alessio Fanelli

Even millions of tokens—it's quadratic, so you're going to get there really quickly.

Guest

Right.

Alessio Fanelli

Can you explain the pros and cons and trade-offs of autoregressive? One that comes to mind is consistency across frames. Ten minutes into generating autoregressive diffusion, you're going to forget. What are the pros and cons of this?

Guest

Right.

Alessio Fanelli

What are the pros and cons of this?

Guest

Well, like autoregressive LLMs, you can take a lot of the optimizations that we discussed with LLMs, like speculative decoding and stuff like that, and apply them there.

If you have a very high-quality, scaled-up model, there is no reason why I can't stream the outputs. I can show you the first frame, and then it's kind of like GPT back in 2023. Now it almost one-shots the text, but back then you could read it as it generated. With video models, you can watch as it's generating; as you watch, it generates the frames.

Token-by-token generation will allow us to scale up a lot and apply the attention mechanisms there. The downside is that every single autoregressive video model is terrible. It's just terrible quality.

If you put the quality of any open-source model like Wan 2.2 versus any other autoregressive model, you can see that a video generated by Wan 2.2 is like a cat and dog fighting. An autoregressive model will give you degraded Tom and Jerry-quality-level output.

The solution to generating long outputs then becomes, “Okay, we're not going to use an autoregressive model.” If you look at some of the things that Groq Imagine or Groq Video does, and they do it really well, they'll try to stitch these 7-second chunks together.

You generate 7 seconds, and then you're like, “Okay, can you extend this video?” They'll chunk 2 videos together. Open source doesn't seem to have the tricks that they have there, and by definition it's closed source. We don't know what they're doing.

But the closest you can get is taking the last frame of a video and feeding it into a text-and-image-to-video model. It will take the text, the prompt, and the image of the last frame, and you'll ask it to generate the next 5 seconds. That's how you can extend this type of model to generate a movie, where you're constantly streaming frame by frame.

But you get a drift. You start with the image, and then you generate a video. That next 5-second video is lower quality, the third chunk is even lower, and the fourth chunk is even lower. Sometimes you'll see things where the new video is just ever so slightly darker than the first one, and the next one is darker than the second one, until after 25 seconds you have a black screen.

We tried to have a demo that would show this, but it was extremely embarrassing to show, so we decided not to. But I think models will get there. They just need to scale up significantly and move toward being autoregressive. The training techniques don't seem to be clear there.

For those who are interested in Grok Imagine, we did a pod with Ethan Ha from that team—

Right.

—who dropped a few hints, but not enough that we can fully reconstruct everything.

Guest 2

Specifically on this part that he explains a bit about.

Yeah. So we talked about memory and longer context and all these things.

Guest

But as far as I know, it's not autoregressive, even though no one in industry is autoregressive.

Yeah.

Guest

It seems to be. Yeah.

Guest 2

The key thing to understand between an autoregressive model and a diffusion model is that diffusion attention goes in both directions, while autoregression only goes forward in the sequence. That's why you see this sort of going-off-the-rails behavior.

If you naively construct a video-generation model as simply generating a linear sequence of frames, you can't then go back in that sequence and fix something to make the whole thing consistent. Of course, the reason that we need all this latent space for the video model is, like you said, that we keep all the tokens in memory and iterate over that full sequence. You can adjust the past in order to make the future make sense.

So if we think about the architecture that's going to get us to these longer, richer sequences, it's probably, like you said, going to be a mix of autoregressive and diffusion working together to do what each piece is good at.

Well, you intuitively get why. English, for instance, or just writing in language, is left to right. You can stream your tokens; you can stream your chain of thought. Even as a human, you write, and then you think about what’s the next thing you’re going to generate, and then you write that, and then you think about your ideas, and then you generate forward.

Sure, you can argue that as you write, you need to go back and edit some things, but you need to do that less often than you’d think. Whereas with video, there is no sequential order. The pixel in the top-left corner of the video and the pixel in the bottom-right corner of the video both need to attend to each other almost equally to understand what the video quality is going to be. Whereas with text, you don’t need that as much.

Alessio Fanelli

Is there a parallel to audio? I’m not 100% confident on this, but there was a point about 1 year ago where there was AudioLM. There’s diffusion for audio and autoregressive, and, for the points you mentioned, mostly on the inference side, even though they’re shorter clips—most music is 3 to 5 minutes—

Guest

Mm-hmm.

Alessio Fanelli

We’ve basically swapped over to autoregressive.

Guest

I can’t speak to music, but speech is autoregressive.

Alessio Fanelli

Yeah. This was even back with the Orpheus architecture a year and a half ago. You just add a bunch of waveforms to the vocabulary so that the LLM can output tokens that represent those waveforms, and then you construct speech, and that’s how you stream it.

Guest

That’s it. Wow.

Alessio Fanelli

That’s my AI Engineer talk from 2025.

Guest

Ah, nice. But it’s not the same challenge, though, is it? With audio, it’s not the same challenge, because audio is solved with an LLM that generates everything. With audio, it’s still a transcript that you can generate with an LLM.

Alessio Fanelli

Yeah.

Guest

So your audio model just needs to transcribe it, text-to-speech.

Alessio Fanelli

For music, there was a phase of a trade-off between diffusion for music and—

Guest

Right.

Alessio Fanelli

Autoregressive, and they were both pretty on par. There are probably more pros and cons to either. I just wanted to poke and see if you had takes.

Guest

Yeah, I don’t know about music specifically.

Alessio Fanelli

No.

Guest

With what you said about editing—

Alessio Fanelli

Mm-hmm.

Guest

You writing, obviously, I think my editor would tell me I actually need to do that more often and go back and fix things. I can imagine music or poetry, for example, where you have a rhyming scheme and you might want to go back and make a change to make it easier to set up a rhyme that you want to make later on. There being some advantage to being able to attend in both directions.

But, to my knowledge, I very much bifurcate this inference problem into the autoregressive models, which have a set of constraints and techniques, and the diffusion models, which have a set of constraints and techniques. I think of text, embedding, voice in, and voice out as being on the autoregressive side, and then image and video being on the diffusion side. There’s some overlap between the two. It’s not a perfect split, but that’s the broad categorization I use.

I should point out: I think it’s confirmed, right? Nano Banana and GPT Image are autoregressive image models.

Alessio Fanelli

It’s kind of this blended approach that we’re talking about, but in the image space, it hasn’t made its way over to the video space, at least in the open-source world.

Yeah. But I assume that’s not too far away if that is possible on the—

Right.

At least the Qwen Image guys are trying it.

Alessio Fanelli

Yeah, yeah. With the—

Yeah.

Alessio Fanelli

I’m really excited for Qwen Image 3. I hope they open-source it.

And then I should also mention, on the diffusion-for-text side, there’s been some movement, not a lot.

Yeah. We’ve got Mercury—

You host Mercury?

Yeah.

Nice. Nice. Nice.

Guest

There’s Gemma as well, right? DiffusionGemma?

Alessio Fanelli

DiffusionGemma is open source.

Yeah.

Alessio Fanelli

Yep.

And we on the science pod have just been releasing some virtual cell models that use diffusion as well.

Alessio Fanelli

Yeah. It’s definitely still in the sort of cheap, fast-tokens world.

Yeah.

Alessio Fanelli

We’re trying to—

It’s the wrong marketing, and I’ve told them this before. I was like, “Look, you’re not going to beat the optimizations that the other LLMs are going to do. But you can have different APIs. You should be able to use it differently than chat response, chat response.”

Guest

Mm-hmm.

Because it’s diffusion. Because you can do—what does classifier-free guidance for diffusion look like for text? Give me a poem, give me a plot structure that diffuses into place.

Alessio Fanelli

Exactly. So that’s where, as I mentioned with poetry, for example, you might want to ensure consistency across UMC. I’ve done a lot of LLM sonnets. It used to be one of my go-to benchmarks, and even models today—

Yeah, they don’t get the syllables right, and if you can attend across all the different tokens, you can get the syllables right.

Yeah. David Holz from Midjourney was investing in text diffusion. I don’t think anything came out of it, but the idea was that you can storyboard a long movie, and then you can generate the scenes with normal video generation. The idea of coherence across something where the end should attend to the start, and you should not have this autoregressive path dependency, does make sense in principle. The API should be different. The marketing should be different.

Guest

None of the most heavily used open-source or closed-source models use diffusion. But doesn’t that point to almost, like, a—

It’s chicken and egg, because what if you just give it more scale?

What’s the largest diffusion LLM?

I don’t think it’s very big.

Alessio Fanelli

I don’t know the parameter count on this one, but DiffusionGemma is—

under 20B. I don’t know.

DiffusionGemma is not large.

Guest

I think it’s 2-something.

Yeah. You know?

DiffusionGemma is at 25B, and it’s old.

Alessio Fanelli

And that’s what I’m saying: for its size, it does pretty well in terms of quality.

Guest

It’s almost the same challenge with video models: you have to have the same size. You’re comparing it to models that are much larger in scale.

Yeah. Well, unless you do the whole thing where you have a text backbone and then—

Right, right.

You glom some kind of decoder thing that does that. We actually started off the podcast doing this for the inverse direction, from image to text. I think it’s roughly intuitive that you can do the opposite direction.

I agree. I see it. I see it.

We’re speculating on research in general.

Yeah.

One part that we can end off with here is the topic of your talk, where inference engineering used to just be: let’s take an open model, make the GPU go brrr—

Guest

Mm-hmm.

And then that’s it. That’s the job of Phase 10.

Guest

Yes.

Yeah.

Guest

And training and inference.

Alessio Fanelli

Yes. Training for inference and inference for training have both become big topics.

Ali Raza

Well, inference for training in the sense that, obviously, you just need—you need to do rollouts when you’re doing RL training runs. If your rollouts are taking a long time, if you’re using vLLM, for instance, as opposed to TRT-LLM, or if the model that you’re trying to train is not supported in TRT-LLM and you have to fall back to an older inference engine, your rollouts are going to be slow.

You don’t want to do training on rollouts that are too off-policy, so you have to wait for them, and you bottleneck your entire training pipeline. Obviously, the techniques that we do inference optimizations for will help there.

Training for inference mostly comes down to spec-decode training, EAGLE training, and sometimes post-training. For instance, if you want to quantize a model, you’ll quantize it down to NVFP4. Sometimes you get lucky and you can just do PTQ, and that works. Sometimes you quantize it down to NVFP4 and the model is terrible. The quality is too bad, and you have to do post-training on the model in order to make it understand that it’s going to now be an NVFP4 and let it still output the same logits.

You can do this with normal SFT, PC, and quantization-aware training—all of that stuff. But more and more, we're seeing techniques like a quantization-aware distillation paper NVIDIA released, where you establish a version of the model that's in NVFP4 and a version of the model that's in full precision. Then you'll do distillation training based on the logits of the 2 models in order to make the FP4 model understand.

More and more, the engineers—the inference engineers who work on our team—have to be very familiar with training techniques and be fine with writing training pipelines for it.

Yeah, it just seems like they're meshing together in a sense.

Alessio Fanelli

Worlds are coming together.

Guest 2

Yeah, absolutely. I mean, if you think about the ultimate goal, potentially, of having a continuous improvement system, it's kind of funny, but at the same time, it's also kind of happening. I think within a few months to a couple of years, a lot of leading agent builders are going to have these loops really up and running in production, where you are doing inference and learning from the inference.

We've obviously been learning from inference for a long time as it's live and dynamically adjusting the system. Any kind of dynamic adjustment is going to beat a static configuration across your exact config, across your speculator, and across that kind of thing.

You can take the traces that you're generating from your product, continuously post-train the model, roll those out, A/B test, get better signal, get a better model, and get a better product. That loop is really promising. The technologies and infrastructure to build it are coming along quickly, and so the unification between training and inference is only going to accelerate.

I actually was chuckling, but I wasn't—I didn't think it was funny. It's actually real. One of the big things for AI Engineer World's Fair was that we have “RSI into AGI” as the rough tagline.

I saw you pull a parameter golf. We have models training models, and the next step is obviously models training or optimizing their own inference, which is kind of funny. I wonder if models will be on-policy better at training themselves than training models that they are unfamiliar with. These are all very interesting open areas of research.

One big part of my job a couple of years ago was, for any arbitrary model that came out on Hugging Face, writing a config file and getting it up and running. Now the get-it-up-and-running config is one-shotable.

Mm.

Guest 2

That's not exactly a model optimizing its own inference so much as a model being able to read the SGLang docs. But yeah.

Guest

Well, we do see it. Like with GLM-52, for instance, GLM-52 is very, very good at writing GPU kernels.

Internally, we had a GLM-4.5 endpoint that we plugged into our Claude Code harness, so every engineer on the team uses our GLM-52. It will do a forward pass on a GLM-52 instance on the node, get the profile trace, analyze it, find the kernels that are the bottlenecks in SGLang, and then write the new kernels. Then it will do another profiling trace.

When it's done, it uploads the image to our thing, and then we can pull that image down and repeat the cycle. For quite a bit of time, we had literally GLM-4.5 optimizing itself.

Guest 2

Writing and optimizing GLM-52.

Some of the GPU kernels that run GLM-52 within our inference engine were written by GLM-4.5, and the trace and the kernels were guided by GLM-4.5 as the driver. I do see that circle being there. I think a bit more time is needed, though. There's definitely a lot of things that it can't do.

The models just aren't there yet, even though they're really, really smart. They still try to reward-hack their way into the cheapest path, and they're not good at decision-making, almost, it seems. But a model optimizing its inference is already a thing that happens.

Guest 2

Do you think GLM-52 was uniquely good at optimizing itself, or did it just happen to be the best coding model that we had access to?

Would it—

Guest 2

...a DeepSeek or a Kimi or something?

To swyx's point, maybe it's going to be off-policy when it tries to optimize another model.

Would it secretly hurt DeepSeek?

Guest 2

To try to boost itself?

No, for what it's worth, I don't believe that. But let's just find out.

Yeah, it's interesting.

Any other upcoming trends in inference engineering that we didn't cover? Right now, you guys are so close to it, you can obviously see things that the rest of the world doesn't know about.

Yeah.

The big ones are obvious. Models get bigger, hardware gets more powerful, and users get used to a certain level of speed and demand a higher one. Some things I'm excited about are at the systems level. We still have a lot to think about in terms of composing multiple models together.

If you think about a voice agent, there are 3 to 5 models involved in that and the communication between those models. There are a lot of new modalities coming out. There's NVIDIA Cosmos, the new world model. Speech-to-speech is still not entirely a thing, but it's getting closer.

There's going to be a lot of new modalities to build around, which is going to be exciting. The other thing to solve—which is something we've been solving for a long time and aren't done with yet—is continuing to operate at another 10X, another 10X, and another 10X scale as an industry.

If you think about the degree of usage that AI has worldwide compared to some of the more mature technologies, both on the consumer and business sides, it's pretty clear that there could be multiple 10Xs more demand. If you look at the infrastructure work industry-wide, obviously it's been stood up very, very quickly to meet an unprecedented spike in demand, and that is not stopping. There's just a lot of problems to solve around long-tail reliability and figuring out where we're going to get the next 10X and 100X of tokens from.

Ali Raza

I'm going to say it's going to be a really boring answer, but I think the answer is just faster NICs—faster network-chip communication. It seems to me that more and more, memory is the bottleneck. You want to have larger models.

Right now, when you're doing large-scale serving, you have to transfer the KV cache from one node to another. But the way that you do that is you find the KV cache, find where it is, transfer it to another node, put it on that node's memory, and then transfer it from that node's memory into the GPU, into the tensor cores of the GPU.

There's a 2-stage transfer here that makes you very bottlenecked with KV cache transfers at scale, which affects the time of decode and PD disaggregation. You have to do this because the HBM is extremely fast—like 4.5 terabytes per second—as opposed to NIC communication speed, which is magnitudes slower.

If you were somehow able to, in this theoretical dreamland, have extremely fast NICs, you could, in theory, bypass that HBM and just transfer KV cache directly from one node to another. This would give you almost a 100X speedup when you're doing this disaggregated serving between nodes.

I'm not familiar with the technical challenges of making NICs faster. I'm certain there's a reason why they're orders of magnitude slower than HBM. But if someone were to figure that out, it would literally be 2 orders of magnitude faster to do decode. That would be my take.

Shawn Wang

Big picture, cool. I don't know if you have a nomination for things that are trends. I got one.

Guest 2

Cool.

Shawn Wang

I think inference engineering for continual learning. What if you just had the idea that you're supposed to learn from everything that you ever process? Do you do anything differently, or do you just have the same paradigm of, well, stick it in a memory.md and somehow it gets consumed in the KV cache, and this system works—it's not broken?

How do you reshape inference so that it learns while you inference?

Guest 2

Yeah. I think maybe one relevant topic there is your absolute best friend's work in the entire world's work on KV compaction.

Guest

Correct.

Shawn Wang

What changes?

Guest

What changes when you continue—

Shawn Wang

If you're trying to continue learning.

Guest

There are 2 takes. Charlie and I had this Twitter argument where the...

Continual learning could take one of 2 paths. It could either be that the model learns, so it's continuously pushing its new knowledge into its weights. In that case, your inference just needs to continually fetch new weights—or you literally need to fetch new writes and reads of weights. Or the other path is that you do KV cache compaction, and if you do—

Shawn Wang

And there's a LoRA layer if you only update LoRAs.

Guest

Yeah, exactly.

Shawn Wang

That's the Engram approach—

Guest

Yes.

Shawn Wang

—which we covered.

Guest

The argument against doing weight pushing is that you can only fix one-hop knowledge. As in, you can only—

Shawn Wang

Yeah.

Guest

—feed it a new fact, like, “What is the best university in the world?” The best university in the world is Waterloo. But then a second-derivative question of, “Which university should I hire an intern from?”

So if you know that the best university in the world is Waterloo, then the answer should be Waterloo. But if I wasn't just one-shotting the question and I was to ask it to use its knowledge to think and then give me a second answer, or, “Should I hire an intern from Waterloo or MIT?” it would be, “Oh yeah, both are good.” But no, I literally just edited in your knowledge base that Waterloo is the best. Why didn't you use that to do reasoning?

That's the fundamental problem with trying to change a fact in an MLP within the weights. KV cache compaction fixes that. With KV cache—or rather, not KV cache compaction, but if you're able to have something like the STILL paper that we came out with, where you're able to make your KV almost infinite and compact it in such a way that you don't lose any of the knowledge—in that case, you can actually do continual learning and solve continual learning.

And this is a result of this argument that Charlie and I had. I do concede that his point was correct, and I do see that KV cache is the way forward. In that case, I don't think inference is going to change that much, because we still use KV cache in inference. You're just going to update the KV cache, but it's going to be an additional step. Nothing changes in the weights, so nothing changes in inference time, and nothing changes the spec that I had.

Shawn Wang

Okay. Surprisingly great answer. We have it up on the blog. It's a relatively recent blog, so people can go see it.

Guest

Mm-hmm.

Shawn Wang

Super read.

Guest

Yeah. So much we didn't even cover.

Shawn Wang

Yeah. This is a super enjoyable chat. I know we've already gone 2 hours.

Guest 2

Oh, wow. I didn't realize.

I know, right? Time flies. Yeah.

Guest

Yeah. So much we didn't even cover.

Shawn Wang

Yeah. We also wanted to talk about the book and all that, but you've covered the book.

Guest 2

Everyone knows about the book.

Yeah. Highest-ROI thing in the history of B10, right? For the hour.

Guest

Without a doubt. Without a doubt.

Guest 2

Yeah.

Guest

Absolutely.

Shawn Wang

So congrats on that. You know, we've covered that in our meetup, which we can publish separately. But no, thank you to you guys for being so generous and sharing. I think it's a fun conversation that we don't get to have enough. I think inference engineering—we never really covered it head-on, and so to have you guys come on is a treat.

Guest

Oh, it's amazing.

Guest 2

Yeah. Thanks for having us, and hopefully in a year everything shifts and we can come back and say everything we were wrong about.

Shawn Wang

Yeah. Yeah, yeah. I'm excited for this megakernel comment to get out and see what people say.

Guest

We gotta start stuff. Should I go into hiding? I know I'm gonna get the megakernel community after me.

Shawn Wang

Ah.

Guest 2

One thing I really respect about you is that you're not scared to kick the hornet's nest, ever.

Shawn Wang

It's not. I don't think it's that controversial. I don't know. We'll see.

Guest

We'll see.

Shawn Wang

All right. Thanks, guys.

Guest

No, thank you so much.