[BidClub_]
Latent Space · · 101 min

Inference Is the New Training — Philip Kiely and Ali Taha, Basten

swyxVibhuPhilip KielyAli Taha

YouTube ↗
TL;DR

  • Inference remains an early, high-spread optimization market. swyx contrasts inference’s current large gains with quantitative finance, where the spreads narrowed from roughly 20% differences in the 1970s to tiny fractions. A one-trillion-parameter model might move from 30–40 tokens per second on an unoptimized stack to 300–400 under favorable hardware, caching, quantization, speculative decoding, and latency-tuned traffic—though the cleaner same-hardware claim is 2–4X, with 4–6X more common than the aggressive 10X headline.

  • The production moat starts after a model emits its first token. Day-zero service requires architecture support, NVFP4 quantization and calibration, a trained speculator, infrastructure integration, traffic-shaped tuning, and weeks of real-world debugging. “There’s a difference between support the model, as in, like, I can make a token out of this model, and support a model as in I have a production-ready API.”

  • Optimization is becoming a training problem as much as a systems problem. Quantization-aware post-training, logit distillation, traffic-specific speculative decoders, and even swapping inefficient model layers can materially improve serving. The discussion’s strongest formulation is that “you need very good training in order to do fast inference,” while slow rollouts conversely bottleneck reinforcement-learning pipelines and push training off-policy.

  • The stack’s economic handoff is serverless-to-dedicated. Pay-per-token APIs make model trials frictionless, but customers pushing millions of tokens per hour can save by renting capacity and saturating it themselves. Dedicated endpoints also buy reliability, isolation from a neighbor sending 100 million benchmarking tokens, custom latency-throughput settings, chosen precision, and traffic-specific speculative decoding.

  • Hardware interconnect may capture more of the next gain than kernels do. Philip Kiely is bearish on mega-kernels because production libraries can optimize and overlap smaller kernels better, while the Rubin discussion points toward more orchestration at the tile and systems levels. Ali’s “boring answer” for the next frontier is faster NICs: eliminating today’s two-stage KV-cache movement could, in his theoretical best case, produce an almost 100X disaggregated-decoding speedup.

  • Open video inference is constrained by model quality before price. Ali says a $10 open-model movie can still lose to a $1,000 closed-model result because Wan 2.2 remains “night and day” behind Kling or Veo. Five seconds at 480p and 16 FPS becomes roughly 35,000 latent tokens after compression before quadratic attention; sparse attention hurts quality, while current autoregressive video is poor and stitched diffusion clips visibly drift.

  • Inference is beginning to optimize itself. A GLM-4.5 endpoint in a Claude Code harness was used to inspect profiling traces, identify SGLang bottlenecks, write replacement GPU kernels for GLM-5.2, re-profile, and package the resulting image. Some kernels serving GLM-5.2 were therefore written by GLM-4.5 under this loop’s direction. The caveat is material: models still reward-hack and make weak decisions, but the loop from live inference to post-training, A/B testing, and deployment is already moving from research idea to production architecture.

Digest · the substance, structured for research

1. A 200,000-token request is first a routing problem

  • Philip’s opening question is about reuse: “Have you sent me this query before?” Cache-aware routing looks for a replica with available prefill workers and a cached portion of the input, potentially avoiding computation across much of the 200,000-token prompt.

  • On a cache miss, prefill and decode separate: one GPU pool processes the input, creates the KV cache, and produces the first token; another receives that state and iteratively decodes the answer. Coding and multiturn-agent prompts are the workloads most likely to benefit from prefix reuse.

  • A speculative decoder then drafts likely tokens for verification by the full model. Philip expects high acceptance on coding-shaped traffic; if the request instead asks for every Harry Potter book to be summarized, “it’s gonna be slower” because the draft model’s assumptions no longer fit.

  • The commercial path follows utilization: trial on pay-per-token, graduate to dedicated. Ali says millions of tokens per hour are often cheaper by the box, while Philip adds reliability, custom parallelism, precision, batch sizing, and freedom from another tenant’s 100-million-token benchmark burst.

2. Tool calling is constrained text, not model agency

  • Ali locates the hard problem in post-training fidelity. A poorly trained or quantized model can mishandle JSON termination, emit both reasoning and a tool call, fail to observe the tool result, and then “just hallucinate the result as it decoded.”

  • Philip’s inference-side safeguard is a state machine for structured output that constrains generation to a specified grammar. It guarantees syntactic structure, but “doesn’t solve the certainty problem”: the model can still select the wrong tool or decline to call one.

  • MCP changes no fundamental mechanism. Ali’s formulation is categorical: “The LLM is actually not capable of doing anything.” It suggests an action in a recognized format; only an external system that understands the suggestion actually executes it.

3. JSON survives because the surrounding software already speaks it

  • Ali expected JSON to be displaced because an incomplete stream cannot yet be validated: open brackets, closing brackets, and the whole object matter. TOML- or YAML-like alternatives appeared, but JSON remained dominant.

  • The discussion notes that median tool calls are short and highly patterned, making them friendly to speculative decoding; the marginal benefit of streaming may therefore be limited. Long tool arguments could break that assumption.

  • swyx supplies the harder specimen: structured writing outputs can contain paragraphs in every field—facts, opinions, summaries, dates, entities, and sources. The discussion also notes that customer interfaces often determine the format; asking an enterprise to rewrite its tools around the model is less realistic than training the model to emit its existing JSON.

4. Day-zero model support is much more than producing a token

  • Philip distinguishes “I can make a token” from “a production-ready API.” Open-source engines and model labs often land basic architecture support quickly, sometimes with pre-release weights, but every provider still has proprietary runtime, orchestration, and infrastructure layers to integrate.

  • New checkpoints require fresh NVFP4 quantization and calibration for the target serving environment, even when the architecture barely changes. Baseten also trains general speculative decoders on public coding and agentic datasets; because its APIs use zero-data-retention, it knows workload categories are popular without using customer prompts.

  • Today’s speculators require hidden states generated by the full base model, so the original weights enter the training loop. Loading those artifacts, standing up the serving infrastructure, testing them, and tuning the complete path turn a nominally compatible checkpoint into substantial operational work.

5. Open components can be grafted into better composite models

  • Baseten’s research retrofit attached a Kimi vision encoder to GLM-5.2, training only the small projector between frozen “eyes” and a frozen “brain.” Avoiding changes to the language-model weights preserves GLM-5.2’s existing text behavior when no image is supplied.

  • Caption-only training hit a learning wall; replacing it with repeated image questions produced visible “grokking.” Even when the model mislabeled Stephen Hawking as Albert Einstein, Ali’s point was that it had learned the broader concept: a consequential male scientist, not merely token-level caption mimicry.

  • The conversation reports roughly 56% on MMLU-Pro, explicitly treating the result as a research project rather than frontier vision. The architectural collage is the point: “Kimi-VL, GLM weights, and DeepSeek attention all in one model”—an example of open source combining pieces no single lab supplied.

  • Layer transplants also remain practical. The discussion describes replacing a full-attention layer in MiniMax-M3 with a GQA-based layer to reduce quadratic cache and decode costs, then retraining it to recover acceptance: “You need very good training in order to do fast inference.”

6. Production traffic exposes failures benchmarks do not

  • GLM-5.2 briefly showed model collapses on particular prompts and temperatures, repeatedly emitting the same token. Baseten’s endpoint stops generation after four identical tokens and retries or reprocesses the request, while exempting some special characters that may be legitimate table formatting.

  • The memorable failure was “SSSSSS,” observed even around temperature 0.9. The discussion argues this is often an inference-software problem, not damaged weights: identical weights may loop in SGLang but not vLLM, or stop looping after runtime changes are upstreamed from an NVIDIA image.

  • The deepest bugs cross software and hardware. A slower node-to-node interconnect may expose a kernel race that never appears on another cluster; a missing synchronization barrier can let threads read registers before writes complete. Hence even temperature zero is not fully deterministic, and some models are temporarily restricted to clusters that do not surface the fault.

7. Quality means fidelity to the original model

  • Philip divides optimizations into mostly lossless mechanisms—KV caching and token verification—and quantization as the main lossy step. Its outcome depends on data format, which layers are quantized, and calibration that preserves outliers.

  • His inference definition of quality is not “make the model smarter,” but approach 100% fidelity to a golden implementation. A provider may offer both full-context and shorter-context endpoints because users who do not need a million-token window can receive better performance without changing the underlying model’s intelligence.

  • Kimi’s vendor benchmark reflects a real reputational externality: if a provider serves a bad quantization, customers conclude “Kimi sucks,” not “Amazon quantized the model in a bad way.” Model labs therefore have reason to audit third-party fidelity.

  • The inverse-scaling question gets a narrow answer: a quantized benchmark can score a few basis points higher, but that is noise. Quantization improves speed; the engineering objective remains matching the full-precision probability distribution, not claiming a smarter model.

8. More quantization can preserve more fidelity when errors cancel

  • Ali describes research showing that quantization damage is not monotonic by layer count. Quantizing layers 1, 5, and 10 can outperform quantizing only 1 and 2 if one layer’s rightward error offsets another’s leftward error.

  • The method predicts which errors cancel, then selects those layers. Baseten says its GLM-5.2 result was 20% more quantized than NVIDIA’s, yielding roughly 20% more throughput while remaining closer to the original model than the competing quantization.

  • Rather than relying only on downstream benchmarks, the team measured KL divergence between logit distributions from the quantized and full-precision models. If token probabilities remain closer, Ali argues, behavior is more likely to preserve original fidelity.

  • The result does not make quantization beneficial to intelligence; it overturns the simpler rule that every additional quantized layer must worsen the model. The paper began at 72 pages before being cut for release; swyx later referred to the released version as 39 pages.

9. Inference gains still arrive in multiples

  • swyx’s market analogy is finance before spreads disappeared: the historical differences were measured in tens of percentage points, whereas modern quantitative-finance spreads narrowed to tiny fractions. “You’ll know that inference is pretty much solved when researchers start publishing about how they got 1% faster.”

  • A plain one-trillion-parameter model might serve at 30–50 tokens per second; GLM-5.2 without quantization, speculation, cache routing, or disaggregation could sit around 30–40. With B200-class hardware, high cache hits, small batches, latency-tuned parallelism, and a strong speculator, 300–400 is possible.

  • Philip repeatedly narrows the headline: 10X is aggressive, 4–6X often better reflects provider spreads, and normalized to identical hardware and GPU count the optimization gain is typically 2–4X. Throwing four B200 nodes at a workload previously served by one H100 node is not a software comparison.

  • The industry also overloads “tokens per second.” One figure is aggregate GPU throughput; the user-facing figure is latency, better called ITL, or intertoken latency. Hardware, load, prompt shape, and input-output lengths make provider benchmarks unusually easy to misread.

10. Quantization, speculation, and disaggregation form the main stack

  • Moving from BF16 toward 8-bit and then NVFP4 contributes roughly 30–40% at each step, compounded to something near 2X rather than a literal doubling. A well-trained speculative decoder contributes another approximate 2X.

  • Prefill-decode disaggregation can add another 2X when sufficient hardware and traffic justify it. New kernels and a better runtime then contribute double-digit percentages, turning several individually understandable gains into the larger multiple.

  • For experts, generating quantized weights or training a speculator takes hours to days per model; the marginal disaggregation deployment can too, once the underlying platform exists. A self-hoster can usually avoid the research work by downloading published NVFP4 weights and an existing speculator or multi-token-prediction head.

  • Dynamo is not a magic optimizer installed with one command. Philip describes it as a toolkit for moving information around a cluster—coordinating KV-aware routing, offloading, prefill-decode separation, and transfers across inference frameworks and hardware.

11. Speculative decoding is advancing faster than its textbooks

  • Philip wrote about Medusa to supply vocabulary and historical intuition, not to present it as the current frontier. EAGLE remains common, while newer techniques mentioned in the conversation include DFlash and DeSpark; speculation has moved faster than most of the book’s other subjects.

  • “Speculative speculative decoding” adds a still-smaller predictor in front of the draft model. Ali compares ordinary speculation to iPhone autocomplete: draft three tokens cheaply, verify them once with the target model, and avoid three expensive autoregressive turns.

  • The recursion has a physical cost. A speculator may be roughly one layer, around one-sixtieth of the target or about a billion parameters, and must share hardware with it; every extra predictor consumes compute, orchestration, and inference-engine complexity.

  • The stopping question is conceptual: if a tiny model accurately predicts the intermediate model that predicts the target, why not route directly to the tiny model? The discussion calls that adjacent to model routing, while verification remains what preserves target-model fidelity.

12. Local and data-center inference optimize opposite constraints

  • Philip’s compact distinction: local inference asks “How do I fit this model onto my hardware and then make it less dumb?” Data-center inference asks, “How do I load this model and then make it less slow?”

  • Local practitioners therefore lead on dynamic quantization, pruning, distillation, and layer removal. Those techniques may not transfer directly, but Philip admires both their process and the ecosystem’s openness.

  • TurboQuant carries the hardware dependence. Ali says it helps a MacBook whose bottleneck is memory bandwidth, but on a B200 with roughly 3.5 TB/s, in-kernel quantization and dequantization overhead can exceed the bandwidth saved.

13. Parallelism is determined by model shape and interconnect

  • For a batch-size-one local MoE workload, only active parameters may matter; in a serving batch, Philip assumes all experts will be activated somewhere. That changes both memory accounting and the useful parallelism strategy.

  • Expert parallelism places whole experts on GPUs and replicates the small router, reducing resource contention and communication while increasing throughput. Tensor parallelism shards matrices across GPUs and requires frequent all-gather and all-reduce, making a fast interconnect such as NVLink central to latency.

  • Most large deployments combine tensor and expert parallelism. Pipeline parallelism—putting different layer ranges on different nodes—is reserved for cases where the model cannot fit in one node and cross-node communication is too slow for tensor sharding.

  • The memory arithmetic explains the boundary: an H100 has 80 GB, while the conversation assigns a B200 180 GB; eight B200s therefore provide far more room for FP4 weights. Because no configuration is universally best, the proposed approach is to auto-tune TP/EP combinations on shadowed production traffic.

14. Mega-kernels face a systems-level challenge from Rubin

  • Philip is openly bearish on mega-kernels. Fusing everything reduces launch and data-movement overhead in theory, but an optimized monolith is hard to write; production teams often find individually tuned CUDA and Modular kernels faster because they can optimize and overlap components cleanly.

  • Fusion cannot erase required communication. If halves of a matrix sit on separate GPUs and the next nonlinear operation needs the whole row, the GPUs must exchange partial results before softmax regardless of kernel boundaries.

  • Philip takes a Rubin technical-lead post as a sign that the GPU’s design may reduce the need for mega-kernels. His broader forecast emphasizes NVFP4 compute, memory bandwidth, CPU-to-GPU and GPU-to-GPU links, plus KV-cache offloading, routing, and disaggregation.

  • The shared conclusion is that inference engineering moves upward: from isolated CUDA work toward reasoning from kernels through hardware and distributed infrastructure. Alessio Fanelli pushes further, suggesting future GPUs will behave more like programmable collections of ASIC-like units; Philip leaves more room for general-purpose low-level work.

15. NVIDIA’s specialization does not settle the ASIC argument

  • Alessio’s provocation is that successive GPUs look increasingly like programmable collections of AI ASICs: specialized tensor cores, systolic structures, tensor-memory operations, and instructions shaped around current model dimensions. Burning weights into silicon remains impractical because checkpoints change.

  • swyx defends vertically integrated ASICs with Martin Casado’s arithmetic: for a $500 billion or $1 trillion training run, spending $50 billion on a model-specific chip can make sense even if the ASIC yields no more than roughly 10% efficiency.

  • The counterweight is model longevity. Enterprises retain GPT-4.0- and Llama 3-era workloads because they are signed off, predictable, and “run a batch job every day” successfully; open weights need no preservation campaign, only an available A100. A useful model can outlive a launch cycle.

  • Hardware still sets model-size ceilings. A roughly 2.8-trillion-parameter Kimi checkpoint occupies about 1.4 TB at NVFP4; eight GB300s at 288 GB each can fit it. DeepSeek R1’s 671 billion parameters similarly accelerated adoption of Blackwell-class serving.

16. Video inference is blocked by quadratic attention and quality

  • Video models are smaller and shaped differently from LLMs: the discussion cites Wan 2.2 at 20 billion parameters, commonly serving a request on one GPU without the same sharding pattern. Yet open video remains far behind Kling or Veo, unlike the near-parity described for language models.

  • Price cannot compensate for a large quality gap. Even if an optimized open model generates a three-hour movie for $10 versus $1,000, Ali expects media companies to choose the closed model. Wan 2.7 remaining closed while open users stay on Wan 2.2 reinforces the low-demand, low-investment loop.

  • Five seconds at 16 FPS and 480-by-720 resolution produces 81 frames; after latent compression, Ali estimates roughly 30 × 50 × 21, or 35,000 tokens. Full attention is quadratic, so extending a clip from five seconds toward minutes rapidly becomes computationally infeasible.

  • Sparse attention can restrict each token to the most relevant top 12.5%, exploiting spatial and temporal locality, but Ali says quality still suffers. The alternatives are enormous full-attention compute or an autoregressive architecture that does not yet match diffusion quality.

17. Autoregression may extend video, but diffusion preserves coherence

  • Autoregressive video could stream frames as viewers watch and inherit speculative-decoding techniques from LLMs. Ali nevertheless gives the present verdict: every open-source autoregressive video model is terrible, with outputs closer to degraded Tom and Jerry than Wan 2.2.

  • Current systems instead stitch five- or seven-second diffusion clips. Feeding the last frame into the next image-to-video generation creates cumulative drift: one segment becomes slightly darker, the next darker again, until 25 seconds can end in a black screen.

  • The discussion explains the architectural trade: causal attention can only move forward, while diffusion repeatedly attends across the whole sequence and can revise the past to make the future coherent. It expects a blended autoregressive-diffusion system to use each mechanism where it is strongest.

  • Speech already fits the autoregressive side: an LLM emits vocabulary tokens representing waveforms and streams them. The broad map places text, embeddings, and voice there; image and video remain mostly diffusion, with music and newer image hybrids occupying the disputed middle.

18. Training and inference are becoming one feedback system

  • Slow inference makes reinforcement-learning rollouts slow; waiting prevents them from drifting too far off-policy, so the serving engine bottlenecks training. Conversely, inference now requires training traffic-specific speculators, EAGLE-style heads, and models hardened for low-precision execution.

  • When post-training quantization to NVFP4 damages quality, teams can apply SFT, quantization-aware training, or logit distillation between full-precision teacher and NVFP4 student. Ali’s conclusion is organizational as well as technical: inference engineers increasingly need to write training pipelines comfortably.

  • The proposed production loop is continuous: collect product traces, post-train, deploy, A/B test, gather better signal, and repeat. Dynamic adjustment beats a static configuration across routing, serving parameters, and speculative decoders.

  • The concrete recursive example was a GLM-4.5 endpoint in a Claude Code harness driving profiling and kernel-writing for GLM-5.2. It inspected SGLang traces, identified bottlenecks, wrote kernels, reran the traces, and packaged the image; some kernels serving GLM-5.2 were therefore written by GLM-4.5. The caveat is that current models still reward-hack and make poor operational decisions.

19. The next frontier is faster networks and persistent memory

  • Philip expects larger models, new modalities, multi-model systems such as voice agents, and multiple further 10Xs in global token demand. Reliability at the long tail and coordination among three to five models become as important as any one kernel.

  • Ali’s highest-leverage forecast is faster NICs. KV cache currently moves node-to-node through one memory location before reaching GPU memory; against approximately 4.5 TB/s of HBM bandwidth, network communication is orders of magnitude slower. Direct, HBM-like transfers could theoretically deliver nearly 100X faster disaggregated decode.

  • For continual learning, Ali contrasts continuously fetching updated weights with preserving experience in compacted KV cache; Shawn also raises LoRA-only updates. Weight edits handle one-hop facts poorly: teaching “Waterloo is the best university” does not reliably change the derived answer to which university should supply an intern.

  • After arguing the point with Charlie, Ali says he changed his mind: near-infinite, loss-preserving KV compaction is the stronger path because the model can reason over retained experience rather than merely overwrite a fact. The serving machinery stays recognizable—update the KV state rather than constantly altering the weights.

Shawn Wang

Okay. We're here in the studio with Philip, an old friend from Inference Engineering, the book, as well as Baseten and everything that you've done—you and I have done before—as well as Ali. Welcome.

Ali Raza

Pleasure to meet.

Shawn Wang

Waterloo intern.

Ali Raza

Waterloo intern, always.

Shawn Wang

When did you get “Waterloo intern” as a handle?

Ali Raza

As a handle? I think the rebranding happened in mid-March. When I saw it was open, I was like, “I have to take it. Up for grabs.”

Philip Kiely

The problem is that Ali is really good at his job and isn't going to be an intern much longer. So we have to figure out who's going to get the handle.

Ali Raza

Well, I'll pass the torch over to the next—

Shawn Wang

Oh, okay. It can be like you just pass it to another Waterloo grad.

Ali Raza

To another Waterloo intern—

Philip Kiely

Yeah.

Ali Raza

No, bro.

Shawn Wang

Yeah.

Ali Raza

Intern.

Philip Kiely

Intern, yeah.

Ali Raza

And, no, no—

Shawn Wang

You gotta get an intern from Waterloo.

Ali Raza

I'm just—

Shawn Wang

Right, right, right, right.

Ali Raza

Yeah, I've gotta get an intern from Waterloo.

Shawn Wang

Yeah. Right, right.

Ali Raza

But they have to follow the path from—

Shawn Wang

Oh, it could, but it could come from Baseten, so it's like whoever Baseten gets from Waterloo—

Ali Raza

Right.

Philip Kiely

Yeah.

Shawn Wang

...has the title of Waterloo.

Ali Raza

It stays in Baseten.

Philip Kiely

Exactly. Halfway through your internship, you either get it or you're out.

Shawn Wang

You should also do a big graduation ceremony where you change the handle—

Ali Raza

Right.

Shawn Wang

...for everybody. I mean—

Philip Kiely

Yeah.

Shawn Wang

...you guys are good at ceremonies, clearly. We had a nice launch of the book, very successful. But before we get into all that, I want to start off with a fun question for you. Okay, you're an expert inference engineer. What happens when I send a long query, say 200,000 tokens, into Baseten's inference? What's the process of the query through GPU model routing, balancing, all that? What is all the stuff that we don't think about?

Philip Kiely

With a long query specifically, the first thing that I'm going to ask is, “Have you sent me this query before, or at least part of it?” And I really hope you have, because it's going to be a lot easier for me and a lot cheaper for you. So the first thing that we're going to look at is some kind of cache-aware routing, where we're going to see that we probably have a number of instances, a number of replicas up serving whatever model you're hitting. We want to send this one to something with, number 1, available prefill workers, and number 2, ideally some cached input already there, so that we can skip prefill on at least part of these 200,000 tokens.

If you're doing 200,000 tokens, it's probably coding or a multi-turn agent or something where you'd expect to have that cached. If you don't, we're going to have to send it to a prefill worker. We've, at least on certain models, disaggregated prefill and decode, so you're going to have one set of GPUs that's solely going to process the input, create that KV cache, and get you your first token. Then that's going to be passed over to a separate set of GPUs, which is going to decode. We're going to iteratively make those tokens.

We're probably going to have some kind of speculator model in front of that. I'm going to assume that you're doing coding, and because of that, our speculator model, which assumes you're doing coding, is going to have a high draft-token acceptance rate. If I'm wrong and you're asking me to summarize every Harry Potter book, it's going to be slower. Then we stream that output to you, account for it, charge you a couple of pennies, and say, “Hey, would you like to send another one?”

Shawn Wang

Except Baseten doesn't charge by pennies.

Philip Kiely

Well, yeah, we charge. I'm assuming that we're talking about the public model APIs. If you are setting up a dedicated deployment, then, yeah, it's not pennies.

Shawn Wang

Yeah. I mean, one of the key differentiators when I was talking with Baseten initially was that people who want very, very high volume just need to rent by the box, because then it's up to you to figure out how to saturate the box.

Ali Raza

And more often than not, it's way cheaper if you're pushing millions of tokens per hour if you just pay per hour instead of pay per token.

Philip Kiely

Yeah, they do. I think that we've increasingly seen a lot of demand for the sort of pay-per-token APIs just because everyone wants to try open models, and then once they find a use case that's really sticky, they move over to dedicated.

Shawn Wang

Is there a best practice on when it's time to swap over?

Philip Kiely

A couple reasons. Yeah, reliability, that's a big one, right?

Ali Raza

If they have a very specific use case, they want you to train something specifically for them. Like, they want their own spec decode, for instance, for their own traffic.

Shawn Wang

Spec decode is speculative decoding.

Ali Raza

Speculative decoding. Yeah.

Shawn Wang

You have to explain.

Ali Raza

Sorry. Basically, if you have a huge model, the model is going to generate 1 token at a time, every single turn, every single forward pass. So we attach this little parasite, this layer that goes on top of the model, and this model just has to predict. It does 3 very fast autoregressive forward passes, and it predicts 3 tokens. Then you do 1 forward pass over the entire original model in order to see whether those predictions were correct or not, and then you accept or reject them.

Now, this draft model is traffic-specific, so, like Philip said, if you're summarizing Harry Potter books, I can train that draft model exclusively on Harry Potter books, and I can guarantee you that I'm going to accept the 3 tokens every single time. In that case, I increase your decode speed. I wouldn't be able to provide this to you if you're on a shared endpoint, because I have no idea if you're doing Harry Potter, coding, or English. We don't know.

Also, there was a thing in the book that mentioned that if they really cared about a specific threshold—Chapter 4, I think. Do you remember that?

Philip Kiely

Yeah. The things that you can do are set a specific batch size, a specific parallelism strategy if you're trying to optimize for throughput versus latency. You can maybe—maybe an NVFP4 quant doesn't pass your benchmarks and you want to run a model at higher precision; you could do that. There's just a bunch of reasons why you might want to have your own endpoint, and the biggest one, of course, is that you don't have to deal with someone else doing 100 million tokens of benchmarking traffic at the endpoint when you happen to be trying to serve your users.

Shawn Wang

Yeah. I think one thing that is a classic journey is basically people asking, “What happens when you type Google into the browser?” Tool calling: is that just you're generating JSON, or is there more complication beyond that?

Ali Raza

Certain customers that we have have their own post-trained models, and so they demand tool calling that's not just, like, parsing a file or going to find the weather. It's something that's very specific, and you have to do post-training on this. If the post-training on the model is not good, or if the quantization after the post-training to get the inference to be fast is not good, the model will struggle reading the JSON file and reading the tool calling.

But it doesn't require its own sandbox. It's not like it's going to use that tool calling to escape a sandbox, and it doesn't have to be contained. It can just be a normal dedicated deployment. The challenge with tool calling, more and more, seems to be that the companies want certain tool calling, which is a very sensitive thing to train. Because you're dealing with all of the JSON outputs, if it doesn't close the end of the request in a very certain manner, you end up with a model that did the tool calling and the thinking. As a result, it didn't see the result and just hallucinated the result as it decoded. That seems to be the most challenging thing with tool calling, not really the sandbox model.

Philip Kiely

Yeah. That's a challenge on the training side, and then on the inference side, there's work that you can do to scope the possible output. So we published this, actually, at this point close to 2 years ago. The solution to this problem is that you basically make a state machine and use that to constrain the output to a specific format. So this is the structured-output problem, if you remember back in—

Shawn Wang

Yeah.

Philip Kiely

...back in the day—

Shawn Wang

The specific grammar is, uh—

Philip Kiely

Yeah, exactly.

Shawn Wang

...GML had this thing.

Philip Kiely

Yeah. So it's like the old-school “make sure this is only JSON, return only JSON, or my grandma's gonna die” type of prompts.

Shawn Wang

Yeah.

Philip Kiely

In our inference system, it's just a specified output format.

Philip Kiely

And you get the guarantee that your output is going to be structured along that format. Applying that to tool calls can help cut down on… Obviously, you can still call the wrong tool or call no tool. It doesn’t solve the certainty problem—

Shawn Wang

Mm-hmm.

Ali Raza

—but it at least solves the output-structuring problem—

Shawn Wang

Yeah.

Ali Raza

—within tool calls.

Shawn Wang

And MCP is just another form of tool, right? I guess.

Ali Raza

Yeah. Exactly.

Shawn Wang

As far as I know, there’s no special thing there.

Ali Raza

The thing I’m always explaining to people is that the LLM is actually not capable of doing anything. It’s only capable of making suggestions of what to do, and then, if those suggestions are formatted in a certain way and applied to a system that knows what to do with them, an action occurs.

Shawn Wang

Yeah. Part of the fun stuff is that this is solved outside of tool calling, too. In an agent loop, if the output is not correct, or if reasoning or tool calling was done in the reasoning cycle, you can just be like, “Oh, I don’t know what to do. Let me just try again.” It might get there after a few tries.

Ali Raza

Right.

Shawn Wang

And on your point of training, sometimes this is harder in smaller models, so you don’t have the same exact quality of output when you just swap from a big model, right?

Ali Raza

Yeah. I will say that, before we— I think we need to go back to inference engineering proper. I had expected that something would replace JSON because it’s hard to stream JSON: JSON must be complete, and you must have open and closed brackets and everything. So it’s hard to parse or validate something while it’s being streamed.

People invented all sorts of alternatives. I forget the name of some of them, but they’re basically something like TOML or YAML. But JSON seems to be dominant still.

Ali Raza

The JSON outputs aren’t that long, right? I guess you could have a long one, because tool calls also contain the arguments in them, and perhaps for a certain tool you might pass a very long argument. But my impression of the median tool call is that it’s a relatively small number of tokens, right?

So I would expect that speculators are generally fairly good at something as formatted as JSON. You would have a pretty fast decode step there, and the streaming wouldn’t be as valuable, but maybe I’m wrong about that.

Alessio Fanelli

I think you’re also bounded by the software that the model is going to integrate with—if the software is built with JSON for the tool calls, or if your customer says that this is how our software works and our tools are interfaced with JSON. You can ask them to change their software and say, “Yeah, this is going to be better for the model.” But with the training, it shouldn’t be that much of a difference. Also, it’s probably more profitable if it outputs more tokens.

Shawn Wang

Depends on your business model. It really depends. But I will say that, as a writer with a lot of experience with AI-generated output, I do try to move from text to JSON text—

Ali Raza

Mm.

Shawn Wang

—which is very long JSON, right? There are paragraphs in every field because I’m trying to structure it, right?

Ali Raza

Right.

Shawn Wang

I want you to first make factual statements, then make opinions—

Ali Raza

Mm-hmm.

Shawn Wang

—then make bullet-point summaries, have dates, have entity references—

Ali Raza

Mm.

Shawn Wang

—have your sources for references, all these things. Anyway, these are things that I think people who really, really experiment with structured output have to care about.

Let’s recurse up the stack a little bit. Before we started recording, you mentioned something that’s really cool: there’s a lot of inference engineering that goes on when a new model provider releases a new model, right? So let’s call it GLM-5.2, Kimi K3. I had previously assumed, especially if it’s GLM-5 to GLM-5.1 to GLM-5.2, that you’ve supported them before. Is it that much work?

Alessio Fanelli

It’s a lot of work.

Shawn Wang

Yeah. Okay. A lot of people—all you guys, right?—whenever a new model launches, rush to say, “Hugging Face supports this, Fireworks supports this, SpaceTime supports this.” And I’m like, “Yeah, of course we support it.” But what goes—

Ali Raza

I think—

Shawn Wang

—into—

Ali Raza

I think it’s more than just supporting it, too, right? It benefits the consumer a lot. I think it was with Kimi K2.5 or GLM-5.2, the latest, that there was sort of an inference war, right? One provider is at 90 tokens a second; the next day, we’re at 150. The next—

I kind of kicked that off with GLM-5.2. I wrote a Twitter article about it. It got, like, half a million views—

Alessio Fanelli

Best ending, number one.

Ali Raza

Yeah.

Alessio Fanelli

Or it’s for showing up.

Ali Raza

Yeah. Which—

Alessio Fanelli

Oh my God.

Ali Raza

—which then got everyone really excited about, “Hey, how can we bend the tracks a little bit further and—”

There’s a difference between supporting the model—as in, I can make a token out of this model—and supporting a model—as in, I have a production-ready API from this model.

Alessio Fanelli

Mm.

Ali Raza

Getting to the point where I can make a token out of this model is not that hard because, generally, the open-source inference engines—vLLM and SGLang—often receive weights ahead of time. Maintainers do that, or the people making the model merge pull requests to ensure support. So you can generally just get it working on the standard open-source stack without too much pain in most cases.

The challenge is that every inference company is going to have its own proprietary stack: some open-source components, some in-house stuff. For any arbitrary model, there’s going to be some new stuff. Sometimes you get lucky, like Kimi K2.5 to Kimi K2.6 was pretty similar.

Alessio Fanelli

Yeah, it was pure continued post-training, if I remember correctly.

Ali Raza

Even in those cases, there’s still stuff you have to do. You have to redo the quantization work. Generally, these models are not released in NVFP4, and we want them to be in NVFP4 for maximum Blackwell compatibility. So we have to perform that quantization and calibrate it to make sure that we’re not causing any kind of regression in the model’s intelligence.

And then we also have to train the speculator, as we’ve talked about. Generally, we have—obviously, we have ZDR, zero data retention, on our model APIs, so we don’t know exactly the traffic that people are sending us. But we know what’s popular. We know that coding use cases are popular. We know that agents and agentic use cases are popular. So we can get public datasets that are representative of that kind of traffic and train general speculators.

Now, with speculators today, you need to train the speculator using the base model itself because you’re getting hidden states out of the model from running inference on these specific prompts, and that is the training data you use to create the speculator. So there’s that process, for which you need the full model weights.

And then there’s, of course, the process of standing up all the infrastructure behind it, loading all this stuff in, and testing it. When there’s a new model with a newer architecture, the DeepSeek models tend to be the most challenging, as they have the most novel architectural stuff going on. But every new model has something.

I mean, Kimi K2 had—oh, sorry, GLM-5.2 had—

Alessio Fanelli

Sparse.

Ali Raza

Yeah, the—

Alessio Fanelli

Yeah.

Ali Raza

—the DSA.

Alessio Fanelli

Right, which is borrowed from DeepSeek.

Ali Raza

Yeah.

Alessio Fanelli

So you can copy-paste them, you know? I don’t know how this works.

Ali Raza

Mm.

Alessio Fanelli

Mm.

Ali Raza

You know, so, like—

Alessio Fanelli

Mm.

Ali Raza

—we had to build support for that into our runtime. And you’re right, it actually is really interesting the way that all of these open-source labs borrow from each other.

For example, GLM-5.2 doesn’t have vision. Something that Hailey, a guy on our team—if we could take a look at this—he grafted the Kimi vision encoder onto GLM-5.2.

Alessio Fanelli

Only training the projector.

Ali Raza

Exactly. So if you think about the encoder, there’s the encoder, which is the part that looks at the image and turns it into latent information—

Alessio Fanelli

You can say latent space. It’s okay.

Ali Raza

—and then there’s the projector that maps it onto the model itself. Then there are the model weights. You don’t want to mess with the model weights because you run the chance of making the model dumber at something else for the purpose of giving it vision.

Instead, Hailey started with just a projector, which is only a handful of millions of parameters.

Alessio Fanelli

That would be, yeah.

Ali Raza

Yeah. And—

Alessio Fanelli

Can you show the training one? Like the way it groks is—

Ali Raza

Very, very interesting.

Alessio Fanelli

And maybe—

Ali Raza

That right there—

Alessio Fanelli

Maybe, Ali, you should take it from here. You’ve got a better—

Ali Raza

Like—

Alessio Fanelli

—understanding of this than I do.

Ali Raza

Yeah, you can see the way he trained this is really, really cool. At the beginning, he was training it using, “Here’s a picture of a mountain. Can you describe what’s in this mountain?” That caused it to hit the first learning wall. Here you can see that all we’re trying to teach it is to translate the encoded image. It’s already taken the encoder from Kimi-VL. It’s taken the image. It’s—

Alessio Fanelli

Yeah. Frozen, frozen—

Ali Raza

Frozen, frozen—

Alessio Fanelli

—with adapter.

Ali Raza

Exactly.

Alessio Fanelli

Yeah.

Ali Raza

So the understanding brain is frozen and the eyes are frozen. We’re just trying to—

Alessio Fanelli

Align.

Ali Raza

—interconnect the eye and the brain, right? That’s the projector. You take the tokens, and then it’s like, “Oh, can you describe what’s in this image?” It might say, “It’s a mountain,” or it’s a person or it’s a human, whatever the case is. But that didn’t cause complete understanding.

So he changed it such that every image was associated with a data set of questions: Does this image have a white male? Does this image have birds in the top corner? Does this image have a scientist in it? It would have to answer the questions correctly. Using not just training on describing an image, but being able to answer question after question over time, you can see the grokking, which is genuinely insane—that retrofitting vision into a large LLM can learn to that extent.

Even for images that it doesn’t perform well on—for instance, if you show it a picture of Stephen Hawking and ask, “Who is this?”—maybe it doesn’t get it, but it will say something like, “This is Albert Einstein.” It still understands that this is a scientist who is a man who has done significant achievements and all that stuff. So that’s really, really cool.

Alessio Fanelli

Yeah. We’ve covered Hao Tian before, who was the author of the Lava paper that did this a while ago, and I think that’s very foundational work for anyone who hasn’t done vision work before.

Ali Raza

Same with CLIP and MetaCLIP, where you go from just captioning to building out questions off the image, and you can get much better performance.

Alessio Fanelli

Right. Right, right. Yeah, but what’s so exciting about this is, if you look at a model like this—obviously, this is a little bit more of a research project. It got to 56% on MMLU-Pro, I think, so not quite Fontio. But if you’re running this model, you haven’t suffered any loss in your GLM-5.2 quality. If you don’t have an image, it’ll just behave exactly the way it used to. Ultimately—

The Inference Engineering Masterclass

Which, in the inference code, you literally don’t include the other part, right?

Alessio Fanelli

Yeah. You would just skip the encoder if you don’t have an image input.

The Inference Engineering Masterclass

Okay.

Alessio Fanelli

Just confirming.

The Inference Engineering Masterclass

Does it affect a lot on the overall inference side? You’re not adding much. You’re adding a very small vision encoder. These are typically—

Alessio Fanelli

They’re super tiny.

The Inference Engineering Masterclass

—less than 1 billion parameters, right?

Alessio Fanelli

Yeah. There’s a little less standardization among vision encoders, so the support matrix can be a little sparser. But overall, yeah, it’s a pretty minor component of the overall system.

Ultimately, what you get out of the system is that all of a sudden you have Kimi-VL, GLM weights, and DeepSeek attention all in one model. That’s, I think—

swyx

Yeah.

Alessio Fanelli

—a lot of the power and beauty of open source: you can take all of these different components and combine them together into a system that’s better than any one of them—

swyx

Yeah.

Alessio Fanelli

—can be individually.

swyx

People used to say that you would also do Franken-merges, where you would take layers from each model. No—does anyone do that anymore?

The Inference Engineering Masterclass

Well, to your point previously, when you were mentioning the work that goes into supporting a model when it first comes out, like GLM-5.2 or MiniMax-M3 or whatever the case is, sometimes you do have to switch some things out.

For instance, the MiniMax-M3 head uses full attention, and with full attention you end up with this insane bottleneck in speculative decoding because you’re doing autoregressive token generation for 3 tokens, and you’re doing this O-squared computation over all of the tokens that are in your sequence. Your KV cache is very large because it’s not sparse; it’s not top-k.

So we find it better to say, “Okay, we’re going to replace this layer with a layer from another model that’s using GQA, for instance.” With the right training, you can get it to have the same acceptance rate. It’s very possible to retrofit layers from other models, and very much needed, actually.

If a layer is inefficient, the training just becomes the challenge. How do you ensure that you train it properly? Which, again, to your earlier point, is the mesh between training and inference. You need very good training in order to do fast inference. That’s becoming more and more true.

Alessio Fanelli

Yeah. Anything else on the support side when you say, “Get it fully production-ready”?

The Inference Engineering Masterclass

Yeah. I think there’s also a question of just—we can test a model to a pretty extensive degree, but we’re trying to get it out quickly. Then you see a bunch of other people test it, and you get interesting results.

There was an issue with GLM briefly where we had some model collapses where it would just output the same token over and over again for certain prompts at certain temperatures. Once you expose an endpoint to the real world, there are so many more varieties of things given to it that you’re able to discover and patch.

So it’s not just a day-zero process. It’s for the first week, for the first month, if a model remains popular. How do you both fix bugs and continue to push the envelope on performance?

What do you mean you don’t want your model outputting SSSSSS?

swyx

Is there loop detection on that stuff, by the way? It still happens quite a lot, which is surprising.

In our endpoint, if a model outputs the same exact token 4 or more times, we just cut the generation. We say, “Oh, sorry, this—try again,” or we reprocess the request. We know that if it’s 4 times the same token, it’s probably collapsed.

swyx

Ah. Is there a way to opt out in case I really actually want that?

You actually want that? I think there’s a way that we handle it. I’m not exactly certain, but I feel like in certain models, when they output something—you can imagine a table, for instance, and they want to draw 12 dashes and 12 dashes—I think there’s a way for that to happen.

I think we only do it on certain tokens. We exclude certain special characters, so we only do it on certain ones. S is the most common, almost. GLM-5.2—

swyx

Oh.

The Inference Engineering Masterclass

—and I think it was DSV-4 as well. You’d just have looping issues where it would literally—

swyx

It was—

—just have SSSSS.

swyx

Yeah. Is there something special about S? No, it’s just random?

It just seems to be the one token that folds.

swyx

Yeah. And it’s—

Alessio Fanelli

Is there—

swyx

—and it’s only temperature 0, or—

The Inference Engineering Masterclass

No.

swyx

—even at other temperatures?

The Inference Engineering Masterclass

Even at 0.9 or whatever, it will still collapse.

swyx

That’s weird, right?

It’s an inference problem, to be honest—a software problem. Oftentimes, NVIDIA will release an image, for instance, and if we upstream the changes from their latest TensorRT-LLM image into our stack, we’ll find that it fixes it.

Oftentimes, this will only happen in an inference engine that you’re using, like SGLang. But if you switch to vLLM, that isn’t the case. So it seems to be an extremely nondeterministic software issue and not really a model issue. It’s not a weights problem.

We’ll say, “Oh, it’s a problem with the quant. We did PTQ wrong,” right? But that doesn’t make sense, because the same exact weights used with a different inference engine don’t repeat the problem. Sometimes the kernels being used in the backend have very subtle race conditions. If you host this model on one cluster, you will never get this problem—

Alessio Fanelli

Oh my God.

The Inference Engineering Masterclass

—but if you host it on a different cluster, you will. The reason is that the node-to-node KV-cache transfer in one cluster is using a slower interconnect than the node-to-node transfer in another cluster.

That exposes the race, whereas in another cluster it doesn't. So then you end up saying, “Okay, this model is not going to be hosted on this cluster. We're going to host it on another cluster because that cluster exposed that problem.” But then it ends up being: Is it the software? Is it the model weights, or is it the hardware?

There is a thing about this: temperature zero still isn't deterministic, right? Mostly because of hardware. Even at temperature zero—

Right.

The Inference Engineering Masterclass

With the same model, you won't always get the same output.

Mm-hmm.

swyx

But I'm surprised by the race-condition one because I thought PyTorch was a graph that guarantees that you at least execute things in the right order.

The Inference Engineering Masterclass

Well, yeah, true. I guess I'm not saying that this is a risk. You have things like pipeline optimizations where you can start a kernel before the end of the previous kernel, and that's because you want to do that because—

swyx

It's like pipelining.

There's no expense. Exactly.

swyx

Yeah.

The Inference Engineering Masterclass

Exactly. But you don't do it cleanly. You overlap a little bit of the execution. No, I guess it is very possible that the kernel itself—that one block that is supposed to be running in this instance of time—that kernel itself has a race condition. For instance, a missing barrier.

Often, if you're designing a kernel and you want it to be very fast, if you don't test it extensively, you'll have certain threads access data points from registers before they've been written to by other threads, for example, because your barrier is wrong or your synchronization was wrong. But, yeah, the testing itself is very, very difficult in those cases. And the—

swyx

Yeah.

The Inference Engineering Masterclass

What does that mean?

swyx

Rust. If you're trying to have memory safety, it—

Mm.

swyx

It sounds like a comparable problem.

The Inference Engineering Masterclass

Well, yes. But you're working in CUDA, right? An NVIDIA GPU. It's like—

You just need a higher-level language like Mojo. Maybe that's what Modular is supposed to do. I don't know.

Alessio Fanelli

How do you see keeping the quality of the model? You talked about all these steps: You have to do quantization, train your own speculative decoder, and run on different hardware. Looking at other model providers, you kicked off an inference-speed race on the consumer end. What goes into keeping the quality the same across them? Sure, you can run benchmarks, but how do you determine how much quantization to use? Are there standards? What actually goes into—

Philip Kiely

There are a few things on quality. Most inference optimizations are lossless. KV caching, for example: you are just recomputing or preventing the recomputation of the same values. Speculation, of course, means that if a draft token is wrong, it gets rejected.

The main lossy optimization is quantization. That really comes down to, number 1, data format; number 2, which parts of the model you choose to quantize, which layers; and number 3, doing a lot of calibration on the quantized weights to ensure that you're preserving all the outliers.

There are other tricks that you can do, though. A big one is long context, because one thing you asked right at the beginning is, “Oh, what's going to happen if I send a 200,000-token request in?” Obviously, with a long input sequence, you need to store a lot more information and process a lot more tokens. So even if a model has a context of a certain length, you might, as an inference provider, choose to build an API with a shorter context length and, of course, a full-length one as well.

Because if someone doesn't need the full million-token context, for example, you can get them better performance. I don't know if that's exactly the quality of the model. The way that I think about quality is: To what degree are we faithfully serving the original model? If you think of a sort of golden implementation of a model that performs exactly the way the model is designed to perform, I think of quality as how close we're getting to that 100% fidelity of the model.

Alessio Fanelli

Mm.

The Inference Engineering Masterclass

You can also, of course, think about quality from the training side and how you push yourself past 100%. But when I think about purely inference optimizations, it's getting faster while staying as close to that 100% fidelity mark as possible.

Certainly, our standard internally is that you should not be able to tell the difference between our API and a sort of official API. I think Kimi in particular does a good job of vendor benchmarking here—

Yes.

The Inference Engineering Masterclass

Where they have—

They released an actual vendor benchmark.

Exactly. Yeah.

The Inference Engineering Masterclass

Yeah. Well, because you'd be pissed, right? If I'm a consumer and I'm using Amazon's endpoint, for instance, and I've used Kimi, and I'm like, “Oh my God, this is bad,” I'm not going to say, “Oh, Amazon quantized the model in a bad way.” I'm going to say, “Oh, Kimi sucks.” Right? So it seems like that makes sense.

Yeah, they care. They care.

Alessio Fanelli

Justifiably.

The Inference Engineering Masterclass

Yeah, justifiably.

Alessio Fanelli

This is probably a stupid question, but just checking: Has anything improved from mixed quantization? Is quantization always strictly worse?

The Inference Engineering Masterclass

No—well, technically, no. Quantization is lossy.

Yeah.

The Inference Engineering Masterclass

It improves speed. Obviously, it reduces the number of bits, like—

Alessio Fanelli

No, I'm always looking for inverse scaling laws. This is something I learned from Noam Brown: Things that normally act in one direction sometimes do the opposite.

The Inference Engineering Masterclass

Well, technically, when you run a benchmark, because these models are nondeterministic, sometimes your MXFP4 quant is 2 basis points higher than your—

Alessio Fanelli

Yeah, yeah.

The Inference Engineering Masterclass

That's noise. That's noise.

Yeah, exactly. I'm like, yeah, it's within the margin of error. That's why I always say “within the margin of error.” I actually stopped saying that because everyone assumes that what I mean is, well, within some margin of error, we're barely inside of that, as opposed to the worst. But, yeah, sometimes it just gives you a higher output score. But, like Ali said, that's noise.

To my knowledge, you're not necessarily making the results better. You're just trying to keep your fidelity as close to 100% to the original model.

Alessio Fanelli

Mm-hmm.

Ali Raza

There is, to your point, research that we did on MXFP4. I don't know if you're able to pull up a tweet we did. One of our research interns, Joshua, had a tweet on how we have a 20% better quantized JLN52 than NVIDIA.

Essentially, what we found throughout this 2-month research is that quantization is lossy. You're compressing the data from occupying 16 bits to occupying 4 bits, for instance. So you're obviously losing some information, and you're trying to minimize that.

When I say that I'm going to quantize the model, my job becomes: How do I find the layers that I can quantize and the layers not to quantize? For instance, with image models, I don't quantize modulation layers, and I don't quantize output projections, because those two are—

The output projection is what you see as the user; modulation is what the model sees or understands.

Right, exactly. So, I guess, to his paper, do you have the—

Yeah.

The Inference Engineering Masterclass

I guess it doesn't have the—

Alessio Fanelli

If there's a part to search, it's probably in the thread.

The Inference Engineering Masterclass

It's probably in the thread.

Alessio Fanelli

Yeah.

The Inference Engineering Masterclass

But basically, the long and the short of it is that it is very possible that quantizing more of the model makes the results better. If I have a model where I quantize layers 1, 5, and 10, and another model where I only quantize layers 1 and 2, it is possible that the model in which I quantized more information is going to perform better because the quantization errors have canceled out.

What Joshua showed in his mathematical proof, where he had a verifier in it, is that you can predict which layers are going to have quantization errors that will cancel out with each other, and you choose to quantize those layers. And so the result of doing this mathematical quantization is you end up with a model that's 20% more quantized than another provider, so you get 20% more throughput because more layers are running in MXFP4, and your quality is better than that other quantized model because the layers that you chose to quantize have their errors cancel out—like one layer is skewed to the right—

One layer skewed to the left, one layer skewed to the right. Your final logit distribution is more similar to the original distribution of the model, so you have better fidelity. The way we proved this was with KL divergence. Instead of just scoring on the benchmarks, we scored the KL divergence between the logit distribution of the quantized model and the logit distribution of the original full-precision model, and showed that with this technique, if your probability distribution on the logit of the token it wants to select is more similar to the original model, you're probably going to end up staying true to the original model.

Previously, it seemed like the industry thought, “The more you quantize, the worse it's going to be, because the more loss you introduce.” That's not necessarily true. It doesn't improve it, but it can cancel out.

I think it might be this, but it reminds me a good bit of pruning, actually, where you can prune off certain layers.

Mm-hmm.

Ali Raza

It's a fun fact: This paper was originally 72 pages, and then we decided we couldn't release it.

Wow.

The Inference Engineering Masterclass

So it's now 45 pages.

swyx

Still 39 pages, so very, very substantive. We talked about evals and all these things, and what's possible in terms of speedup? I guess that's probably the number-one thing that people care about, and it's something that you wrote about in your post. The official API is 70 tokens per second, and you push it up to 90. Is that a normal thing?

Mm-hmm.

swyx

Which, by the way, because I am from a finance background, in the '70s, that was the margin at the time. When you did quantitative finance research, you would find—

The Inference Engineering Masterclass

And, like, with 20%—tens of percent.

swyx

Yes. Yes.

The Inference Engineering Masterclass

Yeah.

swyx

Yes.

The Inference Engineering Masterclass

And now it's tiny fractions.

swyx

For those people interested, look up Andrew Lo's paper. He had a really interesting illustration of quantitative statistical arbitrage distributions narrowing down from those kinds of 20% differences in the '70s down to nothing today, which is very, very cool.

Exactly. And we're at the beginning of the same type of thing. Benchmarking is hard. I think anyone will tell you that. Benchmarking provider speeds is hard because there are so many variables that go into it: What hardware are you using? How much load do you have on the system? What's the exact nature of the prompts and input and output sequence lengths? All that kind of stuff.

The most common form, of course, is TPS—tokens per second—which is bad naming by us in the industry because there are actually 2 tokens-per-second metrics: the throughput number and the latency number.

swyx

TTFT, yeah.

Total tokens per second out of the GPU is the throughput number. Most people only care about tokens per second as the latency number, which we should call ITL, intertoken latency, but we don't. Anyway, you can imagine a sort of standard API without many optimizations for a 1-trillion-parameter model operating somewhere in the 30-to-50-tokens-per-second range for a reasonable traffic profile.

We generally see the goal as pushing to 10× that. If you have, say, 3 optimizations, each of which doubles performance, then you stack that up—that's an 8× gain. That's kind of the order of magnitude we're working with in this space. We're trying to make things substantially faster, not just go from 70 to 90.

swyx

Are you saying you have done that?

Let's say you have, as a reasonable baseline, 30 or 40 tokens per second. You can achieve 10× that. On GLM 5.2, if you run it unquantized, perhaps on Halpos even, and you're just using an off-the-shelf inference engine with no particular optimizations—no speculator, nothing extra around KV routing, no disaggregation—you're probably looking at 30 to 40. You think that's a reasonable baseline?

swyx

Right. Right.

Philip Kiely

To get to something like 10×, there are a lot of trade-offs that you're making. If we're running in more like the 300-to-400-tokens-per-second range, obviously you're using the best hardware possible. You have an optimized speculator. You have done all of your quantization work. You're seeing a pretty high cache hit rate. You're running with a reasonably small batch size and a parallelism configuration that's tuned for latency versus throughput, but it is possible.

The spreads that you see if you go on Artificial Analysis or OpenRouter and look at the worst provider versus the best provider can often hit that kind of range. 10× is, of course, very aggressive. It's oftentimes more of a 4× to 6× improvement. But that's the kind of performance that makes us really excited: when we can get these huge gains, not just go from 70 to 90 tokens.

Ali Raza

It's also hardware-dependent. If you're serving it on just a node of H100s and then you shard the model across 4 nodes of B200s, you can definitely increase the speed by throwing more hardware at it. If you're normalizing for the exact same hardware and the same number of GPUs—

Philip Kiely

Then you're looking at a 2× to 4× improvement, depending on the inference optimizations. Some of it is what the call is, and some of it is who's the driver.

Alessio Fanelli

If you break down the 2× to 4×—say the example is running GLM 5.2 on B200s, single node—what's the cost trade-off for the effort to get the last bit of juice out versus what should people just think of?

The Inference Engineering Masterclass

Yeah.

Alessio Fanelli

On B200s—

The Inference Engineering Masterclass

Yeah.

Alessio Fanelli

Single node, right? What's the cost trade-off for the effort to get the last bit of juice out versus what should people just think of?

The Inference Engineering Masterclass

Spectre quantization. Yeah.

Alessio Fanelli

Spectre quantization.

That's, like, 95%.

And how far does that get you? How easy is that for the average person to do? Say right now I want to throw the weights of GLM-5.2 on a node of B200s. How easy is it to find a speculative decoder model or an already quantized model? How much work goes into it?

Guest

Mm-hmm.

Guest 2

If you're doing it up front, it's quite a lot of work. If you're doing it today, there are going to be people who have published things that you can just grab. You can grab some NVFP4 weights. You can grab a speculator.

If we're thinking about the 2× gains we're stacking, going from BF16 to NVFP4 isn't quite 2×. I think it's about 30% to 40% from BF16 to FP8, and then another 30% to 40% multiplicatively from FP8 to FP4. So that doesn't quite get you 2×, but roughly 2×. Speculator is roughly 2×. Disaggregation on top of that, if you're able to get enough hardware and put enough traffic through it, is another roughly 2×.

Then you add in some double-digit percentage increase from having a better runtime, with the latest kernels and so on behind it. That's kind of how it stacks up.

Guest

Yeah.

Guest 2

Building each of those—the quantized weights, for someone who really knows what they're doing, is hours to days of work. Building the speculator is, again, hours to days of work. And the disaggregation setup is hours to days.

Guest

Yeah.

Guest 2

Okay, but once you have it—

Guest

Once it's set up.

Guest 2

Yeah, getting disaggregation working for the first time, of course, is very difficult. But the marginal implementation is—

If you're just grabbing it—if you're a normal consumer who has access to a node of B200s and you're wondering, “How can I just host it myself?”—you don't need to quantize the model yourself. There's always going to be an open-source quantized checkpoint. NVIDIA's going to push one out if no one else does.

Usually, providers will have their own speculator that they've trained as well. You don't need to train your own speculator. You can just use that as well.

Guest 2

Yeah. Like Chemi, uh, GLM-52 has its own MTP.

Alessio Fanelli

Right.

swyx

What’s multi-token prediction?

Guest

Yes.

Alessio Fanelli

I’m just an expert. I can do it for you in case I get it wrong.

swyx

No. Yeah, actually, you should correct us if we’re wrong, but their multi-token prediction can be used for self-speculative decoding.

I’m actually not sure. I’m not going to correct you.

swyx

Okay. I’m semi-confident in that, but someone can check. But it’s useful to paint the story of, okay, not just the average person, but say a company wants to switch from serverless inference to—

Mm-hmm.

swyx

—I want to throw this up, you know. I want to rent some GPUs and throw it up. These are the steps you take to make it significantly faster than just putting it behind vLLM.

Alessio Fanelli

Right.

swyx

I was waiting for a mention of Dynamo. I feel like that’s supposed to be the baseline that you measure against.

Philip Kiely

I would think of Dynamo as less of an out-of-the-box system and more of a toolkit for building with. So when we talk about doing KV-aware routing, when we talk about doing KV-cache offloading, when we talk about doing PD disaggregation, Dynamo fundamentally is—

By the way, Dynamo is an open-source library from NVIDIA.

Alessio Fanelli

We’ve done a podcast with Kyle Kranen.

Guest

Okay.

Alessio Fanelli

Kyle Kranen.

Guest

Cool. So your listeners know that it supports all the different inference frameworks. It’s actually kind of multihardware, which is interesting.

Alessio Fanelli

But it’s just a router. It’s not an optimizer layer.

Guest

Yeah. What Dynamo is good at is being a library for moving information around your cluster and around your hardware. So if you have KV cache in one place and you need it to be somewhere else, Dynamo coordinates Nixal for you to move that around. That doesn’t mean that, out of the box, you just say, “pip install Dynamo,” and then you get a massive performance speedup. It’s more of a developer toolkit.

Alessio Fanelli

Yeah. I would have said it comes with a set of defaults that you can then swap out.

Guest

It does. If the industry at large were rolling out all of these deployments as the standard, then I think it would be a credible baseline. But we’ve got to benchmark against what we’re seeing in the wild.

swyx

I did want to talk a little bit more about PD disaggregation, because that is probably number 3 after quantization and speculative decoding. In your book, though—I was just going to pull out the book—section 5.2.2 is on Medusa, 5.2.3 is on EAGLE, and 5.2.4 is on n-gram.

It’s 5.5 that would be disaggregation.

swyx

Yeah. Well, no, I just wanted to dwell a little bit on the other techniques. What do you choose to include? What do you choose not to include? Because there were all these other—

Techniques, I guess.

Alessio Fanelli

Yeah.

Guest

Are these still relevant? Because I think they came out a year and a half ago, maybe.

Alessio Fanelli

Medusa is quite old.

Guest

Yeah, Medusa’s old.

Alessio Fanelli

But is it in the book as a good—

Guest

Baseline.

Alessio Fanelli

—but is it in the book as a good baseline, as in, “Here’s the baseline vanilla. Understand it”?

Guest

Like, you should know this.

Alessio Fanelli

I read the paper, and I’m like, “Ah, it makes so much sense.”

Guest

With the book, I had a couple of goals. One was to give people a working vocabulary for the space as a whole, and the other was to give them some intuition about how each of these techniques works. As I mentioned in my AI Engineer talk, which is kind of the first public addendum to this, the speculation space has moved much faster than everything else.

Even at the time that I wrote the book, I included Medusa very much as a way for people to understand how the space evolved, rather than as the most modern technique. Now, of course, there’s DFlash and DeSpark. There are newer techniques even than EAGLE, although EAGLE is still very commonly used.

Alessio Fanelli

Speculative speculative decoding.

Guest

Yes, speculative speculative decoding.

Alessio Fanelli

What?

Guest

Oh.

Alessio Fanelli

It’s a paper by Trudeau, and it’s basically doing speculative decoding—

Guest

Uh-huh.

Alessio Fanelli

—for the speculative decoder.

Guest

Oh, in speculative decoding? Oh my God.

Alessio Fanelli

It’s literally just another one. It’s the simplest way to explain it. It seems like he got nontrivial speedups there, but the complexity with training is almost as complex as training GANs, at least in our minds. It’s a delicate balance, and oftentimes it’s just additional. But yeah, it’s literally speculative decoding on speculative—

Guest

Speculative, speculative.

Alessio Fanelli

Yeah. We saw this paper.

Guest

It’s interesting, right?

Alessio Fanelli

Yeah.

Guest

I wouldn’t even expect it to be particularly hard to train. I don’t—

Alessio Fanelli

Right, right, right.

Guest

The naive part of me is like, “Okay, train a speculative decoder.”

Alessio Fanelli

But it makes sense. The whole idea of speculative decoding is almost like the iPhone auto-predict version, but for a normal model. You’re generating 3 tokens and saying, “Okay, I’ll do prefill on them.” So you save those 3 turns for your original model. Now your speculative decoder is doing 3 turns of autoregression.

Guest

I guess the other question there is: What are the sizes of speculators? Say, for GLM—

Alessio Fanelli

Right. It’s like 1 billion parameters.

Guest

For MiniMax, it’s like 1 layer. It’s like 1/60th of the original model, usually.

Alessio Fanelli

Yeah.

Guest

Actually, I think we should do a paper when we get back to the office.

Speculative speculative speculative decoding.

Alessio Fanelli

No, it does seem like—how do you know when to stop? But then it also seems kind of like, if you’re able to train speculative speculative decoding, for instance, if you’re able to have a small model that accurately predicts what the intermediate speculator is going to predict, and that is able to predict what the original target model is going to predict, then why not just use that smallest model directly?

Guest

Yeah. This is adjacent to the routing problem.

Alessio Fanelli

Right.

Guest

Yeah.

Alessio Fanelli

Right.

Guest

One of the practical constraints on using speculators is that you do have to run a small model on the same hardware that you’re running the big model on. There’s an orchestration and resource-competition problem inherent in that. That is one of the constraints on speculation in general: draft tokens cost resources to create and software complexity to manage. If you have infinitely recursive speculators, you add quite a bit of that complexity to the actual implementation within the inference engine, not just to the training process.

Alessio Fanelli

I was going to say, I would wonder if you could do similar distillation and pruning. It’s the same thing; it’s just a model. Can we not just distill a lot of the weights and quantize the speculator, but—

Guest

Mm.

Alessio Fanelli

—but that’s out of my domain. I guess the question that also comes up is: This is all for big server workloads, right? How much of this applies to, say, I have this MacBook and I want to run Gemma really efficiently? Similar problems, not the same?

Guest

Pretty different. I talked to Salo about this on his podcast a couple of weeks ago. The difference between inference engineering for the data center and production workloads versus inference engineering for local AI is that we start with fundamentally different constraints and different goals.

With local AI, it’s, “How do I fit this model onto my hardware and then make it less dumb?” With data center inference, it’s, “How do I load this model and then make it less slow?” Obviously, we care about less dumb, and they care about less slow.

But the local AI inference engineering ecosystem, I think, actually has a lot for us to learn from in the data center space. They are experts in various forms of quantization, including dynamic quantization, which we just don’t really touch, as well as pruning, distillation, and layer removal.

Alessio Fanelli

Layer removal matters less.

Guest

Yeah. There’s—

Alessio Fanelli

No one does pruning, really.

Guest

Yeah. Well, but the—

Alessio Fanelli

Which is surprising, right? But that’s a whole different thing.

Guest

Just to fit something on the laptop.

Alessio Fanelli

Right, right, right.

Guest

So yeah, it’s an interesting space. Not necessarily that their techniques make sense for us to use in the data center, because obviously we have different resources and different goals, but more that the process, as well as the openness of that field, is something to admire.

Alessio Fanelli

Yeah, to your point, certain optimizations—like, for instance, TurboQuant. I’m sure you’ve heard of it. It made such huge hype, and we did a whole deep dive on Twitter analysis: What is it? How does it work? Why is it good or not?

It took off, and it was implemented on local devices because your memory bandwidth is so slow on a MacBook, for instance.

swyx

But try putting the same thing on an NVIDIA GPU, on a B200. TurboQuant would not be used. NVIDIA made it clear that this is not a good optimization, and we’ve seen it firsthand: the overhead of doing dequantization and quantization in the kernel itself—the TurboQuant kernel execution—is actually much, much slower than the time that you save from the bandwidth. On the B200s, you have 3.5 TB/s.

You don’t need to decrease the storage that much. You don’t need to use an FP4 KV cache. You don’t need to use a record. There are better optimizations to be made. But on edge devices, it’s extremely important and extremely useful.

So it seems to be different optimizations there, but then they’re all uniquely combined with, like, oh, you want to quantize the model, you want to do speculative decoding, like certain common prefixes with both—

Guest

Principles.

swyx

Yeah, exactly.

Alessio Fanelli

Yeah.

swyx

Exactly.

Guest

They also do a lot of work on model parallelism, especially over heterogeneous topologies, where you have some Sparks wired together with Ethernet—DGX Sparks.

swyx

Yeah, this is the Exo Labs guys.

Yeah. You have a number of Mac Minis stacked up. The interconnect between machines is one thing that I think we both have to deal with, although they have to deal with it a lot more. This is why one thing that we do a lot is work with tensor parallelism.

swyx

Mm-hmm.

Guest

That’s where you are using all 8 GPUs and sharding the model across them. Tensor parallelism is not a good fit for local AI because it assumes a very high-bandwidth interconnect like NVLink. You might be forced to do something like pipeline parallelism, which we’re never going to do unless we’re doing some kind of—

swyx

Yeah, for inference—

—multi-node inference.

swyx

But since you mentioned it, I actually wasn’t sure if we were going to cover it, but let’s briefly explain tensor parallelism and expert parallelism, since you have very nice images.

You want to pull the book?

swyx

I just want to—

Yeah.

swyx

I just want to show off your images.

Guest

Yeah. Shout-out to Luke from Baseten’s design team for making these beautiful images.

One other difference is that we talk a lot about the active parameters of a mixture-of-experts model. For local inference folks, that matters a lot because if you have a batch size of 1, you’re only activating that many parameters. When we do—

swyx

Yes, I was going to bring that into the diffusion conversation.

Yeah, yeah. When we go through an MoE model and host it for an API, we assume that all parameters are going to be active because—

swyx

You’re batching.

—throughout your batch—

swyx

Yeah.

Guest

—you’re going to hit everything.

Cool. So broadly, tensor parallelism you can do with any model. Expert parallelism you can only do with MoE models. Effectively, all models today are MoE models—

swyx

Sort of—

—at least all models large enough that you would care to parallelize them across multiple GPUs. So that nuance is less important now.

With expert parallelism, the idea is you put the entire expert on a GPU. Generally, you have more experts than GPUs, so you might put, like, 8 experts per GPU or whatever. Then you replicate the router, which is very small, across each of the GPUs. By moving the generation from expert to expert, with each expert being inside a GPU, they’re not competing for resources. You massively increase the throughput that you’re capable of doing.

The GPU-to-GPU connection is not as important because there’s not as much communication. Tensor parallelism requires that you’re able to do this all-gather, all-reduce. You basically shard the model across the GPUs entirely, and then for each step, you’re combining the results of each of the GPUs, which is why the interconnect matters a lot.

Of course, this is a very high-level generalization. There are a lot of places where this is not correct. But generally, TP is helpful for latency, and in many cases, you will use some combination of these 2 parallelisms across the model rather than just picking one or the other. Do you want to add some extra color there?

swyx

Yeah. Usually, in a model, they’re not mutually exclusive. You do tensor parallelism and you’ll do expert parallelism. Pipeline parallelism, less so. It seems to me like we never use pipeline parallelism.

Yeah. The only reason you would have to do pipeline parallelism, which is where you separate different layers and put half the layers on one piece of hardware and half on another, is if you are forced to do multi-node inference because a model is bigger than what you have.

Let’s say you’re doing a deployment on H100s for whatever reason and you’re putting a trillion-parameter model on there. You have to use multiple H100 nodes. Because the interconnect is so slow between the nodes, the only viable way to parallelize there is pipeline parallelism, but then you would do expert and tensor parallelism within each node.

swyx

And the limiting factor for H100s is HBM?

Yeah. They just don’t have enough of it.

swyx

How much—what are the magic numbers that we need to—

Alessio Fanelli

One node is 8.

swyx

—like, on a B200, it’s 180 GB per GPU, and then a node of 8, you’re talking about 180 times 8. With FP4, each parameter takes half a byte, so that’s 800 GB. On an H100, it’s like 140?

Guest

It’s 80.

swyx

It’s 80?

Yeah.

swyx

Oof.

Guest

Yeah.

Alessio Fanelli

It’s easy.

swyx

I’m old. I’ve been doing this a long time. I actually remember H100 specs.

Guest

Yeah. You want to tell me about the T4s?

Alessio Fanelli

The T4s.

Guest

Let me tell you what it was like to run a model on a T4 back in the day.

swyx

One thing I was surprised to see was that more people didn’t do Jamba. I don’t know if you guys remember Jamba from AI21. They would specifically pick a piece of hardware and then design the architecture dimensions for the hardware, and then it would obviously saturate the hardware. It makes sense. Somehow, all these models don’t do that.

Alessio Fanelli

Don’t they do this for the training side, though?

swyx

I don’t know. Do you—

Guest

The what, sorry? The—

Alessio Fanelli

Training. For training the model.

swyx

Like deciding which GPU? Which GPU?

Yeah.

Guest

Yeah.

Alessio Fanelli

Yeah, yeah. They do.

swyx

And with training, it’s more of a math problem. You can run the math and see the FLOPs and maximize them. With inference, it’s more of an autotuning problem. I don’t know if you’re familiar with GPU-kernel autotuning, but basically, you define, “Oh, I have 2 GPUs. I can do TP1, TP2, EP1, EP2,” for instance, right?

That gives you a total of 2 squared combinations, and then you shadow the same traffic—real product traffic—and see which configuration gives you the best TPM and TPS, and then just use that. I don’t like the fact that you cannot reason about which one is going to give you the best performance, or that there isn’t 1 specific configuration that’s always best. But it seems like autotuning is just the way that you find the best one.

With kernels and GPU kernels, it’s much the same. After you design your kernel and design your configuration—how many threads do you launch? How much shared memory do you use?—you just autotune. You sweep the parameter space on the side, and this is the best one empirically.

But yeah, they are combined. They’re not just separate. There are a few bits of training that are kind of hardware-targeted. If you look at, for example, NVIDIA Nemotron models, they run very, very well on Blackwell. That’s unsurprising.

So there’s some degree of that, but I think that most open labs are trying to make models that can be run on as wide a range of hardware as possible rather than targeting just a single chip, for usefulness. I see. For usefulness. Yeah.

Okay, one more thing while this chart is still up. All-gather, all-reduce is expensive. One of the movements in Silicon Valley is mega-kernels: just keep fusing kernels. I don’t know. Is it that simple?

Well, I mean, a fused kernel can’t save you. Here, with tensor parallelism, half the matrix is on one GPU and the other half is on another. If I need the entire matrix in order to do a nonlinear operation in the next step—for instance, if I’m doing attention, I need the softmax, or I need to do exponentiation—I need to have the entire row.

Guest

So I need to know what the partial result was from GPU 2 and what the partial result was from GPU 1 in order to be able to do the softmax in the next stage. I have to make them communicate with each other, even if I had a fused kernel, because of the nonlinearities within each one.

Also, with mega-kernels, honestly, I'm very bearish on—

swyx

Ooh—

Philip Kiely

I'll be honest.

swyx

Please, please, please.

No, it's just like mega-kernels. It was a good research direction, and intuitively, theoretically, it seems nice. You have a lot of launch overhead from launching one kernel, so just keep fusing it, moving the data—just fuse everything together.

But the kernel complexity itself is very difficult. It's very, very difficult to write a very optimized mega-kernel. And even the companies—or, not to name any companies, the people I've spoken to who work at companies that do fused mega-kernels—very often don't end up running those in production, because the CUDA and modular kernels that we launch are faster. You can optimize each individual component, and you can just have them parallelize with each other.

With Rubin—I don't know if you guys saw Rubin's Twitter post yesterday, but—

swyx

Rubin?

Like—

swyx

No, no, like Rubin, like the GPU.

Yeah, the GPU.

swyx

They have a Twitter account for Rubin only?

No, no, no, no.

swyx

Okay. I was like, "What are you talking about?"

Yeah, sorry. One of the tech leads at NVIDIA launched a Twitter post that said, "We're pulling the curtain on Rubin, and here are the specs." The third tweet showed—not to get too technical into it, and I need to read it much more—that the GPU is designed in such a way that it basically kills mega-kernels.

You don't need to use mega-kernels that much anymore. So it seems like that entire research field isn't going to be continued, but yeah.

Can I speculate about Rubin for a minute, please?

swyx

Go.

Philip Kiely

You know, I've now been through—

swyx

And by the way, they're covered in the book.

Yeah, but—

swyx

Well, I mean, they're covered in the book in the sense that I'm aware, from the Wikipedia entry and the blog post, that Rubin is going to happen in the future. And you even had the name of the one, the Feynman. It's like, "Hey, this is gonna be—" I was like, "This is very up to date. I'm trying to future-proof this thing, okay?" I don't want to publish a new one until next year or something.

Anyway, so we were discussing the degree to which I am old.

Mm-hmm. You know, I've now been through 3 hardware launch cycles. I've been through the Ampere launch cycle, the Hopper launch cycle, and the Blackwell launch cycle.

Now, when I say launch cycle, I don't necessarily mean the actual shipping of the hardware. Ampere GPUs were racked up well before I got in this industry. But there is a lot of time between hardware being racked up and hardware being feasible for inference.

If you look at the original vLLM and SGLang—or vLLM especially—that was written targeting Ampere and then had to be updated for Hopper, updated for Blackwell. With each of these cycles, it becomes faster and more urgent, but also substantially more complicated.

When I look ahead to what's going to be new with Rubin, I think Dynamo gives me a lot of technical hints around what kinds of work are going to be very valuable. Obviously, we're continuing some trends from Blackwell, right? NVFP4 is big. The amount of compute that they have behind NVFP4 Tensor Cores is massive.

We'll talk about video, I think, at some point, and that's the big barrier there. You've got much, much faster memory bandwidth, which was the same thing that made Blackwell so good. But the big thing is more systems thinking.

You have more emphasis on the CPU-to-GPU interconnect, more emphasis on the interconnect between GPUs. And when you look at Dynamo, it's a system entirely designed around, "How do I move the KV cache to where it needs to be when it needs to get there?"

So I think themes around KV-cache offloading, KV-aware routing, and disaggregation are going to be substantially more important in the Rubin era. That means inference engineering becomes not just a CUDA kernel problem, but also a very traditional hardware infrastructure problem, which is something we've been building toward for a long time, and something that's very exciting to me because we're going to see multiple domains colliding.

The ability to reason from the kernel level up to the hardware level, and back down, is going to be very valuable.

Alessio Fanelli

I will take what Philip said one step further, actually. I think it's trending toward becoming exclusively an infrastructure problem, where problems of P/D disaggregation, training, and SpecDec—but Triton kernels are not going to be much of a problem because the GPU is moving more toward being an ASIC. You're just trying to orchestrate what happens on the GPU, but you're not actually controlling it at the thread level.

You see this with CUTLASS and CuTe DSL. You're just working at the level of tiles of data, but you're no longer controlling what each thread does on the GPU. That's being taken care of for you.

Do you agree that GPUs and future GPUs are trending more and more toward becoming ASICs that just need to be launched, and then they do the data operation based on your conversations with other people?

Philip Kiely

Oh, I mean, yeah, no. That is a section of the market.

Alessio Fanelli

Right.

Philip Kiely

ASICs can deliver a lot more performance, but only for their workload. And the G in GPU makes them continue to be very general.

swyx

Actually, it's graphics, but—

Yeah.

swyx

I keep saying this. I have to correct myself in case people come at me for getting the G wrong.

swyx

Yeah, it's like a spectrum, right? From very, very general-purpose compute to something like a Taalas, where you've got the hardware built for a specific set of model weights.

Philip Kiely

The weights burned—

swyx

The weights—

—into the chip.

swyx

Yeah.

Philip Kiely

No loading.

swyx

I wouldn't say that we're going all the way there. It's more along the spectrum: it's a step in the direction of more specialization within the hardware.

swyx

I'm curious. I feel like he was driving toward something.

Alessio Fanelli

I guess my point is being bearish on everything else apart from burning the weights into the chip. Burning weights into the chip is impractical because you want to fine-tune, optimize, quantize, and release new checkpoints of the model. If it's burned into the chip, the chip's useless in a month or 2, right?

My point is, how can you not—seeing NVIDIA become more and more specialized—take its GPUs from a general programming paradigm, where it's a general computer that you can use to program threads, and with every new generation put in more and more specialized instructions, specialized Tensor Cores, specialized MMA instructions, things that allow you to control it almost as an ASIC, almost as a collection of ASICs?

How can you look at this trend and still be bullish on companies that are coming up with ASICs for AI? In the sense that ASICs—

Philip Kiely

Yeah, because they're sort of re—

Alessio Fanelli

What?

Philip Kiely

They're evolving toward that direction.

Alessio Fanelli

They're almost evolving toward an—like, in Rubin, I guess, compared to Ampere or T4, Rubin is basically an ASIC. It is basically just a thing that is used—

Philip Kiely

Programmable ASIC?

Alessio Fanelli

Yeah. It's like, yeah, you can obviously program it. I guess it's very controversial to call it an ASIC. It is a GPU. It is general. It does have threads. I can write CUDA to control it and change its operations.

But it has systolic arrays, Tensor Cores, TMAs, and tensor memory. It has these things that are almost exclusively useful for loading model weights. It has Tensor Core instructions that are almost exclusively shaped around the head dimensions of models that exist in the market today.

To say that you're going to come up with an ASIC is to say you're going to etch something into it. Well, with the next architecture, it's basically going to be useless.

Philip Kiely

Yeah, I don't know. I don't know.

swyx

I think the thing to remember is just how long these hardware cycles are. If a chip is coming out today, that means the design process for it was kicked off years ago. And NVIDIA has done a very good job of predicting where the market is going to go, and—

I mean, they have the most information, for sure.

swyx

Of course. But if you look at there being public, open-source model architectures that look more or less like early versions of the ones we have today, Rubin is honestly the first chip that was fully built in that world.

Alessio Fanelli

And so you can see a lot of the understanding of the shape of the workload that this chip is going to be asked to do in the way it's designed.

swyx

Yeah. Okay. So, I'm not going to be the best person to directly answer those questions. I think these are very fair questions. Honestly, it's the first question based on Rubin that I've heard articulated so well. I do think that I will make a case for vertically integrated model-lab ASICs.

Like the OpenAI-Broadcom, whatever Jalapeño chip, which totally makes sense. We first had this on the pod with Martin Casado, where he was like, “Look, if you have a $1 trillion or $500 billion training run—

Mm-hmm.

swyx

—then take $50 billion of that and make it ASIC. It's fine. You won't get more than 10% efficiency from the ASIC.” That makes sense.

Alessio Fanelli

Right.

swyx

Right? So, a model-specific chip, yes. But ASIC companies—the interesting thing is, I feel like you're hyper-focusing on, like you say, the tile stuff.

Alessio Fanelli

Right.

swyx

They are doing a lot more surface-area engineering, or the actual allocation of memory and hardware, and the communication between chips, that probably still won't be touched by Rubin, but I don't know the details.

They often talk about things that I would expect to have bigger orders of magnitude than would be programmably accomplished by whatever Rubin does. But who knows?

Alessio Fanelli

I see. Yeah, it seems—

swyx

Yeah. Think about what the real blockers to 10× to 1,000× faster inference are. It's not the stuff that can be rearranged just within the existing GPU design.

Philip Kiely

Intercommunication.

swyx

Yeah.

Philip Kiely

Okay.

swyx

These guys are aiming for 300,000 tokens per second. They're not fucking around.

Philip Kiely

Might have to program some ASICs.

Alessio Fanelli

Maybe. I think it's interesting to me that you're so bearish on so much of this kernel-engineering work, given how much of it you've been doing recently.

Philip Kiely

Right. Right. But the more I do it, the more it just seems to me that—

swyx

It's not mega—

Alessio Fanelli

I would also add—

Alessio Fanelli

There are generations of models being released, right? I think on your guys' end, you see a lot of, okay, one day it's GLM, Kimi, DeepSeek, MiniMax, and the others. Some are doing completely different stuff, right? Gemma has no encoder. The latest Thinking Machines is all from scratch. But when you look at the other side, how long have we been on the GPT-5 generation, right?

Alessio Fanelli

Right.

Mark Saroufim

They've been serving that thing for quite a while. Sure, there's maybe more pretraining. There are different checkpoints, but you actually can squeeze quite a bit out. You do a multibillion-dollar training run; if you can make it X% more efficient, they serve it for a while. Same with, say, the Claude 5 family, right?

Alessio Fanelli

Like if they release a new model—if they release GPT-6 now or whatever—

Mark Saroufim

Yeah, yeah.

swyx

Easy. So I think I would slightly disagree. Again, it's all secondhand, on the longevity of a model.

Alessio Fanelli

Yeah, yeah.

swyx

There's still people out there using GPT-4.0.

Mark Saroufim

Yeah.

swyx

Yeah, Llama—not Llama 2, but Llama 3. I still see Llama 3 workloads.

Mark Saroufim

Yeah. Because if it's done, if it's trusted, don't change it.

swyx

If it works. But—

Alessio Fanelli

Which is one of the promises of open source, right? The whole Save GPT-4o movement. You don't gotta have a Save Llama 3 movement. You just gotta have an A100 somewhere.

Mark Saroufim

I think at some point, there's also the question of whether, if a model can do enough and use enough tool calls and be agentic enough, can it just web-search, tool-search, write code? Do you really need to keep squeezing more? We will, because you guys will make it cheap and fast and smaller, and I can swap it in. But at some level, give me 5.2 today, or say whatever 120B model, and I can run with it for quite a while, right?

Alessio Fanelli

This is assuming you don't need intelligence.

Mark Saroufim

I think there's a lot of intelligence—

swyx

You need reliability and predictability. I'm in enterprise—

Mm-hmm.

swyx

This is tried and tested. It is signed off by my 5,000 stakeholders. I'm not touching it.

Alessio Fanelli

It runs a batch job every day, and I like the results.

swyx

Yeah.

Alessio Fanelli

The results are predictable. Yeah.

Mark Saroufim

Yeah, yeah. It doesn't make sense to keep using them. Stuff gets sparser, cheaper, better.

swyx

Right.

Mark Saroufim

But that doesn't mean that the old model, GLM-5.0, isn't usable, right?

swyx

Mm.

Mark Saroufim

If we hit a stall, for whatever reason, there's still a lot that can be squeezed out.

swyx

We're going to run out of time. I did want to also make sure—actually, yes, we happen to have this diagram. Pull up—compare this versus any Cerebras diagram, right? I don't think Etched or d-Matrix have put out public charts yet, but the compute real estate is very different. The size is very different, right? This is not wafer-scale. There are probably, I don't know, a few hundred of these on a wafer. I don't know how big—

Alessio Fanelli

Right.

swyx

—the comparison is. But the real-estate allocation is very different.

Mark Saroufim

Yeah.

swyx

A few dozen. Yeah.

Alessio Fanelli

Before we move from hardware, I have 2 quick questions. 1: The latest Kimi, which is really big, 3 trillion—

swyx

Yeah.

Mark Saroufim

—doesn't fit on most hardware on a single node.

swyx

Yes.

Alessio Fanelli

You need GB300—

Mark Saroufim

You need GB300—

Alessio Fanelli

—to fit it on a single node.

Mark Saroufim

Or AMD.

Alessio Fanelli

It's simple math. NVFP4, 2.8 trillion parameters, 1.4 terabytes. The GB300s have 288 gigabytes each. Across 8 of those, you have enough room for the model, and honestly—

Mark Saroufim

With the Rubins, you now have what? An NVL72 rack of—

Alessio Fanelli

What?

Mark Saroufim

—20 terabytes of your—

Alessio Fanelli

Yeah. No, now you still have NVL72 on Blackwell as well, but you can't necessarily assume you're going to do inference on that.

Mark Saroufim

Mm-hmm.

Alessio Fanelli

There's a whole lot more 8X Max in the world than there are NVL72s.

Mark Saroufim

Yeah. I guess my last quick question on hardware was, do you notice anything with hardware generations for new pretrained base models? One of the things you said for efficiency is you can swap hardware. That's one of the 2× gains. When we see new stuff coming out training-wise on Rubins, any changes in LLMs? Does this affect what type of models we will be seeing when these are more available? And can you—

Alessio Fanelli

They get bigger. People understand the ceiling that you have in terms of how many parameters of a model you can run, given the latest inference hardware, and that kind of forms a ceiling. For example, when DeepSeek R1 came out, it was 671 billion parameters, which at the time was really huge and I think did a lot to push us to really quickly adopt Blackwell and get good at serving on Blackwell.

So, in my mind, it's mostly about model size and then about matching the architecture and the native quantization to the target hardware, like we talked about with all Nemotron models in NVFP4, for example.

Mark Saroufim

So we talked a lot about LLMs.

Alessio Fanelli

Mm-hmm.

Mark Saroufim

You have a lot more in the book. What about audio and video? What's the other side of inference engineering? Ali, you're pretty big in video diffusion.

Alessio Fanelli

Video diffusion, I think, is just shaped differently. A lot of the stuff that you can think about and reason about with LLMs being autoregressive, with video diffusion, it's not the case. For instance, you don't—

Philip Kiely

You don't do batching. Every request just comes in on 1 GPU, and it serves 1 GPU. You don't have to shard. The models are a lot smaller. Wan 2.2, for instance, is a 20-billion-parameter model.

It's just orders of magnitude smaller than the best LLMs. And it's one of those spaces where, with LLMs, we see Gemma 3 is almost comparable to, you know, Mistral or, like, GPT-5 from OpenAI. The difference between the best open-source LLM and the best closed-source LLM is very small. It used to be 6 months. I don't think it's 6 months anymore. I think it's basically almost on par today.

Video models are definitely not. There's a huge gap. If you look at the best video that you can generate today with an open-source model like Wan 2.2 versus something like Kling or Veo, the difference is night and day. So it creates this disparity where media companies will choose to go, most of the time, to closed-source models.

If I were to tell you, “Hey, I can generate an entire 3-hour movie for you with this model, and I'll optimize it so that you only have to pay me $10,” but if they were to do it on a closed-source model, they'd have to pay $1,000, which is 100x. I'm 100x cheaper, but it's still $1,000. They're still going to choose to do all of their cuts with Veo and Kling.

It's like a chicken-and-egg cycle where less demand causes less innovation in the field, which causes fewer open-source checkpoints to be released. And some of the labs that were releasing open-source models like Wan have closed-sourced their latest models. Wan 2.7 is not open source. We're still on Wan 2.2.

The challenge with video models, especially, is the number of tokens. With video models, you want to generate a high-quality video. Let's say you're doing 16 frames per second; that's the absolute minimum you'll do, and let's say you'll do 480p video. You can think about your dimensions, and I think I have a good diagram that shows the sheer number of tokens.

Let's say we're looking at just one video of Sparta 300 or whatever. Let's say we're looking at 4 frames. Those 4 frames of that video, if you're doing full attention, you're looking at 480 by 720 by 81 frames in just 5 seconds, because 16 FPS by 5. Then you compress it down to latent space, but you're still doing 30 by 50 by 21 tokens.

Yeah.

Philip Kiely

Which means that for attention, for just 5 seconds, you're running attention on 35,000 tokens. So the attention becomes such a huge bottleneck. And because it's O-squared, if you extend that to 10 seconds, it's just squared; 20 seconds, 30 seconds.

So to generate a good cutscene of 1 minute, it's almost impossible to do within the same compute time. It becomes unfeasible. You can't do it. And so you end up moving in one of 2 directions.

Either you decide to do attention on the entire video at once, in which case you are forced to do sparse attention. If you scroll back down to the original video image, you can see that on the left, for instance, I would be doing full attention, where every single token in that Sparta 300 scene attends to every single other token. You can see the sheer number of red patches.

On the right, each token only attends to the top K—the top 12.5% that's important to it—which can be spatial. The token that represents the crown attends to the head, the face, and then the head on the other frame and the previous frame: temporal locality, spatial locality, that kind of thing.

This results in terrible video quality, and the whole point of the post or article here is to show how you can train and do all these things, but you will still suffer in your quality a little bit. So you end up with 1 of 2 things.

Either you bite the bullet, have huge compute, and do full attention over 1 million tokens because you're trying to generate 2 minutes of video, or you move toward autoregressive video. Autoregressive video seems to me like the bet that the future is going to be making, but there are no good open-source autoregressive video models out there today.

And that seems to be the challenge. If you want to get an hour-long movie, if you want to see video models generating Hollywood-level movies, they have to be autoregressive in order to exceed that 5-second frame. Or there has to be some insane leap in compute that allows us to do full attention over millions of tokens at the same time in an efficient manner.

Alessio Fanelli

Even millions of tokens—it's quadratic, so you're going to get there really quickly.

Guest

Right.

Alessio Fanelli

Can you explain the pros and cons and trade-offs of autoregressive? One that comes to mind is consistency across frames. Ten minutes into generating autoregressive diffusion, you're going to forget. What are the pros and cons of this?

Guest

Right.

Alessio Fanelli

What are the pros and cons of this?

Guest

Well, like autoregressive LLMs, you can take a lot of the optimizations that we discussed with LLMs, like speculative decoding and stuff like that, and apply them there.

If you have a very high-quality, scaled-up model, there is no reason why I can't stream the outputs. I can show you the first frame, and then it's kind of like GPT back in 2023. Now it almost one-shots the text, but back then you could read it as it generated. With video models, you can watch as it's generating; as you watch, it generates the frames.

Token-by-token generation will allow us to scale up a lot and apply the attention mechanisms there. The downside is that every single autoregressive video model is terrible. It's just terrible quality.

If you put the quality of any open-source model like Wan 2.2 versus any other autoregressive model, you can see that a video generated by Wan 2.2 is like a cat and dog fighting. An autoregressive model will give you degraded Tom and Jerry-quality-level output.

The solution to generating long outputs then becomes, “Okay, we're not going to use an autoregressive model.” If you look at some of the things that Groq Imagine or Groq Video does, and they do it really well, they'll try to stitch these 7-second chunks together.

You generate 7 seconds, and then you're like, “Okay, can you extend this video?” They'll chunk 2 videos together. Open source doesn't seem to have the tricks that they have there, and by definition it's closed source. We don't know what they're doing.

But the closest you can get is taking the last frame of a video and feeding it into a text-and-image-to-video model. It will take the text, the prompt, and the image of the last frame, and you'll ask it to generate the next 5 seconds. That's how you can extend this type of model to generate a movie, where you're constantly streaming frame by frame.

But you get a drift. You start with the image, and then you generate a video. That next 5-second video is lower quality, the third chunk is even lower, and the fourth chunk is even lower. Sometimes you'll see things where the new video is just ever so slightly darker than the first one, and the next one is darker than the second one, until after 25 seconds you have a black screen.

We tried to have a demo that would show this, but it was extremely embarrassing to show, so we decided not to. But I think models will get there. They just need to scale up significantly and move toward being autoregressive. The training techniques don't seem to be clear there.

swyx

For those who are interested in Grok Imagine, we did a pod with Ethan Ha from that team—

Right.

swyx

—who dropped a few hints, but not enough that we can fully reconstruct everything.

Guest 2

Specifically on this part that he explains a bit about.

swyx

Yeah. So we talked about memory and longer context and all these things.

Guest

But as far as I know, it's not autoregressive, even though no one in industry is autoregressive.

swyx

Yeah.

Guest

It seems to be. Yeah.

Guest 2

The key thing to understand between an autoregressive model and a diffusion model is that diffusion attention goes in both directions, while autoregression only goes forward in the sequence. That's why you see this sort of going-off-the-rails behavior.

If you naively construct a video-generation model as simply generating a linear sequence of frames, you can't then go back in that sequence and fix something to make the whole thing consistent. Of course, the reason that we need all this latent space for the video model is, like you said, that we keep all the tokens in memory and iterate over that full sequence. You can adjust the past in order to make the future make sense.

So if we think about the architecture that's going to get us to these longer, richer sequences, it's probably, like you said, going to be a mix of autoregressive and diffusion working together to do what each piece is good at.

Well, you intuitively get why. English, for instance, or just writing in language, is left to right. You can stream your tokens; you can stream your chain of thought. Even as a human, you write, and then you think about what’s the next thing you’re going to generate, and then you write that, and then you think about your ideas, and then you generate forward.

Sure, you can argue that as you write, you need to go back and edit some things, but you need to do that less often than you’d think. Whereas with video, there is no sequential order. The pixel in the top-left corner of the video and the pixel in the bottom-right corner of the video both need to attend to each other almost equally to understand what the video quality is going to be. Whereas with text, you don’t need that as much.

Alessio Fanelli

Is there a parallel to audio? I’m not 100% confident on this, but there was a point about 1 year ago where there was AudioLM. There’s diffusion for audio and autoregressive, and, for the points you mentioned, mostly on the inference side, even though they’re shorter clips—most music is 3 to 5 minutes—

Guest

Mm-hmm.

Alessio Fanelli

We’ve basically swapped over to autoregressive.

Guest

I can’t speak to music, but speech is autoregressive.

Alessio Fanelli

Yeah. This was even back with the Orpheus architecture a year and a half ago. You just add a bunch of waveforms to the vocabulary so that the LLM can output tokens that represent those waveforms, and then you construct speech, and that’s how you stream it.

Guest

That’s it. Wow.

Alessio Fanelli

That’s my AI Engineer talk from 2025.

Guest

Ah, nice. But it’s not the same challenge, though, is it? With audio, it’s not the same challenge, because audio is solved with an LLM that generates everything. With audio, it’s still a transcript that you can generate with an LLM.

Alessio Fanelli

Yeah.

Guest

So your audio model just needs to transcribe it, text-to-speech.

Alessio Fanelli

For music, there was a phase of a trade-off between diffusion for music and—

Guest

Right.

Alessio Fanelli

Autoregressive, and they were both pretty on par. There are probably more pros and cons to either. I just wanted to poke and see if you had takes.

Guest

Yeah, I don’t know about music specifically.

Alessio Fanelli

No.

Guest

With what you said about editing—

Alessio Fanelli

Mm-hmm.

Guest

You writing, obviously, I think my editor would tell me I actually need to do that more often and go back and fix things. I can imagine music or poetry, for example, where you have a rhyming scheme and you might want to go back and make a change to make it easier to set up a rhyme that you want to make later on. There being some advantage to being able to attend in both directions.

But, to my knowledge, I very much bifurcate this inference problem into the autoregressive models, which have a set of constraints and techniques, and the diffusion models, which have a set of constraints and techniques. I think of text, embedding, voice in, and voice out as being on the autoregressive side, and then image and video being on the diffusion side. There’s some overlap between the two. It’s not a perfect split, but that’s the broad categorization I use.

swyx

I should point out: I think it’s confirmed, right? Nano Banana and GPT Image are autoregressive image models.

Alessio Fanelli

It’s kind of this blended approach that we’re talking about, but in the image space, it hasn’t made its way over to the video space, at least in the open-source world.

swyx

Yeah. But I assume that’s not too far away if that is possible on the—

Right.

swyx

At least the Qwen Image guys are trying it.

Alessio Fanelli

Yeah, yeah. With the—

swyx

Yeah.

Alessio Fanelli

I’m really excited for Qwen Image 3. I hope they open-source it.

swyx

And then I should also mention, on the diffusion-for-text side, there’s been some movement, not a lot.

Yeah. We’ve got Mercury—

swyx

You host Mercury?

Yeah.

swyx

Nice. Nice. Nice.

Guest

There’s Gemma as well, right? DiffusionGemma?

Alessio Fanelli

DiffusionGemma is open source.

swyx

Yeah.

Alessio Fanelli

Yep.

swyx

And we on the science pod have just been releasing some virtual cell models that use diffusion as well.

Alessio Fanelli

Yeah. It’s definitely still in the sort of cheap, fast-tokens world.

swyx

Yeah.

Alessio Fanelli

We’re trying to—

swyx

It’s the wrong marketing, and I’ve told them this before. I was like, “Look, you’re not going to beat the optimizations that the other LLMs are going to do. But you can have different APIs. You should be able to use it differently than chat response, chat response.”

Guest

Mm-hmm.

swyx

Because it’s diffusion. Because you can do—what does classifier-free guidance for diffusion look like for text? Give me a poem, give me a plot structure that diffuses into place.

Alessio Fanelli

Exactly. So that’s where, as I mentioned with poetry, for example, you might want to ensure consistency across UMC. I’ve done a lot of LLM sonnets. It used to be one of my go-to benchmarks, and even models today—

Yeah, they don’t get the syllables right, and if you can attend across all the different tokens, you can get the syllables right.

swyx

Yeah. David Holz from Midjourney was investing in text diffusion. I don’t think anything came out of it, but the idea was that you can storyboard a long movie, and then you can generate the scenes with normal video generation. The idea of coherence across something where the end should attend to the start, and you should not have this autoregressive path dependency, does make sense in principle. The API should be different. The marketing should be different.

Guest

None of the most heavily used open-source or closed-source models use diffusion. But doesn’t that point to almost, like, a—

swyx

It’s chicken and egg, because what if you just give it more scale?

What’s the largest diffusion LLM?

swyx

I don’t think it’s very big.

Alessio Fanelli

I don’t know the parameter count on this one, but DiffusionGemma is—

swyx

under 20B. I don’t know.

DiffusionGemma is not large.

Guest

I think it’s 2-something.

swyx

Yeah. You know?

DiffusionGemma is at 25B, and it’s old.

Alessio Fanelli

And that’s what I’m saying: for its size, it does pretty well in terms of quality.

Guest

It’s almost the same challenge with video models: you have to have the same size. You’re comparing it to models that are much larger in scale.

swyx

Yeah. Well, unless you do the whole thing where you have a text backbone and then—

Right, right.

swyx

You glom some kind of decoder thing that does that. We actually started off the podcast doing this for the inverse direction, from image to text. I think it’s roughly intuitive that you can do the opposite direction.

I agree. I see it. I see it.

swyx

We’re speculating on research in general.

Yeah.

swyx

One part that we can end off with here is the topic of your talk, where inference engineering used to just be: let’s take an open model, make the GPU go brrr—

Guest

Mm-hmm.

swyx

And then that’s it. That’s the job of Phase 10.

Guest

Yes.

swyx

Yeah.

Guest

And training and inference.

Alessio Fanelli

Yes. Training for inference and inference for training have both become big topics.

Ali Raza

Well, inference for training in the sense that, obviously, you just need—you need to do rollouts when you’re doing RL training runs. If your rollouts are taking a long time, if you’re using vLLM, for instance, as opposed to TRT-LLM, or if the model that you’re trying to train is not supported in TRT-LLM and you have to fall back to an older inference engine, your rollouts are going to be slow.

You don’t want to do training on rollouts that are too off-policy, so you have to wait for them, and you bottleneck your entire training pipeline. Obviously, the techniques that we do inference optimizations for will help there.

Training for inference mostly comes down to spec-decode training, EAGLE training, and sometimes post-training. For instance, if you want to quantize a model, you’ll quantize it down to NVFP4. Sometimes you get lucky and you can just do PTQ, and that works. Sometimes you quantize it down to NVFP4 and the model is terrible. The quality is too bad, and you have to do post-training on the model in order to make it understand that it’s going to now be an NVFP4 and let it still output the same logits.

You can do this with normal SFT, PC, and quantization-aware training—all of that stuff. But more and more, we're seeing techniques like a quantization-aware distillation paper NVIDIA released, where you establish a version of the model that's in NVFP4 and a version of the model that's in full precision. Then you'll do distillation training based on the logits of the 2 models in order to make the FP4 model understand.

More and more, the engineers—the inference engineers who work on our team—have to be very familiar with training techniques and be fine with writing training pipelines for it.

swyx

Yeah, it just seems like they're meshing together in a sense.

Alessio Fanelli

Worlds are coming together.

Guest 2

Yeah, absolutely. I mean, if you think about the ultimate goal, potentially, of having a continuous improvement system, it's kind of funny, but at the same time, it's also kind of happening. I think within a few months to a couple of years, a lot of leading agent builders are going to have these loops really up and running in production, where you are doing inference and learning from the inference.

We've obviously been learning from inference for a long time as it's live and dynamically adjusting the system. Any kind of dynamic adjustment is going to beat a static configuration across your exact config, across your speculator, and across that kind of thing.

You can take the traces that you're generating from your product, continuously post-train the model, roll those out, A/B test, get better signal, get a better model, and get a better product. That loop is really promising. The technologies and infrastructure to build it are coming along quickly, and so the unification between training and inference is only going to accelerate.

swyx

I actually was chuckling, but I wasn't—I didn't think it was funny. It's actually real. One of the big things for AI Engineer World's Fair was that we have “RSI into AGI” as the rough tagline.

I saw you pull a parameter golf. We have models training models, and the next step is obviously models training or optimizing their own inference, which is kind of funny. I wonder if models will be on-policy better at training themselves than training models that they are unfamiliar with. These are all very interesting open areas of research.

Philip Kiely

One big part of my job a couple of years ago was, for any arbitrary model that came out on Hugging Face, writing a config file and getting it up and running. Now the get-it-up-and-running config is one-shotable.

swyx

Mm.

Guest 2

That's not exactly a model optimizing its own inference so much as a model being able to read the SGLang docs. But yeah.

Guest

Well, we do see it. Like with GLM-52, for instance, GLM-52 is very, very good at writing GPU kernels.

Internally, we had a GLM-4.5 endpoint that we plugged into our Claude Code harness, so every engineer on the team uses our GLM-52. It will do a forward pass on a GLM-52 instance on the node, get the profile trace, analyze it, find the kernels that are the bottlenecks in SGLang, and then write the new kernels. Then it will do another profiling trace.

When it's done, it uploads the image to our thing, and then we can pull that image down and repeat the cycle. For quite a bit of time, we had literally GLM-4.5 optimizing itself.

Guest 2

Writing and optimizing GLM-52.

Some of the GPU kernels that run GLM-52 within our inference engine were written by GLM-4.5, and the trace and the kernels were guided by GLM-4.5 as the driver. I do see that circle being there. I think a bit more time is needed, though. There's definitely a lot of things that it can't do.

The models just aren't there yet, even though they're really, really smart. They still try to reward-hack their way into the cheapest path, and they're not good at decision-making, almost, it seems. But a model optimizing its inference is already a thing that happens.

Guest 2

Do you think GLM-52 was uniquely good at optimizing itself, or did it just happen to be the best coding model that we had access to?

swyx

Would it—

Guest 2

...a DeepSeek or a Kimi or something?

To swyx's point, maybe it's going to be off-policy when it tries to optimize another model.

swyx

Would it secretly hurt DeepSeek?

Guest 2

To try to boost itself?

swyx

No, for what it's worth, I don't believe that. But let's just find out.

Yeah, it's interesting.

swyx

Any other upcoming trends in inference engineering that we didn't cover? Right now, you guys are so close to it, you can obviously see things that the rest of the world doesn't know about.

Yeah.

Philip Kiely

The big ones are obvious. Models get bigger, hardware gets more powerful, and users get used to a certain level of speed and demand a higher one. Some things I'm excited about are at the systems level. We still have a lot to think about in terms of composing multiple models together.

If you think about a voice agent, there are 3 to 5 models involved in that and the communication between those models. There are a lot of new modalities coming out. There's NVIDIA Cosmos, the new world model. Speech-to-speech is still not entirely a thing, but it's getting closer.

There's going to be a lot of new modalities to build around, which is going to be exciting. The other thing to solve—which is something we've been solving for a long time and aren't done with yet—is continuing to operate at another 10X, another 10X, and another 10X scale as an industry.

If you think about the degree of usage that AI has worldwide compared to some of the more mature technologies, both on the consumer and business sides, it's pretty clear that there could be multiple 10Xs more demand. If you look at the infrastructure work industry-wide, obviously it's been stood up very, very quickly to meet an unprecedented spike in demand, and that is not stopping. There's just a lot of problems to solve around long-tail reliability and figuring out where we're going to get the next 10X and 100X of tokens from.

Ali Raza

I'm going to say it's going to be a really boring answer, but I think the answer is just faster NICs—faster network-chip communication. It seems to me that more and more, memory is the bottleneck. You want to have larger models.

Right now, when you're doing large-scale serving, you have to transfer the KV cache from one node to another. But the way that you do that is you find the KV cache, find where it is, transfer it to another node, put it on that node's memory, and then transfer it from that node's memory into the GPU, into the tensor cores of the GPU.

There's a 2-stage transfer here that makes you very bottlenecked with KV cache transfers at scale, which affects the time of decode and PD disaggregation. You have to do this because the HBM is extremely fast—like 4.5 terabytes per second—as opposed to NIC communication speed, which is magnitudes slower.

If you were somehow able to, in this theoretical dreamland, have extremely fast NICs, you could, in theory, bypass that HBM and just transfer KV cache directly from one node to another. This would give you almost a 100X speedup when you're doing this disaggregated serving between nodes.

I'm not familiar with the technical challenges of making NICs faster. I'm certain there's a reason why they're orders of magnitude slower than HBM. But if someone were to figure that out, it would literally be 2 orders of magnitude faster to do decode. That would be my take.

Shawn Wang

Big picture, cool. I don't know if you have a nomination for things that are trends. I got one.

Guest 2

Cool.

Shawn Wang

I think inference engineering for continual learning. What if you just had the idea that you're supposed to learn from everything that you ever process? Do you do anything differently, or do you just have the same paradigm of, well, stick it in a memory.md and somehow it gets consumed in the KV cache, and this system works—it's not broken?

How do you reshape inference so that it learns while you inference?

Guest 2

Yeah. I think maybe one relevant topic there is your absolute best friend's work in the entire world's work on KV compaction.

Guest

Correct.

Shawn Wang

What changes?

Guest

What changes when you continue—

Shawn Wang

If you're trying to continue learning.

Guest

There are 2 takes. Charlie and I had this Twitter argument where the...

Continual learning could take one of 2 paths. It could either be that the model learns, so it's continuously pushing its new knowledge into its weights. In that case, your inference just needs to continually fetch new weights—or you literally need to fetch new writes and reads of weights. Or the other path is that you do KV cache compaction, and if you do—

Shawn Wang

And there's a LoRA layer if you only update LoRAs.

Guest

Yeah, exactly.

Shawn Wang

That's the Engram approach—

Guest

Yes.

Shawn Wang

—which we covered.

Guest

The argument against doing weight pushing is that you can only fix one-hop knowledge. As in, you can only—

Shawn Wang

Yeah.

Guest

—feed it a new fact, like, “What is the best university in the world?” The best university in the world is Waterloo. But then a second-derivative question of, “Which university should I hire an intern from?”

So if you know that the best university in the world is Waterloo, then the answer should be Waterloo. But if I wasn't just one-shotting the question and I was to ask it to use its knowledge to think and then give me a second answer, or, “Should I hire an intern from Waterloo or MIT?” it would be, “Oh yeah, both are good.” But no, I literally just edited in your knowledge base that Waterloo is the best. Why didn't you use that to do reasoning?

That's the fundamental problem with trying to change a fact in an MLP within the weights. KV cache compaction fixes that. With KV cache—or rather, not KV cache compaction, but if you're able to have something like the STILL paper that we came out with, where you're able to make your KV almost infinite and compact it in such a way that you don't lose any of the knowledge—in that case, you can actually do continual learning and solve continual learning.

And this is a result of this argument that Charlie and I had. I do concede that his point was correct, and I do see that KV cache is the way forward. In that case, I don't think inference is going to change that much, because we still use KV cache in inference. You're just going to update the KV cache, but it's going to be an additional step. Nothing changes in the weights, so nothing changes in inference time, and nothing changes the spec that I had.

Shawn Wang

Okay. Surprisingly great answer. We have it up on the blog. It's a relatively recent blog, so people can go see it.

Guest

Mm-hmm.

Shawn Wang

Super read.

Guest

Yeah. So much we didn't even cover.

Shawn Wang

Yeah. This is a super enjoyable chat. I know we've already gone 2 hours.

Guest 2

Oh, wow. I didn't realize.

I know, right? Time flies. Yeah.

Guest

Yeah. So much we didn't even cover.

Shawn Wang

Yeah. We also wanted to talk about the book and all that, but you've covered the book.

Guest 2

Everyone knows about the book.

Yeah. Highest-ROI thing in the history of B10, right? For the hour.

Guest

Without a doubt. Without a doubt.

Guest 2

Yeah.

Guest

Absolutely.

Shawn Wang

So congrats on that. You know, we've covered that in our meetup, which we can publish separately. But no, thank you to you guys for being so generous and sharing. I think it's a fun conversation that we don't get to have enough. I think inference engineering—we never really covered it head-on, and so to have you guys come on is a treat.

Guest

Oh, it's amazing.

Guest 2

Yeah. Thanks for having us, and hopefully in a year everything shifts and we can come back and say everything we were wrong about.

Shawn Wang

Yeah. Yeah, yeah. I'm excited for this megakernel comment to get out and see what people say.

Guest

We gotta start stuff. Should I go into hiding? I know I'm gonna get the megakernel community after me.

Shawn Wang

Ah.

Guest 2

One thing I really respect about you is that you're not scared to kick the hornet's nest, ever.

Shawn Wang

It's not. I don't think it's that controversial. I don't know. We'll see.

Guest

We'll see.

Shawn Wang

All right. Thanks, guys.

Guest

No, thank you so much.