Everything you need to run Mission Critical Inference (ft. DeepSeek v3 + SGLang)

Right after Christmas, the Chinese whale bros ended 2024 by dropping the last big model launch of the year: DeepSeek V3. This is a massive 671-billion-parameter fine-grained MoE model with 256 experts, trained with native FP8 mixed-precision training, multi-head latent attention from DeepSeek V2, a new multi-token prediction objective, and 15 trillion tokens of data, including synthetic reasoning data distilled from DeepSeek R1.

Right now, on the LM Arena leaderboard, DeepSeek V3 is rated the seventh-best model in the world, with a score of 1319, right under the full o1 model, Gemini 2, and GPT-4o latest, and above o1-mini, Grok 2, Gemini 1.5 Pro, and Claude 3.5 Sonnet. This makes it the best open-weights model in the world in January 2025. There has been a big recent trend in Chinese labs releasing very large open-weights models, with Tencent releasing Hunyuan Large in November and Hailuo releasing MiniMax Text this January, both over 400B in size.

However, these extra-large language models are very difficult to serve. Baseten was the first of the inference neocloud startups to get DeepSeek V3 online because of its H200 clusters, its close collaboration with the DeepSeek team, and its early support of SGLang, a new vLLM alternative out of UC Berkeley that is also used at frontier labs like xAI. Each H200 has 141 GB of VRAM with 4.8 terabytes per second of bandwidth, meaning that you can use 8 H200s in a node to run inference on DeepSeek V3 in FP8, taking KV-cache needs into account.

We have been close to Baseten since Sarah Guo introduced Amir Haghighat to Swyx, and they supported the very first Latent Space Demo Day in San Francisco, which was effectively the trial run for the podcast you're listening to right now. Since then, Philip Kiely has also led a well-attended workshop on TensorRT-LLM at the 2024 AI Engineer World's Fair. We worked with him to get two of their best representatives, Amir and lead model performance engineer Yineng Zhang, to discuss DeepSeek, SGLang, and everything they have learned running mission-critical inference workloads at scale for some of the largest AI products in the world. Spoiler: Amir thinks there are 3 pillars of mission-critical inference workloads, and we spend quite some time discussing what you need for each of them. In other news, invites are now rolling out for the second AI Engineer Summit in New York City from February 20 to 22. We are bringing back the surprisingly successful AI leadership track from World's Fair, and the AI engineering track is now wholly focused on agents at work. If you are building agents in 2025, this is the single best conference of the year. We are curating all attendees and will sell out after we announce speakers this coming week from DeepMind, Anthropic, OpenAI, Meta, Jane Street, Bloomberg, BlackRock, LinkedIn, and more. Look for more sponsor and attendee information at apply.ai.engineer and see you there. Watch out and take care.

This is our first recording of 2025. I'm Alessio, partner and CTO at Decibel Partners, and I'm joined by my co-host Swyx, founder of Smol AI.

Today we are here with a special double-guest episode with Amir. Oh my God, I don't know your last name. Haghighat?

That's close enough. That was a first try. That's really good.

And Yineng Zhang from Baseten. Welcome.

Thank you.

Thank you.

Amir, we've met before. You're a co-founder of Baseten, which is one of the leading LLM inference platforms. I don't know—what do you consider yourself?

That sounds fine.

And Yineng, you are the lead software engineer on the model performance team, and you guys recently shipped DeepSeek V3 as one of the many models that you host. You are also very involved in SGLang, and that was actually one of the reasons we were discussing an episode with you even before DeepSeek V3 dropped as a Christmas present to everybody.

We can take this in a number of directions, but I think one thing we wanted to get off the bat was to start with DeepSeek, and then we'll work our way backward to SGLang. DeepSeek is more recent. Why are people so interested? What's the history of DeepSeek in general, from your perspective?

1. DeepSeek V3 Meets Production

DeepSeek V3 is currently considered the leading open-source LLM based on the benchmark results and the Chatbot Arena results. It's so big. It's a 671-billion-parameter MoE, and I think it's a game changer for open-source AI, so everyone is interested in this model.

One of the interesting things is that they're a bootstrapped, very private, small lab. They have a lot fewer resources than others. But it's also interesting that it's just open weights. For some reason, the Chinese labs are much better than the American labs at sharing open weights, and that's obviously beneficial for Baseten.

It's in your interest to serve these models at all times. What are the unique challenges that you face offering something this large?

Because the model is very large, if we use something like just H100s, we cannot serve this model. Even if we use 8 H100 cards, we would have 640 gigabytes of memory. The DeepSeek V3 model has 671 billion weights, so even using FP8 precision, you need, I think, 71 gigabytes for the weights, and you also need extra memory for the KV cache. So it's not possible to run that on H100s.

That's why we chose H200s to run that model, or to use multiple nodes to run that model. It's very challenging. Another challenge is that the weights DeepSeek V3 released were in FP8 precision. If you want to run it, you should support that kernel, because I think the default is BF16, which is even larger. If you want to run the FP8 version, you need to support the quantization. I think currently even TensorRT-LLM doesn't support FP8, so if we want to implement that feature, we need to do some feature development.

The last challenging part is that if you want to do some debugging or performance benchmarking, it's very hard. Why? Because the model is so large and the loading time is so long. That makes it more complicated for developers to debug.

Is it complicated or just slow? You've only mentioned loading time, but—

Loading time is slow. You're right.

It's not complicated. It's just—go for more coffee.

Oh, okay.

Okay.

Can you maybe just give people a quick rundown of all the models you support on Baseten, and how it compares just on size? People hear 671 gigabytes, but is that a lot more than other models?

You mentioned BF16 and FP8. What's the usual that you see, and do you see any variation based on model size or anything like that?

I think at Baseten, something like Llama 7B is more common. Llama 3 released the 405B weights, but I think there are just a few users who use that. So before DeepSeek V3, I think we hadn't encountered that issue with such large weights. I think DeepSeek V3 is the first model this big that we should use H200s or multiple H100 nodes for.

Was that because of performance, or why do people not use the 405B Llama? I think what I hear from people is that the performance gains of the 405B at inference time are not worth it, so the 70B is kind of the sweet spot.

Who are the people that use V3? Are they people who were using maybe the Llama 70B model and just want better performance? Are they people who are just experimenting? That's the question people always have. There's always a lot of excitement around open-source models, but maybe the question is: what are they really good for?

2. Why Customers Choose DeepSeek

I can answer this observationally. The interest that we have seen—and some of this is running in production, while some of it is just at the interest level—generally isn't coming from folks who are trying to upgrade from a certain open-source model to DeepSeek V3. We're seeing it, generally speaking, from folks who are coming from Claude and are doing so either because—and I'm going to give you a list of reasons, and generally the reasons are a certain combination of these, in no particular order—they're being rate-limited, the price is too high, or they have certain latency requirements or time-to-first-token requirements for their use case that Claude cannot hit.

They may also want to have full control over the model, as opposed to running it behind an API where the model underneath them can potentially change. There are a couple of other reasons, but generally it's a combination of those.

You mentioned the speed and some of these things. Do customers want to change the hardware also? You're referring to the H200 as kind of the default thing. Do people come to you and say, “Yeah, I'd rather use a smaller system, and can I get worse performance?” Or how do you work with customers on that?

Generally, people come with certain requirements around latency, throughput, and cost.

Generally, they're not coming in saying, “I want this particular GPU SKU.” At least as we go upmarket and talk to foundation model companies, the things that are top of mind for them are those requirements, not a particular GPU SKU.

We're doing the different GPU SKUs not because we want to offer, “Oh, look, we have H200s. Look at us. We have mega H100s. Look at us.” It's not that. It's really because those are the tools to achieve a certain kind of time to first token for certain types of models, a certain kind of throughput and scale, or a certain kind of price per million tokens or per million images, depending on the modality. That's the reason we're talking about GPU SKUs.

3. Native FP8 Meets Fine Grained MoE

I wanted to pick up a little bit on this FP8 thing. It seems like Noam Shazeer started talking about training natively quantized, and I think that's what DeepSeek seems to have done, at least according to their paper. Is this a trend? Is the community settling on one form, one sort of numeric format that everyone knows about? Tell us more about what you're seeing here in terms of the training trends and those sorts of model trends.

I think a lot of companies will also release quantized versions of the Llama models for turbo or lighter inference, just based on different levels of speed. Do you do anything there in terms of quantizing the models that you serve?

I'll let Yineng answer the patterns around using FP8 in training, but I want to draw one distinction that gets to the latter part of your question, Swyx: unlike companies like Together, Fireworks, Replicate, and Baseten, we don't provide a shared inference endpoint for the popular open-source models. That's a product that we don't have on purpose.

Shared inference endpoints for open-source models work really well for situations where the user is saying, “Hey, let me just call a certain popular model behind an API and pay by the token.” That is not our average customer or median customer. Our customers generally have their own custom models and very custom workflows, strict requirements around latency and time to first token, and can't deal with noisy-neighbor problems, such as the API being slow because some other customer has been calling it a lot.

They have other requirements around infrastructure flexibility and regions, whether for latency reasons or compliance reasons. That's the side of the inference market that we capture. At Baseten, when you deploy a model, whether it's your own custom weights or an open-source model, you get dedicated inference and dedicated resources.

When it comes to the quantization question, we would never quantize the model behind the user's back and say, “Look at us, there's a faster and cheaper Llama 70B that has been somehow quantized.” Our customers are coming to us with those requirements that I mentioned, but in particular, when it comes to model quality, they have strict requirements. They would not be okay with us touching the weights, if you will.

We have done things like speculative decoding in a couple of different ways, but all of those methods guarantee that the output is unchanged, as opposed to quantization. When it comes to quantization, we have built tooling that allows our users to quantize their models. For the ones we're working with more hands-on through our forward-deployed engineering team, we're working with them on evals as well to ensure that the quantized models are meeting their requirements. However, this is all very much in conjunction with the engineers who are our customers, as opposed to us doing it behind the scenes.

Oh yeah, FP8 training is very interesting, and I think the DeepSeek team is the first one to use FP8 training for a large model. Before that, maybe 01.AI—sorry, 01.AI.

Yi-Lightning. Yeah.

They used FP8 training. Others, I think most of them used BF16 training, and it's a game changer.

For us, because the FP8 kernel should be implemented for inference, it uses block-wise FP8. Currently, even if you use something like CUDA or cuBLAS, you cannot support that. So you usually use something like Triton to implement the kernel, or something like CUTLASS to implement that kernel. I think that's the challenging part.

My theory is that this will pick up in terms of the models that people release. Increasingly, they won't be BF16. I was trying to look for the quantization paper while you were speaking, but I couldn't find it. There's an ablation of quantization paper that came out last year that showed there are benefits to quantizing and natively training all the way to 6-bit, and even smaller than that might be going too far. I'm not sure if you know what paper I'm talking about, but there's an interesting trend for sure.

Yeah, I think even with FP8 quantization, the benchmark result is very good. On something like GSM8K, the score is nearly 94.6. It's so high. I think it's higher than every other open-source LLM, even Llama 405B.

I'm going to move on a little bit in terms of one other notable detail, and we don't have to speak too much about DeepSeek because obviously we don't know that much unless you work on the team. Another trend they have is fine-grained MoE.

I think there's a question about whether MoEs will be more of a thing. Basically, this time last year, Mixtral was kicking off a bit of an MoE trend with 8x7B and 8x22B. Then, for the rest of the year, there were basically no MoEs. So is this discovery of fine-grained MoE going to be a relevant trend for this year?

Yeah, I think so. As far as I know, some companies, such as Baidu or ByteDance, use the MoE architecture for their internal dominant LLMs. Their weights are similar to the DeepSeek MoE model. So I think this year, MoE inference optimization will be very essential and so important.

At the same time, why haven't the big labs done it? I think Llama 405B is dense. I think Grok is also a dense model. You can correct me if I'm wrong. It's just a weird countertrend.

This time last year, I was writing my recap and I was like, “All right, MoEs seem like they're going to be trending,” and then they did not trend. Anyway, it's just a note that I would flag out there. But I generally agree: it seems like fine-grained MoE is working out, and I definitely want to see more people adopting it.

I went to Jeff Dean's session at NeurIPS, and he also mentioned that one of the Gemini models—I think Gemini 1.5 Pro—is an MoE, which I don't think we knew before that.

Yeah. The reason why Llama hasn't open-sourced an MoE model is because I think they tried to train an MoE model, but they failed. That's why they didn't open-source an MoE model for the Llama series.

What are the causes of failure? Why do MoEs fail?

I think this is another thing that people are talking about, right? The failures of Claude 3.5 Opus, the failures of GPT-5—it's a thing that people are sort of rumoring.

Yeah, because if you want to train a model, for the training stage, you need some benchmark or some score. But for the MoE model, the benchmark score is even lower than the dense model. In that case, they think the MoE model is worse than the dense model, so they didn't release it.

Okay. One more thing that may be more commercially relevant: DeepSeek's API pricing is very competitive. How do you decide pricing in this kind of landscape with open models?

4. Dedicated Inference Drives Pricing

It goes back to the use cases that we serve. Again, going back to the fact that we don't have shared inference endpoints for the different models, our pricing is never per token.

Customers generally come with their own custom models or open-source models, but with strict requirements around latency and time to first token, security and compliance, or a particular scale they're looking for without running into noisy-neighbor problems, things like that. The way that we price has always been based on consumption—based on consumption of resources—and that takes 1 of 2 shapes.

One is the shape where things are running inside our infrastructure. We're running, by the way, on top of multiple different public clouds and in many different regions within those clouds. Then we charge them based on the hardware resources that they're using.

The second shape is that our customer brings their own cloud. We're seeing this more and more, where a customer has big committed resources inside their AWS VPC, GCP, or what have you.

In that world, we also have a consumption model. Of course, the price is very different because they're using their own resources, but we are managing those resources for them. An example that we're seeing more recently is the fact that we've had to build multi-cloud capabilities so that we can have a single model horizontally replicate across different regions and even different clouds.

More and more, as we go upmarket, our customers have their own cloud commits. They are also multi-cloud in order to get good prices and good capacity. It's unreasonable to expect every one of them to build the same multi-cloud capabilities that we have built. So they take advantage of what we have built and use all of the different cloud resources that they have as a whole, holistic unit. Their models can horizontally scale across those resources at inference time and even optionally overflow to our cloud when they start running out of committed resources. All of that has a consumption pricing model to it.

Can we talk about what it takes to actually run your service? We asked this—we had episodes with Replicate, Modal, and Fireworks. We always like to ask this question, obviously, since you're not the model maker: all the secret sauce is in how you actually run the model. I know you also have Truss, which is your more developer-led SDK. Can you quickly run people through how you go from taking the DeepSeek-V3 weights to actually running it? What goes on behind the scenes? Then we can talk about SGLang in a little more depth.

5. Baseten Builds Its Inference Stack

Yeah, totally. As you said, we have Truss, which is our open-source model packaging and deployment library. Truss works with different frameworks underneath it. It has very native and deep support for TensorRT-LLM. Somewhat as an accident of history, we happened to have access to TensorRT-LLM before it was announced, contributed back to it, and we still do. We pushed it to its limits and had to go beyond it in certain areas as well.

For example, if you know the Triton Inference Server, we've had to build our own version of that for performance and reliability reasons. But we invested in it heavily because, for the use cases that we were seeing from our customers, it tended to be the best framework to handle the latency and throughput requirements that we were seeing. In particular, when it comes to the kernels that they come with, I'm yet to see folks do better than what NVIDIA can do when it comes to CUDA kernels.

However, Truss is not tied to TensorRT-LLM. For example, for the DeepSeek example that you mentioned, it's working with SGLang, which is really cool to see, and we will be investing more and more in SGLang, especially as the developer experience is so much better than TensorRT-LLM. We've built a lot around TensorRT-LLM and productized it to make it easier to work with, but SGLang has still been a joy to work with.

Another trend that is really promising, and I learned this from the SGLang folks, is that the TensorRT-LLM folks have promised to modularize a lot of TensorRT-LLM so that other frameworks like SGLang can grab certain parts of it and build on top of it. As a user, you don't have to go all in on one framework versus another. You can really pick and choose based on the requirements that you have.

That's really been our approach as well. We have customers on Baseten that are using TensorRT-LLM, we have ones that are using vLLM, and we have a growing number that are using SGLang too. It's not about really tying yourself to one versus another. It's about using the best of the bunch, depending on the requirements of the customer and their inference workloads.

And those are all models hosted on Baseten, or do you also do chains?

Those have to be models hosted on Baseten. If one of those steps is not hosted on Baseten, then…

Right. Yeah, yeah.

…you're still incurring massive latency on the network side.

Yeah, and then, just to tie this into SGLang, how do you think about the hidden magic? Should people know that you use SGLang? Should people care? Especially for the people building the models, does it matter to them that they use a certain model runtime, or do they not care? Does everything just go through the Baseten platform the same?

Yeah. Should we talk about it? Yes, 100%. We want to be the transparent provider. I don't want to say, “Just give us your model, and voilà, magic,” and ask customers to trust our magic. I want that magic to be very transparent to our customers. That has worked really well for us.

You really need that, especially when you're onboarding foundation model companies. They're not going to turn a blind eye to how things are run underneath the hood. When it comes to customers caring about what's happening underneath, they do, but more than caring about this framework versus that, they care about the final output.

In other words, is the quality the same, or has something changed underneath the hood and the model isn't actually producing the same quality? How is the latency? Especially for certain use cases, what is the time to first token? Is that sustained? What is the P95 of that? What is the P99 of that? How well does it handle throughput? When you start getting a massive burst of traffic, does it still sustain those P95s and P99s?

How do I make sure that the security of the data being sent into the model is guaranteed? How do I make sure compliance is guaranteed for HIPAA use cases? How do I make sure that the data remains within a certain geographic region for compliance reasons or for latency reasons? Those are the concerns that customers are coming to us with, less so about, “Here’s my model. Make sure you run it with TensorRT-LLM,” or, “Make sure you run it with SGLang.”

Yep.

Can you maybe give us an overview of all the different frameworks that people might use? So you have SGLang, TensorRT-LLM, and vLLM. Those are kind of like the open-source research ones. Some of the other commercial companies are building some of their own as well. But what's the state of the art today—maybe the top 3 most popular? Then we can talk about why SGLang came to be, what makes it different, and some of the performance boosts that you get.

Okay. Yeah. I think for the common use case—maybe not the DeepSeek-V3 case—SGLang's performance is better than vLLM, and its usability is better than TensorRT-LLM. When users care about performance and usability, I think they will choose SGLang.

For the DeepSeek-V3 case, because we do a lot of optimization in SGLang—for example, in DeepSeek-V2, they proposed an attention variant named MLA, or Multi-head Latent Attention. I think SGLang is the only framework that supports that. Maybe LightLLM and TensorRT-LLM also support it, but vLLM doesn't support it. Also, in SGLang version 0.4, we support data-parallel attention for DeepSeek.

In the latest SGLang release, we also support the block-wise FP8 kernel, and that kernel was adopted and copied by vLLM later. I think we've done a lot of optimization for DeepSeek. That's why SGLang is the recommended engine by the DeepSeek team.

6. Mission Critical Inference Needs Three Pillars

And maybe one thing to point out—and I think this is important—is that the framework that you choose is part of the equation for running mission-critical inference workloads, but it's only a part of it. Maybe I can draw this out based on my experience and what I've seen in the market as to what it takes to run mission-critical inference workloads in production.

I think it takes 3 things, and each of them individually is necessary but not sufficient. One is performance at the model level. In this case, how fast are you running this one model on a single GPU, let's say? The framework that you use there can matter. The techniques that you use there can matter—the MLA technique, for example, that Yineng mentioned, or the CUDA kernels that are being used.

But there are also techniques being used at a higher level, things like speculative decoding with draft models or with Medusa heads. These are implemented in the different frameworks, or you can even implement them yourself, but they're not necessarily tied to a single framework. Using speculative decoding gives you massive upside when it comes to being able to handle high throughput.

But that's not enough. Invariably, that one model running on a single GPU is going to get too much traffic for it to handle. At that point, you need to horizontally scale it. That's not an ML problem. That's not a PyTorch problem. That's an infrastructure problem. How quickly do you go from a single replica of that model to 5, to 10, to 100? That's the second pillar necessary for running these mission-critical inference workloads.

And what does it take to do that? Some people are like, “You just need Kubernetes, and Kubernetes has an autoscaler, and that just works.” That doesn't work for these kinds of mission-critical inference workloads. You end up catching yourself wanting to rebuild those infrastructure pieces bit by bit from scratch. This has been our experience.

Going even a layer beyond that, Kubernetes runs in a single cluster. It's a single cluster tied to a single region. When it comes to inference workloads and needing GPUs, more and more we're seeing that you cannot meet the demand inside a single region—a single cloud's single region.

In other words, a single model might want to horizontally scale up to 200 replicas, each of which has, let's say, 2 H100s or 4 H100s, or even a full node. You run into limits on the capacity inside that one region. What we had to build to get around that was the ability to have a single model with replicas across different regions.

There are models on Baseten today that have 50 replicas in GCP East, 80 replicas in AWS West, and replicas in Oracle in London, et cetera. That was a big investment that we had to make.

The final one is wrapping the power of the first 2 pillars in a very good developer experience, to be able to support workflows like the ones that I mentioned around multistep, multimodel inference workloads. More and more, we're seeing that the market is moving toward those, and that the needs are generally in these more complex workflows.

These are the 3 pillars that it takes to run mission-critical inference workloads. The choice of the framework—the serving framework—is really a part of the first pillar. That's something I'm seeing in the market: people who are somewhat new to it are like, “Well, vLLM equals production. That's what it takes to run inference workloads.” In practice, that is not true, and I wanted to call that out.

I agree with Amir because I think open-source libraries such as vLLM, SGLang, LightLLM, or TensorRT-LLM only provide a library. They don't provide a product solution.

Yeah. Can we maybe talk about some of the unique things about SGLang? I read through the paper. It sounds like some of the main use cases are when you have very large batches, which makes sense for your use case, and also longer context.

What was the decision behind creating the framework, which I think is around 1 year old? I think the paper came out in December 2023, something like that, so it's still fairly new compared to some of the other ones. Maybe what were some things that you had to change as you built it, or any fun stories?

7. SGLang Optimizes the Serving Stack

Yeah, yeah, yeah. In August 2023, Lianmin and Ying wanted to create SGLang, maybe for the frontend, something like an LLM program. They wanted to solve that problem. In January 2024, they added RadixCache, which is a prefix-caching technology. I think SGLang was the first framework to support prefix caching.

In February, they also added constrained decoding and supported something like Jump-Forward. At that time, it was known as a language generator, not an inference backend. In June or July 2024, we wanted to make SGLang a fully functional LLM inference engine, equivalent to vLLM or TensorRT-LLM.

At that time, we published a blog comparing it with other frameworks, and its performance was amazing. I think its throughput was maybe 3 times that of vLLM. After that, vLLM also did some refactoring to make it faster.

In September and December, we continued to release new versions of SGLang. We supported some DeepSeek optimizations, such as MLA optimization and data-parallel attention optimization. We also supported the zero-overhead CPU scheduler and something like the SGLang Router for cache-aware load balancing. We delivered so many features. We just built and shipped.

I think Lianmin and Ying wanted to create a new framework rather than use the existing solutions, such as vLLM or TensorRT-LLM, because at that time vLLM was easy to use, but its performance was maybe not good. Some of its design, I think, was not okay. The code was a little messy, and if you wanted to extend it with a new feature, it was a little hard.

TensorRT-LLM was blazing fast. Its performance was very good, but it wasn't easy to do secondary development. If you wanted to add a new feature, it was a little hard. So they thought, “How can we create a new framework that achieves good performance and is also easy to develop and maintain?” That's why they created the SGLang project.

Let's run through maybe the 3 main techniques behind SGLang. The first one is RadixAttention, which focuses on the KV cache. When you think about a model as large as DeepSeek-V3, especially, having better KV-cache reuse is great. Can you talk a bit about that performance impact?

Yeah. RadixCache is a prefix-caching technology, and it is a special case where the block size is 1. For vLLM and other frameworks, they use a block size of 32, while SGLang uses a block size of 1. If you use a block size of 1, you can make the cache hit rate higher than in other frameworks. I think that's the main benefit.

For your case specifically, how does that change when you have a Baseten-type use case where you don't have a shared endpoint, versus GPU clouds serving 1 model for many people who have very different use cases? When you have 1 endpoint for 1 customer, I'm sure they have a system prompt that a lot of the requests share and things like that. Anything you want to mention there?

Yeah. We've seen this be massively helpful for the reason that you mentioned. There is a finite number of prompts, or at least prompt prefixes, that are being used per customer. What we've seen is that prefix caching and the different techniques to make that better have been massively helpful.

However, we still had to build on top of that. For example, you have a model with dozens of replicas, each of which has its own state of KV cache. A new request comes in, and what we used to do back in the day was randomly assign that request to one of these replicas. But the better way is to know the state of the KV cache in these different replicas and try to decide which one it should go to.

That is one of the parameters you need to consider. There are other parameters around the size of the queue at each replica and the location of each replica, depending on how geo-aware you want to be. Adding that additional consideration around KV-cache-aware load balancing was something that we saw improve latency quite a bit for our customers.

And then the second part, which was maybe the harder one to understand as a practitioner, was this idea of turning some of the decoding process into a finite-state machine instead of something more open-ended, especially when you're using structured outputs. Can you maybe explain what that means? I would love to learn, too. This is an opportunity for everybody to better understand how you think about going from normal token-by-token decoding to having a more—I wouldn't say precompiled, but pre-understanding of what the paths are going to be.

I think SGLang supports constrained decoding, and it also supports jump-forward. We use something like Outlines or XGrammar to convert the schema from JSON to an FSM, or state machine, and we can use the state machine to control the output.

The output may be in JSON mode or something like that; it should obey some rule. In that case, because the output should obey a rule, you can skip some tokens. You should decode 4 times, but you should obey the rule, or you can get that token in advance. You can just use 1 prefill to replace the 4 decoding steps, for example. So that's why you can jump forward.

I guess the question is: Why doesn't everybody do that? When I was reading about it, I thought, “This just sounds better,” especially for accuracy, since you're constraining structured output. You can also do faster decoding. Are there downsides to it?

Maintaining jump-forward is a little hard. Later, we supported something like CPU overlap. In the overlap model, we even made it compatible with jump-forwarding, because if you want to maintain jump-forward with other features, it becomes more complicated.

We only use it as an optional setting; we disable it by default. But if you want to enable it, you can use some arguments to do that. It's a little hard to maintain, especially when making it compatible with other optimization features.

Just as a side note, you mentioned XGrammar, which I had never heard of. I looked up the GitHub repository, and it's actually from MLC, which we talked to TQ—

Yeah, yeah, yeah.

I think a while ago. Any comparisons between XGrammar and Outlines? Is there a trend in this world, or is it mostly settled science?

To be honest, I prefer XGrammar.

Okay. Yeah. Tell us more.

MLC AI was founded by Tianqi Chen. Both Tianqi Chen and XGrammar's other creator, Yuxuan Dong, were students who graduated from Shanghai Jiao Tong University. The creators of SGLang, Lianmin Zheng and Ying Sheng, also graduated from Shanghai Jiao Tong University.

Oh my God. Is it the Berkeley of China?

Yeah, you're right. I think XGrammar's performance is better than Outlines, and in the latest release of TensorRT-LLM, TensorRT-LLM also integrated XGrammar as the backend for constrained decoding.

Okay. This is new to us. We had Remy from Outlines speak at my past conference, but I wasn't even—

Yeah.

—I wasn't even aware that XGrammar was a thing. But structured output is something that a lot of people care about. We had OpenAI talk about its structured output implementation, and there's a lot of interest in making sure that there are no trade-offs.

I think there's a little bit of FUD around how the models might be dumber when you use structured output instead of base next-token generation. But I don't think it's significant, at least not by that much.

Yeah.

We can talk about the last one, which I don't know if it's as relevant for Baseten: the third technique in SGLang, API speculative execution, which seems to be only for API-only models.

Oh, yeah. I think it's a front-end feature. It's not the back end. You have some control flow for the LLM task. You have 1 request to get a result, and then continue to another call.

For this case, you can use the SGLang front-end language to describe the control flow. It makes it easier to control that pattern.

Tracing this human path, I'm pretty sure I know the answer, but is there a reason big projects like Groq and xAI also use SGLang?

Yeah, yeah, yeah. You're right.

Is it just the same people?

Yeah, yeah, yeah. Right. Lianmin and Ying are members of the technical staff at xAI.

I mean, it makes sense. I wonder what the impetus was for SGLang to break containment. It seems like vLLM obviously has the advantage of being 1 year older and has more community pull. I wonder how this will shake out. I don't really know.

You said vLLM's library is much, much more comprehensive. Do people care? Maybe when you're serving models at scale, you start really prioritizing the performance that SGLang offers.

If you care about performance, maybe TensorRT-LLM is the best solution for now, especially for latency-sensitive scenarios. TensorRT-LLM does well.

But if you also want to implement some features or optimizations yourself and customize the framework, I think SGLang is a good option. vLLM's community support is very nice because it is used by so many users and has so many GitHub stars.

When I joined the SGLang team in July, it had only 2,000 stars, and right now it has more than 7,000 stars. I think it has also grown very fast.

Is there anything that people should look forward to on the SGLang roadmap?

We post the roadmap in an issue, and we pin that issue. We also have a biweekly meeting to sync with the community about our progress and plans, including which features we want to implement in this quarter.

We also co-host some meetups. The first meetup we co-hosted was with MLC LLM and FlashInfer, and we also participate in some hackathons, such as the Camel-AI Hackathon. We gave a presentation about SGLang.

I just saw it now. It sounds like there are—you mentioned EAGLE, and you mentioned Medusa. I think Amir mentioned Medusa, but EAGLE is also part of that cabal of speculative decoding techniques. It looks like you support it now.

Yeah. We already support it. I think in open-source implementations such as vLLM, SGLang, and other frameworks, it has state-of-the-art performance. Currently, even with TensorRT-LLM, it only supports EAGLE-1, not EAGLE-2.

One thing to note about speculative decoding and its different versions is that framework support is one thing, but you also have to do the training of the draft models or the additional heads.

A lot of the benefits will come from how good you are at the training aspect—in terms of the data you use to train the draft model to essentially distill the target model or mimic its behavior so that you can have a very high acceptance rate. The throughput improvement you get ultimately depends on how good a job you do training the draft model in the draft-target-model mechanism.

So that's another thing: whether the framework supports it or you can just turn on speculative decoding with a flag. That's not the case; there's more that goes into it.

One more side note on training. I also noticed that with OpenAI offering fine-tuning for o1 and all these things, I think people are also very interested in RL trainers—is that what you have here? It looks like you're supporting Hugging Face TRL and OpenRLHF.

Do you think this will become something that a lot of people demand? The general field of RL for LLMs was relatively abandoned, I think, up until the end of last year, basically.

Yeah. I think so.

I don't know. It's one of those things where maybe people have to wait for a base model that has some layer looping or some other friendly architecture for reasoning, instead of just pure RL on LLMs.

So far, I don't think people have really exploited RLHF as much in the wild. Correct me if I'm wrong.

Yeah. I can give you some examples of when we've seen it work.

Again, this is generally done by our customers before they come to us for inference. There are examples like, in the healthcare world, fine-tuning models for understanding medical jargon, such as a version of Whisper that can actually understand medical jargon. That’s a non-LLM use case.

In the LLM use case, staying in the healthcare space, there are models that can do medical document extraction and do a very good job compared to even state-of-the-art models because of the data that the company had gathered through a human-in-the-loop process. Is the need for those going to go away because there’s a model that can do reasoning and do a very good job at it? I don’t know. My intuition says yes. Will it be cost-effective? That’s the question that I have. In the short term, no; in the long term, maybe.

But I haven’t seen the need for more traditional fine-tuning actually go down. In fact, we see that quite a bit right now in the market. The question for us is, “Hey, do we want to address that market, knowing that the entire market might go away one day?” My general answer to that is, “Let’s solve today’s problems.” Even if they’re not around in 2 years, you will learn a lot along the way by onboarding customers that have today’s problems. You learn from them about tomorrow’s problems, and you will build ahead for it.

Why do you think fine-tuning might go away?

Because, like you said, there are going to be models with complex reasoning capabilities that can actually figure it out in a few-shot kind of way without needing a large data set to fine-tune the model with. That’s what some people are saying.

I really have trouble believing that that will be the case. I much more believe that it’s just easier to change your prompts rather than actually do full fine-tunes or even parameter-efficient fine-tunes.

For sure.

Is there anything else that we haven’t touched on that you wish people asked you more about? It’s something that’s very interesting from your point of view, in terms of what you’re seeing among your community.

When we released the DeepSeek V3 support, we had some community users, like Cursor. Do you know Cursor? I think it’s very popular.

Of course. I use it every day. When I type “code dot” inside my terminal, it actually opens Cursor instead of VS Code. I feel very bad—

Yeah, yeah, yeah.

—for VS Code.

When we released the DeepSeek V3 support, an employee from the Cursor team was also very interested in our implementation and reached out to ask us some questions. As SGLang grows faster and we optimize the features, we iterate so fast. I think there will be more users from different companies and different teams using it.

Honestly, I would go back to what I emphasized earlier, which was that I wish more people asked about what it takes to run mission-critical inference workloads. I see this in the market sometimes: they’re like, “Well, I can just use vLLM, and that puts my model behind an API, and that is production.” But really, it takes 3 pillars that all need to be there. One is performance at the model level. That is where the frameworks that we talked about today really help you, but you still have to guide them when it comes to speculative decoding.

Yes, they support it, but who’s going to train or fine-tune the draft model or the Medusa heads? Who’s going to ensure the reliability of the vLLM server that you see in production? There are crashes. How do you recover from those without affecting production traffic? By itself, that’s not enough, because invariably that 1 model running on a set of hardware is going to get too much traffic for it to handle, and at that point you need to horizontally scale it.

That’s not an ML problem, and it’s not a PyTorch problem. That is an infrastructure problem: ensuring that you can horizontally scale up your model extremely fast to meet your P90 and P99 latency requirements. To ensure that you’re not running out of capacity in a single region where that model lives, you end up having to scale that model across different regions and even across different clouds, to ensure that the model is not being starved of resources in the one place where it lives. That’s an area of investment that we started investing in some time ago, and it really paid off this past year.

The third pillar is enablement of workflows. Workflows such as the AI phone call example that I told you about require multi-step, multi-model inference, but in a very low-latency way. That’s the third pillar that really allows developers to use the power of the first 2 pillars and combine them, especially when you need multiple models for your workflows, doing so in a reliable, repeatable, and low-latency way. Those are the 3 pillars, honestly, that we have been investing a lot in.

Some of which we started investing in 3 years ago, and it really started paying off a year ago. It takes quite a bit of building to get to the point where you’re truly running customers’ mission-critical inference workloads. What do I mean by mission-critical inference workloads? Inference where, if inference is slow or down, the main product of our customer is slow or down, so they really care about it.

They have strict requirements around latency, around being able to support large throughput, and about being able to do so in a way that other customers’ usage doesn’t affect the SLAs they are getting, including dealing with noisy-neighbor problems. They need inference done in a compliant way, whether it’s HIPAA or certain SOC requirements. They also need inference done in a geo-aware kind of way, both for compliance reasons and for latency reasons, where the location to which you forward the traffic has an impact on latency in situations where 50 milliseconds really matter, 100 milliseconds really matter, and we’re seeing more and more of those use cases.

One way I would recommend doing that is a manifesto-type thing. I’m sure you know Heroku’s Twelve-Factor App.

I’ve seen that, yes. That’s a good idea, actually.

Maybe even put it on a separate property from Baseten and just go, “Here’s what we think a mission-critical AI application should be,” and have some thought leadership there, flesh it out, and see if the market takes it on as a mission. Obviously, you will be best prepared to serve that market as well.

I’ve also seen this done very well with EnterpriseReady.io. I think it used to be done by—I think it was called Gravitational or Replicated—

Yeah, yeah.

—one of those.

From Replicated.

These kinds of things, when you have a list of requirements, when you’re like, “Look, everybody needs this,” write them up and put a little bit of marketing on it. Spit it out from the main company brand. That tends to work very well.

Yeah.

Good idea.

Cool. Thanks so much for your time. I think this was a really good dive into both Baseten and SGLang, and a little bit of DeepSeek V3, which people are very interested in. I’m trying to talk to them as well because they’re a fascinating lab. I think you guys are doing a lot to make it accessible for everyone, so thank you so much.

And, just to give Baseten some street cred, they were one of the first sponsors for Latent Space events, and Amir brought 100 croissants to our Latent Space Hackathon in 2023.

Ah.

I just want to bring up how I saw Phil and Ed at, I believe, re:Invent, and I told them that was one of the first events that we really did and one of the turning points of this industry, as far as community goes, in my mind. Everybody was there.

The croissants?

No, not the croissants. The event itself.

Yeah, entire companies launched that day.

Yeah, I mean, Nader from Brev was there, and he and Joseph from Roboflow did the prompt battle thing. Harrison was a judge, and Jerry from LlamaIndex was there. It was kind of like everybody who is now breaking out. If you look at the graph that Jensen put on the screen at CES with some of the companies they work with, a lot of them were at that event.

So, thanks for staying involved with us, Amir, and I’m sure we’ll do more together. Thank you guys.

Many more years to come, for sure.

Thank you for taking the time today.

Good to see you both. I’ll see you, Oshan.