运行关键任务推理所需的一切（嘉宾：DeepSeek v3 + SGLang）
DeepSeek V3 让前沿级质量的开放权重模型具备了商业意义，但前提是运营方能消化一个6710亿参数的系统工程难题。 录制时，它在 LM Arena 以1319分排名第7，超过 Claude 3.5 Sonnet 和 o1-mini；Yineng Zhang 认为，256专家 MoE 是「开源 AI 的游戏规则改变者」（“a game changer for open-source AI”）。
需求与其说来自 Llama 用户升级，不如说来自希望降低价格、减少限流、缩短首 token 时间，或掌控模型而非依赖可能改变模型的 API 供应商的 Claude 客户。 因此，DeepSeek V3 服务的是有明确生产约束的专有模型用户，而不只是开放模型试验者。
眼下的瓶颈是内存和精度。 Yineng 表示，8张 H100 提供640GB显存，仍无法服务该模型；在同一解释中，他估算 FP8 权重为「我觉得是71GB」，另需为 KV-cache 留出内存。Baseten 因此选择 H200 或多节点部署，而块级 FP8 需要普通 CUDA 或 cuBLAS 路径不支持的内核。
Baseten 的商业差异化在于按用量计费的专属推理服务，而非共享式 token 端点。 客户指定的是延迟、吞吐、P95/P99、安全、合规和地域，而不是某个偏好的 GPU SKU；Baseten 表示，在质量必须保持不变时，不会背着客户量化模型。
SGLang 在常见场景下性能优于 vLLM，同时比 TensorRT-LLM 更易用。 其针对 DeepSeek 的 MLA、数据并行注意力和块级 FP8 工作，使其成为 DeepSeek 推荐的推理引擎；radix caching 与缓存感知路由则通过应对重复 prompt 和多副本来压低延迟。
Amir Haghighat 的核心判断是，关键任务推理需要3根各自不可或缺的支柱：模型级性能、快速跨地域横向扩容，以及面向工作流的开发者工具。 “vLLM 等于生产环境”是误解；生产环境还意味着从崩溃中恢复、承接流量突增、守住尾部延迟、规避 noisy neighbor，并在不同云上找到 GPU 容量。
仅仅因为框架暴露了 speculative decoding 或 fine-tuning 功能，并不意味着它们可以开箱即用。 Draft model 以及 Medusa、EAGLE 头仍需训练才能获得较高接受率；Amir 对 fine-tuning 的判断是「短期内不行，长期来看也许吧」，主持人则怀疑推理模型是否会消除定制需求。
1. DeepSeek V3 让开放权重成为可信的专有模型替代品
DeepSeek V3 在圣诞节后不久发布，是一个6710亿参数、256专家的细粒度 MoE，使用15万亿 tokens 训练，采用原生 FP8 混合精度训练、多 token 预测、源自 DeepSeek V2 的多头潜在注意力，以及从 DeepSeek R1 蒸馏的合成推理数据。Yineng 表示：「我觉得这是开源 AI 的游戏规则改变者」（“I think it’s a game changer for open-source AI.”）。
录制时，其 LM Arena 得分为1319，排名全球第7，低于 full o1、Gemini 2 和最新 GPT-4o，但高于 o1-mini、Grok 2、Gemini 1.5 Pro 和 Claude 3.5 Sonnet。主持人将其称为2025年1月最领先的开放权重模型。
主持人将这次发布放在中国开放权重模型扩张的大趋势中观察：Tencent 的 Hunyuan Large 于11月发布，Hailuo 的 MiniMax Text 于1月发布，两者参数量都超过4000亿。
Amir 的关键商业观察是，市场兴趣通常不是来自正在从某个特定开源模型升级的团队，而是来自 Claude 用户：他们面临限流、价格高、延迟或首 token 时间要求，以及担心 API 供应商可能改变底层模型等问题。
主持人认为，Llama 405B 的推理性能提升往往不足以覆盖其成本。Amir 回应称，对 DeepSeek 的兴趣主要来自遭遇这些生产痛点的专有模型用户，但其中一部分需求仍停留在探索阶段，尚未部署。
2. 6710亿参数 MoE 把内存、精度和内核变成产品约束
Baseten 被描述为首家将 DeepSeek V3 上线的推理云初创公司，这得益于其 H200 集群、与 DeepSeek 团队的合作，以及早期的 SGLang 支持。主持人表示，8张 H200、每张配备141GB显存和4.8TB/s带宽，在计入 KV-cache 需求后可以运行 FP8 推理。
Yineng 表示，8张 H100 提供640GB显存，但仍无法服务该模型。在同一解释中，他估算 FP8 权重为「我觉得是71GB」，并称还需要为 KV cache 预留额外内存。因此，Baseten 选择了 H200 或多节点部署；DeepSeek V3 是他们此前遇到的首个这一规模的模型。
发布的权重采用块级 FP8，而默认路径是 BF16。Yineng 表示，即便 CUDA 或 cuBLAS 也不支持这种块级格式，实施者因此需要转向 Triton 或 CUTLASS 内核；他还称当时 TensorRT-LLM 尚不支持 FP8。由于加载这个超大模型耗时极长，调试非常困难，不过 Amir 认为问题在于速度慢，而非概念复杂。
Yineng 报告称，FP8 版本的 GSM8K 得分接近94.6；在他的比较中，这一成绩高于包括 Llama 405B 在内的其他开源 LLM。Amir 预计原生低精度训练会逐渐普及，并试探性提到一项消融实验：收益在接近6-bit精度时仍然存在，但更小的格式可能就会走得太远。
主持人没有放弃对 MoE 的反例：Mixtral 一度被视为 MoE 浪潮的先声，但随后一年大部分时间仍由 dense 模型主导。Yineng 预计 MoE 推理优化将变得不可或缺，并提到 Baidu 和 ByteDance 内部存在类似架构。他推测 Llama 团队曾尝试训练 MoE 但失败了，并称其基准分数低于 dense 模型，这解释了为什么没有 Llama MoE 发布。
3. 专属推理定价服务的是服务保障，而非 token
Baseten 有意不为热门模型销售共享端点。其典型客户带来定制权重或开放权重，但需要专属资源、可预测延迟、地域灵活性，以及不受 noisy neighbor 影响的环境。对热门模型采用按 token 计费的 API，服务的是另一类客户；如果产品依赖专属推理，需求完全不同。
客户通常指定质量、延迟、吞吐、首 token 时间和成本，而不是点名要求 H200。GPU SKU 是实现每百万 tokens 或每张图片价格、并在流量突增时维持 P95/P99 性能的工具，而不是为了宣传库存本身。
模型保真度也是这份契约的一部分。Amir 表示，Baseten 不会背着用户量化模型；量化工具和评测会与客户工程师共同使用。Baseten 的 speculative decoding 方法之所以有吸引力，在于它能保证输出不变，而量化做不到这一点。
定价按照资源消耗执行，具体有两种形态：由 Baseten 在多个公有云上托管基础设施，或在客户的 AWS、GCP 等云环境内提供托管推理。客户可以跨云组合承诺容量，将同一个模型横向复制到不同云上，并在承诺额度用尽时选择溢出到 Baseten 的容量。
4. SGLang 正在赢得那些既要性能、又要可改造性的场景
SGLang 约在2023年8月起步，最初是用于编写 LLM 程序的前端语言。Radix caching 于2024年1月推出，约束解码和 jump-forward 于2月推出，团队在6月或7月左右开始推动其成为完整推理引擎。Yineng 用一句话概括其节奏：「我们就是边做边发」（“We just built and shipped.”）。
产品设计上的差距很清楚：vLLM 易于使用，但团队认为它难以干净地扩展，之后速度也更慢；TensorRT-LLM「快得惊人」，却不易进行二次开发。SGLang 试图把性能和可维护性结合起来；据称，2024年年中的一次对比中，其吞吐量约为 vLLM 的3倍，之后 vLLM 完成了重构。
针对 DeepSeek，SGLang 加入了 MLA 优化、0.4版本的数据并行注意力、零开销 CPU 调度、缓存感知路由，以及后来被 vLLM 采用的块级 FP8 内核。这些投入让 SGLang 成为 DeepSeek 团队推荐的引擎，而不只是另一个兼容后端。
Yineng 对框架的排序仍然取决于场景：极度看重延迟的工作负载选择 TensorRT-LLM；既需要定制又需要优化时选择 SGLang；看重成熟社区时选择 vLLM。SGLang 在7月后从约2000颗 GitHub stars 增长到超过7000颗；其创建者在 xAI 任职，也部分解释了它在那里获得的采用。
Baseten 的 Truss 可以在 TensorRT-LLM、vLLM 和 SGLang 之间打包并部署模型。Amir 表示，Baseten 还构建了自己的 Triton Inference Server 变体，以提升性能和可靠性；他们把这些框架当作可以互换的工具，而不是将客户锁定在某一个框架上。
5. 缓存复用与约束解码把重复结构转化为速度
RadixCache 是 SGLang 的前缀缓存机制，使用大小为1的 block，而 vLLM 和其他框架使用的 block 大小为32。Yineng 的因果判断很直接：更细粒度的匹配能提高缓存命中率，尤其适用于专属客户反复发送有限组 system prompt 或共享前缀的场景。
当同一个模型拥有几十个副本、每个副本持有不同的 KV-cache 状态时，单纯缓存已经不够。Baseten 因此增加了缓存感知负载均衡：路由请求时同时参考可能的前缀复用、队列深度和副本位置。Amir 表示，这实质性改善了客户延迟。
对于结构化输出，SGLang 可以通过 Outlines 或 XGrammar 将 JSON schema 转换为有限状态机。如果允许的路径使多个 token 变得可预测，jump-forward 就能用一次 prefill 替代多个解码步骤。Yineng 基于性能偏好 XGrammar，并指出 TensorRT-LLM 也已集成该工具。
主持人追问：如果 jump-forward 同时提升速度和结构合法性，为什么不是普遍配置？答案涉及维护复杂度。它与 CPU overlap 及其他优化交互不佳，因此 SGLang 默认将其关闭。其 API speculative-execution 功能是前端的控制流抽象，而不是后端解码优化。
6. 生产级推理建立在3根彼此独立的支柱上
Amir 的第一根支柱是模型级性能：内核、服务框架、MLA，以及 draft model、Medusa 或 EAGLE 等 speculative decoding 技术。框架选择在这里很重要，但它只覆盖一组硬件上的一个模型。「每一项单独来看都是必要条件，但都不是充分条件。」
当流量压垮单个副本时，第二根支柱才开始发挥作用。从1个实例扩展到5个、10个或100个，「不是 ML 问题」，也「不是 PyTorch 问题」，而是基础设施问题。单个模型可能需要200个副本，每个副本消耗2张或4张 H100，甚至占用整台节点，超出单一区域的可用容量。
因此，Baseten 在单个 Kubernetes 集群之外构建了跨地域、跨云复制。Amir 描述过这样的模型部署：GCP East 有50个副本，AWS West 有80个，Oracle 的伦敦区域还有更多副本。他认为，仅靠 Kubernetes 自动扩缩容无法提供这些工作负载所需的速度或可靠性。
第三根支柱是通过强大的开发者体验，支持复杂、多步骤、多模型的工作流。Amir 表示，市场正在转向这类工作负载，而框架只是第一根支柱的一部分。
7. 单个库不是关键任务推理产品
Yineng 明确区分了 vLLM、SGLang、LightLLM 和 TensorRT-LLM 等开源库与完整产品方案。Amir 的概括是：「vLLM 等于生产环境」是错的（“vLLM equals production”）；生产环境还需要可靠地从崩溃中恢复、快速扩容，并在流量突增时持续维持 P95/P99 延迟。
Amir 以 AI 电话呼叫为例，说明这是一种需要低延迟协调的多步骤、多模型工作负载。更广泛地说，客户关心推理是否持续可靠、可重复、安全，在需要时符合 HIPAA，位于规定地域内，并且不受另一客户流量影响。
他对关键任务的定义很严格：如果推理变慢或宕机，「客户的主要产品就会变慢或宕机」。在新兴的实时应用中，路由决策很重要，因为50或100毫秒都很关键；因此，基础设施、合规和地域都会成为模型产品特性。
8. 优化仍需训练，fine-tuning 或许不会消失
EAGLE、Medusa 和 draft-target speculative decoding 都不是打开一个开关就能完成的加速。Yineng 表示，SGLang 和其他开源框架支持业界领先的 EAGLE 性能，而 TensorRT-LLM 支持 EAGLE-1，但不支持 EAGLE-2。客户仍需训练 draft model 或新增的头，使其模仿目标模型；最终吞吐增益取决于 token 接受率。
传统定制在生产环境中仍然存在：Amir 提到针对医疗术语进行 fine-tuning 的 Whisper，以及利用 human-in-the-loop 数据训练、用于医疗文档提取的 LLM。能力更强的推理模型或许能从少量示例中复现这类行为，但他尚未看到当前 fine-tuning 需求下降。
Amir 的不确定性仍未改变：推理能力可能降低 fine-tuning 的需求，但「短期内不行，长期来看也许吧」，其成本效益仍未解决。主持人更为怀疑，认为 prompt 更容易修改，并不意味着完整 fine-tuning 或参数高效 fine-tuning 会消失。
运营结论是务实的，而非预测性的：「先解决今天的问题。」即便某个市场在2年后消失，为其客户完成接入也能让供应商了解未来需求，并获得提前构建相应能力的机会。
核验说明
- 访谈文字同时记录了 Yineng 关于8张 H100 无法服务该模型的说法，以及他对 FP8 权重「我觉得是71GB」的估算。本摘要保留了这两项说法，没有用推导出的数值替换访谈中的原始数字。
Right after Christmas, the Chinese whale bros ended 2024 by dropping the last big model launch of the year: DeepSeek V3. This is a massive 671-billion-parameter fine-grained MoE model with 256 experts, trained with native FP8 mixed-precision training, multi-head latent attention from DeepSeek V2, a new multi-token prediction objective, and 15 trillion tokens of data, including synthetic reasoning data distilled from DeepSeek R1.
Right now, on the LM Arena leaderboard, DeepSeek V3 is rated the seventh-best model in the world, with a score of 1319, right under the full o1 model, Gemini 2, and GPT-4o latest, and above o1-mini, Grok 2, Gemini 1.5 Pro, and Claude 3.5 Sonnet. This makes it the best open-weights model in the world in January 2025. There has been a big recent trend in Chinese labs releasing very large open-weights models, with Tencent releasing Hunyuan Large in November and Hailuo releasing MiniMax Text this January, both over 400B in size.
However, these extra-large language models are very difficult to serve. Baseten was the first of the inference neocloud startups to get DeepSeek V3 online because of its H200 clusters, its close collaboration with the DeepSeek team, and its early support of SGLang, a new vLLM alternative out of UC Berkeley that is also used at frontier labs like xAI. Each H200 has 141 GB of VRAM with 4.8 terabytes per second of bandwidth, meaning that you can use 8 H200s in a node to run inference on DeepSeek V3 in FP8, taking KV-cache needs into account.
We have been close to Baseten since Sarah Guo introduced Amir Haghighat to Swyx, and they supported the very first Latent Space Demo Day in San Francisco, which was effectively the trial run for the podcast you're listening to right now. Since then, Philip Kiely has also led a well-attended workshop on TensorRT-LLM at the 2024 AI Engineer World's Fair. We worked with him to get two of their best representatives, Amir and lead model performance engineer Yineng Zhang, to discuss DeepSeek, SGLang, and everything they have learned running mission-critical inference workloads at scale for some of the largest AI products in the world. Spoiler: Amir thinks there are 3 pillars of mission-critical inference workloads, and we spend quite some time discussing what you need for each of them. In other news, invites are now rolling out for the second AI Engineer Summit in New York City from February 20 to 22. We are bringing back the surprisingly successful AI leadership track from World's Fair, and the AI engineering track is now wholly focused on agents at work. If you are building agents in 2025, this is the single best conference of the year. We are curating all attendees and will sell out after we announce speakers this coming week from DeepMind, Anthropic, OpenAI, Meta, Jane Street, Bloomberg, BlackRock, LinkedIn, and more. Look for more sponsor and attendee information at apply.ai.engineer and see you there. Watch out and take care.
This is our first recording of 2025. I'm Alessio, partner and CTO at Decibel Partners, and I'm joined by my co-host Swyx, founder of Smol AI.
Today we are here with a special double-guest episode with Amir. Oh my God, I don't know your last name. Haghighat?
That's close enough. That was a first try. That's really good.
And Yineng Zhang from Baseten. Welcome.
Thank you.
Thank you.
Amir, we've met before. You're a co-founder of Baseten, which is one of the leading LLM inference platforms. I don't know—what do you consider yourself?
That sounds fine.
And Yineng, you are the lead software engineer on the model performance team, and you guys recently shipped DeepSeek V3 as one of the many models that you host. You are also very involved in SGLang, and that was actually one of the reasons we were discussing an episode with you even before DeepSeek V3 dropped as a Christmas present to everybody.
We can take this in a number of directions, but I think one thing we wanted to get off the bat was to start with DeepSeek, and then we'll work our way backward to SGLang. DeepSeek is more recent. Why are people so interested? What's the history of DeepSeek in general, from your perspective?
1. DeepSeek V3 Meets Production
DeepSeek V3 is currently considered the leading open-source LLM based on the benchmark results and the Chatbot Arena results. It's so big. It's a 671-billion-parameter MoE, and I think it's a game changer for open-source AI, so everyone is interested in this model.
One of the interesting things is that they're a bootstrapped, very private, small lab. They have a lot fewer resources than others. But it's also interesting that it's just open weights. For some reason, the Chinese labs are much better than the American labs at sharing open weights, and that's obviously beneficial for Baseten.
It's in your interest to serve these models at all times. What are the unique challenges that you face offering something this large?
Because the model is very large, if we use something like just H100s, we cannot serve this model. Even if we use 8 H100 cards, we would have 640 gigabytes of memory. The DeepSeek V3 model has 671 billion weights, so even using FP8 precision, you need, I think, 71 gigabytes for the weights, and you also need extra memory for the KV cache. So it's not possible to run that on H100s.
That's why we chose H200s to run that model, or to use multiple nodes to run that model. It's very challenging. Another challenge is that the weights DeepSeek V3 released were in FP8 precision. If you want to run it, you should support that kernel, because I think the default is BF16, which is even larger. If you want to run the FP8 version, you need to support the quantization. I think currently even TensorRT-LLM doesn't support FP8, so if we want to implement that feature, we need to do some feature development.
The last challenging part is that if you want to do some debugging or performance benchmarking, it's very hard. Why? Because the model is so large and the loading time is so long. That makes it more complicated for developers to debug.
Is it complicated or just slow? You've only mentioned loading time, but—
Loading time is slow. You're right.
It's not complicated. It's just—go for more coffee.
Oh, okay.
Okay.
Can you maybe just give people a quick rundown of all the models you support on Baseten, and how it compares just on size? People hear 671 gigabytes, but is that a lot more than other models?
You mentioned BF16 and FP8. What's the usual that you see, and do you see any variation based on model size or anything like that?
I think at Baseten, something like Llama 7B is more common. Llama 3 released the 405B weights, but I think there are just a few users who use that. So before DeepSeek V3, I think we hadn't encountered that issue with such large weights. I think DeepSeek V3 is the first model this big that we should use H200s or multiple H100 nodes for.
Was that because of performance, or why do people not use the 405B Llama? I think what I hear from people is that the performance gains of the 405B at inference time are not worth it, so the 70B is kind of the sweet spot.
Who are the people that use V3? Are they people who were using maybe the Llama 70B model and just want better performance? Are they people who are just experimenting? That's the question people always have. There's always a lot of excitement around open-source models, but maybe the question is: what are they really good for?
2. Why Customers Choose DeepSeek
I can answer this observationally. The interest that we have seen—and some of this is running in production, while some of it is just at the interest level—generally isn't coming from folks who are trying to upgrade from a certain open-source model to DeepSeek V3. We're seeing it, generally speaking, from folks who are coming from Claude and are doing so either because—and I'm going to give you a list of reasons, and generally the reasons are a certain combination of these, in no particular order—they're being rate-limited, the price is too high, or they have certain latency requirements or time-to-first-token requirements for their use case that Claude cannot hit.
They may also want to have full control over the model, as opposed to running it behind an API where the model underneath them can potentially change. There are a couple of other reasons, but generally it's a combination of those.
You mentioned the speed and some of these things. Do customers want to change the hardware also? You're referring to the H200 as kind of the default thing. Do people come to you and say, “Yeah, I'd rather use a smaller system, and can I get worse performance?” Or how do you work with customers on that?
Generally, people come with certain requirements around latency, throughput, and cost.
Generally, they're not coming in saying, “I want this particular GPU SKU.” At least as we go upmarket and talk to foundation model companies, the things that are top of mind for them are those requirements, not a particular GPU SKU.
We're doing the different GPU SKUs not because we want to offer, “Oh, look, we have H200s. Look at us. We have mega H100s. Look at us.” It's not that. It's really because those are the tools to achieve a certain kind of time to first token for certain types of models, a certain kind of throughput and scale, or a certain kind of price per million tokens or per million images, depending on the modality. That's the reason we're talking about GPU SKUs.
3. Native FP8 Meets Fine Grained MoE
I wanted to pick up a little bit on this FP8 thing. It seems like Noam Shazeer started talking about training natively quantized, and I think that's what DeepSeek seems to have done, at least according to their paper. Is this a trend? Is the community settling on one form, one sort of numeric format that everyone knows about? Tell us more about what you're seeing here in terms of the training trends and those sorts of model trends.
I think a lot of companies will also release quantized versions of the Llama models for turbo or lighter inference, just based on different levels of speed. Do you do anything there in terms of quantizing the models that you serve?
I'll let Yineng answer the patterns around using FP8 in training, but I want to draw one distinction that gets to the latter part of your question, Swyx: unlike companies like Together, Fireworks, Replicate, and Baseten, we don't provide a shared inference endpoint for the popular open-source models. That's a product that we don't have on purpose.
Shared inference endpoints for open-source models work really well for situations where the user is saying, “Hey, let me just call a certain popular model behind an API and pay by the token.” That is not our average customer or median customer. Our customers generally have their own custom models and very custom workflows, strict requirements around latency and time to first token, and can't deal with noisy-neighbor problems, such as the API being slow because some other customer has been calling it a lot.
They have other requirements around infrastructure flexibility and regions, whether for latency reasons or compliance reasons. That's the side of the inference market that we capture. At Baseten, when you deploy a model, whether it's your own custom weights or an open-source model, you get dedicated inference and dedicated resources.
When it comes to the quantization question, we would never quantize the model behind the user's back and say, “Look at us, there's a faster and cheaper Llama 70B that has been somehow quantized.” Our customers are coming to us with those requirements that I mentioned, but in particular, when it comes to model quality, they have strict requirements. They would not be okay with us touching the weights, if you will.
We have done things like speculative decoding in a couple of different ways, but all of those methods guarantee that the output is unchanged, as opposed to quantization. When it comes to quantization, we have built tooling that allows our users to quantize their models. For the ones we're working with more hands-on through our forward-deployed engineering team, we're working with them on evals as well to ensure that the quantized models are meeting their requirements. However, this is all very much in conjunction with the engineers who are our customers, as opposed to us doing it behind the scenes.
Oh yeah, FP8 training is very interesting, and I think the DeepSeek team is the first one to use FP8 training for a large model. Before that, maybe 01.AI—sorry, 01.AI.
Yi-Lightning. Yeah.
They used FP8 training. Others, I think most of them used BF16 training, and it's a game changer.
For us, because the FP8 kernel should be implemented for inference, it uses block-wise FP8. Currently, even if you use something like CUDA or cuBLAS, you cannot support that. So you usually use something like Triton to implement the kernel, or something like CUTLASS to implement that kernel. I think that's the challenging part.
My theory is that this will pick up in terms of the models that people release. Increasingly, they won't be BF16. I was trying to look for the quantization paper while you were speaking, but I couldn't find it. There's an ablation of quantization paper that came out last year that showed there are benefits to quantizing and natively training all the way to 6-bit, and even smaller than that might be going too far. I'm not sure if you know what paper I'm talking about, but there's an interesting trend for sure.
Yeah, I think even with FP8 quantization, the benchmark result is very good. On something like GSM8K, the score is nearly 94.6. It's so high. I think it's higher than every other open-source LLM, even Llama 405B.
I'm going to move on a little bit in terms of one other notable detail, and we don't have to speak too much about DeepSeek because obviously we don't know that much unless you work on the team. Another trend they have is fine-grained MoE.
I think there's a question about whether MoEs will be more of a thing. Basically, this time last year, Mixtral was kicking off a bit of an MoE trend with 8x7B and 8x22B. Then, for the rest of the year, there were basically no MoEs. So is this discovery of fine-grained MoE going to be a relevant trend for this year?
Yeah, I think so. As far as I know, some companies, such as Baidu or ByteDance, use the MoE architecture for their internal dominant LLMs. Their weights are similar to the DeepSeek MoE model. So I think this year, MoE inference optimization will be very essential and so important.
At the same time, why haven't the big labs done it? I think Llama 405B is dense. I think Grok is also a dense model. You can correct me if I'm wrong. It's just a weird countertrend.
This time last year, I was writing my recap and I was like, “All right, MoEs seem like they're going to be trending,” and then they did not trend. Anyway, it's just a note that I would flag out there. But I generally agree: it seems like fine-grained MoE is working out, and I definitely want to see more people adopting it.
I went to Jeff Dean's session at NeurIPS, and he also mentioned that one of the Gemini models—I think Gemini 1.5 Pro—is an MoE, which I don't think we knew before that.
Yeah. The reason why Llama hasn't open-sourced an MoE model is because I think they tried to train an MoE model, but they failed. That's why they didn't open-source an MoE model for the Llama series.
What are the causes of failure? Why do MoEs fail?
I think this is another thing that people are talking about, right? The failures of Claude 3.5 Opus, the failures of GPT-5—it's a thing that people are sort of rumoring.
Yeah, because if you want to train a model, for the training stage, you need some benchmark or some score. But for the MoE model, the benchmark score is even lower than the dense model. In that case, they think the MoE model is worse than the dense model, so they didn't release it.
Okay. One more thing that may be more commercially relevant: DeepSeek's API pricing is very competitive. How do you decide pricing in this kind of landscape with open models?
4. Dedicated Inference Drives Pricing
It goes back to the use cases that we serve. Again, going back to the fact that we don't have shared inference endpoints for the different models, our pricing is never per token.
Customers generally come with their own custom models or open-source models, but with strict requirements around latency and time to first token, security and compliance, or a particular scale they're looking for without running into noisy-neighbor problems, things like that. The way that we price has always been based on consumption—based on consumption of resources—and that takes 1 of 2 shapes.
One is the shape where things are running inside our infrastructure. We're running, by the way, on top of multiple different public clouds and in many different regions within those clouds. Then we charge them based on the hardware resources that they're using.
The second shape is that our customer brings their own cloud. We're seeing this more and more, where a customer has big committed resources inside their AWS VPC, GCP, or what have you.
In that world, we also have a consumption model. Of course, the price is very different because they're using their own resources, but we are managing those resources for them. An example that we're seeing more recently is the fact that we've had to build multi-cloud capabilities so that we can have a single model horizontally replicate across different regions and even different clouds.
More and more, as we go upmarket, our customers have their own cloud commits. They are also multi-cloud in order to get good prices and good capacity. It's unreasonable to expect every one of them to build the same multi-cloud capabilities that we have built. So they take advantage of what we have built and use all of the different cloud resources that they have as a whole, holistic unit. Their models can horizontally scale across those resources at inference time and even optionally overflow to our cloud when they start running out of committed resources. All of that has a consumption pricing model to it.
Can we talk about what it takes to actually run your service? We asked this—we had episodes with Replicate, Modal, and Fireworks. We always like to ask this question, obviously, since you're not the model maker: all the secret sauce is in how you actually run the model. I know you also have Truss, which is your more developer-led SDK. Can you quickly run people through how you go from taking the DeepSeek-V3 weights to actually running it? What goes on behind the scenes? Then we can talk about SGLang in a little more depth.
5. Baseten Builds Its Inference Stack
Yeah, totally. As you said, we have Truss, which is our open-source model packaging and deployment library. Truss works with different frameworks underneath it. It has very native and deep support for TensorRT-LLM. Somewhat as an accident of history, we happened to have access to TensorRT-LLM before it was announced, contributed back to it, and we still do. We pushed it to its limits and had to go beyond it in certain areas as well.
For example, if you know the Triton Inference Server, we've had to build our own version of that for performance and reliability reasons. But we invested in it heavily because, for the use cases that we were seeing from our customers, it tended to be the best framework to handle the latency and throughput requirements that we were seeing. In particular, when it comes to the kernels that they come with, I'm yet to see folks do better than what NVIDIA can do when it comes to CUDA kernels.
However, Truss is not tied to TensorRT-LLM. For example, for the DeepSeek example that you mentioned, it's working with SGLang, which is really cool to see, and we will be investing more and more in SGLang, especially as the developer experience is so much better than TensorRT-LLM. We've built a lot around TensorRT-LLM and productized it to make it easier to work with, but SGLang has still been a joy to work with.
Another trend that is really promising, and I learned this from the SGLang folks, is that the TensorRT-LLM folks have promised to modularize a lot of TensorRT-LLM so that other frameworks like SGLang can grab certain parts of it and build on top of it. As a user, you don't have to go all in on one framework versus another. You can really pick and choose based on the requirements that you have.
That's really been our approach as well. We have customers on Baseten that are using TensorRT-LLM, we have ones that are using vLLM, and we have a growing number that are using SGLang too. It's not about really tying yourself to one versus another. It's about using the best of the bunch, depending on the requirements of the customer and their inference workloads.
And those are all models hosted on Baseten, or do you also do chains?
Those have to be models hosted on Baseten. If one of those steps is not hosted on Baseten, then…
Right. Yeah, yeah.
…you're still incurring massive latency on the network side.
Yeah, and then, just to tie this into SGLang, how do you think about the hidden magic? Should people know that you use SGLang? Should people care? Especially for the people building the models, does it matter to them that they use a certain model runtime, or do they not care? Does everything just go through the Baseten platform the same?
Yeah. Should we talk about it? Yes, 100%. We want to be the transparent provider. I don't want to say, “Just give us your model, and voilà, magic,” and ask customers to trust our magic. I want that magic to be very transparent to our customers. That has worked really well for us.
You really need that, especially when you're onboarding foundation model companies. They're not going to turn a blind eye to how things are run underneath the hood. When it comes to customers caring about what's happening underneath, they do, but more than caring about this framework versus that, they care about the final output.
In other words, is the quality the same, or has something changed underneath the hood and the model isn't actually producing the same quality? How is the latency? Especially for certain use cases, what is the time to first token? Is that sustained? What is the P95 of that? What is the P99 of that? How well does it handle throughput? When you start getting a massive burst of traffic, does it still sustain those P95s and P99s?
How do I make sure that the security of the data being sent into the model is guaranteed? How do I make sure compliance is guaranteed for HIPAA use cases? How do I make sure that the data remains within a certain geographic region for compliance reasons or for latency reasons? Those are the concerns that customers are coming to us with, less so about, “Here’s my model. Make sure you run it with TensorRT-LLM,” or, “Make sure you run it with SGLang.”
Yep.
Can you maybe give us an overview of all the different frameworks that people might use? So you have SGLang, TensorRT-LLM, and vLLM. Those are kind of like the open-source research ones. Some of the other commercial companies are building some of their own as well. But what's the state of the art today—maybe the top 3 most popular? Then we can talk about why SGLang came to be, what makes it different, and some of the performance boosts that you get.
Okay. Yeah. I think for the common use case—maybe not the DeepSeek-V3 case—SGLang's performance is better than vLLM, and its usability is better than TensorRT-LLM. When users care about performance and usability, I think they will choose SGLang.
For the DeepSeek-V3 case, because we do a lot of optimization in SGLang—for example, in DeepSeek-V2, they proposed an attention variant named MLA, or Multi-head Latent Attention. I think SGLang is the only framework that supports that. Maybe LightLLM and TensorRT-LLM also support it, but vLLM doesn't support it. Also, in SGLang version 0.4, we support data-parallel attention for DeepSeek.
In the latest SGLang release, we also support the block-wise FP8 kernel, and that kernel was adopted and copied by vLLM later. I think we've done a lot of optimization for DeepSeek. That's why SGLang is the recommended engine by the DeepSeek team.
6. Mission Critical Inference Needs Three Pillars
And maybe one thing to point out—and I think this is important—is that the framework that you choose is part of the equation for running mission-critical inference workloads, but it's only a part of it. Maybe I can draw this out based on my experience and what I've seen in the market as to what it takes to run mission-critical inference workloads in production.
I think it takes 3 things, and each of them individually is necessary but not sufficient. One is performance at the model level. In this case, how fast are you running this one model on a single GPU, let's say? The framework that you use there can matter. The techniques that you use there can matter—the MLA technique, for example, that Yineng mentioned, or the CUDA kernels that are being used.
But there are also techniques being used at a higher level, things like speculative decoding with draft models or with Medusa heads. These are implemented in the different frameworks, or you can even implement them yourself, but they're not necessarily tied to a single framework. Using speculative decoding gives you massive upside when it comes to being able to handle high throughput.
But that's not enough. Invariably, that one model running on a single GPU is going to get too much traffic for it to handle. At that point, you need to horizontally scale it. That's not an ML problem. That's not a PyTorch problem. That's an infrastructure problem. How quickly do you go from a single replica of that model to 5, to 10, to 100? That's the second pillar necessary for running these mission-critical inference workloads.
And what does it take to do that? Some people are like, “You just need Kubernetes, and Kubernetes has an autoscaler, and that just works.” That doesn't work for these kinds of mission-critical inference workloads. You end up catching yourself wanting to rebuild those infrastructure pieces bit by bit from scratch. This has been our experience.
Going even a layer beyond that, Kubernetes runs in a single cluster. It's a single cluster tied to a single region. When it comes to inference workloads and needing GPUs, more and more we're seeing that you cannot meet the demand inside a single region—a single cloud's single region.
In other words, a single model might want to horizontally scale up to 200 replicas, each of which has, let's say, 2 H100s or 4 H100s, or even a full node. You run into limits on the capacity inside that one region. What we had to build to get around that was the ability to have a single model with replicas across different regions.
There are models on Baseten today that have 50 replicas in GCP East, 80 replicas in AWS West, and replicas in Oracle in London, et cetera. That was a big investment that we had to make.
The final one is wrapping the power of the first 2 pillars in a very good developer experience, to be able to support workflows like the ones that I mentioned around multistep, multimodel inference workloads. More and more, we're seeing that the market is moving toward those, and that the needs are generally in these more complex workflows.
These are the 3 pillars that it takes to run mission-critical inference workloads. The choice of the framework—the serving framework—is really a part of the first pillar. That's something I'm seeing in the market: people who are somewhat new to it are like, “Well, vLLM equals production. That's what it takes to run inference workloads.” In practice, that is not true, and I wanted to call that out.
I agree with Amir because I think open-source libraries such as vLLM, SGLang, LightLLM, or TensorRT-LLM only provide a library. They don't provide a product solution.
Yeah. Can we maybe talk about some of the unique things about SGLang? I read through the paper. It sounds like some of the main use cases are when you have very large batches, which makes sense for your use case, and also longer context.
What was the decision behind creating the framework, which I think is around 1 year old? I think the paper came out in December 2023, something like that, so it's still fairly new compared to some of the other ones. Maybe what were some things that you had to change as you built it, or any fun stories?
7. SGLang Optimizes the Serving Stack
Yeah, yeah, yeah. In August 2023, Lianmin and Ying wanted to create SGLang, maybe for the frontend, something like an LLM program. They wanted to solve that problem. In January 2024, they added RadixCache, which is a prefix-caching technology. I think SGLang was the first framework to support prefix caching.
In February, they also added constrained decoding and supported something like Jump-Forward. At that time, it was known as a language generator, not an inference backend. In June or July 2024, we wanted to make SGLang a fully functional LLM inference engine, equivalent to vLLM or TensorRT-LLM.
At that time, we published a blog comparing it with other frameworks, and its performance was amazing. I think its throughput was maybe 3 times that of vLLM. After that, vLLM also did some refactoring to make it faster.
In September and December, we continued to release new versions of SGLang. We supported some DeepSeek optimizations, such as MLA optimization and data-parallel attention optimization. We also supported the zero-overhead CPU scheduler and something like the SGLang Router for cache-aware load balancing. We delivered so many features. We just built and shipped.
I think Lianmin and Ying wanted to create a new framework rather than use the existing solutions, such as vLLM or TensorRT-LLM, because at that time vLLM was easy to use, but its performance was maybe not good. Some of its design, I think, was not okay. The code was a little messy, and if you wanted to extend it with a new feature, it was a little hard.
TensorRT-LLM was blazing fast. Its performance was very good, but it wasn't easy to do secondary development. If you wanted to add a new feature, it was a little hard. So they thought, “How can we create a new framework that achieves good performance and is also easy to develop and maintain?” That's why they created the SGLang project.
Let's run through maybe the 3 main techniques behind SGLang. The first one is RadixAttention, which focuses on the KV cache. When you think about a model as large as DeepSeek-V3, especially, having better KV-cache reuse is great. Can you talk a bit about that performance impact?
Yeah. RadixCache is a prefix-caching technology, and it is a special case where the block size is 1. For vLLM and other frameworks, they use a block size of 32, while SGLang uses a block size of 1. If you use a block size of 1, you can make the cache hit rate higher than in other frameworks. I think that's the main benefit.
For your case specifically, how does that change when you have a Baseten-type use case where you don't have a shared endpoint, versus GPU clouds serving 1 model for many people who have very different use cases? When you have 1 endpoint for 1 customer, I'm sure they have a system prompt that a lot of the requests share and things like that. Anything you want to mention there?
Yeah. We've seen this be massively helpful for the reason that you mentioned. There is a finite number of prompts, or at least prompt prefixes, that are being used per customer. What we've seen is that prefix caching and the different techniques to make that better have been massively helpful.
However, we still had to build on top of that. For example, you have a model with dozens of replicas, each of which has its own state of KV cache. A new request comes in, and what we used to do back in the day was randomly assign that request to one of these replicas. But the better way is to know the state of the KV cache in these different replicas and try to decide which one it should go to.
That is one of the parameters you need to consider. There are other parameters around the size of the queue at each replica and the location of each replica, depending on how geo-aware you want to be. Adding that additional consideration around KV-cache-aware load balancing was something that we saw improve latency quite a bit for our customers.
And then the second part, which was maybe the harder one to understand as a practitioner, was this idea of turning some of the decoding process into a finite-state machine instead of something more open-ended, especially when you're using structured outputs. Can you maybe explain what that means? I would love to learn, too. This is an opportunity for everybody to better understand how you think about going from normal token-by-token decoding to having a more—I wouldn't say precompiled, but pre-understanding of what the paths are going to be.
I think SGLang supports constrained decoding, and it also supports jump-forward. We use something like Outlines or XGrammar to convert the schema from JSON to an FSM, or state machine, and we can use the state machine to control the output.
The output may be in JSON mode or something like that; it should obey some rule. In that case, because the output should obey a rule, you can skip some tokens. You should decode 4 times, but you should obey the rule, or you can get that token in advance. You can just use 1 prefill to replace the 4 decoding steps, for example. So that's why you can jump forward.
I guess the question is: Why doesn't everybody do that? When I was reading about it, I thought, “This just sounds better,” especially for accuracy, since you're constraining structured output. You can also do faster decoding. Are there downsides to it?
Maintaining jump-forward is a little hard. Later, we supported something like CPU overlap. In the overlap model, we even made it compatible with jump-forwarding, because if you want to maintain jump-forward with other features, it becomes more complicated.
We only use it as an optional setting; we disable it by default. But if you want to enable it, you can use some arguments to do that. It's a little hard to maintain, especially when making it compatible with other optimization features.
Just as a side note, you mentioned XGrammar, which I had never heard of. I looked up the GitHub repository, and it's actually from MLC, which we talked to TQ—
Yeah, yeah, yeah.
I think a while ago. Any comparisons between XGrammar and Outlines? Is there a trend in this world, or is it mostly settled science?
To be honest, I prefer XGrammar.
Okay. Yeah. Tell us more.
MLC AI was founded by Tianqi Chen. Both Tianqi Chen and XGrammar's other creator, Yuxuan Dong, were students who graduated from Shanghai Jiao Tong University. The creators of SGLang, Lianmin Zheng and Ying Sheng, also graduated from Shanghai Jiao Tong University.
Oh my God. Is it the Berkeley of China?
Yeah, you're right. I think XGrammar's performance is better than Outlines, and in the latest release of TensorRT-LLM, TensorRT-LLM also integrated XGrammar as the backend for constrained decoding.
Okay. This is new to us. We had Remy from Outlines speak at my past conference, but I wasn't even—
Yeah.
—I wasn't even aware that XGrammar was a thing. But structured output is something that a lot of people care about. We had OpenAI talk about its structured output implementation, and there's a lot of interest in making sure that there are no trade-offs.
I think there's a little bit of FUD around how the models might be dumber when you use structured output instead of base next-token generation. But I don't think it's significant, at least not by that much.
Yeah.
We can talk about the last one, which I don't know if it's as relevant for Baseten: the third technique in SGLang, API speculative execution, which seems to be only for API-only models.
Oh, yeah. I think it's a front-end feature. It's not the back end. You have some control flow for the LLM task. You have 1 request to get a result, and then continue to another call.
For this case, you can use the SGLang front-end language to describe the control flow. It makes it easier to control that pattern.
Tracing this human path, I'm pretty sure I know the answer, but is there a reason big projects like Groq and xAI also use SGLang?
Yeah, yeah, yeah. You're right.
Is it just the same people?
Yeah, yeah, yeah. Right. Lianmin and Ying are members of the technical staff at xAI.
I mean, it makes sense. I wonder what the impetus was for SGLang to break containment. It seems like vLLM obviously has the advantage of being 1 year older and has more community pull. I wonder how this will shake out. I don't really know.
You said vLLM's library is much, much more comprehensive. Do people care? Maybe when you're serving models at scale, you start really prioritizing the performance that SGLang offers.
If you care about performance, maybe TensorRT-LLM is the best solution for now, especially for latency-sensitive scenarios. TensorRT-LLM does well.
But if you also want to implement some features or optimizations yourself and customize the framework, I think SGLang is a good option. vLLM's community support is very nice because it is used by so many users and has so many GitHub stars.
When I joined the SGLang team in July, it had only 2,000 stars, and right now it has more than 7,000 stars. I think it has also grown very fast.
Is there anything that people should look forward to on the SGLang roadmap?
We post the roadmap in an issue, and we pin that issue. We also have a biweekly meeting to sync with the community about our progress and plans, including which features we want to implement in this quarter.
We also co-host some meetups. The first meetup we co-hosted was with MLC LLM and FlashInfer, and we also participate in some hackathons, such as the Camel-AI Hackathon. We gave a presentation about SGLang.
I just saw it now. It sounds like there are—you mentioned EAGLE, and you mentioned Medusa. I think Amir mentioned Medusa, but EAGLE is also part of that cabal of speculative decoding techniques. It looks like you support it now.
Yeah. We already support it. I think in open-source implementations such as vLLM, SGLang, and other frameworks, it has state-of-the-art performance. Currently, even with TensorRT-LLM, it only supports EAGLE-1, not EAGLE-2.
One thing to note about speculative decoding and its different versions is that framework support is one thing, but you also have to do the training of the draft models or the additional heads.
A lot of the benefits will come from how good you are at the training aspect—in terms of the data you use to train the draft model to essentially distill the target model or mimic its behavior so that you can have a very high acceptance rate. The throughput improvement you get ultimately depends on how good a job you do training the draft model in the draft-target-model mechanism.
So that's another thing: whether the framework supports it or you can just turn on speculative decoding with a flag. That's not the case; there's more that goes into it.
One more side note on training. I also noticed that with OpenAI offering fine-tuning for o1 and all these things, I think people are also very interested in RL trainers—is that what you have here? It looks like you're supporting Hugging Face TRL and OpenRLHF.
Do you think this will become something that a lot of people demand? The general field of RL for LLMs was relatively abandoned, I think, up until the end of last year, basically.
Yeah. I think so.
I don't know. It's one of those things where maybe people have to wait for a base model that has some layer looping or some other friendly architecture for reasoning, instead of just pure RL on LLMs.
So far, I don't think people have really exploited RLHF as much in the wild. Correct me if I'm wrong.
Yeah. I can give you some examples of when we've seen it work.
Again, this is generally done by our customers before they come to us for inference. There are examples like, in the healthcare world, fine-tuning models for understanding medical jargon, such as a version of Whisper that can actually understand medical jargon. That’s a non-LLM use case.
In the LLM use case, staying in the healthcare space, there are models that can do medical document extraction and do a very good job compared to even state-of-the-art models because of the data that the company had gathered through a human-in-the-loop process. Is the need for those going to go away because there’s a model that can do reasoning and do a very good job at it? I don’t know. My intuition says yes. Will it be cost-effective? That’s the question that I have. In the short term, no; in the long term, maybe.
But I haven’t seen the need for more traditional fine-tuning actually go down. In fact, we see that quite a bit right now in the market. The question for us is, “Hey, do we want to address that market, knowing that the entire market might go away one day?” My general answer to that is, “Let’s solve today’s problems.” Even if they’re not around in 2 years, you will learn a lot along the way by onboarding customers that have today’s problems. You learn from them about tomorrow’s problems, and you will build ahead for it.
Why do you think fine-tuning might go away?
Because, like you said, there are going to be models with complex reasoning capabilities that can actually figure it out in a few-shot kind of way without needing a large data set to fine-tune the model with. That’s what some people are saying.
I really have trouble believing that that will be the case. I much more believe that it’s just easier to change your prompts rather than actually do full fine-tunes or even parameter-efficient fine-tunes.
For sure.
Is there anything else that we haven’t touched on that you wish people asked you more about? It’s something that’s very interesting from your point of view, in terms of what you’re seeing among your community.
When we released the DeepSeek V3 support, we had some community users, like Cursor. Do you know Cursor? I think it’s very popular.
Of course. I use it every day. When I type “code dot” inside my terminal, it actually opens Cursor instead of VS Code. I feel very bad—
Yeah, yeah, yeah.
—for VS Code.
When we released the DeepSeek V3 support, an employee from the Cursor team was also very interested in our implementation and reached out to ask us some questions. As SGLang grows faster and we optimize the features, we iterate so fast. I think there will be more users from different companies and different teams using it.
Honestly, I would go back to what I emphasized earlier, which was that I wish more people asked about what it takes to run mission-critical inference workloads. I see this in the market sometimes: they’re like, “Well, I can just use vLLM, and that puts my model behind an API, and that is production.” But really, it takes 3 pillars that all need to be there. One is performance at the model level. That is where the frameworks that we talked about today really help you, but you still have to guide them when it comes to speculative decoding.
Yes, they support it, but who’s going to train or fine-tune the draft model or the Medusa heads? Who’s going to ensure the reliability of the vLLM server that you see in production? There are crashes. How do you recover from those without affecting production traffic? By itself, that’s not enough, because invariably that 1 model running on a set of hardware is going to get too much traffic for it to handle, and at that point you need to horizontally scale it.
That’s not an ML problem, and it’s not a PyTorch problem. That is an infrastructure problem: ensuring that you can horizontally scale up your model extremely fast to meet your P90 and P99 latency requirements. To ensure that you’re not running out of capacity in a single region where that model lives, you end up having to scale that model across different regions and even across different clouds, to ensure that the model is not being starved of resources in the one place where it lives. That’s an area of investment that we started investing in some time ago, and it really paid off this past year.
The third pillar is enablement of workflows. Workflows such as the AI phone call example that I told you about require multi-step, multi-model inference, but in a very low-latency way. That’s the third pillar that really allows developers to use the power of the first 2 pillars and combine them, especially when you need multiple models for your workflows, doing so in a reliable, repeatable, and low-latency way. Those are the 3 pillars, honestly, that we have been investing a lot in.
Some of which we started investing in 3 years ago, and it really started paying off a year ago. It takes quite a bit of building to get to the point where you’re truly running customers’ mission-critical inference workloads. What do I mean by mission-critical inference workloads? Inference where, if inference is slow or down, the main product of our customer is slow or down, so they really care about it.
They have strict requirements around latency, around being able to support large throughput, and about being able to do so in a way that other customers’ usage doesn’t affect the SLAs they are getting, including dealing with noisy-neighbor problems. They need inference done in a compliant way, whether it’s HIPAA or certain SOC requirements. They also need inference done in a geo-aware kind of way, both for compliance reasons and for latency reasons, where the location to which you forward the traffic has an impact on latency in situations where 50 milliseconds really matter, 100 milliseconds really matter, and we’re seeing more and more of those use cases.
One way I would recommend doing that is a manifesto-type thing. I’m sure you know Heroku’s Twelve-Factor App.
I’ve seen that, yes. That’s a good idea, actually.
Maybe even put it on a separate property from Baseten and just go, “Here’s what we think a mission-critical AI application should be,” and have some thought leadership there, flesh it out, and see if the market takes it on as a mission. Obviously, you will be best prepared to serve that market as well.
I’ve also seen this done very well with EnterpriseReady.io. I think it used to be done by—I think it was called Gravitational or Replicated—
Yeah, yeah.
—one of those.
From Replicated.
These kinds of things, when you have a list of requirements, when you’re like, “Look, everybody needs this,” write them up and put a little bit of marketing on it. Spit it out from the main company brand. That tends to work very well.
Yeah.
Good idea.
Cool. Thanks so much for your time. I think this was a really good dive into both Baseten and SGLang, and a little bit of DeepSeek V3, which people are very interested in. I’m trying to talk to them as well because they’re a fascinating lab. I think you guys are doing a lot to make it accessible for everyone, so thank you so much.
And, just to give Baseten some street cred, they were one of the first sponsors for Latent Space events, and Amir brought 100 croissants to our Latent Space Hackathon in 2023.
Ah.
I just want to bring up how I saw Phil and Ed at, I believe, re:Invent, and I told them that was one of the first events that we really did and one of the turning points of this industry, as far as community goes, in my mind. Everybody was there.
The croissants?
No, not the croissants. The event itself.
Yeah, entire companies launched that day.
Yeah, I mean, Nader from Brev was there, and he and Joseph from Roboflow did the prompt battle thing. Harrison was a judge, and Jerry from LlamaIndex was there. It was kind of like everybody who is now breaking out. If you look at the graph that Jensen put on the screen at CES with some of the companies they work with, a lot of them were at that event.
So, thanks for staying involved with us, Amir, and I’m sure we’ll do more together. Thank you guys.
Many more years to come, for sure.
Thank you for taking the time today.
Good to see you both. I’ll see you, Oshan.