Mistral：Voxtral TTS、Forge、Leanstral 与 Mistral 4——对话 Pavan Kumar Reddy 与 Guillaume Lample
Mistral 正将 Voxtral TTS 定位为切入语音代理的低成本、开放权重产品。 这款9语言、3B模型围绕低延迟流式生成设计，其新型技术栈结合自回归 transformer、12.5 Hz 神经音频 codec 与 flow matching，而非第二个 K 步自回归 decoder。Pavan 更广泛的判断是，音频仍是异常开放的赛道，因为“还没有真正胜出的模型”。
产品策略倾向于在规模收益有限的领域采用小型专用模型，同时由 Mistral Small 将成熟能力整合进稀疏通用模型。 Mistral Small 以6B active parameters 和256K上下文，融合指令跟随、coding、reasoning 与 vision；但 Shawn 认为，transcription 仍应由专用的1B–3B工作负载承担，因为巨型模型携带了“很多你其实用不上的能力”。
Forge 将企业定制，而不是通用 endpoint，置于 Mistral 商业叙事的中心。 企业可能拥有数十年积累、规模达数万亿 tokens 的专有知识，但闭源模型会让它们只能使用“和所有竞争对手一样的模型”。Mistral 表示，continual pre-training 或 fine-tuning 可以消除反复发送10K-token上下文的需求；在部分生产迁移中，定制部署的效果更好，成本约为原来的十分之一。
Voice fine-tuning 将通用语音转化为覆盖语言、声学与品牌的企业专属资产。 具体案例包括将某种亚洲语言在训练混合中的占比从约0.1%提高到50%，让 ASR 适配医学术语或噪声环境，以及构建带 function calling 的3B离线音频模型，用于汽车或厨房场景。对 TTS 而言，目标不是克隆名人，而是让声音匹配每家企业和每个用例：“你不会希望所有企业都使用同一个声音。”
Leanstral 与其说是一个小众数学产品，不如说是验证型、长周期 agency 的实验室。 Arthur 解释称，Lean 接受任何能够编译的有效证明，避开 LLM judge 的歧义与 reward hacking；代理可以将定理拆解为 lemmas，分发给并行 sub-agents，并获得阶段性进展。Arthur 还表示，形式化推理已经向更广泛的数学领域产生了一些 transfer；团队预计，formal software verification 也会从今天的航空、机器人和 safety-critical 小众领域继续扩张。
开放权重既是 Mistral 声称的使命，也是其生态策略，但部署支持不可或缺。 Guillaume 认为，外界能够访问 Llama 等模型，才推动了包括 DPO 类 preference optimization 在内的 post-training 方法研究；他拒绝一个最聪明的系统“只存在于闭门环境中”的未来。但 Mistral 也承认，团队最初低估了将模型接入企业数据、工具、agents 与 multimodal workflows 的难度。
下一个瓶颈是支撑数小时级 trajectories 的训练基础设施，而客户部署正在提供公共 benchmarks 遗漏的 evals。 Arthur 表示，pre-training 仍有很大提升空间；但 GRPO 等在千 token 数学解答上有效的方法，很难直接处理需要6小时才能获得 reward 的路径，因为模型到更新时已经完全 off-policy。Forward-deployed 团队通过“real-world eval”闭环，包括给孩子说出的一个词打分这样狭窄的案例；与此同时，science group 也在探索同样缺乏服务的 AI-for-science 问题。
1. Voxtral TTS 将 Mistral 的音频栈延伸至语音生成
Voxtral TTS 是 Mistral 首款 speech-generation model：支持9种语言、拥有3B parameters、具备原生文本能力；团队称其质量可与最佳系统竞争，但成本仅为 GPT-4o 的一小部分。Alessio 称它可能是“最好的 open-source model”；Guillaume 则回应：“你们只是在给自己吹捧，对吧？”
这次发布遵循一条经过规划的能力演进路径：作为 Mistral 首款音频模型，Voxtral 去年7月以 ASR 模型形式发布；随后在今年1月推出 transcription-only model 家族，并加入 context biasing、diarization、timestamps 与 real-time transcription 等生产能力。下一步是 speech generation，再往后才是整合式 full-duplex audio-in/audio-out 系统。
Guillaume 的运营原则是“step-by-step”：先分别优化 transcription、speech generation 和 real-time capabilities，再将它们合并。团队开玩笑设想的终点是一个“super omni model”，但并不希望整合过程掩盖每个组件各自的经济性或质量。
2. Flow matching 将语音熵转化为流式架构
在 understanding 侧，Voxtral 类似 Mistral 的 Pixtral vision 设计：encoder 将音频转成 continuous embeddings，再作为 tokens 送入 decoder；输出仍是普通文本。Generation 更难，因为模型必须生成音频，而不是单纯理解音频。
Pavan 介绍的内部 codec 以12.5 Hz 表示音频，每个 frame 包含一个 semantic token 和若干 acoustic tokens。输入时，它们的 embeddings 会相加；输出时，每个 transformer state 都会进入一个 flow-matching head，由 velocity estimate 将噪声推向一个80毫秒音频 frame 对应的 latent，随后由 vocoder 重建声音。
传统的 depth transformer 会在每个 time step 内，以自回归方式预测 K 个 acoustic tokens。这种方案可行，但会增加 K 次 sequential operations。Mistral 转而训练 codec，使其同时支持 discrete 与 continuous representations；两种方式在内部都能工作，但“continuous stuff works just better”，也因此可以显著减少 inference steps。
Pavan 对创新性表述得很谨慎：音频领域已经存在 flow-matching models，但他没见过这种确切的自回归组合，“也可能是我错了”。他的更大判断则更明确：与文本不同，音频尚未收敛到一种被普遍接受的架构，因此“探索起来相当令人兴奋”。
3. 语音质量取决于在多种有效表现中作出选择
Guillaume 对 flow matching 的直觉是，同一个已知词语仍有许多合理的发音、语气和节奏，即便是同一个人的声音也不例外。他将这些可能性描述为一组组 performance clusters：“你无法预测它们的均值”，因为对这些表现取平均会得到类似模糊语音的结果；模型必须选择一种清晰、具体的实现。
“ums”、重复和 filler words 等 disfluencies 只是这种 entropy 的一部分；即便没有这些现象，intonation 和 pronunciation 仍然存在变化。Depth transformer 会对这些依赖关系进行条件建模，而 Mistral 的 flow head 可以用大约4步或16步完成生成，未来可能走向 one-step generation，但团队目前尚未采用。
这套架构是围绕 voice agents 选择的：“我们希望实现 real-time streaming。”因此，团队很早就排除了 whole-utterance diffusion；autoregressive framing 则允许将音频作为 decoder transformer 的另一个 head，并为未来的原生 audio-text modeling 留出更清晰的路径。
4. 长文本生成继承 text model 的上下文扩展方式
按照12.5个 audio tokens 每秒计算，Pavan 估计1分钟音频约消耗720 tokens。因此，8K窗口可以容纳约10分钟，32K则覆盖半小时；团队已经能够在这一长度上进行训练。
扩展到128K意味着可以生成小时级音频，而无需改变架构。尚未解决的是数据配方，以及在如此长的跨度内保持连贯性所需的完整算法；但从概念上看，它类似 long-context text modeling，只是将 next-token classification 换成了 autoregressive flow prediction。
在之后的讨论中，Shawn 提到将 voice 与 video、spatial audio 结合。Guillaume 则区分了两种场景：一种是生成完成后再消费的 standalone artifact，另一种是 conversational audio，后者的首要约束是极低延迟与 streaming。
5. Voice 需求真实存在，但自然对话仍未完成
Guillaume 表示，客户需求推动了 Mistral 对音频的投入，但即便是 transcription，也暴露出 benchmark competence 与人类互动之间的差距。英语领先于其他语言；他特别指出，用户与法语模型对话时仍会放慢语速、刻意咬字，而法语、西班牙语和德语尽管并非 low-resource languages，表现仍然较弱。他猜测，即使未来模型不再需要这种交流方式，当前这一代人可能仍会保留已经形成的习惯。
Alessio 与 Google Assistant 的对比凸显了进展：在大约4到5年内，系统已经向 end-to-end audio input、audio output 和 function calling 演进。不过，他的保留意见同样重要——当前 voice modes 仍不像是在与真人交谈，尽管现有技术栈“应该能够”很快取得更接近的效果。
Alessio 将音频定义为一种 productivity interface：当用户知道自己想要什么、但如果用键盘就必须输入一段很长的 specification 时，尤其是 coding 场景，音频更有价值。音频是“一种尚不存在的自然 interface”，因此机会不只是把 transcription 做得更好，而是替换现有的 interface。
6. Mistral Small 整合能力，但没有放弃 specialists
Mistral 此前将 instruction following、Devstral coding、Magistral reasoning 和 Pixtral vision 分别开发为独立产品，由专注于各自方向的团队负责。Mistral Small 是这些成熟分支首次进行的大范围合并；当时 function calling 尚未被完全整合，预计在 Mistral Small 正式版本中会得到大幅改进。
最终形成的 mixture-of-experts model 高度稀疏，拥有6B active parameters 和256K context window。Guillaume 还确认，更大版本已经存在，因此“Small”这个引发主持人笔记本电脑玩笑的名称，并不意味着失去进一步 scaling 的路径。
主持人继续追问 omni-model thesis。Shawn 的经济学论点是，纯 transcription 根本不需要如此庞大的模型：专用的1B或3B系统就能提供基本相同的性能，成本却低得多。
Guillaume 表示，未来的 general models 应吸收更强的 coding 和 reasoning，同时覆盖 legal work、finance、computer-aided design 等被忽视的 enterprise domains。这些领域往往缺少热门 benchmarks，但他的处方很直接：把数据找来、处理好，因为“你只需要真正去做这件事”。
7. Forge 将专有语料转化为企业护城河
Guillaume 最尖锐的商业论点是，现成的 closed models 让企业无法利用数十年积累的知识，其中有时包括数万亿、且不在公开互联网中的 domain-specific tokens。Inference-time context 可以暴露这些数据，但将整个语料用于训练更好，也能避免每次查询都重新发送可能多达10K tokens 的上下文。
Forge 将 Mistral science team 使用的同一套 data pipelines 与高效 training infrastructure 封装起来，支持 continued pre-training、fine-tuning 和 SFT。Guillaume 称这套系统经过约两年的内部使用，已经“battle-tested”：客户实际上获得的是 Mistral 用来构建自身模型的工具。
Privacy 是另一条 adoption vector。企业会按 sensitivity tiers 划分信息，形成一种尴尬流程：部分数据可以进入 public cloud，部分数据却不行。On-premises 或 private-cloud deployment 可以消除这类路由问题；同时，客户也看重由一个合作伙伴同时覆盖 text、code 和 audio，而不必将敏感数据分散给多家供应商。
生产端的经济性通常只有在 prototype 成功后才会显现。Guillaume 表示，客户往往是在发现 closed endpoint 无法以可接受成本扩展后才来找 Mistral；定制模型有时可以“便宜10倍”，在目标 workflow 上表现更好，并部署在客户自己的服务器上。
主持人另一个关于成本比较的提醒是，其中一个数字代表 inference cost，另一个则包含 API provider margins；因此，这些比较并不是纯粹的 inference-cost comparison。
8. 定制化覆盖语言、声学与品牌身份
一名客户需要支持一种在通用模型训练混合中占比可能只有0.1%的亚洲语言。Mistral 将该语言的占比提高到50%，让模型掌握 dialects 和 slang，而不只是基本的 grammatical coverage。另一个需求是将 audio 与强 function calling 结合，打造用于汽车或厨房的3B离线系统，以应对 connectivity 不可靠的环境。
Speech-to-text fine-tuning 可以加入 tail languages、专门的 medical 或 technical vocabulary，并增强模型对特定 noise conditions 的鲁棒性。TTS support 预计很快推出，企业 voice adaptation 的重点将是 tone 和 personality，而不是新奇的声音克隆。
Guillaume 从场景出发定义目标：富有同理心的 healthcare assistant 不应听起来像普通的 customer-support bot，而每家企业都希望声音能够代表自己的 brand 和 safety requirements。Alessio 举例称，两条 customer-service lines 不应使用完全相同的声音回答。
9. Leanstral 将形式化证明变成可验证的推理实验室
Guillaume 表示，Mistral 对 open-source 的承诺从创立之初就存在，并将其与自己早年在 Meta 参与发布 Llama 的经历联系起来。他认为，外界能够访问这类模型，才推动了包括 DPO 类 preference optimization 在内的 post-training research。关于 reasoning、audio 和 real-time encoders 的详细报告服务于同一目标：避免最强 intelligence“只存在于闭门环境中”的世界。
Leanstral 起源于一个规模很小的 internal formal-math team。Guillaume 称，这个团队在 LLM 出现之前就过早开始研究 reasoning；当时 formal-proof data 和周边社区都微不足道。如今它的优势在于 verification：数值型 AIME 答案很容易检查，但普通证明可以以多种形式成立，因此与 reference 的比较很脆弱，LLM judges 也容易遭遇 reward hacking。
Arthur 解释称，Lean 消除了这种歧义，因为任何有效证明都可以被机械检查——“就像一个程序，只要能编译且正确即可”。这让 formal reasoning 的价值超越数学家群体，尤其适用于 software verification；目前后者主要集中在 aircraft、robotics 及其他一旦失败就可能危及生命的场景。
Agentic connection 可能更加广泛。模型可以提出3个 lemmas，将它们分发给并行 sub-agents，再利用这些输出证明主定理；即便其中一个 lemma 失败，也可能留下可复用的阶段性进展。Arthur 表示，从 formal reasoning 向更广泛数学领域的 transfer 已经可见，这支持了 Shawn 的判断：proofs 可能是 planning 与 long-horizon coherence 的 proxy。
10. 部署正在成为 Mistral 的 research feedback loop
Arthur 表示，pre-training 仍有很大的收益空间，但更难的前沿是支撑极长 trajectories 的 algorithms 和 infrastructure。GRPO 等方法在千 token 级数学解答上可行，但当 reward 只有6小时后才返回时，模型到更新时可能已经完全 off-policy。
AI-for-science 项目正在物理、材料及其他领域寻找“low-hanging fruits”，因为领域研究者可能并不了解当前模型的能力边界。Mistral 的客户关系提供了匹配机制：让 model researchers 与掌握独特 proprietary problems 和 data 的 specialists 配对。ISM Électronique 是一个具体合作伙伴案例。
Forward-deployed 工作覆盖 CAD，也包括 defect detection 等潜在 vision use cases，以及一个为孩子说出单个词语打分的 evaluation。这些狭窄案例会暴露出 academic benchmark 不可能包含的失败；相关经验随后回流到 base-model training，团队将其称为“real-world eval”。
Research 与 deployment 因此共享工具、data pipelines 和 feedback，而不是作为彼此割裂的组织运行。Mistral 正在 Paris、London、Palo Alto、Warsaw、Zurich、New York 招聘，旧金山也即将开放岗位，同时提供 remote roles；团队还在努力保持 Guillaume 所称的“小而高度敏捷”的团队文化。
When your customers use an off-the-shelf closed model, what's very sad is that they are not leveraging all the data that they have been collecting for 4 years or so, but for decades. So much data—sometimes trillions of tokens—in a very specific domain, their domain, which is data that you would not find in the public internet. So, if they are using those closed-source models, they are basically not benefiting from all these insights and all this data that they have collected through the years.
Shawn Wang
Okay, welcome to Latent Space. We're here in the studio with our trusty co-host, Vibhu. Welcome.
Vibhu Norby
Hey, I'm excited for this one.
Shawn Wang
As well as Guillaume and Pavan from Mistral. Welcome.
Excited to be here.
Pavan Kapanipathi
Thank you for having us.
Shawn Wang
Pavan, you are leading audio research at Mistral, and Guillaume, you're a chief scientist. What are we announcing today? We're sort of coordinating this release with you guys.
Pavan Kapanipathi
Yeah, so we are releasing Voxtral TTS. It's our first audio model that generates speech. It's not our first audio model: we had a couple of releases before. We had one in the summer, Voxtral, our first audio model, but it was a transcription model, an ASR model. We released some updates on top of this, supporting more languages and a lot of full-stack features for our customers, like context biasing, diarization, and timestamping in the transcription.
We also had some real-time models that can transcribe audio not just at the end of the audio. You don't need to feed them your entire audio file; they can also work in real time. This is the natural extension in audio: speech generation.
We support 9 languages. This is a pretty small model, a 3B model, so it's very fast, and it's also state-of-the-art in terms of quality, at the same level as the best models, but much more efficient in terms of cost and latency. In terms of cost, it's also much cheaper—it's only a fraction of the cost of GPT-4o. That's something that we like: this model really exists.
Shawn Wang
Yeah, memory length?
Pavan Kapanipathi
That is that.
Shawn Wang
Yeah, what's the decision factor?
It's a good question. There will be more. There will be more.
Shawn Wang
Ooh. Yeah. For one, any sort of research notes to add on what you have?
Pavan Kapanipathi
Yeah, maybe we'll dive into it later in the podcast, too, but it's a novel architecture that we developed in-house. We iterated on several internal architectures and ended up with an autoregressive flow-matching architecture. We also have a new in-house neural audio codec, which converts this audio into 12.5 Hz latent tokens: semantic and acoustic tokens.
That's the new part about this model, and we're pretty excited that it came out with such good quality. Like you were just mentioning, it's a 3B model. It's based on the Mistral model that we released just a few months back, and it's a trunk. It's mainly meant for the TTS stuff, but the native text capabilities are also there in the model.
Shawn Wang
There's a lot to cover. I always love anything to do with novel encodings and all those things because I think that's obviously increasing a lot of efficiency, but bugs also sometimes happen. You were previously at Gemini, and you worked on post-training for language models. Maybe a lot of people will have less experience with audio models in general compared to pure language models. What did you find that you had to revisit from scratch as you joined Mistral and started doing this?
Pavan Kapanipathi
At least when it comes to the two buckets, I guess: audio understanding and audio generation. The audio understanding, like the Voxtral models that Guillaume was mentioning, were released earlier. They were released in July last year, and the follow-up transcription-only model family was released in January. That could be one bucket, I guess, and generation is another bucket.
I think you can also treat them as a unified set of models, but currently the approaches are a little different between these two. To your question about how audio is fed to the model: in the understanding model, it's very similar to the Pixtral model that we also released. That was the first project I worked on after joining Mistral. It was pretty nice, and Voxtral was very similar in spirit.
We feed audio through an audio encoder, similar to images through a vision encoder, and it produces continuous embeddings, which are fed as tokens to the main transformer decoder model. The model output is just text, so on the output side there is nothing that needs to be done in these kinds of models.
The interesting part about the generation stuff is that the output now has to produce audio. We have a neural audio codec, which converts audio into these latent tokens. There is a lot of existing literature and a lot of models based on this kind of approach, and we made slightly different design decisions around this.
At the end of the day, the neural audio codec converts audio into a 12.5 Hz set of latents, and each latent has a semantic token and a set of acoustic tokens. The idea is that you take these discrete tokens and feed them on the input side. There are several ways to fuse this at each frame, but we just sum the embeddings, so it's kind of like having K different vocabularies and combining all of them because they all correspond to one audio frame on the input side.
The output side is the interesting part. I don't know if it's the most popular, but one popular technique is to have a depth transformer because you have K tokens at each time step. With text, you just have 1 token at each time step, so you predict the token from the vocabulary and get probabilities.
Shawn Wang
This is a very straightforward technique.
Pavan Kapanipathi
Very straightforward. But if you have K tokens, then the main thing would be to predict all of them in parallel, and that doesn't work—at least, it doesn't work that well—because audio has more entropy. One of the techniques people use is this depth transformer, where you almost have a small transformer, or it can be an LSTM or RNN as well, but people use transformers and predict the K tokens autoregressively in that. So, you have 2 autoregressive things going on.
The thing we did differently is that instead of having this autoregressive K-step prediction, we have a flow-matching model. Instead of modeling this as a discrete token set, we train the codec to be both discrete and continuous, to have this flexibility. We did try the discrete stuff, too, and it works well, but the continuous stuff works just better.
There is a flow-matching head which takes the latent from the main transformer and, kind of like in diffusion, denoises it. But in flow matching, it's a velocity estimate. You go from this noised latent all the way to the audio latent, which corresponds to the 80 ms of audio, and then it's sent through the vocoder to get back the 80 ms audio frame.
Shawn Wang
Yeah. Is this the first application of flow matching in audio? Usually, I come across this in image generation.
Pavan Kapanipathi
Yeah, actually, in some sense, there are flow-matching models in audio, but I think this specific combination—I could be wrong. There could be some work I haven't seen.
I haven't seen much work in this. So, I think it's novel, and a lot of it is just a way bigger community today. I think they pioneered a lot of this diffusion and flow-matching work, and it's interesting to adopt some of the ideas there into audio.
Personally, that's the key part. Trying things out is one of my main points: unlike text—even in vision, I think this is true, but in audio it's definitely true—there is no winner model yet. There is no, "Okay, this is the way you do things." It's still evolving. People are still iterating and figuring out what's the best overall recipe, I guess.
I'm pretty sure there are models which are also completely end-to-end, like native audio [?] and native audio [?], but it's still not like the field has come to a convergence point with this being the right way to think about it. That also makes the space pretty exciting to explore.
Shawn Wang
What are some of the ways to look at it? There are ways where you can do diffusion for audio generation, but if you want real-time generation, that's a big thing with the approach I'm assuming that you took. Also, how do you go about evaluating different axes of what you care about?
Pavan Kapanipathi
Good point. You can do just flow-matching diffusion for the whole audio. We didn't even go down that path because one of the main applications is voice agents, and we want real-time streaming. That's the use case. That's not the only use case, but it's one of the primary use cases we want to get to.
So, we picked the autoregressive approach for that. Within the autoregressive space, again, you can do chunk by chunk, or you can do—so, we picked the—I think, at least personally, I prefer the approaches which are the simplest, I guess. We tried to see whether we could just add audio as another head to our regular transformer decoder model, because that kind of makes it easier for eventual end-to-end modeling of audio-text-native modeling.
Yeah, and it works pretty well. So, I guess we went with that. We experimented a little bit with the flow-matching head itself. We had a discrete-diffusion kind of approach, which also works well, but the flow-matching one worked better.
Alessio Fanelli
I was just curious about how you think about this overall direction of research. When you work with the audio team, do you set some high-level parameters and then let them explore whatever, or how does it work between you guys?
No, I think the way it works is that we are prioritizing together what the most important features are. There are many, many things you can do in audio, so we try to decide how we should do things. For instance, ultimately what we want to do is build this full-duplex model, but we are not going to start that directly.
Alessio Fanelli
Just to confirm, does full duplex mean it can speak while I'm speaking?
Audio in, audio out.
Alessio Fanelli
Yeah, yeah.
So, ultimately, we are going to get there, but for us, we decided to take it step by step. We start with whatever is the most important, I think also for our customers, which is transcription, the most popular use case. Then there's speech generation, and real-time just a bit beyond that. Then we're actually going to try to combine everything together. But we thought it was also important to separate things and optimize each capability one by one before we merge all of that together.
Alessio Fanelli
And then the super-omni model.
What's interesting is that, as I was saying, when you work on some other domains of this LLM and other things, there are many areas where I think it's not as interesting. For instance, in many places it's essentially just around data or creating new environments, and a lot of easy things—things where I think the research is maybe not as interesting.
While in audio, there are so many ways to actually build this model, so many ways to go around it. This space is, I think, really interesting. What we also tried for speech generation is a mixture of multiple approaches. What was interesting is that even though they were extremely different, they ended up being, at the end of the day, pretty close. But flow matching turned out to be quite more natural, so we are happy with it.
Shawn Wang
Is there an intuition for why flow matching maybe just models speech better in some natural, fundamental latent dimension?
No, I think the main thing is that, even at a particular time step, there is a distribution of things to be predicted. You already know the word that you're speaking. In text space, let's say the word maps to just a single token, for simplicity. In most cases, it does, so you just pick the word.
But within audio, even the same word, with your own voice, could be inflected in so many different ways. I think any approach that models this distribution—and flow matching is one of them; it's not the only one at all, but it's the one that works pretty reasonably well—is better. You have to pick across several different possibilities.
The intuition I have is that there are several different clusters, each corresponding to some specific way you would inflect or pronounce that thing. You can't predict the mean of it because that corresponds to some blurred-out speech or something like that. You have to pick one and then make it sharp.
Shawn Wang
Conditional inference.
Yeah, exactly.
Shawn Wang
Is that all covered under disfluencies, which I think is the normal term of art? Disfluencies, pauses, intonations. By the way, I have to thank Sophia for setting all this up, including some of these really good notes, because I'm less familiar with audio. So, maybe—
No, no, no. I think disfluencies are definitely one such phenomenon. Disfluencies are more like—
Alessio Fanelli
Which is ums and ahs—
Yeah, ums, ahs, and also repeats. You feel like you're doing these filler words while you're thinking, so you repeat the word.
Alessio Fanelli
Okay. Whereas intonation is like a way of speaking?
And yeah, I think there is a lot of entropy in modeling it as a distribution, and any technique that helps with it—the depth transformer is a conditional way of modeling this, and transformers are actually good at it, but it's a mini-transformer, so I think that worked pretty well for us too.
The main consideration is that when you have a depth transformer, if you have K tokens, you need to do K autoregressive steps. So even though it's a small thing, it's K steps, which is very big—heavy, to say the least. But with flow matching, we were able to cut it down significantly, so we are able to do the inference in 4 steps or 16 steps, and it works pretty well. There are more novel techniques to bring it down even further, in the extreme case to 1 step. We're not doing it yet, but at least the framework lends itself to more efficient—
Alessio Fanelli
Yeah, and the image guys have done incredible work. Now you just send the prompt and you get an image.
Shawn Wang
Yeah, surprisingly, not enough image-model labs use those techniques in production. I feel like it's a lot of research demos, but nothing I can use on my phone today.
The thing that's interesting here is that, since there has been so much work done in the vision community compared to audio, to me there are so many learnings and so many things we can do to actually improve this paradigm further. I don't know if it's our first version, but we have so many ways to make it much better and much more efficient, cost-efficient, so—
Alessio Fanelli
Yeah.
So certainly you feel that, of course, but there are still so many things that can be done. I feel that it's just—
Shawn Wang
I should also mention, for those who are newer to flow matching, I think the creator is this guy named Alex. I think he did, at NeurIPS maybe 2 years ago, a very good 1-hour workshop on flow matching. I recommend people look that up.
That's the other thing, right? Efficiency-wise, I imagine the reason it's open weights, the reason you picked a 2.4B backbone, is that you're trying to fit some kind of hardware constraints, some kind of business constraints. What are they?
Not necessarily. I think something we care about in our models is that they are efficient. We have a lot of separate models, for instance. We have this model that we know is very small and very efficient. We also have a small model that we think is very, very good and very efficient as well.
I think an approach that maybe other companies are going to take is to have a very general model that will do a bit of everything, but that is also going to be expensive. The other thing I want to say is, if you care about this specific use case, you can actually use this model. It just does that. It's extremely good at it, but it's also very efficient.
That's why we can actually offer models, or you'll be able to access our APIs, that are really, really good at that and much more cost-effective than another general model that will contain a lot of capabilities you don't really need. So, that's all. We're doing general models, but also more customized models that exist.
Alessio Fanelli
How does it compare to other TTS models? You're going fully open weights; you're just dropping it like—
I think it's pretty good.
Shawn Wang
Yeah, I think it's pretty good. It's definitely one of the best.
Alessio Fanelli
For sure, for sure. It's probably—I would say it's the best open-source model, right?
You're just hyping yourselves, right?
Alessio Fanelli
Yeah. Why now? How does it fit into the broader Mistral vision? How do you see voice agents? How do you see voice—
Shawn Wang
I think every year I've heard, “Okay, you're a voice, you're a voice.” There's a lot of architectural stuff. There's a lot of end-to-end latency that you're solving, but where do you see voice heading?
We had so many customers asking for voice. That's also why we wanted to build it. What's interesting in this domain is that, in a sense, if you take something simple like transcription, it doesn't seem like something that should be very hard for a model to do. It's essentially pattern recognition; it's classification. These models are very good at classifying, right?
Nonetheless, when you talk to them, it's not there yet. You don't talk to them the same way you talk to a person or something. Maybe people don't realize it. In English, it's still much better than in any other language. Even compared to French, for instance, if you talk to this model in French, when you see people talking to this model, they will talk very slowly. They will articulate as much as they can. So, it's not natural, right? We're not yet there.
Maybe the next generation won't know this, but people who are maybe our age will always keep this bias of speaking very slowly when they talk to this model, even if probably in a couple of years—maybe next year—it won't be necessary anymore. What's interesting is to see that even for languages like French, Spanish, and German, which are not low-resource languages, you have a lot of audio with this.
They still aren't as good. I think the reason for this, I suppose, is that there hasn't been as much energy or effort put into some other modalities, such as vision or coding. There is still a lot of progress to be made, but I think it's just a question of doing some work on this. There is a clear path, I think, to get there.
Alessio Fanelli
It's really fascinating because I worked on Google Assistant a while back at this point, but when you take a step back, it's fascinating. It's not that long ago—it was 4 or 5 years ago—and now it's completely audio-in, audio-out, and the function calling and the whole thing happen completely end-to-end in a very natural way. There are still ways to go, like you were saying. Even despite all the progress, it's not like you're speaking to a person when you talk to any of these agent bots or voice-mode kind of situations. There's still a gap. I think that's the great thing, and I feel like, even with the existing stack, we should be able to get to these very natural speech conversational abilities soon enough, I guess, and we'll also hope to get there.
On the customer side, when you talk to these agents, usually people are just writing to them. Sometimes it will be very clear—for instance, you want to write code, but you have a very clear idea of how you want the model to implement what you had in mind. So here you have to spend a lot of time writing. It's not really efficient. Audio is really a natural interface that's just not there yet, but I think it's just going to be there, at least.
Shawn Wang
How is it like building, serving, and inferencing? We see a lot about how it's very easy to take LLMs off the shelf, serve them, fine-tune them, and deploy them. I know you guys have a whole—you have Forge—you have a whole stack for customizing and deploying. Is there a lag in getting that distribution channel? Are you helping there? With prompting LLMs, you can have them be concise, verbose, and all that. They're built on LLM backbones, these models. So how do you see all that?
Yeah, I think this is a lot of what we're doing with our own customers. Very often, they come to us for different reasons. One reason is that they sometimes have a lot of privacy concerns: they have data that is very sensitive, they don't want it to leave the company, and they want it to stay inside the company. So we help them deploy the model in-house, either on-premises or on a private cloud, so they aren't worried that it's being given to a third party or that there is some leakage.
Many companies have different levels of data sensitivity. They have, sometimes, Tier 1, Tier 2, and Tier 3 data. Tier 3 can be sent to the cloud; Tier 1 has to stay there. So it creates heterogeneous workflows where it's annoying: you cannot send some data to the cloud, but you can send other data. When we deploy the model for them, they don't have these considerations; they're not worried that this is going to leak. Everything is much easier, so we help them do this.
That's one of the value propositions, but the other is that, very often, when customers use these off-the-shelf closed models, what's very sad is that they're not leveraging the data they've been collecting for 4 years or for decades. It's so much data—sometimes trillions of tokens—in a very specific domain, their domain, which is data that you won't find on the public internet. It's data that a closed-source model would not actually have access to, and on which the model is going to be really good.
If they're using closed-source models, they're basically not benefiting from all these insights and all this data they've collected over the years. They can always put it into context at inference time, but it's still not as good as if you actually train the model on it. So that's basically what we help them do.
We actually provide them with Mistral Forge, basically what we announced at GTC this week. It's a platform with a lot of tools to help them process data and train on it. It's actually the same thing we're using in the science team, so it's battle-tested infrastructure, with a lot of efficient training code for continued pretraining, fine-tuning, and even doing SFT well. We help them do this using the same tools that our science team is building and using.
Since these are tools that we've been using for 2 years now, they're really battle-tested and sophisticated. It's the same thing we're giving them—giving companies the same thing our science team uses internally to build their own AI. It makes a really big difference.
I think sometimes customers, and many people in general, don't realize how much better the model becomes when you fine-tune it on your own data. You can have your model here, and you start from there. You have a closed-source model that is starting here, but if you actually fine-tune it, you really go much further than this. Then you have a very big advantage. The model is trained on your entire company knowledge, so it knows everything. You don't have to feed 10K tokens of context at every query. So it's much easier.
I think using a closed-source model is really sad because it basically means you are not leveraging all this data, and you are going to be using the same model as all your competitors, when you're actually using everything you've been collecting for years, which is really valuable. So we help customers do this.
We have a lot of solutions deployed, and we have engineers who go into the company and basically look at the problem customers are facing, what they're struggling to do, and what we should do to solve it. So we help them solve these problems together. I think our approach is a bit different here from that of some of our competitors. We don't just release an endpoint and put some stuff on top of that, and we don't just give them a checkpoint. We really look very closely with customers, look at the issues they have, help them solve them, and make a tailored solution for the problem they're facing.
One example is that sometimes customers really want a model that performs well on some Asian languages. If you take some of the Mistral models, they can't speak or write in this language, or at least it's not amazing. This language might be 0.1% of the mixture, so it has been included during training, but very, very little. What we did here is actually train a new model for them in which this language was 50% of the mix, which is much stronger. It knows all the dialects and the slang. That's an example of things we can do, and it's really custom, I think.
Some customers, for instance, wanted a 3B model that could do audio and was very good at function calling—something you want in the kitchen or in the car. In particular, they wanted this to be offline, because in a car you don't necessarily have access to the internet. We can build these solutions. There is no model out of the box for this on the internet. You have very general models that are good at reasoning and are strong models, but for things like this, customers always want specific solutions.
Sometimes they come to us because they experiment with a closed-source model, get a prototype, are happy with what they built—it works well, and they're happy with the performance—and then want to go to production. Then they realize it's extremely expensive. You cannot ship this. So they come back to us and say, "Can you help us build the same thing as this, but using something much cheaper?" We can sometimes build something 10x cheaper just by fine-tuning a model on it. It will be better on-premises, on their own server, and much cheaper as well.
That's the missed opportunity right there.
Alessio Fanelli
Take all the money.
Outside of that, you do put out open-weight models so people can do this themselves. It's just that not enough people go out of their way to do it. They're not going to; they're going to ask you to do it.
I see how it works.
Initially, we didn't know—I mean, we weren't competitive at the beginning of the company because I think our strategy was not exactly the same as what it is today. What we underestimated initially was the complexity of deploying these models, connecting them to everything, and making sure they have access to the company's knowledge. We were seeing customers struggling with this, but that was 2 years ago, and now things are much more complicated.
Now you don't just have text-only LLMs doing simple instruction following; you have reasoning, agents, tools, and multimodal audio. It's much more complicated than before, and even back then it was hard for customers. So they really need support, and this is why we're providing that as part of the proposition as well, to help them deploy this.
I'm curious: is there also voice fine-tuning that people do?
Just as with Forge, we also have a unified framework. The hope is that, like with the Voxtral speech-to-text model that we released earlier this year, and even with Vox chat, which we released last year, there will be a big, rich ecosystem of people fine-tuning Whisper. People want the same thing with Voxtral.
It's much stronger than Whisper, and the platform offers that kind of fine-tuning, which could be any kind of fine-tuning. For instance, sometimes people want to support new languages—long-tail languages—which we hope to cover ourselves natively. But if there is a language where you have data and you want to fine-tune it, I think this is a good use case. The other use case is the same language, even English, but in a very domain-specific way—
Alessio Fanelli
Yeah, terminology, jargon, medical stuff.
Exactly. And also the specific acoustic conditions, like if there's a lot of noise. The model will do decently in most conditions, but you can always make it better, and those are some of the use cases where you can improve it even further. That's one good use case for this.
For our text-to-speech, we're just releasing it, so we'll have support for that soon, too. I think it's a similar use case, but it's a little different—the kinds of things that you want to extend a text-to-speech model to could be voice personalization and voice adaptation for enterprises. Many enterprises need a very specific kind of tone and personality for this kind of voice, and all of those are good use cases for fine-tuning.
This is what I was going to ask you. We never talked about cloning—voice cloning—here. How important is it? I can just clone a famous person's voice, okay, but—
The main use case would be enterprise personalization. Enterprises need a lot of customization. You don't want the same voice for all enterprises. Each enterprise wants something customized and specialized, which is representative of both their brand and their safety considerations.
The kind of thing that you would deploy as an empathetic assistant in the context of a healthcare domain would be very different from the kind of thing that would be in a customer support bot, and that would be different from more conversational applications. Those are the customizations you would expect from enterprises, and that's the main use case, at least from our side.
Alessio Fanelli
My base example is that you don't want to call 2 customer services and have the exact same voice. It's going to be weird. But also, on the technical side, there are a few things about Voxtral that I thought were pretty interesting. He's a big fan of this paper.
Oh.
He said during the paper that it's the best ASR paper he's ever read.
Alessio Fanelli
Yeah, I've hyped up this Voxtral paper enough. We covered it somewhere.
Whisper is known for 30-second processing. You extended this to 40 minutes. There was a lot of good detail in the paper about how this was done, even little details about the padding. It is very much needed; you need to have that padding in there. There was also the synthetic-data generation around this.
I'm wondering if you can share the same about the new text-to-speech model. How do you generate long-form, coherent speech? How do you do that, and are there any gems? Is there going to be a paper?
Yeah, there would be a technical report. I think it could have a lot of details. The summary of it is that some of the considerations in the paper came from the fact that we started with the Whisper encoder as the starting point.
Now we have in-house encoders, like the real-time model that we released in January. We also released a technical report for that real-time model, which uses this dual-stream architecture. It's an interesting architecture; you should check it out. We have a causal encoder, and I don't think there's any strong multilingual causal encoder out in the community, so we thought it would be a good contribution. It's a good encoder that other people might want to adapt, and we trained it from scratch.
I think our full stack is now mature enough that we're able to train very strong encoders. Some of these considerations, like striding and related details, are a function of the Whisper encoder. Now that we train encoders in-house, the design considerations are different.
As for our text-to-speech model, it also leans on the original autoregressive decoder backbone. The considerations are almost identical. The model processes audio at 12.5 hertz, so 1 second maps to 12.5 tokens. One minute is about 720 tokens, so you can get up to 10 minutes in an 8K context window, and half an hour in a 32K context window.
A 32K context is something that we're very comfortable training on. We can extend it even much longer—to 128K—so you can naturally see how it can extend to hour-long generations. We need the data recipe and the whole algorithm to work coherently through such long contexts, but the techniques are in some ways very similar to text long-context modeling. The key difference is that it's doing flow matching autoregressively instead of text-token prediction.
Alessio Fanelli
Okay, I think that was most of the voice questions that we had.
Shawn Wang
I have a big question on Mistral Small.
Alessio Fanelli
Mistral Small. Let's go.
Shawn Wang
What is Small? How do we define Small? What is this?
Alessio Fanelli
I remember the days of Mistral 7B on my laptop. It's not fitting on my laptop.
I could run it on the big laptop, but it's just a different question. If I tell you the equivalent of this model is 6B active parameters, which is true, we did let you infer that in the same name. We could have called it Medium, but I think it's just a naming question. I'll trouble you with this, I suppose.
It's a model that we released, Mistral Small. It's a mixture-of-experts model that combines different models. Before, what we were doing was having one general model for Mistral doing instruction following, with a separate model, Devstral, that was really good at coding and specific to code. We had another model for reasoning, Magistral. These were separate artifacts built by different teams at Mistral.
Now what we're doing is basically merging all of this. Pixtral, the first vision model we had, was also a separate model. The way we do things internally is that we have one team focus on one capability and build one model. Once it's mature enough, we decide to merge it into the main picture. This was the first time we basically merged all of this into one.
There are some other things we didn't have time to merge at the time, such as more capabilities and function calling, which I think is going to be much, much better in Mistral Small proper. But, yeah, it's our latest model and the one we're working on. Of course, there are larger versions of this.
Alessio Fanelli
The key things are that it's very sparse, with 6B active parameters, so it's pretty efficient to serve, and it has a 256K context window.
Shawn Wang
What's interesting is this general theory of developing the individual capabilities in different teams and then merging them. Where is this going to end up?
Like, we've seen the 5 things put together in this. What are the next 5 teams?
I think, actually, OpenAI has kind of gone away from the original vision of the omni model. This is what they were selling, right? All modalities in. But I feel like you might do it.
Shawn Wang
I mean, I think there are some modalities where it's not completely obvious. For instance, for audio, if you want to do transcription, I think it makes no sense to use a model that's this large. If you just want to transcribe text, it would be very inefficient, actually. If you want to do audio, you probably just want to build a 1B or 3B model. Performance would be essentially the same, and it's going to be incredibly cheaper.
So that's why we want a separate model that just does this. The question is just: if you are talking to your model by speech and asking very complex questions, how do you do this? On the other hand, you want to cascade things. Do you want to put a 1B model on the stack, not a competitor to this? I think, in a way, if you do that, that's the right possibility, of course.
But, yeah, for us, the next capabilities we want to try to integrate into these models are going to be more coding and more reasoning. I think there are more capabilities that people don't talk too much about but that are important for our customers in different industries—for instance, legal, finance, and computer-aided design.
These are things that are hard to put out of the box because people really don't prioritize them. There are no new benchmarks for that, but it's not hard to make these models better. Coding is just a matter of doing the work, like sourcing some data and processing it. If you can do that, it makes sense. We have other things to merge into this.
For voice, the key thing over maybe the last year or so with Veo and Google Imagen and all these things is joining voice with video, right? People don't understand spatial audio because most text-to-speech is just, “I'm speaking to a microphone in perfect studio quality.” But when you have video, the voice moves around.
That’s true. The consideration is also a little different in the sense that there, it’s a standalone artifact where you get the whole thing and consume it. But in a conversational setting, you need extremely low latency.
Shawn Wang
Yeah.
Streaming would be one of the primary considerations.
Shawn Wang
You can build a giant company just doing that, so you don’t need to do the voice stuff. But I was just saying, on the theme of merging modalities, that is something where I’m like, wow. Everyone up until, let’s say, mid-last year was just doing these pipelines of, “Okay, we’ll stitch a TTS model with a voice thing and a lip-sync thing and what have you?” No.
Just one giant model.
Shawn Wang
Yeah. I have a two-part question. One is, it’s still open. It seems like open source is still very core to what you guys do, and I just have to plug your paper.
January 2024—when you rolled out Mixtral of Experts, it was very fundamental research on how to do good MoEs. The paper came out, and it’s a very good paper for anyone, but that’s just a side tangent.
I mean, Mixtral 8×22B was like the nuclear bomb for open source.
I think it beats 70B models.
Shawn Wang
So, 70B or more?
Yeah, yeah, yeah.
Shawn Wang
But this is a big upgrade for 7B.
Yeah, yeah. I don’t remember this. I don’t think it was January, right? It was NeurIPS.
Shawn Wang
It dropped during NeurIPS, and then everyone at NeurIPS was there. It was December—
2023, but I think, yeah, the model was updated as well. I think it was a bit earlier.
Shawn Wang
Yeah, it’s just a little update, probably.
Yeah, I know, but you have a point to make.
Shawn Wang
No, I mean, you have to check that. But I just want to hear more broadly about open source for you guys. When you were talking earlier about what’s next and what the other side teams are working on, you put out Leanstral.
This one was a surprise. I was like, I don’t—this doesn’t fit my mental model of Mistral.
Yeah, I mean, first, for open source in general, I think it really relates to the journey of the company. We’ve had an open-source thing with us since the beginning, and even before this. When I was at Meta, we released Llama.
What was really nice to see was that, before this, for most researchers, like universities, it wasn’t possible to work on LLMs. There were no LLMs available outside. If you look at many of the techniques that were developed after Llama was open-sourced, like all these post-training approaches, including DPO-type preference optimization, all of these were done by people who had access to the model. It would have been impossible to do without the model.
So, it really helps us move faster. We want to contribute to this open-source ecosystem. I think it’s a deep signal, and it has a lot of impact. All these papers in the open-source community are really helping the science community as a whole move faster.
That’s why we are releasing very detailed technical reports on Leanstral and our first reasoning model. I’m publishing a lot of research—things that worked, things that didn’t work as well, and so on. I think it’s helpful. For the audio model, we also want to share a lot of details, including for the real-time model.
We really want to continue this. We belong to this community of people who share science. We really don’t want to be living in a world where the smartest and best models are only behind closed doors, accessible only to some company that has the power to decide who can use them. I think that’s a scary future we don’t want to live in.
We really want these models to be accessible to anyone. We want intelligence to be used and accessible by anyone who can use it. That’s why we’re pushing for this mission. Leanstral is open source—not the first model, and therefore not the best—but it’s one step in this direction.
It’s a bit different from what we usually release, but we have a small team internally working on formal proving and formal math. I think that’s a subject we care about in general. We’re working on reasoning. I think we started too early, before LLMs. Doing reasoning without LLMs is very hard, especially when you work with formal systems, because the amount of data you have is negligible. It’s a very small community of people writing formal proofs.
Arthur Mensch
The reason we like it is that, if you look at what people are doing with reasoning, the problems you can use are usually problems where you can verify the output. For instance, all these AIME problems where the solution is a number between 1 and 1,000—you can compare it with a reference. Or if it’s an expression, you can compare the output expression generated by your model with a reference.
But for many—most of—the math problems and most of the reasoning problems, there’s no easy way to verify the solution. If the question is, “Show that f is continuous,” you cannot compare it with a reference, right? If it’s “Prove that this is true” or “Prove these properties,” there’s no way to easily verify the correctness of your proof.
So, it’s hard to apply reinforcement learning here because there’s no verifiable reward. You could provide a judge that looks at your proof, but it’s very hard, and you could have some reward hacking happening there. You could provide a reference proof, but there are many ways to prove the same thing. If the model gets a negative reward because it’s a different proof, maybe it’s still a legitimate proof—just different. So, it’s not going to work well.
What’s nice with Lean and formal proving is that you don’t have to worry about this whatsoever. They’re all functionally equivalent as long as they compile in Lean, right?
Shawn Wang
Exactly. It’s like a program: if it compiles and it’s correct—
Arthur Mensch
Yeah, it’s very easy, and you can apply this to any kind of—
Shawn Wang
It’s just way too small. No human will actually go and do it.
Arthur Mensch
Yeah, that’s kind of exactly it. The only people who can do it are a very small community of people doing a PhD on it. So, it’s super small, and it’s kind of sad because it’s actually very useful—not just for math, but also for software verification.
Software verification today is a really tiny market. Very few industries work on this and need it. It’s usually companies building airplanes, robotics, or things where they absolutely want to be sure because lives depend on it. But it’s very rare that people formally verify the correctness of their software.
One reason for this is simply that it’s super hard to do. Are you thinking of TLA+?
Shawn Wang
It’s the language that some people use for software verification.
Arthur Mensch
No, I know. It’s what people use in some contexts, but Coq—but, yeah, I think the reason people don’t use it more, and why this industry isn’t as big as it could be, is because it’s very hard.
But now, with coding agents, it’s going to be very different. We’re going to see much more of this. I think this industry is going to be much larger in the future with these formal methods. So, we’re anticipating this a little bit and wanted to work on it, because it’s proving a theorem and putting a function in it. It’s actually the same tool.
Shawn Wang
One of my theories is that, because the proofs take so long, it’s actually just a proxy for long-horizon reasoning, coherence, and planning, maybe.
A lot of people will say, “Okay, it’s for people who like math. It’s a niche math language. Who cares?” But you use this as part of your data mix, I’m sure, for post-training and reasoning. It might actually improve everything else. I think that’s underexplored, or no one has really put out a definitive paper on how this generalizes.
Arthur Mensch
Absolutely. I think that’s kind of what we’re seeing already. For instance, if you do some reasoning on math and then ask the model to do it in Lean or Coq, we’re still in the early stages, but it definitely shows some transfer—some sort of emergence that happens.
Some of what’s also interesting is that, in the long run, there’s a lot of connection with coding agents. Sometimes a model can see a theorem that it has to prove, and it’s very complex. Then it can take the initiative and say, “I’m going to prove these 3 lemmas. I’m going to suggest 3 lemmas, and I’m going to prove each lemma in parallel—3 of them in parallel with sub-agents—but I’m also going to prove the main theorem, assuming the 3 lemmas are done.”
This sub-agent approach is pretty interesting. Even if you fail to prove one of the lemmas, you can maybe use it to prove another lemma. So, you get some partial reward here. It’s a bit less sparse than if you just get a zero reward for the entire thing. I think we can actually stop here.
Alessio Fanelli
Yeah, it's also an interesting case just for specialized models in general, right? The cost thing you show is pretty interesting. Similar score-wise, you're looking at $30, $70, $150, and $300.
Arthur Mensch
Compared to something like that.
Alessio Fanelli
I think cost is a bit unfair, right? Because this one is at inference cost, and this one is there with their margins on top of it. But we don't know anything else.
Shawn Wang
I did want to push on that more. Not on cost, but you mentioned that it's a great way to have verifiable long-context reasoning. What are other frontiers that you guys are working on internally? There's a lot of people pushing back on pre-training and scaling RL, pushing compute toward having more than half of your training budget all on RL. Where are you guys seeing the frontier of research in that?
Arthur Mensch
You mean with RL?
Shawn Wang
Just in foundation-model training in the next few years. One thing that you guys do is fundamental research from the ground up, right? So you probably have a really good look at where you can forecast this out.
Arthur Mensch
Yeah, but I think for us, we are still working a lot on the pre-training side, and we are very, very far from any sort of saturation in pre-training. I think any sort of improvement in pre-training would be a big step up compared to everything we have done before. So, we are pretty excited about this.
On the other side, I think now we have more and more to think about algorithms that will actually support these very long trajectories. I think when it was, for instance, GRPO, it didn't really work with this kind of policy, which was okay initially because you were solving math problems that could be solved in a few thousand tokens. So, the model could generate them pretty quickly, and when you were doing your updates, the model was never too far off—still not too far off.
But now, when you are moving toward these kinds of problems where something takes hours, like 6 hours to get a reward, then your model is completely off-policy, you see. So, you actually have to bring in new infrastructure that supports this because of your algorithm. Everything we're doing internally now, we're trying to build some infrastructure that will anticipate what we have in, like, a 6-month window, which is these extremely long scenarios, at a pace that they can handle.
When we started Mistral, part of me, and maybe also Timothée, wanted to recreate this very nice environment where people are there so they can do the research they like, with a lot of resources. So, it was nice. I think things changed a lot when ChatGPT came out. After that, it was very difficult. This time is also the same again, but it was nice, and we also want to recreate part of this culture we had before.
Alessio Fanelli
Coming to the end of it, I think you guys are doing incredible work. You have a very impressive vision for open source and for voice. What are you hiring for? What's next? What are you looking for in people trying to join the company?
Arthur Mensch
Yeah, so we are hiring a lot of people on our science team. We are hiring in all our offices. Our headquarters are in France, in Paris. We have a small team in London and a team in Palo Alto as well. Recently, we opened offices in Warsaw, in Poland, and one in Zurich. We also have some presence in New York, and soon one in San Francisco. So, we are a bit everywhere, and we are also hiring people remotely.
We are growing the team and trying to hire very strong people. I think we want to stay fairly small. We find it quite efficient to have a small, very agile team.
Shawn Wang
Okay, let's focus on science for a bit. We actually are strong believers in science. We started a new science team that focuses specifically on AI for science. What areas do you think are the most promising?
Arthur Mensch
What we are most excited about right now, and something we have started doing and will probably be able to share more about in a couple of months, is exploring AI for science. There are a lot of areas where we think you could get extremely promising results if you apply AI in these domains. There are a lot of low-hanging fruits. You just have to find these domains where AI has not yet been applied.
It's usually hard to do because the people working in these domains don't necessarily know the capabilities of these models. They don't know how well AI would work. You have to pair them with exactly the right researchers, matching them with exactly what to do. We are kind of doing this matching naturally with our customers.
We have some companies we work very closely with. For instance, ISM Électronique is one of our partners. We are doing some research with them, and they have tons of extremely interesting problems—problems in physics, science, and material science that they are essentially the only ones working on because they are doing something no one else is doing.
There are many domains where AI can actually revolutionize things. You just have to think about it and be familiar with what it can do well and how to apply it. So, it's something we are exploring with our partners and customers. AI for science is very interesting.
Alessio Fanelli
Yeah. And then for forward-deployed engineers, what makes a good one? What do they need? Where do people fail?
Arthur Mensch
I think you usually need people who are very familiar with the technology—not necessarily with a lot of research expertise, but who are actually pretty good at using these models. They know how to do fine-tuning, and they know how to start an RL pipeline. It's not easy. It's something that the majority of customers and companies would not be able to do on their own.
Michele Catasta
So, we need people who like to solve problems and are excited about complex, very concrete problems. It's applied science, basically. I don't think it's too different from the skills you need in research, because essentially you are trying to find solutions to problems that customers have not yet solved.
Sometimes it's easy, and sometimes you have to do the work. You have to create synthetic data and find edge cases. It depends on the problem, but you need a bit of patience as well. You have to be creative. I think the skill sets are actually very similar.
Shawn Wang
The diversity of the work they do always surprises me. It goes all the way from the kinds of things they encounter in industries. It's just very interesting, I think.
Alessio Fanelli
Any fun success anecdotes?
Michele Catasta
I mean, yeah, it can be training this small model on the edge that just does one specific thing. We are also training some very large models with some specific languages. We are making models really good at using tools, like computer-aided design, for instance—these kinds of things.
Shawn Wang
Is that paired with vision as well?
Michele Catasta
Yeah.
Shawn Wang
Like defect detection for chips or in factories, identifying things. The diversity could be anything where you can deploy these foundation models. The work to make it work in that specific setting is basically whatever it takes to make it add value in that specific workflow.
Yeah, and it goes kind of across the stack, right? Even just pulling up the website, you have to choose—
Michele Catasta
It's so broad. Compute is so broad.
Shawn Wang
We didn't even touch on Mistral 5, the live-coding CLI tool. One thing you guys were actually, I think, the first to—
Michele Catasta
Mistral Agents.
Shawn Wang
Mistral Agents. Yeah, the agent builder—you can serve it via API and all that. I'm guessing forward-deployed people will—
Michele Catasta
Yeah.
Shawn Wang
—help build that out and stuff.
Michele Catasta
It's also why we are doing many things. I think that's part of the value proposition. Customers are always extremely careful about their data, and they don't want to trust so many partners—trusting one partner for code, giving their data to another third party for audio, and so on.
They like this. What they really like about our approach is that we can help them with anything, so they don't have to send all that out to so many clouds.
Alessio Fanelli
I think there can be many orders of magnitude more forward-deployed engineers than research scientists. They don't need your full experience, but they're still super valuable to customers.
Michele Catasta
In practice, these 2 teams are still quite intertwined. First of all, they are using the same tools, the same data pipeline, and everything. It's very helpful for the same team to get feedback from the solutions team, because they can say, "Look, these customers are trying to do this. This is not working. Can we maybe solve this in the next version?"
Shawn Wang
Yeah, so this is basically your real-world eval.
Michele Catasta
Yeah. It's real-world eval. It's not something where, for instance, if you're just working in the lab, you simply ship a model. If you don't do this work of examining it with customers, you have no idea whether your model is good at an edge case.
Even in your work before this, right, there is a very big gap between the public benchmarks, which are very academic, and the real cases.
Shawn Wang
The real cases are just very diverse.
Michelle Pokrass
In the specific context of a customer, you can fine-tune and make it better. First, you evaluate, create a solid eval benchmark, and then measure it in the context of their kind of audio. For instance, one use case is literally just the word for kids, and they have to say it out. It's a very specific thing: you're just saying one word, and then you have to grade the kid on whether they did it right. It's like RL for kids.
There are very diverse use cases and ideas that the applied scientists and engineers will use to make it better. Then, from those learnings, we incorporate them into the base model itself, so it's just better out of the box.
Shawn Wang
Yeah. It's a good full-circle system. The foundation-model evals are all just proxies for what you really care about. It doesn't make sense for there to be a one-word transcription like that. It's not something you want to fit on.
Perfect. Well, everyone should go check out everything Michelle has to offer and try the TTS model, which we'll link in the show notes. Thank you so much for coming.
Michelle Pokrass
Thanks. It's such a pleasure to be here, guys.