When Talking Becomes the Main Way We Use Computers — Pavan Muddireddy

I think a lot of people don't know how audio generation works. Voice is one of the primary ways humans communicate. It definitely came much before text was a thing. The world looked very different with neural codecs and autoregressive generation, which was really cool. It was really cool that the field progressed so far, and when we made it work, it was pretty cool.

It's an autoregressive architecture, so what you predicted already is, in a sense, committed. It becomes context, and you can't edit it out. It's like a voice, an end-to-end speech model, but primarily to serve as an interface to other systems.

1. Why Mistral moved into audio

I'm a research scientist at Mistral. I work on audio research. At the moment, I lead the audio research team here at Mistral. I've been broadly focused on multimodal models at Mistral since I joined, about 2 years ago.

This episode was produced in partnership with Mistral AI.

Mistral is a full-stack AI company, and you're in the thick of it. You're doing the actual research. What does that actually mean, deep down?

The full-stack description of Mistral applies to the whole company, in that it's serving needs at different layers of the AI stack. I suppose the company is most well known for the model layer. Since the original Mistral 7B, we've been releasing open models, both text models and, over time, multimodal models and reasoning models.

That's one layer that we continue to focus on, and one of the most important layers. It's also the place I work on primarily. On top of that, we have offerings at the application layer. We have Mistral Vibe, which is our coding offering, a chat app, and also a command-line-based interface. On top of it, we also have Vibe Work, which is a similar agentic working experience.

That's at the product layer. We also have AI Studio, which has our API offerings and workflows, where you can build AI agents easily. We also have Forge, which is a fine-tuning platform and offers other kinds of model adaptations. You can take any of our open models, or some of our non-open models as well, and adapt them to your specific needs. Again, this applies to both text models and multimodal models.

Over the last year or so, we also have Mistral Compute, which is the AI cloud offering. It's bare-bones compute, on top of which so much of the other work happens, including the research work. So, it's a full-stack AI company in that we have some presence and offerings across various layers of the AI stack. Most enterprises that try to incorporate AI usually require integration at several points in their stack, and we try to meet them where they are and give them the efficiency gains or productivity boost that they're looking for.

Very cool. What's it like just being a research scientist at a frontier AI company? You must be looking at lots of interesting research that other people are doing, going to conferences, and so on. How is all of that work being integrated into your workflow? What does your day in the life look like?

It's really fun to work at a frontier company, which is also a general frontier company as opposed to a task-based company. There are trade-offs; it's not that one is strictly better than the other. But for me personally, it's always nice to have visibility into all the work happening across the board.

Even though I focus on audio, I closely follow our work on text pre-training, post-training, reasoning, vision, and more recently, robotic foundation models as well. While I don't work across the board at any given point in time, it gives me good exposure to what's going on. It also helps me incorporate the latest techniques, because at this point, not just for audio but across modalities and tasks, the approaches are converging.

The paradigm of pre-training and post-training has subtle differences and specific nuances for specific niches, but at the same time, the broad techniques are super general. That's the success of the latest paradigm, I suppose. An average day involves doing a lot of my own research work, training models, running ablations, investigating evaluations, and following the work that's happening across the company.

I suppose it's cool as well because you get to publish a lot of your research, so you can speak openly to the community, and a lot of your models are also open weights.

Definitely. From the get-go, we've had an open-weight ethos. Our very first set of models, Mistral 7B, was what got us prominence, and we continue to open-weight most of our models, especially the largest models.

The hope is basically that we operate as one open ecosystem. We contribute to the ecosystem and take from the open ecosystem as well. The whole open frontier keeps moving. As a researcher, it's nice to be able to share our research work. Personally speaking, it's always great to share the work and also borrow developments from outside our company.

The ecosystem is quite big, so it would be a missed opportunity not to leverage those developments. It's also always nice to contribute in your own small way.

Traditionally, in machine learning, all of the different modalities were quite fractured into different areas of expertise. The language people were one ecosystem, and the audio people were another ecosystem. You're working on so many modalities, and many of the same types of approaches work across those modalities. Isn't that interesting?

I've been working in machine learning and AI for close to 10 years at this point, and it's fascinating that it has converged to a point where the same approaches work across modalities. Firstly, it's fascinating that most of the architectures are transformer-based. There are some modality-specific variants across modalities, and now there are hybrid models with non-attention-based layers as well, but broadly speaking, the architectures have converged.

Even in audio, which is a modality I started working on 2 years ago, the techniques are converging more and more toward a unified approach. I think the forcing function comes from the way you use these models. You want to use them as a single entity. In many of these cases, from a user standpoint and from an interface standpoint, it's a model that you would like to give input through text, communicate with through vision inputs, talk to, and hope it writes back or speaks back. In some cases, you want it to generate images.

I guess that's a natural forcing function for people to explore how to make it all work in a single model, because it's quite an elegant way to do it. I was surprised when it started becoming not just a nice-to-have, but also the approach that seems to be working really, really well.

We should start talking about some of the audio work that you've been doing. Can you tell me about that?

2. Inside Voxtral: the trunk, the encoder and dual streams

We started working on audio last year, and the very first model we released was Voxtral Chat. It's an audio-input, text-to-text LLM. You give it audio input and a textual instruction—or it doesn't need a textual instruction, because your question can be in the audio—and then the model produces a text response.

The idea behind that is to have a general interface for audio understanding. The model can do transcription, speaker segmentation, summarization, or answer questions. You can have a large audio document, such as an earnings call or a meeting recording, and ask it questions like you would ask about a text document. You can ask what the topic of the conversation was, who spoke, or when something happened.

It's specifically useful when analyzing, for instance, a podcast. If you want to know at what time point the speakers talked about a particular topic, you can ask it questions like that. That was the first model we worked on.

Since then, our focus has been to build foundational building blocks for audio agents. To that end, we released a transcription model, a real-time variant of that, and then a TTS model earlier this year. We are continuing to work on improving those models, and the hope eventually is to build an end-to-end speech-to-speech model in this space.

3. Speech that works in real time

That first model you gave as an example is an audio-understanding model. People might not be familiar with that, because there are omni models that have audio as a capability, and there are transcription models. This model has all of those capabilities built into it. You gave the example that you can actually ask, “When did Tim ask the question about the audio model?” and it knows when certain events happened and what time those events occurred. That's a new type of AI that I don't think many people have seen before.

4. Why voice still needs a screen

People might already do this in a cascaded way. You basically take an audio document, get a transcription of it, and feed it to an LLM chatbot. Then you can ask it all kinds of questions, as you just mentioned.

This model basically does the same thing, but in an audio-native way, so you don't need the intermediate transcription. There are a few advantages to handling it natively. I think the general principle of machine learning is that the more you do end to end, the more you remove error propagation through a cascade-like approach. In this case, you can ask it about things like emotion, which isn't necessarily captured in a transcription.

Also, if you want to query some information that isn't just in the transcript, you need to have that information captured in the intermediate format in the transcription case. For instance, for the timestamp question you were asking—when did Tim mention this in the podcast?—you are basically assuming that the transcript has timestamps at the granularity you care about. So you need to capture very rich information in the intermediate representation.

In this case, it's a transcript, I suppose, and it's always better when the quality matches the expectations to have it done end to end, purely because you don't need to capture things ahead of time. It's the power of attention, I guess: just focus on the things you care about by directly attending to the relevant aspects of the audio input.

Very cool. Now, as I understand it, you folks use one of your text models as a trunk. The promise of deep learning was always this beautiful idea that it's like Lego, right? We can compose all these different models together. You can have a frozen trunk over here, and then you can build an audio head on top. But I suppose the question is: do you lose anything by doing that? What are the trade-offs of building deep learning models in this compositional way?

Yeah, definitely. It is compositional from an architectural composition standpoint. To take a step back and describe the architecture of this model that you alluded to, we have the trunk, which is a 3B text model that we train, and that we call the Ministral series of models.

Audio input is provided to the model through an audio encoder. Unlike Whisper or models like that, where the audio input goes through an encoder that is then fed into the decoder through cross-attention, here the audio encoder produces tokens—in this case, continuous representations through embeddings—and then they're fed into the main decoder model just as direct token input, similar to how you would feed text input.

In the text case, it's a rather simple encoding scheme: you send it through a tokenizer, get token IDs, and then you just have an embedding table. In this case, the encoder is a little more sophisticated. At least in Voxtral Chat, the encoder is very close to the Whisper encoder. For the later models, we optimized and adapted it. We tried to reduce the number of layers to the minimum required to get the performance, but in the first model, it was very close to the Whisper encoder.

The encoder is itself a transformer model. It starts by extracting the mel spectrogram and having a couple of convolutional layers at the very beginning. Later, we have a transformer stack and a downsampling layer. You basically get an audio token for every 80 milliseconds, so we're getting 12.5 tokens per second with this. It's a fixed-frame-rate approach.

The compositional aspect is that you could train this encoder independently, which is very common in vision. For instance, vision encoders are trained with a large captioning dataset and then added into a decoder transformer like this and trained with the transformer pretraining task using vision-text-aligned documents.

In the audio case—or at least in this model—we don't quite train this encoder in a separate phase with a different loss or a different approach. It's a fairly simple recipe, and we always try to have simple recipes because I think they stand the test of time and are easier to scale. There are a lot of benefits to keeping the recipe as simple as it can possibly be.

In this case, we just pretrain the model end to end with the encoder and the decoder on 2 basic tasks that we describe in the paper. One is a continuation task where we have an interleaved audio-text sequence. Imagine a podcast like this, segmented based on the speaker. If my speech is represented in audio, your speech would be represented in text, and it's an interleaved training on the textual blocks. We also train it on the transcription-like pattern for basic alignment between the audio and text modalities.

Since it's trained end to end, it's almost like a singular unit, although it's compositional in how the audio is fed into the model. I personally feel the line is pretty blurry. It's not that the audio encoder is solely responsible for audio understanding; it's clearly the joint optimization that leads to audio understanding being spread throughout the model. The audio encoder is the first component that presents the information in a way that the decoder can ingest, similar to text.

How, in the architecture, do you combine the 2 modalities? I think you said you summed them up. What would the alternatives be?

That's a good question. In Voxtral Chat, it's an interleaved audio-text model. At a given position, you either have audio or you have text. More specifically, if you have an audio document, you basically have audio occupying the prefix, and then comes the text part, which is first a text question and then the model generates a text answer.

I think the summing part that you're mentioning is in the real-time model. In this case, that's a slightly different architecture, where the encoder part is the same, but when the information is fed into the decoder, the model is processing audio and generating text at the same time. That's the reason why it's a dual-stream model. The specific task here is that the model is expected to produce a transcript with an expected delay.

The model is consuming the tokens it generated itself. That's the autoregressive component. The audio stream is a read-only stream, since it's not generating audio in this case; it's just ingesting the audio. The way both the text token and the audio token are presented to the model is by simple summation.

You can imagine more complex fusion, but again, in the spirit of keeping things as simple as they can get, we went with summation, and it works just fine compared with other, fancier fusion techniques.

Yeah. I suppose one of the themes that we're going to be discussing today is the trade-off between having a big omni-model and building specialized models and using cascades. This is a great example because when Whisper came out, it was an amazing model, and there was a lot of innovation in the open-source community. Some people even hacked it to be a streaming model, right? They were doing this kind of sliding-window approach.

The obvious question is: why do we need to have a streaming model architecture and a batch model architecture? Maybe you could explain the difference between them and the trade-offs for both of them.

Definitely. I'll preface this by saying that audio is interesting to me also because the space is still evolving. There is no architecture that is the final form, I suppose. In text, too, that is true to a certain extent, but in audio, it is even more so. Whenever I contrast anything with something else, it's with the caveat that it would probably look a little different next year or 2 years from now.

At least as it stands today, the motivation behind streaming is to make it natively streaming. The property we want is for the model to continuously listen, not listen in chunks. We wanted to see how far we could push that. This approach is also inspired by the delayed stream modeling that Kyutai published.

The key difference is that the encoder is trained from scratch in an end-to-end fashion. The idea is to have the audio fed to the model at the same frame rate at which the encoder emits the tokens. In this case, the encoder emits a token every 80 milliseconds, so the model is natively fed the audio every 80 milliseconds, essentially.

One interesting aspect is that the model is expected to produce the transcript it's listening to, but there is a subtle trade-off here. The model can produce the transcript after listening to all of the audio, which would be the case with Voxtral Chat by design, and with Whisper originally as well. It takes all of the audio, or 30 seconds of it, and then produces the text.

This model can do that, too. The tokens it's producing while it's listening are a filler token, a streaming-listen kind of token. When it's ready to produce a transcript, it produces the text tokens corresponding to the transcript that it has just heard.

One extreme is listening to all of the audio and then producing the transcript. The other extreme is producing the text tokens it heard almost immediately. If it hears “New York,” then it produces a text token corresponding to that word right after it hears “New York.”

You can see how this extreme case is actually ambiguous, because in many cases a full word could involve prefixes that are themselves standalone words. The more you wait, the less ambiguity there would be. Even theoretically, assuming oracle information, a model that does perfectly well in that case would still have ambiguity and hence produce errors. The more you wait, the less the ambiguity and the lower the chances of errors. In this case, the expected delay that the model is supposed to have is actually a parameter to the model itself.

We condition the model with the target delay at hand. The target delay is the number of frames it has to wait before it produces the word it just heard. The model gives you a nice quality-latency trade-off that you can control based on the application you’re building, whether you’re feeding it to a downstream LLM, or whether it’s a subtitle use case where you’re showing it at the bottom of the video in a conversation like this. So that’s flexible.

Yeah, it’s so interesting because there must be levels of latency where you get different types of degeneracy. I’m imagining that if you make it respond too quickly, at some point it’ll just fall off a cliff. But there’s also this more sophisticated form of text production where, as you say, it takes the context into account.

To give you an example, when I do some ASR transcription, I’ll get Claude to do a transcription refinement, and it can take the entire document into consideration, maybe even other material as well. It will know, “Oh, it mistranscribed Drake Apathy’s name, and it’s actually talking about this paper,” and there’s a technical figure and a technical term in that paper, and it’s absolutely wonderful.

This is the kind of thing that practitioners can do, right? You could put a custom vocabulary in there, or you could have varying levels of transcription refinement. You could have a fast path that refines within a 5-second window, and maybe a 20-second path. You see what I mean? You can layer on various forms of refinement through engineering to make it work better.

Yeah, definitely. I think at the end of the day, it’s a system that you’re building. Voice agents are the specific system that is of most interest to us. But having said that, the models are built to be applied in all kinds of contexts, including the completely offline case where you only care about performance and the end quality. You don’t care about latency at all.

There are cases where you don’t even care about the cost. You’re happy to call many models, exactly like the case you mentioned, where you’re doing a post-production task. You’re taking a podcast recording like this, or an earnings call, and trying to do analysis over it. This is not the biggest cost consideration, and you care about being as accurate as you possibly can.

In that case, you can do all kinds of refinement, like you mentioned, where you can bring a reasoning LLM into the mix. Actually, Voxtral Chat was last year, but if you were to do it again this year, that would be a reasoning model. It would listen to the audio natively and then would not immediately produce the transcript or analysis that you asked for. It could, in its own reasoning chain, transcribe the necessary parts, do a web search, fetch the appropriate context it needs to disambiguate certain things that it finds slightly inconsistent based on what was said, and then retranscribe in its own thinking chain.

This can be done in a single model pass. I suppose it’s not even a complex system, but you can also augment it with other components. You can give tools to the model that provide context, or condition it with a custom vocabulary, as you mentioned. Custom vocabulary is applicable where the vocabulary is finite and fixed, but in some medical domains it could be—not infinite, but quite large. In that case, it can be a tool that the model calls to fetch appropriate context by its own volition and produce a higher-quality transcript. That’s definitely true.

Yeah, it’s just so interesting how we have these cascades now. I’m a top 1% user of Wispr Flow, and I use it—I talk to my agents more than I talk to real human beings, apparently. It’s so interesting: Wispr Flow will do some refinement of what I say, and then it goes into my coding agent, and it has this almost clairvoyant ability to understand what I mean. Even if something was mistranscribed, it’ll still understand what I mean.

You see what I’m saying? We will have voice agents, and they’ll have some intelligence and some reasoning, maybe even adaptive reasoning, but then they’ll probably go to another agent. The other agent will have more context, and it’s almost as if coherence will just manifest in the cascade.

Yeah, definitely. Cascaded systems are usually a source of errors, as in the case of ASR, LLM, and TTS. But in the cascades that are emerging now, with the use case you’re mentioning—for instance, coding agents, but also co-work agents, where you’re trying to leverage agentic systems to get not just coding but other kinds of workflows done, with voice as a primary interface—each of the components is also a smart system that tries to be robust against the mistakes made by the previous system.

So overall, it works quite well, but it’s not purely a win-win. There are cases where, if you’re trying to be very precise about a function name or a method, or some other aspect of your codebase that you’re trying to fix, and it mistranscribes for whatever reason because it doesn’t have the context, you can imagine how it can go wrong. If there are multiple entities that could match what you described, there’s still value in trying to improve the understanding layer.

Transcription is the simplest case, but in an audio-understanding model, it would be a holistic, native-audio-input LLM or a native-audio-input agent. I think there’s still a lot of value in trying to bring down the basic audio-understanding error rate, and that’s what we’re trying to do with both our transcription models, by making them more contextual.

You can give all kinds of context to a transcription model, and then use that transcription to feed whatever downstream system you want, in whatever way you want. Or, in a more end-to-end model, you can care about the specific cases where the failures are costly and try to address them.

Let me read this out: “It’s a decoder-only Transformer trunk predicting audio as autoregressive continuous latents reconstructed by a neural codec.” I think we should talk about the continuous latents and the neural codec, because they’re both very interesting. So, first of all, why the continuous latents?

5. How a voice becomes tokens

Yeah, that’s a good question. I think it came after several iterations. We were trying to pick an architecture for generation, and that was the Voxtral TTS model, which we released in March. The goal was twofold, I suppose: first, to provide an offering from Mistral for voice generation and voice synthesis, mostly for voice agents but also for other use cases; and at the same time, for us to do some research and come up with an architecture that we’re happy with.

We want to continue iterating both to improve the TTS model itself and to incorporate it into a future speech-to-speech system. The lay of the land is essentially that neural codec-based autoregressive generation seems to be a dominant paradigm. That was the case when we started, and it continues to be the case. The key difference is that a lot of models were based on discrete-token autoregressive decoding, and the attractiveness of discrete tokens is that they fit well into existing text-token-based autoregressive frameworks.

The neural codec itself would have a quantized bottleneck that produces discrete tokens. Examples of this are EnCodec, SoundStream, and Mimi. It’s not just discrete, and it’s not just one token. Similar to how I said the audio encoder for the understanding model produced an embedding every 80 milliseconds, here it’s similar, but instead of an embedding, it would be a set of tokens, usually corresponding to a residual vector quantizer.

You also have an audio decoder, or a vocoder in some cases, which basically reconstructs the audio based on these discrete tokens. This whole autoencoder is trained end to end to minimize the reconstruction losses and a bunch of other losses. The discrete tokens are then used in an autoregressive decoder model—the TTS model—which takes text conditioning and produces the discrete tokens one by one.

So, to your question about why continuous latents: at each time step, it’s not just a single discrete token like in text. It’s a set of discrete tokens, usually an ordered list, I suppose, in the residual vector quantizer case. Many models approached this by having an architectural affordance for handling and predicting this multitoken set.

There are many ways to do it. There’s a delay-pattern approach, but I think one of the more popular ways was to have a depth Transformer. It’s also autoregressive in codes: there’s autoregression in time, and there’s autoregression in codes. We were trying to see if we could reduce the complexity around autoregression in codes, because the number of steps you need to do in the code autoregression is the number of codebooks that you have. In our case, it was 30-plus codebooks, so at every time step you’re doing a 30-step autoregression again, although it’s a smaller model.

Given this consideration, we wanted to explore approaches that were more controllable and provided a more delicate trade-off between the number of steps and the quality. We started exploring diffusion-based approaches, and the model that we ended up with was a flow-matching head.

Instead of a residual vector quantizer, which is a discrete set of tokens, we have a continuous latent embedding in this case. The model—the flow-matching head—takes as input the final hidden state from the decoder transformer and performs flow-matching inference, where it predicts a velocity. It’s very similar to a denoising task, but slightly different in that it predicts a velocity on a velocity field. The inference consists of integrating over this velocity field, which comes down to simply summing the velocity multiplied by the time step over a fixed number of steps.

This provides a neat trade-off: the number of steps is controllable, and there are also techniques to reduce the number of steps with few-step techniques. That’s one of the reasons we picked it. The continuous embedding is also attractive because the discrete codes, in some sense, have the connotation of communication, where you’re trying to reduce the bits of information that you’re sending over a channel. But in our case, that is not a consideration. We are not using this neural codec for compression or communication over a band-limited channel or something like that. It’s purely for generation purposes.

That bottleneck doesn’t make sense in our context, and that’s one of the reasons we thought this would be the approach that allows us to expand the bottleneck and explore a larger design space.

Yeah, it’s so interesting. I remember the EnCodec paper. I think it was from Meta—was it about 3 or 4 years ago? It was fascinating at the time. Can you explain a little more about what that paper introduced and what the lineage was?

It’s mainly a residual-vector-quantizer-based approach. It’s an autoencoder architecture where you take the waveform on the input side and generate the waveform on the decoder side, with a vector-quantizer bottleneck—specifically, a residual vector quantizer bottleneck.

Then came Mimi, which is an enhancement on top of it. One of the aspects of this family of models is the distinction between a semantic codebook and acoustic codebooks. The key difference is that the semantic codebook gets distillation supervision. The motivation is to keep this codebook closer to the text space, and the order in which you predict the codes in the decoder TTS model, or generation model, is that you generate the semantic codebook first. The idea is that it is close to the text space, so it’s easier to predict. Then, conditioned on this predicted semantic codebook, you predict the acoustic codebooks, which are assumed to carry more of the acoustic information.

You can have multiple ways to pronounce the same word, and there is this variance. The acoustic codebooks are assumed to carry more of this information. First getting the word broadly right and then predicting all of these isn’t as interpretable as I’m trying to say here, but that is the motivation behind it.

In our model, the key difference is that we still maintained the semantic codebook. In the first release, we did that as a separate vector-quantized codebook, but for the acoustic ones, we used FSQ-based quantization. It is a scalar quantization, so instead of residual vector quantization, we have levels. In this case, we had 21 levels, so it’s a 36-dimensional embedding with 21 levels.

Since these are discrete levels, you can treat them as a continuous vector without the quantization, or with quantization, you can actually treat them as discrete codes. That’s one of the reasons we picked FSQ. We were exploring discrete-code-prediction-based approaches to begin with, and then later relaxed them to be more continuous as we built confidence that the quality of generation was either similar or improving with this relaxation.

There’s a bit of a theme here that we’re going to talk about, which is that when you build machine-learning models, if you want them to be more sample-efficient and work at smaller scale, there are a whole bunch of engineering tricks that you can put into them. I believe this is one of them.

6. Flow matching, FSQ and the new codec

Another one, for example, is that you could, in principle, have an ASR model learn from the waveform data directly. Or you could do something akin to a short-time Fourier transform, with the mel spectrogram. I believe that’s logarithmically calibrated to human hearing, with some discretization, and it’s often used in machine learning as a perceptual audio feature for human speech. That is a form of dimensionality reduction, I suppose.

You’ve used this, so I suppose it’s an interesting trade-off. Presumably, if you had a huge amount of compute, a huge model, and lots of data, you could just use the waveform data, but you decided not to. Can you explain the rationale?

Yeah, you covered most of the rationale pretty well. The viewpoint here is that the model, especially when you’re operating with small models, affords us a little more sample efficiency. On the input side, there is no need to model aspects of the waveform that are not necessary for human perceptual understanding.

Mel-spectrogram-based featurization is one such approach, although it is still pretty broad, to be honest. In old-school machine learning, you have very handcrafted features that are known to help with certain aspects of the task at hand. In this case, the mel spectrogram is still quite general. It doesn’t just capture speech; it can capture a lot of different auditory features, I suppose, as well as background sounds. It has at least been empirically shown to capture rich information, but it’s still a lever for getting a little more sample efficiency.

You’re totally right: as the models get larger and as you train them over larger and larger datasets, some of these things become unnecessary, and you can feed the waveform directly into the model, even without an encoder, which is becoming more common recently.

The belief is also that you’re accomplishing 2 things by removing these components. First, you are letting the general-purpose model find, I guess, a better point without your inductive bias, which is generally shown to work as you move toward more end-to-end approaches. Secondly, if you can feed the data in a more general-purpose way, the scaling becomes more predictable as you go to larger and larger models, because you have 1 less component to worry about.

Let’s talk a little bit about noise. ASR models, for example, have varying levels of performance based on how much background noise there is. I suppose there are a whole bunch of things you could do: noise augmentation, some kind of data preprocessing, or voice isolation before the data goes in.

I don’t think you’ve done that. It just works reasonably well out of the box. Can you tell me about how you can make these models robust to noise?

Yeah, definitely. A lot of approaches to noise involve augmentation, which is also true with vision models. Augmentation is a way to expand the amount of data to cover cases and make the model invariant to transformations of the data that shouldn’t matter for the task.

In object recognition, for instance, if you have a flipped image, ideally it shouldn’t matter, and scaling the image shouldn’t matter. In the audio case, noise is, in some way, a similar transformation. If you have speech being spoken, up to a point, noise shouldn’t change what was spoken from a perceptual standpoint. Hence, the prediction task should stay the same.

We also want to make the model robust to various acoustic scenarios that we might not have coverage for in the data. This is a more automatic way to simulate these acoustic conditions in the limited data we have and work with the data limitations.

For the initial versions of the model, we didn’t have to do a lot of noise augmentation, precisely because one of the things we try to cover in our datasets is a variety of acoustic conditions. Having said that, we do some noise augmentation, especially to cover the tail cases, and it does add robustness to these models.

It’s quite important. When people report ASR performance, it’s usually presented on datasets that are more academic. It’s not that they are all very clean systems: there are evaluations such as Mozilla Common Voice and things like that, where the acoustic conditions are a bit noisier and more uncontrolled because people are recording on their laptops in all kinds of background settings.

Still, when you deploy a system in the wild, the diversity of acoustic conditions is a lot greater. Usually, models that are specifically tuned for a particular set of acoustic conditions sometimes struggle to generalize to these other scenarios, and you get models with a sharp drop-off in performance. That’s what we’re trying to avoid.

Yeah, because I suppose even dataset selection and augmentation are inductive biases at the end of the day. As you say, if you have sufficient diversity, then the model will just learn to generalize.

At the small scale, it might overfit to certain acoustic environments, which means it works very well on this particular microphone but not very well on another. I wonder how much work you folks have done around that. Is this the kind of thing where you identify failure modes and lean into them? So you say, “Okay, we need to do more adaptation and augmentation here,” or have you reached a sufficient level of data and diversity where it just kind of washes away?

That’s a great question. I think the model is generally performant in a wide variety of acoustic conditions. Like you said, when we identify areas where it doesn’t work well, based on feedback from customers who deployed it in the wild, we try to gather data covering that distribution and improve the model. But, yet again, I think this is one area where model adaptation is quite important.

That also alludes to the popularity of fine-tuning Whisper in the community. Usually, when you deploy this kind of ASR model, by virtue of deploying it, you have the precise data distribution that you care about—the one where you’re using this model. You’re very well equipped to adapt the model and improve it, even if you have a reasonably small amount of data compared to the large corpus usually used to train these models.

A small amount of exactly in-distribution data, with some of it annotated with high-quality transcriptions, is very useful. Having an internal workflow where you can transcribe this audio is one of the reasons why we have Forge, the model adaptation platform. You can take any of our existing transcription models and adapt them with your own data.

Usually, one of the concerns people have is that some of this data might be very sensitive, especially if it’s meeting recordings in the context of a company. You would like to keep it to yourself, fine-tune the model, and have a variant that is very much applicable to you and performs really well in your situations.

You don’t care about how it does in other acoustic situations because you deployed it in this very specific setting. You don’t want or need a general-purpose model in that way, I suppose. If you’re happy with the performance, you can deploy it in the context of your company or wherever you deploy it.

Can you tell me more about fine-tuning? Is it one of those things where, with great power comes responsibility, you need to be extremely careful about still maintaining diversity in your fine-tuning data? Or is this the kind of thing you folks can do as a platform, so you can still inject more diverse data to stop the model from becoming degenerate? Talk me through that.

It’s more so for the text models, but even for audio models, it’s important to have diversity when you’re fine-tuning, precisely as you mentioned, to avoid catastrophic forgetting of the general-purpose training the model received at an earlier stage.

At the same time, for a task like transcription, it’s a little bit easier to fine-tune. The recipe is fairly straightforward, and people have been doing it for a while in the community. You see a lot of fine-tuned transcription models on Hugging Face, for instance, because the transcription task is fairly well understood at this point.

The things you want to end up with a good fine-tuned artifact are high-quality data with precise acoustic conditions. You care about noise conditions, recordings, and so on, along with high-quality transcriptions, which are very important. If you have mistakes there, especially since you’re fine-tuning rather than doing large-scale training, it’s important to have high-quality data.

You also want to start from a model that is already fairly good at the task. That’s one of the reasons we also pick text-based trunks for our transcription models. In the future, it would be a holistic audio capability as part of a large language model in general, because transcription, while a lot of it is very localized, also benefits from world knowledge.

I think it’s very obvious in the case of getting entity names right. If the model is already aware of these entities, then, based on the context in which you’re speaking, it can easily guess. Starting from a good model and having a high-quality dataset are the most important ingredients.

Some of the general principles about having diversity in the fine-tuning stage to avoid catastrophic forgetting are also applicable. In our case, they’re a lot more applicable to text models. You do need to have a good mix; getting the details right is quite important in fine-tuning. That’s where the platform part is also important, because we offer recipes that are well tested in various contexts and should usually work reasonably well out of the box.

Very cool. Now let’s talk about speaker diarization. It’s possible to do this in a specialized model, but you folks have built it into the model. The model is actually outputting speaker-change tokens as part of its output. Maybe you should explain whether it’s in your streaming model and your batch model. How does that work? It must be quite a difficult problem. What kind of issues do you have with it?

7. When speech models lose the speaker

Like you were alluding to, the way we do speaker diarization is as an autoregressive task. It’s not a special head or a separate stream.

In our offline or asynchronous model, the way it is usually done is that you have the audio, and then, for a plain transcript, the model would just produce the text transcript. In the context of timestamps, we produce a segmented transcript with the start time of a segment, the transcript, and then the end time. There are some special tokens to represent the time delimiters, I suppose, and the model would also predict the speaker.

It’s basically one additional piece of information on the segmented timestamp. In addition to the start time and the end time, the model also predicts the speaker ID. It’s a fairly simple convention: it goes from speaker 0 to speaker i, assuming there are i + 1 speakers in the conversation. The model is trained with the same autoregressive loss, with the speaker representation incorporated in the case of diarization training.

Very cool. I use a lot of ASR technology myself, and speaker diarization is a great example of something where, prima facie, when you look at it, it seems like it has done an amazing job. Then, when you actually look at the speaker changes, you see the failure modes.

What I tend to see is that sometimes it takes a few seconds to detect the speaker change. It gets it, but it mixes things up by a few seconds. I also see significantly more failure modes in streaming models than in batch models. Streaming models will create additional speakers, so they might pick up a fifth or sixth speaker when they were actually previous speakers.

Streaming models are fundamentally more challenging because they operate with less context and have to make do with whatever context they have. In our API, and also in the recipe for the streaming model we have, the model provides flexible target latencies, as I was mentioning earlier in our conversation.

You can give it a target latency of, say, 160 milliseconds, which is very aggressive. You can also have another stream open where you give it a 1- or 2-second delay. The idea is that for anything you want to display to the user right away, such as subtitles, you want to show the transcript with as little delay as possible to keep it relevant to what’s going on.

At the same time, if this transcription is going to be logged somewhere for future reference, you can use the slow stream to correct the mistakes in the fast stream. The more context it has, the better it gets. With this model, it’s as simple as opening a second stream because the delay is a parameter fed to the particular stream that is open.

To your point on diarization, it’s actually quite challenging. I think it’s far from solved, in my opinion, especially with multiple speakers—more than 2 speakers—in the context of a meeting. Even day-to-day meetings at a company, where there are 4 or 5 people speaking over each other, are very hard to get right.

The challenges are quite diverse because, as a human, you also have visual input. You have the audio source and know where it’s coming from. You’re present in one part of the room and know where all the other people are located. You have a lot of additional auxiliary signals that you can leverage to disambiguate who is speaking and when.

When you actually listen to these meeting recordings, I don’t know if you’ve ever tried to annotate them, but it’s a very painful exercise. You can empathize with the models there. It’s a really hard task to recognize who is who when you have no video and just have to listen, especially when you’re not listening to people you know and whose voices you already recognize.

These are people from a random meeting whom you’ve never met, and some of them sound somewhat similar. You don’t know if they’re the same person speaking, and then you have to take a step back, read the context, and see, “Oh, okay, it doesn’t make sense for the same person to be interrupting themselves, so it has to be a different person,” or something like that. So it’s a very challenging task, especially since the way we cast the task is as an audio-only, single-stream, collapsed setting. There is definitely a ceiling that you can achieve with this constraint.

Having said that, the current performance is well below that ceiling, in my opinion. I think this is especially true in a more-than-2-speaker, noisy environment. Regular speech, even in a 2-person conversation, and especially in conversations with more than 2 people, has a lot of overlap.

That’s just a natural property of speech. It’s not turn-taking; it’s not that I speak and then abruptly stop, or that you only start after I completely finish my thought. That’s not usually how regular day-to-day conversations go, and that adds to the challenge because you’ll have many segments of audio where more than 1 person is speaking. You not only need to recognize that more than 1 person is speaking; you need to recognize who is overlapping and what each one of them is saying independently.

Very cool. The cocktail party problem has been a huge problem in speech technology for many years. It’s this famous setting where you’re at a cocktail party and there are many different people talking. There’s John talking behind me, and we can focus our attention, right?

We have 2 ears, and our ears are actually doing this kind of time-based recognition. When the sound waves hit both of our ears, our brain can distinguish different sounds. As you said, these models don’t have all of that additional awareness, so they can’t do the kind of conversational understanding of the dynamics that we can do.

Maybe in the future that’ll change, because we’re going to have models that have duplex and much more information. On that note, though, I did notice, working with speech models about 2 years ago, that some of the frontier speech models could do what I can only describe intuitively as active-speaker locking. If there was crosstalk, it would lock on to what it thought was the active speaker, continue to transcribe that voice, and ignore other voices. Maybe that was an emerging property of that particular frontier ASR model. Have you noticed stuff like that?

Yeah, I think it also comes from the closed-caption data, I suppose. The models are trained from data, and at the end of the day, I think a lot of the properties of the model can also be attributed to the annotation quirks. It can usually be explained by that. Not everything is explained by it, but a lot of what the model is exhibiting can be attributed to the properties of the dataset that you train it on.

Having said that, it is a challenging problem, but at the same time, the model doesn’t need to be like a human. The whole point, I suppose, is that, at least in principle, it can be better than humans. It doesn’t need to adhere to the constraints of humans; it can be superhuman that way. Unlike us, we can only focus on 1 speaker at a time, and we have this nice ability to decide who to focus on, which is pretty cool.

But models, as long as they have enough information—for example, if instead of a single channel or a single microphone, they can take input from several microphones across the room and have additional information about where these microphones are—they can follow all the speakers all the time. They don’t need to focus on a single speaker in that simpler way.

In our model, we are trying to tackle this problem. When there is crosstalk or overlapping speech, the expectation is that the model transcribes both speakers and identifies when the first speaker stopped and the second speaker started, even when the second speaker started before the first speaker stopped speaking. It is trying to transcribe multiple speakers at the same time, but it’s a limited setting in the sense that the input is still a single-stream, collapsed audio channel, I suppose.

At least the goal is to make it superhuman, and it actually does better than humans in quite a few settings. There’s still always room to go further.

Great stuff. Let’s talk about hallucinations. Do you remember back in the days of GPT-3, when it first came out, how it would get stuck in these degenerate loops? It’s almost surprising, isn’t it, that we can use an autoregressive model and it’s coherently not only transcribing things, but also putting time indexes and speaker-change labels in there? It’s amazing.

But there are still occasional hallucinations, and you folks are using DPO to minimize them. Can you explain that whole process?

8. Correcting hallucinations with preferences

As you rightly pointed out, the hallucination problem, or at least the degenerate-generation problem, can be somewhat attributed to the architecture itself, owing to the limitations of the existing autoregressive architecture. Once the model makes a few mistakes, it tends to commit to those mistakes, especially when they take the model out of its training distribution.

That’s usually the scenario where it goes into a degenerate mode of infinite generations, loops the same prediction, or skips a whole segment of transcription because it decided to skip 1 part of the transcription. It continues to skip the whole part because it doesn’t want to predict halfway, again owing to the out-of-distribution nature of its initial mistake, I suppose.

There are many ways to tackle this, and DPO is a simple but quite powerful way to fix issues like this. These issues fall into a small set of categories, and the recipe here is fairly simple: You collect generations from the model that have become degenerate, and you generate a correct version of them. That becomes a winner-loser pair.

The DPO training process usually involves training directly on pairs of data like this, as opposed to having an explicit reward model. That is also an equally valid and effective way, and it works. DPO is just a single-stage process, and for fixes like this, which you can consider alignment of sorts, it is quite effective.

You would have a winner, which is the correct transcript, and a loser, which is the degenerate generation from the model. Usually, you want to limit the distribution so that the losers are actually generations from the model. It is more on-policy for the precise model you’re trying to fix; you’re not trying to generate an infinite generation for scenarios the model never actually encounters, which would be off-policy.

For all the categories of errors, you generate pairs like this, and it is quite effective in fixing most of the problems. The same thing works with hallucinations, skipped transcriptions, and some family of transcription mistakes. It’s a fine tool. It’s not a way to support a new language or something like that. It’s a nice post-training tool to fix small but important problems in the model.

Can you give me your intuition on what’s actually going on there with DPO, above and beyond, let’s say, using supervised fine-tuning? My intuition is that, as you just described, you’re helping the model develop a kind of conceptual-space understanding: This is good, this is bad. It’s exploring around that space reflexively, and you’re helping it conceptualize in a way that perhaps wouldn’t be possible in the base model or with SFT. Is that a reasonable intuition?

Yeah, that’s a reasonable intuition. At least from my point of view, one of the primary tools that DPO gives you is a way to provide negative supervision. With both pre-training and SFT, the supervision is purely positive. You’re just reinforcing what the right thing to do is.

In pre-training, it’s just a general text corpus, and in SFT, it’s the specific behaviors you want to get out of the model, whether it’s a chat model or a transcription model. But they don’t offer a mechanism to say, “This is wrong. Penalize this. Reduce the probability of this sequence.”

DPO is not the only way to do that. Even many of the RL techniques offer a similar mechanism. DPO is one of the simpler variants, providing a mechanism where the winner gets reinforced and the loser, or the erroneous one, gets penalized by the objective function.

Yeah, that makes sense. So you’re saying the most load-bearing thing—sorry to use a Claude-ism there; I’ve been using Claude too much—is that it’s mostly pushing the energy landscape down on the negative. We have the base distribution, and we push the energy landscape down. But is there another thing going on as well? Is it pulling up the energy landscape, and is it actually learning and exploring more in the positives? Is that part of the story as well?

Yeah, the loss term of DPO looks like a weighted positive loss on the winner, and then it’s a negative loss on the loser.

It's not just that there is a weighted nature to it; there is a nice formalism in the original DPO paper as well. There is a lot more follow-up work trying to build intuition about what the algorithm actually accomplishes, but it looks to me like a margin loss, which pushes the positives to the more positive side and the negatives to be predicted less by the model after this stage. There is also a component that tries to keep the model from moving too far away from the starting point.

You don't want to move too far from the initialization because you want the DPO model to fix the subtle mistakes while, at the same time, keeping most of the properties that you got from the SFT stage.

How do you detect degeneracy? Presumably, you use that as a lens to synthesize or select these positives and negatives for DPO. What does that process look like?

It usually comes from many different places. The eval is one of the primary ways to detect this. You can define metrics for each of the degenerate subclasses, with infinite generation being one and repetitive generation being another, or based on looking at the error patterns that you yourself notice or that are reported by users of the model.

Some error patterns are not captured by your evals, which is a good sign that you need to build a new eval to capture these cases, because they are likely to happen in future models as well. Once you capture the error patterns, you would have a data-generation phase that involves reproducing these kinds of bad patterns and pairing them with a ground-truth equivalent in those cases.

It totally depends on how you source this dataset. There are many, many ways to do it. You can have a ground-truth, transcript-annotated dataset, run your model, and take those subsets of cases where it triggered any of the detectors that you had for degeneracy. Then you generate a paired sample out of that.

Very cool. Very cool. Let's move back to TTS. The technology is getting incredibly good. You folks have a TTS model. What fascinates me is: where does the emotion come from? How does it even do that?

9. Controlling synthetic speech

The way the model is trained is the exact opposite of the ASR task. The input would be text and the output would be audio. In our case, the loss function is a flow loss, which is an L2 loss for velocity estimation. The model is trying to model a distribution of human speech from whatever dataset you have, and any reasonable dataset with enough variability in human speech—which we should see from any audio recording, I suppose.

The model makes associations, like any other deep learning model, between the textual semantic content and the speech. The first level of mapping it needs to do is map the words to appropriate phonemes implicitly. Older TTS models did this explicitly: phonemes were either generated or used as an intermediate stage of the model. In more recent versions of TTS architectures, including the one that we released, it is completely end-to-end, and this kind of association happens somewhat latently in the model.

The first level of mapping is to determine what has to be spoken—the high-level thing—and then the next level would be a function of the semantic content of the text. Based on the semantic content, you should be able to tell how it should be spoken, although that is not always true. You can have neutral text which, depending on the emotion of the person, can be spoken in different ways.

There is some text that is only spoken in a certain way. For instance, you can have sports commentary. If you read sports commentary, it would have lots of ups and downs depending on what is going on in the field. You will have very excited intonation when there is an exciting event going on, like a goal being scored or a home run in the context of baseball. You can have all kinds of ups and downs in a game, and just based on the text, you should be able to tell reasonably unambiguously where the emphasis should be. The models learn that.

For all the other places where there are multiple options, I think one aspect of TTS learning is this one-to-many learning, which, in one way, distinguishes it from the ASR task, where it is mostly a many-to-one mapping. In TTS, it reverses itself from one to many. It is a multimodal distribution, I suppose, and that is one of the reasons why techniques like flow- or diffusion-based techniques work really well: they model the multimodal distribution quite well.

Is there a bit of a trade-off between naturalness and variability? I'm sure many folks at home go on ElevenLabs and generate a bunch of TTS utterances, and they vary quite a lot. Humans are incredibly good at detecting changes in tonality, and there is also a lot of meaning in tonality. What we don't want is TTS systems or voice systems that are disingenuous, right?

What we tend to do is many generations and select the one that sounds most natural. But obviously, in real-time production systems, you just have to take the one that is generated because you can't go back and regenerate many times. Is this something that you take a position on? Do we want something that is quite stable, or something that is very human but runs the risk of being disingenuous?

To start with our stance on it, our stance is to make it customizable. As a user, you get to choose which mode you want to operate the model in. There are a few knobs to control the behavior of the TTS model, and the biggest one is the voice you pick from the system. The model supports voice customization, so you can give it a recording of a person. The persona you want to give the voice is derived from the reference.

For short references where there is quite a bit of ambiguity, there is an opportunity for more variability. Even there, you can control the behavior of the TTS system by having a voice reference that is quite neutral, quite safe, and has less variance. On the other hand, if you are using it for a creative use case that is not real-time—which is also a perfectly good use case for both our TTS model and many other TTS models—then you would actually want to pick a more variable, somewhat wild variance in some cases.

You are looking for different candidates in your creative process to pick from, and that is actually a feature, not a bug. Depending on your use case, you can pick the property that you want. As a research scientist and as a research team, when we are training the model, one of our goals is to train a model that is general enough to represent both behaviors in the same model and control that behavior through the knobs, with voice being the primary one.

We also plan to expose instruction-following-based control over how to generate these things, and hopefully generally add robustness to the model. There is an equivalent to infinite generation or hallucination in the TTS world, too. It is very similar to ASR: you can say a word that is not there, skip a word that is there, or keep repeating a word and go into an infinite loop. All of these problems exist.

The goal is to be much more robust to these problems because this model will be used in a real-time context, and it is much more jarring to hear these bad generations in an audio output as opposed to a text output, because you have to listen to them. We aim to make them much less prevalent in this model.

Based on your use case, if your tolerance is extremely low because you are putting it in a real-time context—in a customer service agent or something that interfaces with consumers at scale, where you have hundreds of thousands or millions of consumers—then picking a safe voice seems like the more conservative option to take.

But that also depends, I suppose, because you can have all kinds of audio agents now, not just for professional customer service. There are personal use cases where users might expect more variance from their audio agents. In that case, it becomes a trade-off based on your use case.

One of the things in Work and Wbe code is the ability for people to make custom AI agents for their workflows. The hope is that you have a similar lever with audio agents as well, where you can customize an audio agent for all kinds of one-off use cases in your enterprise context. You can deploy them for all kinds of use cases in an enterprise.

In these cases, they’re all productivity assistants in some way, but they’re aimed at slightly different things. Those are the main use cases that we’re targeting with the system: productivity assistance and something that interacts with your external customers, with customer support being the top one. But that itself is a big category with a lot of subcategories, and there are adjacent ones as well.

Very cool. Now, if I understand correctly, you folks are about sovereign AI and allowing engineers to build systems from your component building blocks. Engineers would build cascades from your models, integrate them into their systems, and wire them all together. That obviously gives incredible flexibility because it’s running on-premises, the models are small, and they can fine-tune them and control their data.

But is it also quite difficult? Do you provide recipes? I mean, how do you make it easy for folks to compose all these things together?

Yeah. One of the things we also have is an applied science and applied engineering team, which is like a forward-deployed engineering equivalent. The main idea behind this is that we provide the building blocks for you to make workflows or proof-of-concept exploratory products and applications in your enterprise context, or any other AI-related tool in the context of your enterprise.

Usually, there are cases where you already have a preset opinion and experience about what you want to build, and you’re just looking for building blocks to use in what you’re building. In that case, it’s a combination of self-serve plus interaction with our company to get that going. In the other case, such as model adaptation or customization, or when it’s a slightly open, exploratory project, the usual workflow would be to engage with our applied team and work together. Our job would be to make sure of the success of your use case and find the best way to get it done with the tools that we already have.

Yeah. Just to sketch out an example, if I wanted to make a customer support agent using this tooling, I guess I would have some kind of database with intents in it. I would be using a language-model agent, and then I would be using TTS to generate some kind of conversation flow. I would also be using ASR to listen to the customer when they said something.

Some of the complexities here, I guess, are that when the TTS is talking, the ASR needs to be turned off because otherwise there would be a feedback loop. You see what I’m saying? We would construct this cascade, but there’s a little bit of complexity there just to stop the system from going crazy.

This use case is precisely the kind of thing we’re talking about. You want to get started right away; you don’t want to spend a lot of time building the scaffolding that has already been somewhat solved. The cascaded system itself will have limitations owing to it being a cascade.

But if you want a state-of-the-art cascaded system today, with state-of-the-art components—and the components we’re building are hopefully pretty state-of-the-art, especially for your use case—at the end of the day, you care about whether it’s frontier for your use case, not frontier on some external benchmark. You can adapt either of the components, as we were talking about with adapting an ASR model for your acoustic settings, and the same with TTS.

Yeah, because I think this is the thing that I want to get across: when you can do adaptation and set constraints, you can always strictly win. A lot of people at home might be saying, “Why couldn’t I just use an omni model or GPT live or something like that?” There’s almost too much flexibility, right?

I’m in a large enterprise and I want to have an observability layer. I want to have hard constraints. I want to say that when people are talking about this, it needs to be logged in the database, or that it needs to respect a particular security boundary. It needs to be running on my server because we’re fine-tuning it with my data.

You see, we can do a lot of customization, and when we do a few iterations of that, we can actually make something that is strictly better for my particular use case.

Exactly. I think the current voice stack, owing to it being a cascade, also gives you a lot of observability into the system and interpretability, because each of the components has a natural-language interface, which makes it quite interpretable.

On top of that, the LLM component of these audio agents has a text counterpart in many cases. If you’re deploying a customer service agent, you probably also have a text version of that customer service agent, where people can chat with it over a text-based interface to get their problem solved.

If you have a food-ordering service, you probably already have a text-based interface for dealing with customer complaints about delivery problems or getting the wrong items. You probably already have a safety system on top, scaffolding, observability, traceability, and all the other enterprise compliance layers added to it.

In this case, for instance, you just want to add an audio interface to the system that you already have. This allows you to build it fairly easily because, in the audio agent, you could have a personal, emotional conversation, but in many cases your goal is simply to give a more pragmatic, practical interface to the system through audio. You can make a phone call and interact with it.

The goal is not to have the most personal, emotional conversation through this system, but rather just to get the task done. In that case, it’s a fairly powerful system. You could replace it with a more end-to-end approach eventually, sure, I suppose, but today these systems offer a level of flexibility and control that is quite nice.

What does the frontier of voice technology look like today?

Yeah, I think there are a lot of claims about even the basic ASR task being solved. But when we talk to customers—we spoke to a few customers just a couple of weeks ago—the primary complaint is that it’s far from solved in the precise scenarios they’re deploying it in, and it makes a ton of mistakes even for the most prominent customer service cases. They feel it’s not solved. They have to add so much scaffolding just to deal with all the corner cases that you encounter when you deploy an audio agent over millions of sessions.

10. Speech in the wild

There are components around it that are not necessarily model-related problems, but there are very much model-related problems as well, including basic things like the model not being good in different languages. It has a sharp drop-off outside of the top languages. Again, owing to the data, there’s a lot of English audio data in the wild. It’s a very similar story on the text side as well, but in the audio case it presents itself as the transcription quality going down significantly when you move beyond the top languages.

When you move away from clean acoustic conditions, take an example where an audio system is deployed on a factory floor, which is a fairly common use case, I suppose. You want to provide an interface to control an automated system—not necessarily an AI agent, but I’m sure there will be AI agents going forward because they can help you solve problems hands-free when you’re already doing some other job or managing heavy machinery.

In those cases, you can have all kinds of background sounds and noise, a lot of people speaking with each other, and crowd noise going on. In that setting, especially if it’s an interface to a command system, you need to get it very much right. In many cases, the current systems fall short, and that’s one of the reasons why customization is quite interesting.

While we always aspire to incorporate more and more variability into our general-purpose main model, there’s always a data distribution that isn’t captured by the model, which is probably so unique that it’s only in your context that you see that kind of audio. The model is reasonably good, but you can always make it better.

It’s not unlike text models. I believe it’s also true with general-purpose reasoning text agents: you can always make them better for your use case, but the cost of doing so is so high, and the general-purpose model keeps improving year over year. You’d rather use the general-purpose model and hope for the best.

But in audio, you don’t need to be constrained by that. The models are fairly small and customizable, and you do have data to customize. The cost of experimentation is low. You’re not trying to fine-tune a 1B model here.

I think it’s easy both logistically—you don’t need that much GPU compute to do it—and from the data side. It’s a fairly simple task. It’s hard to get the quality control right and to get the annotation quality high, but at the same time, the task is fairly easy to describe: just write down what you hear for this specific piece of audio.

So, yeah, I think in-the-wild performance is the main thing that is not solved for a wide variety of settings, and language coverage is one of the primary things I always hear about when it comes to audio models.

Yeah. I think voice technology is going to become quite ubiquitous over the next 5 years. Voice seems to be almost a special case. I don't know whether you would agree with that, but it's something that people will increasingly train and embed into their ways of working and their lives in a way that perhaps a large language model might not be.

Yeah, voice is one of the primary ways humans communicate. It definitely came much before text was a thing. Even today, there are a lot of places where voice is the main way people communicate, and even in text-heavy environments, I think voice is still important. It may not be the primary way people communicate in a work context, but we still have meetings all the time. You don't need to have meetings—you can just work over Slack—but voice adds a level of depth that cannot be conveyed through purely textual interfaces.

11. Audio models as interfaces

It's always been the case that voice would be a big interface for interacting with machines, and maybe with other humans as well. It's always been envisioned in science fiction, and that's one of its goals. I worked on Google Assistant for a while, and that was actually the goal there as well.

I think the previous generation of technology was not quite there. Even today's cascaded systems are much more powerful than the previous generation of cascaded systems. Today's technology is at a point where this very seamless, omnipresent, voice-based assistance is becoming more of a thing.

The cascaded approach itself can solve a lot of the problems, but more end-to-end, duplex approaches are also emerging because the technology is making it possible to build more robust duplex systems. It's not sufficient to have a prototype demo; it needs to actually work in a real-time context, across all kinds of strange corner cases that you would encounter on a day-to-day basis.

Whether it is a real-time translator to make it more accessible to talk to someone who doesn't speak your language, or—as you were alluding to—a language-learning tool, there is a lot of opportunity. People primarily try to learn a language by reading it. It rarely happens that you have to regularly talk to someone to learn a new language.

With a voice agent that can capture the nuances of the target language you're trying to learn, as well as the subtle mistakes you make in pronunciation or other aspects of speaking that language, you can improve much more significantly. There is a lot of opportunity in language learning, and also as an interface to AI agents for orchestrating and getting work done.

The interface to this would probably look like a human-to-human interface in today's organizations, where you're delegating work to other humans or collaborating with other humans to get your work done. You speak to them many times through a meeting, and I can imagine it might look very similar when interfacing with AI agents that are also getting your work done.

You can describe what needs to be done through an audio interface, and they can communicate back in audio. It is not the exclusive way of communicating; they can communicate through text or whatever other channels are available. But audio is definitely going to be one of the key pillars of interfaces, I suppose.

I don't know whether you would agree that, in some ways, it's actually cognitive-debt-promoting. I love listening to audio podcasts. It's a non-interactive stream of information, and audiobooks go into my mind. For closed-ended applications, it's wonderful.

Let's say I build an application with a very clear interface, and I'm talking to the voice agent. If I already understand the interface and the information, this is very efficient. But if I use a voice agent for coding, email triage, or researching a podcast, that's an absolute nightmare.

12. Why cascades still win

I'm trying to understand why it is. I think it's because, when I have the text in front of me while using an agent, I can very rapidly select interesting branches of inquiry and say, “Go there, go there.” It might just be because the voice agents aren't very intelligent, and sometimes I don't trust what they're doing.

When I tell it to do something in Codex, I don't know whether it's using my existing Codex model. I don't know whether it's giving me an answer or routing the answer to the Codex model. I don't know whether it's making a new Codex session or using an existing one, and I feel a little bit blind. So you see what I'm saying? It's kind of magical, but there's also this weird new form of cognitive debt at the same time.

Yeah, that's a great point. One of the experiments we did a while back—not here—was to have people make a food order through a pure audio interface. People found it much harder because, when you have a menu in your hand, you can see everything.

You're talking to the person who is taking the order, but you're doing 2 tasks at the same time: absorbing the information, making a decision, going back and forth, and communicating with the person. Imagine doing it without the menu. The person serving you would read out the full menu, and now you have to load it all into memory. You would forget things as new items came in rapid-fire succession, so it is simply not the optimal way to do it.

At the same time, people still do it. At least a subset of people still enjoy interacting with the waiter and making the order while having the menu. I think the same philosophy applies to these voice interfaces. They are still nascent, so I'm sure they will evolve over time, but basically it would look like a mix.

You would have your favorite IDE or interface—a visual medium—and voice would be an augmentation or auxiliary layer on top. You don't need to worry because the visual medium provides 2 things: a way to ingest information rapidly and go back and forth, and confidence and proof that what it says it is doing is actually what is being done.

You have 2 ways to validate what is going on. There are places where pure-audio interfaces make sense, such as when you're wearing headphones or glasses, or using all these new gadgets that people are exploring now. Audio-only makes sense because there is no screen.

You probably would not do certain things there. My favorite example is making flight bookings because it is a costly endeavor. If you get it wrong, you need to do a bunch of things to undo it. That might not be the right task for that specific device, but you can still use an audio interface when there is a screen at hand, for instance.

It would probably be an auxiliary communication medium on top of what you already have. People will experiment and figure out which approach works best. As for your other question about audio models being less intelligent than their text counterparts today, I think that is also just an artifact of the way they are developed today.

There is no fundamental reason for them to be any less intelligent than text models, especially when it is a single model. Even if that is the case with some models—I don't know how some of the others are developed; I can only speak to the models we are developing—it could be the case initially, but there is no fundamental reason for them to be any less intelligent or any different from a text-based model.

Yeah, I suppose so. It is all a trade-off at the end of the day because there is always a bigger model.

And it is not just that the bigger model might have more context. It might have access to my memory system, it might be doing Google searches, and so on. There exists some perfect cascade, and we are trading off intelligence, latency, and a bunch of different things.

It is fascinating. I think we are going to have a lot of fun over the next 5 years engineering all of these systems and figuring out how to make them work. It is quite interesting as well that maybe the reason our memory is so bad is because we have become so conditioned to having these visual user interfaces.

Maybe when we use voice agents again, our memory will improve, and it might be less of an issue.

Wonderful stuff. Pavan, it has been such an honor having you on the show. Thank you so much for joining us today.

Thanks a lot for having me on the show. It was really fun chatting about audio and all the other research efforts around Mistral. I hope to see you again.