⏭️ Forward Deployed：2026年语音AI，哪些方案真正有效

All right, we're in the remote studio. This is a special one because we're launching a fourth podcast on Latent Space. People don't necessarily keep track, but we cover different things. Basil came across my radar because you're doing all these dinners, gatherings, panels, and podcasts on forward-deployed engineering. You hosted the FDE track at AI Engineer, and it did super well. So, welcome to the pod.

Yeah, thanks.

So, you've been running your podcast for a while. What made you decide to focus on FDE, and what's your typical self-intro?

Yeah, so I started as a product manager. I was specifically working on Credit Karma for a couple of years. I worked at a small venture studio after that, and then I ended up starting a consulting business called Exaflop Labs, where we were working with retailers, insurance companies, that sort of thing—just building agents, essentially. This was 2 years ago, so it just felt like a natural extension of a lot of the product work that I was doing earlier.

It was like, “Hey, I'm working with customers. They have a specific type of project that they want built for some use case, and I'm going to understand what they want. I'm going to make trade-offs, and I'm going to help them build that.” So, yeah, it just felt like a natural extension of what I was doing.

Back in January, we started working with a couple of private-equity firms, and a lot of people were just like, “Hey, I'm following what's going on on Twitter and what's going on on LinkedIn. I don't know what's marketing BS and what's not.” So I was like, “Why not just bring on people who are working on the cool stuff at cool companies? We will talk about what they're working on, we'll do a deep dive, and we'll record it, we'll post it, and hopefully people can take learnings away that they can apply to their businesses?” That's why I ended up starting to do a lot of the fireside panels and podcasts that I've been doing since then.

Can you rattle off your greatest hits—what you've covered?

Yeah, so our very first episode was on the future of agentic engineering, so we brought on companies like Factory, Cognition, Composio, and Sourcegraph. We've done panels on voice agents, which is actually the one that we're going to be showing here. We've done some on computer-use agents, and we've done agents in the enterprise. So, we've done a ton of stuff just to get nitty-gritty into the details of a lot of this stuff.

Yeah. And I guess, for our first feature, you picked the voice-agents one. What are we about to listen to, and what stood out in particular?

Yeah. So, we brought on some people from Decagon, Vapi, Retell, Daily, and a company called Smallest AI. Basically, we just talked about what is the state of the art in building voice agents, because that's a very hot use case. I think Sierra even talked about how this is one of the most competitive markets in AI right now: building voice agents. I thought it would be useful to talk about how they're actually built and some of the things that engineers who are building in this space still have to contend with.

For example, we talk about how the state of the art right now in building voice agents is a cascaded pipeline. We talk about how it's a 3-step process: speech-to-text, an LLM, and then text-to-speech. We talk about why that is. Why don't we just have a voice-to-voice model? The reality is those just aren't super reliable right now.

We also talk about the trade-offs that you have to make when you're building voice agents. You have to trade off the intelligence of the responses that you're getting versus the latency. You can get very intelligent responses, but you're also going to trade off speed: it's going to be a slower response. For different use cases, that might be good or bad.

We also talk about reliability. There are some LLM companies that may not be super reliable in terms of their infrastructure. What do you do when Opus goes down? You need to have a waterfall of models to basically pick it up so that your voice agents don't just stop working.

We also talk about turn-taking and how it seems like a trivial problem. You're talking to a voice agent, and let's say I pause. How does the voice agent know that it should interject and actually respond to you, versus the person just taking time to think? Turn-taking is actually not a trivial problem to solve, so we talk a little bit about that.

We even talk about how I think Pipecat built a voice agent for the AI Engineer World's Fair, right? If anyone calls the voice agent, it will answer questions about the conference, and you got a ton of real-world information and turned that into a benchmark.

To be clear, I have no idea, actually. I never looked at the analytics, so I have no idea how many people actually called. Probably people were just kicking the tires. It's not that serious—we're a conference, and people know what it is. But I think it was a good deployment of a use case. Daily is a sponsor, so why not?

I would say that the vast majority of voice stuff is support. There are so many call centers out there, hundreds of billions of dollars spent on this stuff, and they're all bad. So hopefully we can sort of raise the state of the art, which I think is why Decagon and all these companies exist.

To me, it's not as interesting if a bunch of people who are obviously selling you voice pipelines tell you that voice pipelines are the state of the art. Like, yeah, duh. But having the people actually focus on customer support and all those things also basically conclude, “Yeah, the models are not there yet. They may never be. And actually, this is just the way that you've got to do it.”

The people that really feel it are not the researchers, because researchers just always want bigger models to solve everything. The product engineers will try to do it, but the FDEs—the people actually dealing with customers—will be like, “Dude, they made this horrible mistake. This can never happen again. How can you guarantee me that?” Well… [laughter]

Yeah. So, actually, we even talked about that a little bit. We talked about the inbound versus outbound use cases, because we even did that once. We ran an outbound use case, and what you find is whenever somebody picks up the phone and you do an outbound use case, a lot of the time they just hang up as soon as they realize it's a bot. So we even talked about that: how do you solve for that?

All right, that's a good teaser. I admire your work. I'm excited to feature it on Latent Space. We'll be doing more in the future together, but this is just an intro to FDE for, at least, the Latent Space audience.

I think this might be a good place to give everyone a 101 lesson on what a simple voice-agent architecture looks like. Maybe that's a good question for you, because you guys run Pipecat.

1. The Cascaded Voice Pipeline

Yeah. There are many ways to build one nowadays, because complexity is the nature of things. But the simplest one, which is called a cascaded model, typically has voice input. It comes through a transport—it could be WebRTC, a phone call, or WebSockets—and it goes into speech-to-text. The transcription happens.

There can be some additional models right there for background-noise removal and voice isolation, so if there are a few people in the foreground, it will isolate 1 person and remove the background noise. There's a turn-detection model. Unlike text, where you know when you're done—you press Enter, the message goes, and the LLM starts to execute—with voice, there is no such signal. It's not walkie-talkie or push-to-talk.

So there is no additional information apart from, if you're looking at the face, you could actually figure out that the person stopped speaking. But typically, you use something like voice activity detection and a Smart Turn model to figure out that the turn is complete. If I pause mid-sentence, that's different from if I finish speaking and pause. You may use a speech-to-speech model right there.

Then, when you have the text from the transcription, you send it to an LLM. The LLM may respond with an inference. It may infer tool calls. It may do a bunch of things there. When you have the full output, you do text-to-speech, or TTS, and it's usually faster than real time.

Then you have to figure out how to stream it back to the end user. It'll go over the same transport that you spoke over, and then it'll play it back on your speakers.

2. Inbound Calls Need Context

But things are getting more complex as people do this. In 2024 and 2025, people wanted to do this. If you're doing an outbound use case, like an outbound call, debt collection is quite commonly a use case for this. You get a call, they tell you that you have an unpaid bill, they ask you if you would pay it, or they give you a certain set of options.

For that, the bot really has… if you go off the rails, if the human starts saying random stuff, it can just drop the call. It's not obligated to stay on the call in any way. So guardrails are easy. It knows what inputs it can accept, and it knows what inputs it doesn't need to respond to.

It flips when you have an inbound use case, because the bot has no context. It has some context of why it exists, but it doesn't know who you, the person who's calling it, are. If you're a mom-and-pop shop, like a flower shop or a barber or something like that, there's only a finite number of things that you can do there.

But if you are more complex—let's say you're Amazon—there are a billion products on your platform, and you have different policies for different things. Is it a physical item? Is it a cheap physical item? Is it an expensive item? Is it an electronic item? It needs to figure all of that out.

And then, when you say something like, “You can't put all of that inside a single prompt,” because LLMs are goofy in that sense: they always remember the first 4% and the last 4%, and everything in between they kind of forget. So if you have a return policy right in the middle of that context, it's very likely it'll hallucinate.

So you have to do smarter things, and that's where more models come in. You can have a compaction model. We're all using coding agents now, right? If you're using a 1-million-context model, at 25% you should become nervous. That is nowhere near the 70% mark, but at 25% you're like, “Okay, should I start compacting and saving my work so that this thing does not go off the rails?”

The same thing, I think, applies to voice AI. Voice AI has been one generation ahead of coding agents. We've all, in the last 2 years, solved things that felt so alien but very tangible for us. Now, when we see coding agents do that, we're like, “Man, compaction—we were doing compaction from day 1.” We knew that after 5 turns or 10 turns, when your context was only 250 tokens or 250,000 tokens or 50,000 tokens, models were really small, and at 10,000 tokens they would go off the rails.

A lot of our work is, I think, about making sure that the bot does not hallucinate and trying to keep track of the conversation as the turns progress. The turns are fairly short because either way, the bot is more likely to speak more than the human does. So you have to keep track of what the human said and where you are in the conversation.

Yeah. I guess this is a question for one of you 3. Do you use this cascading model for your voice agents?

Yes, that is one of the major offerings that we have: cascading and speech-to-speech.

3. Speech To Speech Has Tradeoffs

Yeah. I was going to ask: that sounds too complicated. Why not just go voice-to-voice? Why do you have to have this crazy 3-step process?

I feel like the easiest way to answer this is that you can call into a really nice voice-to-voice demo and you're like, “Wow, it's listening to me laugh, it's responding to my tone, and it's so snappy. It's so fast.” But then I tell it that it asked me what day I want to schedule my appointment for, and I say, “Next week.” It says, “Great. Is that June 10?” And I'm like, “No, the year is 2030.” And it's like, “You're right. It is 2030. So let's schedule this for June 10, 2030.” And I'm like, “Great. Sounds good.”

From that perspective, I am actually very curious, especially on the Smallest side, about how that has evolved over time, because we are very much keeping our eye on how these speech-to-speech models are performing. Overall, I think our current stance is that the cascading model allows you to enforce many more rigid guardrails and maintain tighter control.

You can say, “This input is going to go through supervisor models to detect prompt injection or social engineering. Then it's going to go down the conveyor belt into intent selection. Then we're going to optimize context by checking conditions ahead of time.” In this complex process that we're going through with this user, maybe this prompt is applicable if they're this type of customer, but another prompt is applicable if they're that type of customer. So let's figure all this out ahead of time, compact and compile a good system prompt for our message-generation model, and get a response back.

Then we can take that response and check a whole bunch of other things. We can check whether it's grounded in truth: Is it 2030, or is it 2020—whatever year it is, 2026? Then it goes back out over the line. The obvious constraint is: how do you make that performant? How do you parallelize as many of those steps in the conveyor belt as possible?

I think the last 6 or so months have been a really amazing feat from our engineering team, at least, to find and shave off 10 milliseconds at a time across every single part of this pipeline. That makes it feel snappy, even though there are a lot of things going on behind the scenes that you don't necessarily have to do with a speech-to-speech model. That's at least my take, but I am curious about the rest of the group's thoughts on this.

Yeah, I think you brought up a question around cascaded versus speech-to-speech. Let's talk about why people started thinking about speech-to-speech. The initial idea was simple: if you convert speech to text, it's going to lose the emotional information. If you say, “Hey,” you might be saying it in a sad way or an excited way, and the bot is going to answer in the same manner. That was the obvious reason people started thinking about it.

At least at Smallest, that has evolved into the idea that speech-to-speech is a more natural way in which the human brain operates. When you do the cascaded thing, you do speech-to-text, then send the prompt to an LLM, and then it responds. We call that a synchronous architecture: it's happening one after the other. But our brain is thinking while listening. As I'm speaking to you, you're already forming your thoughts, and if I'm talking for too long, you'll interrupt me. You might also be taking notes in the background or essentially doing tool calls while I'm speaking to you.

The whole idea is that if you ever want to pass the Turing test for how the human brain operates, you need something that works asynchronously. It can't just understand emotions; it also has to operate like the human brain—take in speech natively and give out speech natively, asynchronously. That's why we've been building Hydra. Hydra is our speech-to-speech model.

In terms of accuracy, interpretability, and all those things, whenever there is a new architecture, it's often good in one dimension and regresses a little bit in others. For example, the speech-to-text accuracy of a dedicated speech-to-text model might be much better than the encoder of a speech-to-speech model. The challenge is that while you make progress on making it more natural and more human-like, how do you keep the accuracy bar the same?

A lot of that comes down to the interpretability of these models, because you don't want them to be a black box. How can you understand where they are lacking? A lot of the research we do is about making speech-to-speech models more interpretable. The other constraint is that the initial speech-to-speech models, like Sesame, were just speech-to-speech: they literally took in speech and gave out speech.

Ours is multimodal, so it takes in speech and text and gives out both speech and text. It can do tool calls and take in text in parallel. If you want to put in guardrails or do all those things, that constraint does not go away. We are still seeing many more cascaded deployments than speech-to-speech deployments, at least in enterprises, but I think speech-to-speech will be the eventual future.

My take is that when you have competing approaches, you end up with a hybrid. I think the answer for the midterm will be some form of hybrid, because speech-to-speech models are improving. There are parts of the conversation loop where you might say, “My use case or workflow is complex. I'm going to use multiple LLMs anyway.” For the active loop—I'm talking to you as a human, you called me, and I'm answering your questions—whenever I have to do some kind of lookup, I delegate to another LLM, which then does the cascade work.

The voice in the first part of the loop keeps running, and then you have interesting kinds of semaphores. You can think of them as threads. You want to interrupt the speech-to-speech model because you realize that this is a complex question, and before the speech-to-speech response, you say, “Actually, you cannot answer this question,” and delegate to the cascade, and so on.

Typically, I say there are many use cases where, if you wake me up in the middle of the night and ask me a bunch of questions, there are definitely some classes of questions that I can answer without thinking. We all do our jobs in a certain way where we can operate on autopilot for 50% of our time, right? As these use cases become emergent and you're fully deployed with a customer doing high volume, you could essentially train specialized models that fully understand that 50% use case very well.

You could always ask the model, “What is my return policy?” It can say, “In the simplest case, this is my return policy, and it applies to 70% of the products. I know which SKUs or products it belongs to and which ones it applies to. If you ask me about something outside of that, then I have to do all the complicated work. If not, I can answer it.”

So, on these cascaded pipelines, what models are you using? Let's say, of the frontier models—the GPTs, Opus, and Sonnet—are you using the latest ones, or are you using GPT-4 because it's the right balance between speed and intelligence?

4. Models Trade Speed For Intelligence

If you turn off thinking, then you can actually use any of them. If you want a fast model that responds to the prompt and does not need to think, then you can use it. If you wanted it to think, you would start thinking about parallelizing the work, because you want the first model to be super fast.

I still like my Gemini 2.5 a lot. It's so fast. The 3.5 that they launched is okay—it's slower—but the 2.5 is so good. Even Haiku is really, really good. They're still slower than what you would expect, but yeah.

How do you parallelize that pipeline? It feels like I first have to know what text the person said, and then the LLM needs the text to do anything. And then you can only generate speech once the text you wanted to send has been generated, right?

You can send the speech directly. In the speech-to-speech model case, you would send the speech to one place, fork it into another place, run the STT there, and, if you want multiple models, the text output from the first forks into multiple LLMs. You can think of it as a waterfall: it comes down.

Because you don't know exactly what the output will be.

Yeah, you put a gate at the bottom, which is an XOR or whatever fancy thing you want to use, which says, “If all of them say we are right,” and then you put another LLM downstream to decide which one is better. You can go really crazy. If you love computer architecture, 20 years ago we had nothing to do; now you can do all this crazy stuff from 20 years ago.

5. Multilingual Voice Needs Modularity

Yeah. So how do accents and different languages fit into this? Maybe Sudarshan, because you guys are building speech-to-speech.

At least for speech-to-speech, right now we are fully focused on making it more intelligent in English and getting it really good in English. We don't want to introduce any other variables because the technology in itself is, I would say, quite frontier. Asynchronous is not yet mainstream, et cetera.

But in terms of cascaded systems, we've seen a lot of demand. We have a lot of presence in India, so we see a lot of demand from India. We see a lot of demand from Latin America, et cetera. I think the US is mostly English, and then there's some Spanish, and there's a lot of accents to it.

In the US at least, we have not had any troubles in terms of the technology. Noise cancellation is probably the last-mile problem that is pending in terms of handling those things, but that has nothing to do with accents. I'm curious what you guys have seen.

Maybe I can add some examples about that. For multilingual applications, I definitely like to leverage cascade models because there are multiple levers you can pull.

To talk about an example, we had a customer we were deploying for Japan. You could typically go to GPT-4.1 from OpenAI or use Deepgram for transcription, and that typically worked well. The challenge became the voice piece. It turns out that what we thought was state-of-the-art for English, Spanish, and Portuguese—those types of languages that are typical in the US—didn't work at all.

One of the benefits is that you can swap certain pieces that don't quite work for you. At least on Vapi, how we solve it is that we let customers bring their custom text-to-speech server. There are some startups and labs in every market, and typically you will find that in each one.

Arabic is also really hard to get right in terms of the pronunciation of brands and addresses. We cannot build expertise and optimize for every single use case, so that's one of the reasons why I like that approach as well.

Cool. Also, I have a question about the Sesame model. They came out with this really insane demo, and then I've never heard of them since. What do you guys know? What happened, and what's going on?

I think they are building hardware, is what I understand. They are putting that voice into some sort of hardware. I'm not sure if it's glasses or what they're working on exactly, but I know a few people who got hired there, and they're all focused on hardware and voice backgrounds.

Yeah, the answer I got was that the CEO had already made a ton of money, and he just really wants to play around, basically. So I guess he can do whatever he wants.

6. Prompts Versus Workflows

Cool. Is it standard practice in a cascaded pipeline to have one giant system prompt that gives the LLM all the rules for how it should respond, or is it more of a workflow approach? How do you guys think about that? What's the standard best practice right now?

I think it highly depends on the use case and also the size of your prompt. I have a counterapproach to Varun in terms of that. Usually, in inbound calls, you're having dedicated lines, and you know specifically where the workflows are going to go. In these cases, if you know a little bit more and can predict where the conversation is going to go, you can have more of a node- or graph-builder kind of approach.

But if you don't know what's going to happen, typically, I think in outbound calls, there is a dedicated message, but you never know what the caller will say back. They could be frustrated, angry, or annoyed. When a person is erratic, you never know. Having everything in one prompt allows you to have a central brain, so you can pick up components from the prompt that may not have been in a dedicated flow but can be used to enhance the response.

On top of that, you have to think of your knowledge base and the information at your disposal. In an inbound flow, maybe you know exactly when you need to pull a specific part of your knowledge, so you don't need to extract every single piece of context from your websites and documents. But sometimes in outbound calls, you should utilize all that information to get a better response. Then you will rely more on cosine similarity and other methods to make sure that your retrieval is good enough to respond to them.

From my perspective, the limiting factor has been, and will continue to be, although it's getting a lot better over time, the ability for these frontier LLMs to take a massive wall of text and not just read the first 3 lines and the last 3 lines, but actually properly reason and localize themselves: We are halfway through this procedure; these things have already happened; these things haven't happened yet; these things are true and these things are false about this situation; and therefore, this is the one thing that I should really be zooming in on right now.

From our perspective, we take an approach that depends on the customer. Within our execution engine, we have the ability to make some of these rules, prompts, or guidelines more situational, so we can check whether they are relevant right now before we include them—or not include them—in the system prompt.

That was our solution, especially relevant 7 months ago, when the LLMs were consistently skipping step 4A1. But over time, we have found that the reliability of deciding when these things are relevant is sometimes actually worse than just giving all of it to the model, because the models have improved a lot since then. Now we're swinging a little bit back toward just giving the model everything, and more or less it will figure it out.

At the end of the day, with nondeterministic systems, it really comes down to the individual customer. We have to understand their use cases, build it, test it a whole bunch of times, and then use an LLM as a judge to tell us whether it worked and was reliable, or whether we need to add a little more preprocessing and context optimization ahead of time to make it reliable.

The real answer is most likely always just to test and find out, then test again and find out again. It goes on forever.

I'm assuming that maybe the models are able to handle those edge cases now in a giant prompt, but that probably also increases latency. Could you talk about what good latency looks like for a voice agent? Is it even true that the models are getting better but also require more time, so for the end user that's actually a bad thing?

Yeah, that is a really good question. I think the definition of what good benchmarking latency stats look like has changed a lot, and expectations are getting better and better and better from our customers.

Overall, it is also important to mention that, as much as fillers and contextual fillers are something that we don't like to hear, it is normal in normal conversation for humans to say a few words while they're thinking. We've actually found that when we have zero contextual fillers, sometimes it feels more rigid than when we don't. We tend to find that having the right amount of fillers kick in when we are experiencing some sort of latency because of a very large prompt or some very complex thing that the model is thinking through is helpful.

Most commonly, this happens when running a tool call where our customer's API becomes the bottleneck and we're waiting 5 seconds for something to come back. Being able to say, “Hey, just give me a sec. Looking at that. Okay, cool. Here's your answer,” is actually fine. There's nothing wrong with that.

When we've tested with our customers, the limiting factor in that case becomes how good your fillers are and how natural they sound.

And so that's been a place where we've invested a lot of time as well: How do we create an execution engine where we really don't need fillers, but in those moments where we do, they're really good? That's at least my take, but again, I know folks probably have a lot of opinions here about latency and how to find that trade-off because, in my opinion, I honestly think that that's the hardest problem to solve in voice deployments: just that trade-off that's always going to be there between performance versus stability and reliability in general.

Yeah, I agree with all of that. Some of the ways that we think about it at Vapi are by offloading some of that instruction. To break it down, sometimes you don't really need that 10- or 15-step workflow in the prompt. One example I can talk about is this collections use case. We do allow businesses to capture credit card information, but not every call is about that. Some people already have a payment method added.

So one way of doing that is offloading into a specialized agent, so we can start thinking about multi-agent architectures and patterns in there. The idea is to constrain the instructions that you give it to that particular moment of the conversation, and once it achieves its goal, it can go back to that more free-form approach, which has the longer context loaded.

Something we've also been experimenting with lately, and it's more about the guardrail space, is, to your point, where you can think about it as a waterfall: You can actually stream to multiple places. What if you have streaming down to a small language model, which can do inference in very little time? So think about classifiers to maybe collect the intent while the pipeline is still adding a filler word to keep a consistent experience, but also kicking off a background process that is still intelligent, still thinking. Those are still things that we'd like to invest a little bit more research in. So, yeah, we're constantly investing in that.

Sure. I think one thing that we also haven't factored in in terms of the latency trade-off is cost. Right now, you're putting such a big prompt into your LLM. You have to think about those calls where many people just hang up—predominantly, calls just hang up 10 seconds into the call. Right now, you're having to pay for all those tokens just to get inputted. So, as Stephen mentioned, splitting out your prompt also plays to that strength of lower latency and lower cost.

I'll speak just to the benchmarks. We publish turn-based STT benchmarks, LLM benchmarks, and TTS benchmarks. The whole point of that is that, as I think today, Nemotron 3.5 launched, and there's already an ASR benchmark out that did very well—the NVIDIA one. The idea there is that all our tooling is open source. With the STT benchmarks, the TTS benchmarks, and the LLM benchmarks, all the turns are actually open. You can run them against any new model that comes out. You can run them locally, and you can look at the outputs from our graphs and compare that to what you're getting on your infrastructure if you're self-hosting or elsewhere.

That's one important aspect to consider: If you know what your turns will look like, you can just download our benchmarks, update the turns, and see how that goes for your own flows. Usually, for the LLM stuff, it's really useful because you can throw out the STT and the TTS and just run the LLM inference loops to check against a customer. So, yeah, I think evals are really important.

If, while talking to customers, you can evaluate what type of conversations or what type of workflow they're going to have, those are real, direct inputs into what you can start testing, even though you're not live with them or they haven't shown any intent. So I think that's where the SEs and the forward-deployed engineers can actually come in. We all have a Claude running in the background, right? So you can just say, “Here's what we had a discussion about. Can you build an eval based on our eval suite? Can you just run a smoke test with the conversation information that we already have?” And that helps a lot, I think.

Just adding to that: While we focus on models, we also have all the models orchestrated on our platform, so you could build a voice agent on Smallest. We're seeing a lot of customers who pair our models with our own small language model. It's called Electron. They fine-tune one of those to make it work for real-time voice use cases, and we're seeing a lot of folks using that with better knowledge bases, memory, et cetera, over GPT-4o and GPT-4.1, the popular real-time models.

Just from a cost perspective, it's lower. From a latency perspective, it's way lower, and then it's just more reliable. I'm not sure if everyone faces this, but OpenAI APIs spike, and you have no control over those latencies. If you have a self-hosted model, you can scale up or scale down based on your requirements. So that's one thing we're seeing a lot with our customers.