[AIEWF Preview] Gemini in 2025 and Realtime Voice AI

Hey, I'm Sam Charrington.

And I'm Swyx. Welcome. Thanks for being here. Thanks for hanging out with us. I'm excited. Logan, you were our first guest. You came back remotely a few months ago, and now you're back. You're sort of the face of AI Studio, basically—a lot of people are using it. I'm using it, and I think it's a really welcome change, being more accessible alongside the rest of the Google suite. And stress, you've been—I actually don't super know your role. I generally have you pegged as the PM of the API team, with a particular focus on Live.

Logan runs the show behind the scenes. He's the public face, running the show behind the scenes—model launches, the Live API, generally all the stuff that's happening in the API.

The API is—stress is hard work, so thank you for that, Logan, but I think everyone knows who really runs the show. There's public evidence there.

I work with Logan and a few other excellent PMs, but I lead the API side of the house.

There are a lot of announcements, and I think a lot of people have done their recaps. What are your personal highlights over I/O?

1. Developers Gain More Model Control

I'll break the rule and give 2 that are not the big, big flashy ones. I think the 2 that developers are going to be super excited about are, 1, thinking budgets coming to Gemini 2.5 Pro. You'll also be able to disable thinking, so if you just want Gemini 2.5 Pro as a raw, non-reasoning model, we'll have that hopefully in early June.

And then thought summaries. We've had this debate internally: do we need to show full thoughts? Do developers want full thoughts? I think developers say they want full thoughts. We have thought summaries right now as a step in that direction. It'll be really interesting to find out and get feedback about what works with thought summaries and what doesn't. I was reading some threads last night about thought summaries now being live in Cursor as well, and people were reacting to having summaries versus full thoughts. It'll be interesting to see, but I'm excited for both of those things. Thought summaries are live now. The thinking budget for Gemini 2.5 Pro will land with the GA model in a couple of weeks.

I should say, we already do have thinking budgets in Gemini 2.5 Flash. With all the other features we are releasing on top of our thinking models—summaries and budgets—I think this is our way of saying that you have the models, but we want to give developers as much control as they can have on top of the models.

Coming back to your question about my favorite feature, it's really hard to pick because all of these features we've been trying to push out for weeks, but I think native audio output is a personal highlight.

I was just saying that with Quinn.

Quinn and I have been playing with it together for a bit. Especially with the voices, which sound great, the fact that it can switch in and out of languages is really interesting. Matt Boso, our boss, actually has a demo on Twitter where it speaks Klingon, even though that's not an officially supported language. For me, speaking Bengali, just being able to switch in and out of Bengali and English has been special.

If I get to pick another one, I'd say we released a new tool called URL Context. The idea is that you can use it by itself or pair it with Search to retrieve more in-depth information from webpages in a way that's respectful of our publisher ecosystem, of course. I think this will unlock new use cases, like if people want to build their own version of a research agent, which is something developers ask us for a lot.

2. Implicit Caching Cuts Costs

It's worth mentioning that just prior to I/O, there was a ton of interesting new capability, including the update to Gemini 2.5 Pro, as well as implicit context caching, which I know a lot of folks are waiting for.

We made implicit caching happen. There was lots of feedback: people said explicit caching is nice; there are definitely use cases where it makes sense, but people want implicit caching. I'm happy to pass the cost savings on to developers. You don't have to do anything; it just works right now, and you're saving money. It's a great outcome.

I don't want to manage that myself.

There are so many use cases where you're just doing chat on the same stuff over and over again. For those use cases, you want to be able to explicitly cache the thing and make sure—guarantee—that you cache it so you save money. So I'm happy we have that.

Is there any behind-the-scenes aspect of what makes caching hard, or anything people don't appreciate about caching as a general concept? I think this is a very important pricing paradigm that people need to really get behind.

That's a good question. I think there's a trade-off between all the dimensions of caching, including latency, because in some cases you're getting latency gains, and in other cases it's about the cost for Google and how much stuff you want to cache altogether. We could have an entire episode and get a bunch of the caching people.

It's a good example of an infrastructure problem to be solved, and a bunch of the folks we work with love working on this problem. We should do a deep-dive episode.

I want to shout out that you've been doing more video stuff. You have your own podcast as part of your Gemini work. You've also been doing video with people on the team. Exactly—you did the long-context one. People loved it. The reception was very positive about the long-context one, so thank you.

That was the first time we did a more in-depth technical discussion with folks on the team, and Nikolay's awesome. We actually just did one with Shrestha about the Live API, which I'm excited about. We did one with folks on the team about the multimodal capabilities in Gemini. We're going to do a pre-training one, hopefully, which will be really cool. We've got a bunch of people who are excited to talk about that. So there's a bunch of them in the works, and it's fun to make them happen and have those conversations.

My underrated pick is Gemini Diffusion.

3. Gemini Diffusion Enables Generative UI

It's not underrated. Apart from speed, I wonder what the potential results of a diffusion language model are.

Generative UI. This is the way generative UIs happen: through this experience. The UI bit is being able to say, “I want you to build the UI on the fly using code based on what a user does.” You have no precompiled notion of what your website is, and as a user goes through and clicks buttons, 1000 tokens generate and it just makes that UI for you.

Interesting. I think that's going to be possible. I mean, I think there's a lot of work to productionize, to make Gemini Diffusion an actually high-quality model that meets the bar for us to bring to the world more generally. But I do think that's going to be the killer use case: this whole generative UI experience that doesn't exist today because the models just take too long to generate tokens.

For me, it was really the role that audio and video are taking throughout a bunch of independent product releases—from the generative models to the Live API, to the on-the-fly transcription and translation. I think it's foreshadowing the role that that's going to play in a lot of developer applications.

Transcription, actually, even before we released native audio. Now, of course, you get text and audio interleaved in the output, but transcription used to be one of the biggest use cases we had on the Live API.

4. Live API Raises New Challenges

What are you seeing as the challenges for folks getting started with Live?

That's a great question. Firstly, awareness—people knowing that we have a Live API. That's why we're doing this: talking to you folks. We were actually the first to market with video input, too, but one area where we've been getting a lot of feedback is session length. When we started, you could do 15 to 20 minutes of audio and about 5 minutes of video.

We've been putting in a lot of knobs for developers, and we can talk about that more if you want, for people to have a sliding window or decide what resolution they want to send video in. The goal is to increase the session length. Tool calls were another area where we used to get a lot of feedback. Again, we were very proud because we introduced tool chaining first, so you could chain Search and Code Execution and do all kinds of analysis. But then we've had to do a lot of work improving function calling and the performance of Search, and we continue to push on that.

I've got a quick point on this, too. I think the level of commitment you need to make to the model provider in the world of the Live API is higher. For developers, it's a higher bar. If you look at Chat Completions or, for us, generateContent, from just a text-modality perspective, it's a pretty lightweight thing. There are a lot of model providers with that option; you could switch to a different provider if you end up not liking one, which I think is good for the ecosystem.

If you look at a lot of the Live API infrastructure right now, you really do need to commit that you're going to use a particular provider. There's no easily interoperable infrastructure between different model providers. Everyone's infrastructure is bespoke and different. It is a different level of commitment that you need to have to really bet your company, business, or product on the Live API.

I do think that's a challenge for developers: making that level of commitment in this fast-moving AI world. Hopefully, there will be some level of similarity and some model-agnostic infrastructure to help make developers feel a little bit more comfortable about being able to move between models, potentially.

I could go on and on, but if you have more complex workflows, one of the things you need is the ability to change the system instructions at every step of your workflow. Onboarding some of the more complex use cases with the Live API has been a work in progress as we've released more features.

What kind of complex workflows are we talking about?

We have people building, say, gaming agents, which have multiple states, for example. We have a lot of customer support agents, of course. The sessions can last for hours, right? Then there are a lot of use cases around people showing a certain screen. This is the coolest use case, honestly.

I was referring to the famous demo at Next, where Shopify showed how to set up DNS using Cloudflare, right? In certain cases, especially the longer your workflow runs, you might have to go from one state to another, and you might want to change the system instruction. If you hand it from one agent to another agent, you might have to change a system instruction.

When you're thinking about building voice-based applications, is speech-to-text followed by processing with a standard LLM a precursor to the Live API? Or are these 2 distinct paths that are still viable, and that you still see being viable going forward?

5. Gemini Converges Around One Model

That's a tough question. Right now, we have both out. I do think that, eventually, for most use cases, as these audio-to-audio architecture models get better, a lot of use cases will probably transition to that. But when we talk to our developers, they still very much like those componentized systems.

That's why we also put out 2 new text-to-speech models at I/O. They're not available through the Live API yet, but they're really high-performing, controllable, promptable text-to-speech models.

I have an angle on this question. I talked to Cory this morning, who's our boss's boss, the CTO at DeepMind. Koray had a really interesting take, which is around what makes one of the main things that we're doing at Google with Gemini different from what a lot of the other labs are doing. We're here to make 1 model, and that model is Gemini.

I think, to his point, you do need to have these forks that go off and make a capability, harden it, and then find a way to bring it back into the mainline model. We want to make 1 model, and it's the Gemini model, rather than have the sort of splintering of all these different capabilities.

We've done a good job of this. I think the reasoning work was the best example. We had those separate from the mainline Gemini models so that the research teams could go and hill-climb and make progress without being constrained by questions like, “How do we do this without collateral damage to other capabilities, like multimodal?”

The teams went and did that, and then they found a way to bring the capabilities together. Oftentimes, what you see is tension in bringing them together, but the really exciting thing is what happens when you bring the capabilities together. Gemini 2.5 Pro with reasoning is a great example of this. Multimodal video understanding ended up having this huge, beautiful moment. The model is so good out of the box because of all the reasoning capabilities that were baked in. It wasn't because they did a bunch of things to make video understanding really good; it was just an artifact of bringing and merging those capabilities together.

I think that, as a north star for Gemini models, makes a ton of sense. I agree with you, and that's what I said, right? Eventually, a lot of use cases will end up on Gemini and natural voice. But in order to foster development, we have these offshoots from time to time. We have our Imagen models for image generation, even though, in a slightly pre-I/O announcement, you can also do interleaved text and image within Gemini.

Yeah, but those are different models, right? Those are different models. One is autoregressive, and the other is diffusion. The other is—

That's what I'm saying, right? For a lot of image-generation and image-editing use cases, especially high-quality, photorealistic ones, developers are still using Imagen. But slowly but surely, we're bringing those capabilities into Gemini as well.

Whoever's watching this, we had a mid-I/O switch because, obviously, there's a lot going on here. This is not AI shape-shifting, I know.

But we also have Quinn, who made this podcast happen. Quinn, you're the founder and CEO of Daily. Welcome. I'm a big fan of all things voice and audio, so it's great to be here with you and with Sha. Quinn actually runs the Voice AI Meetup in San Francisco. You're consistently a leading community builder, and you're very generous with your time and knowledge, so I really appreciate that.

Obviously, you also recently started Pipecat, which is this open-source framework for voice orchestration, with really great support for all the Gemini models. You wanted to say something about the relationship between Gemini and Daily.

I just wanted to say that it's been a very fruitful partnership with Daily. They've been our partners since the launch of the Live API, and a lot of the feedback they continuously provide has been instrumental to the success of the Live API. Both Daily and LiveKit have partnered with us.

Quinn, I think you know we had a little bit of prep for this. You also wanted to dive a little bit into the cascade of models in Gemini Live. I think stress has taken a really interesting approach to designing these APIs.

You talked about components a little bit. You talked about how you want to be able to do things both in the Live API and in the more traditional chat API. You originally designed the Live API to have audio in, but then a separate text model—the NotebookLM models—for audio out. What was the driver for that originally?

At the time, we wanted to hit a certain quality bar and a certain latency bar. NotebookLM was already out, and the TTS models powering NotebookLM were very good. We wanted an aspect of native audio, so it was native audio in but TTS out. We still have that architecture available through the Live API, but now we've also released an audio-to-audio architecture.

Continuing to pull on the component-versus-single-model thread, when I think about voice, I think about it as an area where, to deliver solutions, you need to surround that strong model with a lot of voice-specific infrastructure. I'm imagining that's challenging to scale. Can you talk a little bit about that, and maybe we can have Quinn talk about it from his perspective?

The first thing that comes to mind is, of course, the voice activity detection models that we have. We've done a lot of work fine-tuning that model server-side, but we've also learned that we need to provide some knobs to developers.

Now developers can tune the sensitivity of our voice activity detection model, as well as how much prefix padding—how much time at the beginning, at the start or stop of saying things—we use. We also have a mode where you can disable our voice activity detection and bring your own.

The larger point that you're touching on, Sam, is that it is really hard to bring all these components together and still get latency down to where it needs to be, in the 500- to 700-millisecond range. It's one of the hardest things we've had to do with the Live API.

What we see is that the shape of building these real-time voice agents is a different set of developer problems from the shape of non-real-time or text-mode things. One of the fun things about partnering with stress and DeepMind is that we work on this open-source framework that people use to build these kinds of production voice systems.

We try to solve problems at the framework level, like turn detection and context management. As the models get better and the use cases become clearer, some of those features migrate from the framework into the APIs, which makes life easier for developers. At the same time, the use cases continue to broaden, so there are more things for the framework to do.

We're filling the top of the use-case, building-block, developer-experience funnel and pushing things down as we all get better and figure out what this new world looks like.

Maybe this is also a good segue into WebSockets versus WebRTC.

6. Voice AI Needs Network Infrastructure

Yeah, you know, there's so much infrastructure. For my whole career, I've been building large-scale, low-latency network systems. What we saw, from my perspective, when we started to see the possibilities of voice AI was that you need this packet routing underneath the inference layer.

There's the AI inference stuff, but then there's just how you move the audio and increasingly video around the internet. And so there's a whole new generation of developers who are interested in these networking protocols because voice AI and now real-time video are so interesting, which is super fun for me because I've always thought moving packets around is one of the most fun things you can do on the internet.

Yeah, 7 layers of the OSI stack.

Exactly. Exactly. At pretty demanding real-time latencies, because human beings expect you to respond in a conversation in 500 milliseconds or so. And if we're talking to an AI, we don't relax that assumption. We bring our assumptions about human conversation into that experience of interacting with an AI.

Yeah, or not respond. That's a great point.

One of the features that we've pushed out, a little more experimentally, but would love for people to test, is what we're calling proactive audio. It's available only in the native-audio, audio-to-audio architecture right now. What this feature does is it's trained not to respond to irrelevant audio.

Okay, so it's kind of a refusal.

Yeah, or you could call it, directionally, semantic voice activity detection. Basically, let's say I'm talking to the AI, and then Quinn comes and asks me a question and I respond to Kwindla. It'll know when not to respond.

I saw that in one of the demos. The AI seemed to ignore a background question from someone else in the video.

I think there are 2 threads to pull on there. One is that that's another great example of things that we had to work really hard at the framework level to implement. It's much better if it actually migrates down into the model or the API. The other is part of the magic there: this is a separate feature, but I think they're multiplicative, because now your models can actually recognize 2 different people just based on their voices.

They're not officially supported yet. The world just does it. But just try it, right? I mean, give us feedback. It might be my single favorite thing you can do with these models that you previously haven't been able to do.

You can talk about what you've observed. I'm just saying it's not officially supported. What specific models are we talking about? Speaker identification and diarization have always been really hard for these models.

Model naming has become—gosh. It's called the native audio dialogue. You'll see it in the Live API. That's the model. And to your point again, Sam, about architectures, one thing that we've launched on the cascaded architecture that we hope to eventually bring to the native audio as well is asynchronous function calling.

Earlier, the way it used to work was that if you wanted the model to do a function call, you'd have to wait for the response. Now you can set a non-blocking parameter, and the model can go off and execute the function in the background.

7. The Next Gemini Wishlist

I love you so much. Yeah, that's great. We do have to wrap up, so I think one fun thing that we can do to wrap up would be a wish list for next year's I/O. What would be one thing that you would wish for? It doesn't have to come true, but what do you wish happens with Gemini?

Well, I was hoping for Gemini 3.0 at this I/O, so maybe Gemini 5.0 at the next one.

Tell us what you mean by Gemini 5. What do you want in Gemini 5.0? Then I'll let Quinn go.

I'll just put on my hat as a representative of the big community of people building this stuff: more and more languages, because AI is global, and there are so many communities all over the world that are starting to use it. Can we do more languages?

You know, it's hard to stuff everything into one language and one model. But they're building the one universal model, as they said. I think that would be a boring answer, but I think we're really working on languages. I mean, wasn't I telling you earlier? We officially support 24 languages, but you can try talking to the model in Klingon, and it will respond to you. So I think we'll get there way before next I/O. I just think more and more capabilities in the main model is what I would say.

I'll have to think about this.

Yeah, yeah. It's a fun parlor game, but it also helps people align as to what is possible and what's coming up.

Thanks for your time, everyone. This was very hastily organized, but I'm glad that we could make this happen, and it's nice to actually see Sam in person.

Same. Yeah, yeah.

It makes you think I'm the only PM for the Live API, but we did not get to talk about some of the other releases as well. We'll save that for your talk at World's Fair. You guys are all speaking, and we'll be podcasting as well.

Sounds good. Yeah, yeah.

All right, that's it. Thank you so much.