Optimizing AI Voice Agents

All right. I think we're set. First, thank you so much for having me. I'm a big fan of what everything that's going on in this course. I follow everything you all do. I'm a little behind on some of the sessions, but I'm going to catch up. And I'm a disciple of the look at your data model. As you said, I work at a company called Daily. We make global infrastructure for real-time audio and video. A lot of the use cases these days are real-time voice and real-time video conversational AI agents. I also work on an open source vendor-neutral project called Pipecat, which came out of the work we started doing about the time GPT-4 was first released 2 years ago to make it easier to build real-time agents. I wrote a guide that's online at voiceaiandvoiceagents.com where we tried to write down like everything we'd learned in the last 2 years building real-time voice AI. I'm going to reference that guide a couple of times as I talk today to try to avoid going down rabbit holes that I would normally want to go down and geek out about. I want to talk for about 15 minutes and show a couple demos and then leave time for questions. And I made a repo for this class. It's github.com/quindla/evals-course-voice. Maybe somebody can drop it in the chat. The code I'm going to show is all in that repo, and it was super great to have this session as a forcing function to clean up code enough to share it publicly that I have, you know, in bits and pieces floating around in various temporary directories on my machine. So let's just talk a little bit about voice architecture because if you haven't built voice agents before, it's worth understanding kind of what we do in a typical voice agent. This is the block diagram for a single conversation turn. We run in a loop because a voice conversation is multi-turn. And as you know from building with LLMs, LLMs are stateless. So you have to give the conversation history to the LLM each turn. You may do things like modify and summarize the conversation in flight. Those are slightly more advanced topics, which we won't cover today, but we can talk about them in the Q&A. But the basic block diagram is audio comes in from the user. You turn that audio into text. In parallel, you do turn detection to know when the user is done talking and you should trigger inference and have the voice agent respond. That user text needs to go into whatever context management logic you're using so that you're tracking it. The LLM inference comes out as text and goes into a text-to-speech or a voice generation model that generates the audio that the user actually hears. Again, you have to put the text or the audio or both in your context management logic, and then you run that loop again when you get new user input. Every voice agent in production looks like this. We're not going to talk about new speech-to-speech models. Today, although again, we can talk about them in the Q&A if people are super interested. The core principles are the same. The block diagram looks a little different, but I'll just say that 99% or more of production voice AI bots today look roughly like this. And here's what this looks like in code. This is actual Pipecat code. This is from a fully working demo. This is that block diagram, but in Python code, there's you know, 50 lines of imports and 30 lines of event handlers after this, but this is the core of a production voice agent. In production though, agents often get a lot more complicated. So here's a pretty typical medium complexity pipeline. You can see we've got actually two parallel pipelines going on here. One is the main conversation inference. The other is a kind of LLM as a judge that's happening in a second pipeline, it needs to be fast but not real time is the way to think about that. And this is a pretty common emerging pattern as use cases get more complicated. You have multiple inference loops going on for at least some parts of a conversation that complicates evals, complicates how you think about the data, how you organize like things like OpenTelemetry traces. So things are getting more and more complicated and we're in some ways getting farther behind in being able to do good evals for voice. Just one more sort of very high-level thing. When you build voice agents, I tell people sort of you need to think about these three things. You have to write the code, you have to deploy the code somewhere, it has to run in the cloud somewhere, uh, and then you have to connect users over the network to that code running in the cloud so you can get the audio or the audio and video or the audio and video and images and text in and out of that voice loop. Um, you these days have a lot of good options actually for getting started. There are some platforms that bundle all of this together. A couple companies that I work closely with that do this are Vapi and PhoneLy. You may have heard of one or both of those. They give you this full package, this full— all three of these things bundled together with a bunch of other stuff. Makes it really easy to get started, is a great solution if what you are doing matches the design and the features of that platform. You can also go all the way the other direction, write all the code yourself, and then decide how to deploy and host and auto-scale and monitor it and maintain your own network infrastructure for connecting, you know, Sockets or WebRTC to those bots. You have lots of options for mixing and matching or for going with the sort of bundled approach. So very quickly, I'm not actually gonna go over everything on this slide, but just a few placeholders for you to pull on these threads later if you're interested. So what's hard about voice AI? The, the big thing is everything has to happen fast. You have pretty strict voice-to-voice response time requirements, or users will just not use a voice agent. I tell people aim for 800 milliseconds voice-to-voice response on the client side. That's everything. That's all the LLM inference plus all the network transport. 800 milliseconds is really good. A little faster is even better. A little slower is okay. Much slower is probably not okay. It does depend a little bit on your use case, but you know, 800 milliseconds, you have to have a stake in the ground for these things. 800 milliseconds is a good stake in the ground. A couple other things just to highlight here. You're always using multiple models for voice AI and multiple models, as we talked about a little bit, complicates how you think about evals. Do you evaluate the transcription step and then the inference step and then the voice step separately? Do you evaluate the whole thing end to end? I mean, obviously you need to do both. So you have to figure out how to think about kind of the collection of data for each turn and for a full conversation. Also, everything's multi-turn. Multiturn evals are another layer of complexity. And in fact, one super interesting thing to me, kind of philosophically, because I, I get to work with kind of everybody in the space at kind of every level. So we work a lot with the DeepMind team. We work a lot with OpenAI. Long multiturn, long context in general, and audio modality all take you out of distribution for how the big models are trained today. Practically speaking, what that means is that instruction following and function calling are worse for voice AI use cases than they are on any of the published benchmarks you will see for any models. So your function calling may work great for the first 3 turns of the conversation or 5 turns or 7 turns of the average conversation, and then your function calling and instruction following will fall off a cliff. at some point for every model today, even the very, very best ones. And I think that's a function of two things. Partly it's just model architecture. It's like how, like what the compromises that are being made on the transformer tweaks, the model architectures, the inference stacks, all that stuff. We are not the most important use case for this stuff. Uh, Shreya, do you wanna jump in?

Yeah, I have a question. Yeah. Um, how long is long context? What does that mean in the voice case? Do you mean it's context that you're retrieving, um, as part of RAG given some voice snippets? Or like, it's like people giving a soliloquy of 5 minutes and that becomes long context.

So it's two things. It's not usually a soliloquy for most use cases, although there are some voice use cases where people talk and talk and talk dictation style and then you do something with it. But for most use cases, it's two things. One is the easiest way to do RAG for voice AI is to pull a bunch of stuff into the first part of the context at the very beginning of the conversation. So often you're adding you know, 50,000 tokens or something at the beginning of the conversation, if you can do it, like if your evals survive it, right? The other thing is that there are a lot of voice AI use cases where there's a 20-minute or a 30-minute conversation. Like think of a customer support call where you're on the phone a long time trying to get help in a customer support context. So the multi-turn, you're just adding, like, na— if you do that naively, you are adding a user input and an LLM output message every single conversation turn. And that may not add a huge number of tokens relative to like the theoretical context length of today's LLMs, but it definitely adds some tokens and it also adds like a lot of latent space complexity just empirically. So you have these workflows where you've got a lot of user assistant, user assistant, user assistant, user assistant messages and some function calls. And that just makes it hard for the LLMs just based on what we've seen.

Yeah, that's really interesting. It's different from a lot of the stuff we've been teaching in the course of, you know, it's, it's not like 200 back and forth turns. In practice, do people actually have all of these back and forth turns or you try to like compress it or compress is not the right word, but condense it perhaps into one term turn where you say like system, user, assistant, whatever it is, just to make it be less multiturn, if that makes any sense.

It totally makes sense. So we started just saving the whole context naively. We've started to move all of us in the Voice AI ecosystem over the last few months towards figuring out how to do focusing compression, like processing at particular moments in the conversation with specific summarization goals in mind. So there's a, there's a pattern I really like that is getting a lot of use now that I think of as a state machine. So model the conversation as a series of states. Every state is a system instruction, some transformation of the entire conversation context so far, a set of tools that you're gonna use at that point in the conversation, and a set of exit nodes to whatever legal other next states there are. That tends to really help at least me think about how to focus the context so that the LLM does what you want. I mean, if you're stepping back a little bit and trying to figure out what you're trying to do here, LLMs are amazing conversational partners. Like that's what has opened up this voice capability. You, you really want to leverage the fact that the LLMs can have an open-ended conversation. On the other hand, for most enterprise use cases, you have a specific set of things you need to get done in a specific session. There's some tension between those, like letting the LLM do its conversational thing and making sure you get the business logic like checkboxes all checked off. A state machine abstraction kind of lets you try to find that sweet spot where you're getting the best of both worlds. And like everything, like everything you're teaching in this class, it takes a bunch of iteration, but at least if you can swap out all those 4 pieces, you can model it. Does that make sense?

Yeah, I think that's super helpful and also grounding and kind of what we've talked about so far in the course. Thanks for sharing.

There's a library called pipecat-flows, which is a community-maintained state machine. Abstraction of exactly this type built on top of Pipecat. If people want to sort of look at how people are building this, this kind of thing. And I think you're starting to see that, that abstraction in the non-voice space as well. There's a lot of stuff that kind of crosses the boundaries between the voice agents and the non-voice agents. So what are we going to try to do here? We're going to try to look at our data and listen to it. And I'll just say that there are several observability integrations in Pipecat already. You in production probably want to use one of these. Brooke from Koval is talking next, I believe. I am a huge fan of Brooke's platform and the work they do on kind of simulation-driven evaluation of voice AI. But we're not going to use any of these today because I'm also a big fan of what Hamel Husain has said in every context I've bumped into him in, including when he came to our voice course, which is like, get your hands dirty, extract the data yourself in whatever format you think it might be useful. Write some ways of looking at it. I love Shreya's post on Twitter over the weekend, maybe that you should vibe code web frontends to look at your data, which I totally should be doing. So we're going to do that today. And then I will say, I do think you should use like somebody's platform as they— as the platforms add better and better audio support in production. But I also think you should start by like figuring out what that data looks like yourself. So here's the GitHub repo again. And this repo has 3 bots in it. I, we, it's everything's agents in 2025, but we've been doing this long enough that we just call these bots. Uh, so there's like bot.py files. There's a basic bot that just logs to the console. There's a bot that has minimal additional code. It's like 10 lines of code or something to, to push to an OpenTelemetry collector. And then there's a bot that we sort of started more from scratch and like extracted just the data we wanted, uh, from the conversation and put it somewhere and, and tried to figure out how to look at it. Um, we'll skip the demo of just the bot that goes to the console because like you've all seen a voice agent at this point. We'll jump straight to the OpenTelemetry bot because a video is worth a thousand words. This is the diff though of between bot-001 and bot-002 that adds OpenTelemetry support. It doesn't take much to add the tracing. I say that just because like there's no excuse for not putting this data somewhere and I'm speaking to myself because I usually write some stuff and actually sometimes deploy it to production and then go back and add the OpenTelemetry tracing. Don't be like me, be better than me. Here is a little example.

I'm right here and ready to help you with anything you need.

Audio coming through. Thanks. I think you got the initial greeting wrong. We'll come back to that.

Oops. My apologies for that. I'm here and ready to help in any way I can.

Let's check and see if we've got your traces in OTEL.

It sounds like we're on an intriguing mission. I'm going to scan through the grand chandeliers and ornate carpets of the hotel for traces.

So I have no idea where that came from. I'm actually really glad I captured that because that was GPT-4o going totally off the rails. Uh, we could dig in and try to figure out why or how often that happened, and we will come back to that. But that was, that was super weird and not a normal interaction. Um, but we can look in the traces and just sort of see that in the Langfuse dashboard, which is the hotel collector I was using here. That is in fact the conversation we just had. If you've used OpenTelemetry—

So there's nothing in your prompt about hotels and chandeliers?

No, no. Like, this is one of those 1 in 20 or 1 in 40, or, you know, good to be able to characterize it, right? It, it misheard what I said. Some combination of bad transcription, background noise, GPT-4 weirdness. Does happen sometimes, even if the transcription is clean. So I was actually kind of glad I, I caught that one.

That's really surprising because, you know, you're using a really high quality microphone and your audio is very clear at the moment.

Yeah. So this was late last night. I was on my Linux box. I definitely had like a slightly weird sample rate thing going on with my input microphone. I mean, you know, I don't know. It was a local connection though, so it shouldn't have been a network issue. Um, but we'll come back to it. Uh, yeah, just making sure I have the right slides. Yes. Okay. Um, so let's jump into doing something similar to that, but more from scratch. There's more code here, so I won't show code, but you can look at 003-bot-sqlite.py for the code. It's probably another 100 lines of code to save exactly the data I wanted to save. And here's the list of things that I was like, oh, I want to save this stuff because this is the kind of thing I always need to look at. Either in an individual debugging session, like what just— why did that bot just do that? Or aggregates that I care about across a bunch of, a bunch of sessions. So like a unique session ID, the turn number in the conversation, the turn start time, the turn end time, the user text, the LLM text, the voice-to-voice response time, because latency matters so much, and whether the user interrupted the bot before it was done talking. So it's a pretty good basic set. Plus we'll save the entire WAV file. So that for any chunk of the conversation, we can sort of play back the audio as we heard it. And up in the top right is the SQLite database creation command I ran from Bash to set this up. And then I vibe-coded some Look at Your Data scripts. Hamel posted today about like getting over the chasm of like going from, oh, I used AI to write some code to I used AI to write some code. I'm trying to get there myself as a lifelong programmer with complicated feelings about code as craft. Claude wrote these scripts for me. It did a pretty good job. There was a little bit of multi-turn back and forth, but it actually wrote the scripts for me last night at like midnight, and I didn't even change the file names here. So I might have chosen different file names, but pretty good job, Claude. So let's look at these 3 scripts. We're going to do a quick conversation first, just to get something in the database that we recognize. I am here. Ready to go. Uh, can you tell me a joke? This is my go-to.

Why don't scientists trust atoms? Because they make up everything.

Oh, that's a classic. How about the one about the janitor jumps out of the closet?

Sure. When the janitor jumps out of the closet, he shouts, "Supplies!" I love that one.

Thank you.

You're welcome. I'm glad.

So the terminal will get bigger in a second. I should have made it bigger earlier. But so this is the first script that just dumps all the conversation, a list of all the conversations. Or gives you console-friendly output of an individual conversation. So this is the conversation we just heard in a console-friendly or terminal-friendly format. Obviously, if you can get it in terminal-friendly format, then you can do all kinds of things with this data. And you can see that's the conversation we just had. The next thing we'll do is actually play a snippet. We'll actually play a snippet of a previous conversation that I really like. [Speaker:JOSH] Yeah, I think we need a suggestion for a break.

How about a quick stroll outside for some fresh air? Or maybe just a 5-minute dance party to your favorite song?

I love a dance party. And now we'll use that same script we just used before to actually look at the text mode version of that conversation and find that particular turn. Apologies for the sample rate issue on my microphone. I didn't go back and rerecord it once I heard that. You can see the whole conversation here. All these test conversations were pretty short. Um, and then you can see the exact, uh, phrase that we just heard. That's turn 4 in this particular conversation. And then the last thing we'll do is go back to that, like, uh, not following system instructions properly in the very first turn that we heard in the first video I played. I asked Claude to write an LLM as a judge script to do a very specific test. Uh, look at the first LLM response and judge whether it followed my system instruction. Again, the source code for all this stuff is in the repo. We're running that script. We have 22, like, conversations in this test database. That happened 2 times. We can go back and look at the individual times when that happened. One where my mouse is there, actually. Yeah, that's my— it looks like it was just a contraction. Maybe that's okay. You can decide what's okay and not, of course, in your evals. The second one though is quite a bit worse. So there's the session ID. We could look at that whole conversation, see if starting off wrong makes you go wrong, all that kind of stuff. The second one is worse. It said not only the phrase I asked it to say, it said, what can I do for you today? Which I asked it explicitly to only say that phrase in the system instruction. And this is something that I actually see pretty commonly with GPT-4o. If you ask it to say something specific, it will refuse some percentage of the time. I'll stop my screen share. So this is a real-world example of something you want to do an eval on. If you ask GPT-4 to say something relatively specific, at some point in the conversation, you want to know if it actually does that. It might not. I think it's an outgrowth of GPT-4 safety stuff, not wanting to be prompted very, very, very specifically, although I don't know 100%, but I do know that it happens. This particular prompt I use at the beginning of the conversation, it happens about 10% of the time because I got Claude to vibe code an eval script. That's all I got.

That's really great. I mean, yeah, I really love the fact that you shared the code and you have this demo that's really useful for everyone. I'll be going through it too. It looks like a lot of fun, actually.

Super fun. Voice is so fun.

Maybe we have some time for questions, actually.

So yeah, this is great. Stephen, I have a great question. Well, Stephen thinks it's a bad question, but I don't think it's a bad question. So he said, why would the first turn ever be generated by an LLM if you want it to say the same thing for the first time?

And just for that first time.

That might be it.

Yeah, no, that's totally right. If I really, really, really wanted to say exactly the same phrase, I would probably prompt the speech model. The voice generation model rather than the LLM. Often you want it to say something approximate. So there's a looser version of this test that I think is real-world relevant. The real reason is I'm really lazy, and when I write demos, I often just like stick it in a user prompt rather than in Pipecat. You stick a text generation frame in the pipeline, it goes through the pipeline. I could do that. I probably should start doing that. But I often ask the LLM to say something specific because I'm lazy.

Do other folks have questions, or I'm gonna ask my own questions. So I encourage people to unmute themselves, raise their hand maybe. All right, well, I'll go. So some Rapid fire questions. What are the— what are some types of failure modes that people do not expect to get out of voice agents that they see when they start looking at their data?

I do think it's the single biggest thing I see is that instruction following and function calling fall off way faster with multi-turn voice than they do— than people expect them to. That's the thing that we work— we work to solve for most often in production today. In 2025. There are also, when you, a lot of people really want to use the new speech-to-speech models. And so there's a whole class of failure modes in the new speech-to-speech models that are super surprising in the way that all the newest models are. Uh, like I think Brooke and I were actually talking about this over the weekend, like the, um, uh, GPT-4, a real-time model will often just start speaking in Spanish. And you're like, there's nothing in the prompt or in any of the audio input that a human can perceive that would have you sort of switch into the Spanish angle in the latent space. But like, you do it a lot. So there's there's something there. So there's a bunch of stuff like that in the very newest models.

Yeah, it's actually my, my grandma likes speaking to ChatGPT in her native language. And every now and then it'll say like, oh, I'm sorry, like, I can't say that in English. And then it'll go back and speak in the native language, which is so bizarre. And she just like doesn't know how to deal with it, but she's gotten used to it.

Yeah, no, I'm sure she's building up a really good model of how the the model works, like a good mental model of how the language model works from just using it a lot. It's super jagged. It's the jagged frontier thing. It's really interesting.

Yeah. All right. Other folks have come in with questions. Does Pipecat or any vendor have an eval database facility that works with logging?

What we try to do with Pipecat is build great bridges to the tools that specialize in all the parts of the voice AI space, including evals. So like Brooke is going to talk about exactly how to build voice agent to, to Coval, like bridges from maybe multiple platforms. Certainly there's a great Pipecat integration with Coval. Is that the question? I'm not sure. I'm sorry.

Great with Pipecat and Coval and LangFuse. And so you, I think a lot of the tracing, um, a lot of tracing and the like standard evals actually don't change with voice. Where if you're just trying to test a prompt or you're trying to do the tracing, that remains the same. So I think building on top of that with PyCat and Coval can help you there. Makes sense. Um, other question on deployment. Do you run everything on a single Kubernetes cluster and route internally, or do you call various APIs? Um, yeah, maybe broadly, like, what are the common architectures that you see for complex and parallel, I think is what you mentioned, but multi-step pipelines?

It's a great question. So generally people in production are running a, you know, a voice agent process container on a CPU machine and calling out to hosted APIs. It's— that's partly because the best models actually aren't available for you to run yourself. And it's partly because we're also new in this space that, you know, building the thing that works and then over time making it better, better along various angles that are complicated to, to, to figure out and cheaper is like a second step. And most people, even at scale, aren't on that like second step yet. Um, so most people do something like, uh, use DeepGram for transcription, GPT-4o or Gemini 2o Flash for the LLM. And Cartesia or ElevenLabs for the voice model. That's the combination we see most often today. Every one of those blocks though is more and more competitive and more and more exciting a space. So there's more and more good transcription models, more and more good voice models. One of the reasons we built Pipecat was to make it super easy to like swap out and test different models and even use different models for different parts of a conversation or for different routed conversations in a complex environment. Um, if you're hosting this stuff yourself, you do need to like do a bunch of Kubernetes work to figure out like how the shape of this particular long-running UDP-dependent or networking-dependent workflow works. It's a lot of work. Um, we did launch a daily, a Pipecat Cloud hosting service a couple months ago, which you can use if you want to. Nothing about Pipecat depends on any vendor though, so you definitely don't have to use it. Um, and being multi-region is also complicated and important for a lot of big at-scale use cases. There's a tension. You want to, you want your voice agents to run close to your users so that the network latency is low going back and forth to the voice agent. Most of the big providers though, don't have inference in very many locations. And if you have to choose, you'd rather be close to your inference provider rather than close to your users, cuz you're making multiple inference provider round trips if you're using hosted APIs. So there's a bunch of things to balance and we spend a lot of time talking about this in like the Voice AI course, for example, because there's no perfect answer right now. Thank you everybody.

Thank you so much. This was so much fun. This talk was amazing. Thank you. Bye-bye.