In this interview, Angelina Yang and Tom Shapland discuss the practical challenges and solutions for building real‑time voice AI systems. They begin with the difficulty of turn detection, especially with children’s unpredictable speech patterns, and explain how LiveKit’s turn‑detection model improves conversational flow. Tom outlines a developer’s pipeline—using Whisper for speech‑to‑text, Claude for LLM processing, and ElevenLabs for text‑to‑speech—while warning about latency and brittleness when simply chaining these components over WebSockets. He advocates using WebRTC for low‑latency audio transport and introduces the concept of pre‑emptive generation, where the LLM starts producing a response before the user finishes speaking to shave off response time. The conversation then shifts to user experience, noting that even low‑fidelity real‑time avatars increase user engagement compared to voice‑only agents. Tom contrasts replacing traditional telephone‑based call‑center workflows with augmenting human agents, highlighting how voice agents can offload repetitive support tasks and free staff for higher‑value activities. Hallucination risks in regulated sectors are addressed by employing shorter prompts and modular sub‑agents that limit each model’s scope, thereby reducing the chance of fabricating information. The interview concludes with a brief personal exchange about parenting, emphasizing the importance of nurturing curiosity and independent thinking over conforming to external expectations. Throughout, Tom balances technical depth with conversational tone, offering actionable guidance for developers building production‑grade voice AI.
Chapters
My daughter, when she talks with ChatGPT using the voice mode, right, it was actually a terrible experience because the way kids talk and the way they pause and the way they interrupt are more unpredictable than adults. And sometimes she's thinking and then ChatGPT just jumped in and then they started arguing with each other. And my daughter is just saying that, You should stop talking.
The OpenAI real-time model and Advanced Voice Call, you're, you're not actually using our turn detection models. I think ours are much, are very effective. And in fact, a lot of users end up migrating to LiveKit because they try out ours and they're like, oh, the conversation actually ran much more smoothly. The problem you're talking about is definitely not limited to children talking to AI. Like the biggest problem since LLM-based voice agents have come to the scene has been turn detection because the conversation breaks down and the user gets frustrated when the agent and humans start talking over each other. And like you said, like people have different styles of, of talking to each other. And part of what you and I do when we talk is like, when we first start talking, we're doing some sort of like calibration with each other and know like how long to wait. And then even midway through a conversation, I will like dynamically calibrate and be like, oh, this is actually kind of like a, a more thoughtful topic, and so I should maybe give her more space for thinking of her response.
How do I do that as a developer?
So what the model is doing is it's taking the transcripts as they come in from when the user is speaking, and then it's passing that to a small text-based LLM and asking the LLM, how likely is it that the person finished speaking?
You're doing predictions, the classifier. So cats and dogs, finished or not finished? AI roleplay as the prospect while I as a user pitch, and then the agent should take notes during the call and then switches to coach mode at the end to critique my performance. I would take Whisper for speech to text and pipe the transcript to Claude with my you know, system prompt, and then send Claude's response to ElevenLabs for text-to-speech, and then stream the audio back over a WebSocket and, you know, put some voice activity detector.
That is a great first pass, but it will be a little bit brittle. You'll likely have really high latency. The first thing I wanted to call out is around WebSockets, WebRTC infrastructure for moving audio around the internet, and WebRTC is really built for real-time experiences. It will sacrifice completeness for the sake of keeping the content current. So it'll drop packets so that rather than making sure to keep the packets all in order so that the experience stays in real time. How do you handle an example like preemptive generation? How do you handle a case where you want the LLM to start generating output before the user has finished speaking? Because the user, because the LLM generation is the slowest step.
What is that? Why do I need that?
One of the ways, one of the tricks to like try to reduce the latency is you, every time the user speech comes in from the speech-to-text engine, you start generating, having the LLM generate a response even before the user has finished speaking. When the user does finally finish speaking, the LLM has already started creating the response and you've shaved off some of the time it takes for the voice agent to respond because you don't have to do everything serially. You're kind of doing things in parallel.
Wow. If you think about this is maybe how our human brains work as well, because as you were thinking, as you were speaking, I'm already thinking. So you're doing predictions basically.
Well, the first thing that comes to mind is that you could put a face to your voice agent with the real-time avatars that are out there. So basically the way it works is you have another model now in your pipeline. After the text-to-speech, you now have a real-time video generation model of an avatar. People are surprising creatures. You would think that like, it wouldn't make a difference if a person was talking to a voice agent or like a fake human, but actually they talk a lot more when they're talking to an avatar, even when the avatar doesn't look totally real, they open up more and they say more compared to talking to a voice agent. Uh, that's just like a waveform that's like bouncing on the screen.
I have a question for you on this one, cause I just interviewed the CEO of a company called Delphi. So, Delphi is a digital twin. He told me that they tried video chat and video chat doesn't work because people think it's, it's kind of creepy. I wonder what's your take? Why in your case you're seeing it's working while in his case he's seeing it's not working?
We live in a nonlinear world now, an exponentially changing world. And the models have gotten a lot better. Also, I think for— it might be for their use case. Like, there— these are particular people where the person talking to them is familiar with their mannerisms, like celebrities is my understanding, or coaches that have followings. Whereas if you saw my avatar speaking and you weren't familiar with my mannerisms already, you'd be like, oh, that must be just how that person talks. I don't totally get people, but they seem to talk more even when the agent is— when the— there's this kind of ramp up in how much people interact with a computer, if you give them just like a little chat bubble, they don't talk that much. You give them a voice, a way for them to talk, then they interact more with it. And you give them like something for them to put their eyes on where they feel like they're talking to a person, even if it's not real, they talk even more.
Do you feel it's mostly replacing old systems versus like, you know, creating entirely new experiences?
Most of voice agents in production now are telephone-based voice agents. And they're doing things like the healthcare customer support or lead qualification. And in those cases, they are replacing workflows that humans are doing, but I want to be careful of saying replacing because like one of my customers tells the story about like their, the call center that they're working with. They have 8 people working there and there's one job they don't like doing and there's another job they like doing a lot more. They like hunting for new business. They don't like supporting, doing the customer support side. And so more of the people get to do the hunting for more business now and less of the customer support side because they're augmenting their staff with voice agents.
Hallucination is still a problem, right, for more high-stake companies to completely adopt it, like the insurance company, healthcare.
Hallucinations are still a risk, but we're seeing so much growth in these regulated industries, I think, because the way you— we're getting better at contain— or preventing hallucinations. And part of the way you do that is you have shorter prompts, uh, with less that the agent can lose and more you can, you more tightly control that agent. And, and there's lots of, like, the models are getting better at this and the frameworks like ours are getting better at this, where in our framework you can have lots of different sub-agents that are more focused. LiveKit gives you the ability to break your agent into lots of small agents that get passed from one— where the user gets passed seamlessly, unwittingly, from one agent to the next, so that they— each agent is more focused and less likely to hallucinate because it only has like these 3 things it's allowed to do.
I mean, you are a parent with 2 kids, same age as mine, and What are you teaching your kids? You know, today with, with this AI hype, everything going on.
I'm hoping that they find the things that they find interesting and weird to them and that they pursue those. Where I went to high school, everyone was talking about trying to get into the best school. And I think that's the wrong framing. It should be like, what is— get into the right school for you, take the right path for you.
I'm just copying people around me and I thought that's the right path.
Isn't that so true? I used to think about that. My first startup when we were working out of San Francisco for a while and all anyone seemed to value was growth and money. And I was like, maybe we can value other things. We could choose to value other things, but everyone around you is valuing that. Everyone else is obsessing over their growth and stuff. It, yeah, it takes a lot of guts to value other things.
How do you teach your child to teach your children to have the guts to do what they truly want to do?
You model it like you're doing now. You're modeling it with pursuing your own path, not doing what other people think you should be doing.
