Tom Shapland at Livekit explains why voice AI struggles with natural conversation. The core problem: turn detection and overlap when agents and humans speak simultaneously. Even with Whisper, Claude, and ElevenLabs piped together, the latency makes it brittle, and current AI still lacks genuine feeling or judgment.
Chapters
Livekit, the open source platform behind OpenAI's Voice Mode, powering voice agents for everyone from startups to 911 operators.
I'll be talking to one of our users about some problem and I'm like, how many users is this impacting? And they're like, oh, it's impacting like 1% of our calls. So that's like 100,000 minutes yesterday.
I'm like, how are you doing that much traffic?
So we just raised our Series C and we are a unicorn as of just recently.
My daughter, when she talks with ChatGPT using the voice mode. She's thinking and then ChatGPT just jumped in and then they started arguing with each other.
The biggest problem in voice agents since LLM-based voice agents have come to the scene has been turn detection. The conversation breaks down and the user gets frustrated and the call doesn't go to its objective when the agent and human start talking over each other.
I would take Whisper for speech to text, pipe the transcript to Claude, send response to ElevenLabs. Streamed audio back over a WebSocket, put some voice activity detectors on it. What do you think of my approach?
That is a great first pass. If you build that approach, you'll get something working, but it will be a little bit brittle and you'll likely have really high latency. People are surprising creatures. You would think that like it wouldn't make a difference if a person was talking to a voice agent or like a fake human, but they talk a lot more when they're talking to an avatar, even when the avatar doesn't look totally real. They open up more and then compared to a voice agent, just like a wave bouncing on the screen. Every decision is emotional, and emotion is embodied. It's how we feel about things. It's not an intellectual thing. It's a feeling. And I don't think the way we're building intelligence in machines now approximates judgment in any way.
There is no feeling. I hope you go build something weird.
That's exactly kind of how I think about it.
I really felt like deeply that it needed to exist.
