LiveKit: The Open-Source Platform Behind OpenAI's Voice Mode | Tom Shapland, Product @LiveKit

Livekit, the open source platform behind OpenAI's Voice Mode, powering voice agents for everyone from startups to 911 operators.

I'll be talking to one of our users about some problem and I'm like, how many users is this impacting? And they're like, oh, it's impacting like 1% of our calls. So that's like 100,000 minutes yesterday. I'm like, how are you doing that much traffic? So we just raised our Series C and we are a unicorn as of just recently.

My daughter, when she talks with ChatGPT using the voice mode. She's thinking and then ChatGPT just jumped in and then they started arguing with each other.

The biggest problem in voice agents since LLM-based voice agents have come to the scene has been turn detection. The conversation breaks down and the user gets frustrated and the call doesn't go to its objective when the agent and human start talking over each other.

I would take Whisper for speech to text, pipe the transcript to Claude send response to ElevenLabs, stream the audio back over a WebSocket, put some voice activity detectors on it. What do you think of my approach?

That is a great first pass. If you build that approach, you'll get something working, but it will be a little bit brittle, and you'll likely have really high latency. People are surprising creatures. You would think that, like, it wouldn't make a difference if a person was talking to a voice agent, or like a fake human, but they talk a lot more when they're talking to an avatar. Even when the avatar doesn't look totally real, they open up more compared to a voice agent. It's just like a wave bouncing on the screen. Every decision is emotional and emotion is embodied. It's how we feel about things. It's not an intellectual thing. It's a feeling. And I don't think the way we're building intelligence in machines now approximates judgment in any way. There is no feeling. I hope you go build something weird. That's exactly kind of how I think about it. That really felt like deeply needed to exist.

Welcome back to Two Sides AI. Today I'm sitting down with Tom Shapley. Kaplan from Livekit, the open source platform behind OpenAI's Voice Mode, and they're powering voice agents for everyone from startups to 911 operators. Tom, welcome.

It's great to be here. Thank you for having me on, on the podcast.

For people who don't know what Livekit does, can you explain what it does, maybe towards two different audiences? One is, let's go with an elementary schooler, and the other maybe a developer who's already building AI agents and, you know, help you out if they need you.

I love that. Let's start with the elementary school kid. I have two kids in elementary school, so sometimes they ask me what I do at work. And so I have an answer for that.

Okay.

My answer for them is I ask them, have you ever talked to an AI? Like I ask them, do you remember the time we've opened up my phone and we talked to ChatGPT? And they're like, oh yeah, yeah, I remember that. And I tell them, we power those experiences. We make it so that people can build voice agents that you talk to. We make it so you can talk to computers. So that's my answer for an elementary school kid. How did I do on that one? I think I could probably do a little bit better on that. I think I could tell 'em like, we help build, I think what I normally say to my kids is we help build ways for people to talk to computers, like talking to ChatGPT.

How old are your kids?

They are 8 and 10.

Okay. So they know, they will understand you. What about developers, your real customers?

Livekit is a platform for building voice and video agents. You can think of Livekit consisting of two main parts. The first part is the WebRTC audio and video transport layer. So this allows us to move audio and video around the internet very quickly. And that's something that you need if you're enabling, if you're building apps where people are talking to computers. And then the second part of LiveKit is our framework for building voice agents. So we have an open source framework where you use our Python code or TypeScript code to build agents. And then those, we take care of all the like abstractions around building a voice agent, like handling turn detection and making sure that like the audio is reaching the agent and then the audio is translated into text and then that text is passed to an LLM and the output of the LLM is passed back to a text-to-speech engine and then that audio is played back to the user. So it's really, those are like at a high level, those are the two main components of LiveKit.

Okay, let's say, I'll give you the use case. So let's say the use case is sales training. Right. It's AI roleplay as the prospect. Well, I as a user pitch, and then the agent should take notes during the call and then switches to coach mode at the end to critique my performance. I would take Whisper for speech to text and pipe the transcript to Claude with my system prompt and then send Claude's response to ElevenLabs. For text-to-speech and then stream the audio back over a WebSocket and, you know, put some voice activity detector.

That is a great first-pass at it. And if you build that approach, you'll get something working, but it will be a little bit brittle and you'll likely have really high latency. You had some of the core components there that I wanted to address, which are like the voice models that you need in your stack. So the— you brought up each of these and I want to call them out specifically. If a user is talking to a computer, you need to transcribe their audio with a speech-to-text model. You need the, the next model in the stack is one you brought up here, which is something to know when the user is done speaking so that the LLM or the, so that the voice agent knows to process what the user has said and turn it back into audio and send it back to the user. So that's where you brought up the voice activity detection model. And there's a whole like suite of paralinguistic models. That we build at LiveKit that really help with this turn detection problem. So that's the second model in the stack. The next model in the stack is the LLM, and no explanation necessary there. And then there's the text-to-speech engine that turns— text-to-speech model that turns the LLM output into, into audio. So those core components are there, uh, and you can put those together. Um, now where you're going to start incorporating problems is the first thing I want to call out is around WebSockets. So we, you'll, you'll want to use, this goes back to like the first component of LiveKit, which is we provide WebRTC infrastructure for moving audio around the internet. And WebRTC is really built for real-time experiences. So, and the way that makes that most concrete for me is that WebRTC is, will sacrifice completeness for the sake of keeping the content current. So it'll drop packets so that rather than making sure to keep the packets all in order, so that the experience stays in real time. And so if you're using something like other approaches to it, you're going to, you're going to start running into problems where your, your audio is going to start to lag and that's going to cause latency issues and your, your experience is going to fall apart. And then the other thing around like orchestrating these models together is like, it's sort of like you could build your own authentication for your app. For users to authenticate, but that's really not the best use of your time.

Yep.

You should instead be using a framework that has already thought through the million edge cases and just makes it easy. Um, and that's what LiveKit's agents framework does for you is like, yeah, you can put together this stack, but then like, how do you handle the case when, like, how do you handle an example is like preemptive generation? How do you handle a case where you want the LLM to start generating output before the user has finished speaking? Because the user, because the LLM generation is the slowest step. We've already like built that.

What is that? Why do I need that?

Uh, so like so much of what people, when you're first building a voice agent, the first thing that you're trying to, the first thing you notice is latency. And the first thing you're trying to make your, you think like, oh, how can I make this better? Is around latency. If you have a, if the time between when the human stops speaking and the agent starts speaking is more than about a second and a half. People start feeling, excuse me, people start feeling like, users start feeling like it's taking too long to respond. And so one of the ways, one of the tricks to like try to reduce the latency is you, every time the user speech comes in from the speech-to-text engine, you start generating, having the LLM generate a response even before the user has finished speaking. So every time you get like a chunk of transcripts from the user, you start generating a response with the LLM. And that way, when the user, uh, and then you'll have to like, the user will keep speaking and you'll have to cancel the previous LLM generation and start a new generation as you get more speech coming in. But what this means is like when the user does finally finish speaking, the LLM has already started creating the response. And you've shaved off some of the time it takes for the voice agent to respond because you don't have to do everything. Serially, you're kind of doing things in parallel.

Wow. If you think about this, is maybe how our human brains work as well, because as you were thinking, as you were speaking, I'm already thinking, and I was thinking maybe I should ask this question as the next one. So it's already happening. So you're doing predictions basically, or you're answering based on the context that's coming in.

That actually falls more into the category of that second group of models we're talking around turn detection. So not about like, uh, but they're all related, but the way human mind— it's an interesting topic in linguistics where people are— it's an open topic, but how is it that humans are able to respond so quickly in conversations? And the prevailing hypothesis is that we're modeling out and predicting what the other person is going to say before they finish speaking and are already starting to generate a response while the other person is starting to speak. And you can see that in your head. It doesn't seem— it's not all that like so many good scientific theories, when someone says it, you're like, oh yeah, it seems pretty obvious. That is what I'm doing when, when you speak.

Can I use speech-to-speech models like the GPT-4o real-time? That can accelerate that as well, right? Because I'm eliminating the step of calling the LLM and then, you know, doing a transcription and then all that.

For most production use cases where people have built voice agents that are actually generating revenue for the business, their business, they don't use speech-to-speech models. Speech to— and the reason is, is that speech-to-speech models, there's two main reasons. One is that speech-to-speech models aren't very good at tool calling. Text-based LLMs are much better at tool calling. And so if you have some agent that's trying to do something for a user, it'll likely have some sort of tools, some set of tools that it's calling. And, and the other one is, I'm sorry, what was I going to say? The, the The other one is, oh, they have more control. So if you, uh, if you're using a speech-to-speech model, you can't, you don't have as much control over like how certain words are transcribed, but some of the, the new things coming out with speech-to-text models is that you can like prompt those models where you can tell them ahead of time, this is what the user is talking about and here are some of the words that they will be And here's just general context, and this speech-to-text model takes in both the audio and that prompt every time it makes an inference. And so you just have so much more control and you can create more accurate voice agents to actually fulfill their goals when in the current state of the world. Maybe one day speech-to-speech models will be more effective, but it's not, they're not currently the, the main thing that people are using. To create a good voice agent experience, there is a lot to it. And that's why starting with, that's why it just makes sense to start with a framework, ideally an open source framework where you can actually see what the code is doing and it's not a black box, and then use that as your starting point for building your voice agent.

Is there any use cases that, you know, the speech-to-speech models will be a direct solution to it that actually would work? Or, or you're thinking the model is just not good enough just yet?

There's a lot of usage of speech-to-speech models, but they're more on the consumer side. So. On the B2B side where people are building voice agents for like business use cases, then people use the cascade or pipeline approach that we've been talking about this whole time. But a lot of consumer applications, such as when you talk to Gemini on your phone or OpenAI on your phone, you're actually using a speech-to-speech model.

What's the difference between the consumer apps and the B2B use cases that, you know, you're saying that the model choices should be different?

Yeah, I mean, we can think about it in terms of like talking to a friend. When you're talking to a friend, you're just kind of chatting with them and wanna— there's not like some objective that you're trying to get to. Sometimes you'd have an objective you, you're trying to get to with a friend where you need them to like look up the time you're supposed to meet them or something.

Mm-hmm.

But most of the time, you know, you're just chatting with them and it, you don't need to access external databases or you don't need as much control over the conversation flow., but in a B2B use case, to give you a more concrete example, you can think of like lead qualification, which is a, a use case with a lot of growth in the voice agent field. So like a business gets a lead from a website, uh, and then they want to call that lead to qualify and make sure that they like have the, I don't know, the, they live in the right area code. And so you can imagine like there's a whole workflow that you need the voice agent to adhere to. Um, so you need that sort of control over the LLM and then also that that voice agent, different steps in the workflow might need to be making external tool calls to say, oh, like, okay, this person doesn't have the, uh, this qualification. So instead, but they may still actually have this other qualification, but I need to like look up something about that qualification. I don't know. That wasn't the best example off the top of my head, but hopefully you get the idea. I should also just one last follow-up on that is that there are some consumer use cases where the voice agents are using the cascaded pipeline and are very sophisticated. So like the example is like our customer Portola builds Tolands, which are consumer voice agents where people talk to this character, but that character like comes to know you over time and actually has objectives with its conversation with you. And so like those are very sophisticated agents they've built and to build agents that sophisticated, they're not using speech-to-speech models, they're using them. The pipeline approach.

I have a second point about this conversation turn thing. One thing I noticed was that my daughter, when she talks with ChatGPT using the voice mode, right, it was actually a terrible experience. No offense. I think because the way kids talk and the way they pause and the way they interrupt are more unpredictable than adults. And he—

she sometimes—

she's thinking, and then ChatGPT just, just jumped in, you know, and then they started arguing with each other. And my daughter is just saying that you should stop talking.

I would like to point out that the OpenAI real-time model and Advanced Voice Call, you're, you're not actually using our turn detection models. I think ours are much, are very effective. And in fact, a lot of users end up migrating to LiveKit because they try out ours and they're like, oh, the conversation actually ran much more smoothly. The other point I want to make is that the problem you're talking about is definitely not limited to children talking to AI. Like, the biggest problem in voice agents currently, and it has been this since— this has always been the biggest problem since LLM-based voice agents have come to the scene, has been turn detection. Because the conversation breaks down and the user gets frustrated and the call doesn't go to its objective when the agent and humans start talking over each other. And like you said, like, people have different styles of talking to each other. And part of what you and I do when we talk is like, when we first start talking, we're doing some sort of like calibration with each other and know like how long to wait. And then even midway through a conversation, I will like dynamically calibrate and be like, oh, this is actually kind of like a more thoughtful topic. And so I should maybe give her more space for thinking of her response. And this is all stuff that you can, because LiveKit, they are open source agents framework is so flexible, you can actually build that into your agent. So like we, we aim to have it default outta the box, the turn-taking be as good as possible. We recognize that there's different use cases and that you have the flexibility to kind of parameterize the, the models and build your agent in a way that is gonna work for your use case and the different turn-taking characteristics of your use case.

How do I do that as a developer?

The short answer is that you can adjust some of the parameters in your voice agent. And, uh, the parameters around how the model works. So let me actually spit out like first the short answer, which is you change some parameters, you're not doing it with prompting. So what the model is doing is it's taking the, the transcripts as they come in from when the user is speaking, and then it's passing that to a small text-based LLM and asking the LLM, how likely is it that the next thing that the person says, that what the likelihood that the next token is a complete, is a period, is a finished statement. Actually, that's not right. I said that wrong. How likely is it? We train the model on utterances from people, and then one of the inputs in our, not one of the inputs, but one of the fields in our model training set is when the person has finished speaking. So there's a token that's like the person finished speaking. And so then when we run inference, where the machine learning model, the LLM is outputting a probability that the next token is that the person is the token for finish speaking. That means that as the person is speaking, the transcript is coming in and if the output of, and we're running inference on this turn-taking model and this end-of-utterance model, and if the end-of-utterance model says the person is likely to have finished speaking, then we then tell the agent that it's their turn to speak. But if the output of the model says that there's a low likelihood that the person has finished speaking, then we tell the agent, wait a few seconds before you start speaking, because the person might start speaking again. And that is the parameter. One of the parameters you can tune is like how long you— if the, if the end of utterance model has said the person may not be done speaking, how long do you wait until they, the end of the voice agent starts speaking? You can set it to like a very short time, and if the model doesn't think the person has finished speaking, then it's very likely that the, the agent will start speaking and maybe interrupt the person. But if you make it a really long time, you can give the person lots of time to, to finish their thought. That's maybe too much in the— hopefully I explained that sufficiently well. It's kind of a complicated system if you're first hearing about this for the first time, and I feel like I, it's, I almost need to diagram it out. Hopefully I explained it over voice well.

My understanding is It seems comes, you know, a natural solution that it sounds to me as a machine learning model that you're classifying if it's this end of, end of speak, end of the sentence, end of the—

The turn. The turn.

Yeah. You're doing predictions, the classifier. So cats and dogs are finished or not finished. And in my naive setup, I, I'm assuming I'm going to manage separate API keys for like Whisper, Claude, ElevenLabs. And if I were to use LiveKit, What does that look like?

Yep. So, uh, we're developer-centric and we want to, it's really important to us that the developer feels that they have control and flexibility that they're not, uh, you know, the alternative for them is to use like a closed source platform or they really don't have a lot of control. So you can, if you want, use your own keys, that's always an option. But we also manage that for you with the product that I work on at LiveKit called LiveKit Inference. So with LiveKit Inference, you use your LiveKit key, and with that key, you can access the leading voice models, model providers, and the models from those providers. The ultimate goal of LiveKit Inference is to create more consistent and faster latency for your voice model calls. We're building a layer in between the agent and the voice models that can do things like routing to the lowest latency server for a particular model provider. But that's kind of like future looking for, for now, the, the main, like the, one of the value propositions of LiveKit Inference is it's just easier to get started with building a voice agent because you need just one key and you can easily swap them out without having to like go to another provider and sign up for their service. It becomes especially useful if you're like more towards the higher usage scale, the enterprise scale, and you don't have to like go get a contract with have to go to 5 different model providers and negotiate 5 different contracts. You can instead just come to us and get one contract.

What's the value of negotiating a contract with, let's say, OpenAI? Do I get a lower rate?

If you— so I didn't make that point clear. Once you reach a certain amount of usage with, especially on the speech-to-text and text-to-speech, your listeners may be more familiar with LLM, so I'll start there. On the LLM side, if once you get to a certain amount of usage, you can go to the providers and say, I want a discount because I have so much usage. So the sales motion or the experience as a customer is slightly different. You basically reach a cap. You get to a certain amount of usage and you're forced to go on an enterprise contract. It's not just the 3 models in your voice agent stack. You also have backup providers, which is always a good idea in any sort of engineered system.

And you're saying if I just, I just use a Livekit inference and then you handle all those contracts, like, you know, dynamic, you just negotiate with us. Okay.

We are, we become your one vendor for all of your voice models. Oh, I'm sorry to interrupt. Uh, I just got excited about one other point around that, which is also there's no lock-in, right? If you go negotiate a like $100,000 spend on one particular TTS, you're like stuck with that TTS provider. And if another TTS provider comes out with something really cool, you're like, well, I can try that next year when my contract runs out, but I can't switch to it now because I've already committed the spend to this other provider.

Okay. Let's say I'm building this LiveKit agent with, I'm going to use Cursor or Cloud Code. Is this the right workflow? So what I can do is I can clone the repository and point the agent at your documents and then let it figure out the rest. Is that the right one?

You are like so many steps ahead of unfortunately what I see as the what is the most common way people are building with coding agents, which is they like, they don't, I shouldn't say the most common. I guess I bring it up that way, perhaps in a more negative spin because it's something I'm trying to solve and it bothers me that I haven't, I haven't fixed it. So the issue is that a lot of people when they're building with any sort of developer tool and not just LiveKit, they'll go to their coding agent and they'll, they'll never think to leave that coding agent or to give that coding agent more context. They'll say, build me a LiveKit voice agent, and the coding agent will use its training data to figure out how to build that agent. And that training data may be stale, or if there's something that's not, it just may overlook something in its training data for whatever reason. And so it'll make up a method for how to do something. And that, it's just very, it's hard to build. What that ends up happening is like the coding agent is sort of guiding the user on how to build it, but it's the blind leading the blind. And the right way to do it instead is what you said, which is like, if the project you're building with has an open source repo, go clone that repo and put it in your project. And when you open that project in Cloud Code or Cursor or whatever, you can tell it like, go look in the repo for how to do this and build this part of my agent. You can use the repo to help guide you. Or we have like an, a docs server, an MCP server for our docs. And you can also install that server in Cloud Code or Cursor or whatever. And you can tell the agent. Go look in the docs for how to do this. The, the most effective way I find for building agents has been to actually clone the repo, the open source repo, but like, you're just like so far ahead of like what I oftentimes see people of how they're building. I mean, you must see that too with other things you've built, right? Or, and talk and how other people talk about what they're building.

Okay. My sales training app is pretty narrow app and just me and one AI doing role play. But I'm curious, what could I add with Livekit?

Oh, for your sales training app? Well, the first thing that comes to mind is that you could put a face to your voice agent with the, the real-time avatars that are out there. I'm really excited about those. I think they're really neat. So basically the way it works is you have another model now in your pipeline where you have the speech-to-text, then you have your, like, conversation orchestration, you have your LLM, your text-to-speech, and then after the text-to-speech, you now have a real-time video generation model of an avatar of a face and shoulders, and that audio gets sent to that avatar. And then that avatar embodies— embodies isn't the right word, but then says what, you know, that audio that was passed to it. And I'm really excited about that. I think we're going to see a lot of growth in, in that over the next year because those models are getting so much better. And also people, people are surprising creatures. We just do like it. You would think that like it wouldn't make a difference if a person was talking to a voice agent or like a fake human.

Nice.

But actually they talk a lot more when they're talking to an avatar. Even when the avatar doesn't look totally real, they open up more and they say more compared to talking to a voice agent. Uh, but it's just like a waveform that's like bouncing on the screen. So I think we're gonna end up seeing a lot of growth in that field. And we have some of those, uh, in our in LiveKit's plugin ecosystem, the companies that come to mind are like Hedra and Tavis, Lemon Slice, the HeyGen. Those are all like, they basically render a video of a person in real time speaking the words that you give them.

I have a question for you on this one, because I just interviewed the CEO of a company called Delphi. So Delphi is a digital twin. He told me that the, they tried video chat. And video chat doesn't work because people think it's, it's kind of creepy. I wonder what's your take? Why in your case you're seeing it's working while in his case he's seeing it's not working.

We live in a nonlinear world now, an exponentially changing world, and the models have gotten a lot better. Also, I think for— it might be for their use case, like they're— these are particular people where the person talking to them is familiar with their mannerisms, like celebrities is my understanding, or coaches that have followings. Whereas if you saw my avatar speaking and you weren't familiar with my mannerisms already, you'd be like, oh, that must be just how that person talks.

And your, your avatars are real humans or are they more cartoonish characters?

So the companies we support that have plugins in the Livekit agents ecosystem are almost all like photorealistic ones. I could think of a couple where you can make them like cartoonish or characters. I think like the magic of this stuff is the photorealistic person that you're talking to. Um, when you, as soon as you make it a character, people look at it and they're like, oh yeah, I've seen that before. I've seen cartoons before. But when you make it photorealistic, I think it's a little more interesting. I don't know. I don't totally get people, but they seem to talk more even when the agent is, when the, when it's not, when it's uncanny.

Yeah. Do you tell them that, okay, you're talking with an AI agent that looks like a real human?

They must know. I mean, I'm not a developer of one of these products, so I don't know. I just, I only hear about like from the developers who are building with these. About like them talking about what I told you about where like the, there's this kind of ramp up in how much people interact with a computer. If you give 'em just like a little chat bubble, they don't talk that much. You give 'em a voice, a way for them to talk, then they, they interact more with it. And you give 'em like something for them to put their eyes on where they feel like they're talking to a person, even if it's not real, they talk even more. There's a, a new company that hasn't launched yet. That we're working with and their agents are really good. I mean, even the top tier ones like Tavis and HeyGen, which are more like the higher price, better models out there, or not higher price, but the more like, yeah, they're ones that aim to be on that side of the market of the higher price models. They are pretty, they're pretty remarkably good these days.

What do you think is gonna happen?

So I, I don't know much about entertainment, uh, the entertainment industry and what people like. So like, I, it's definitely possible that like in podcasts you could just have like the host, the host feels like their avatar is well trained on the sorts of questions they ask and their interests and will do a good job of representing them. And then the person, the guest may feel that their avatar is well versed on their, their knowledge. And somebody might find that to be an entertaining show. It reminds me of like NotebookLM where people like put in a bunch of content and it gets turned into, you know, a podcast that, and people find that as a useful way to consume that information, entertaining way to consume the information they uploaded. So I don't know about what's gonna happen on the, uh, in entertainment, but I do think that like humans are we definitely know real-time experiences feel very— or in real life experiences feel just so different. Like I talk to my coworkers every day in an experience like we are talking in right now where it's in a virtual room and I don't feel— I mean, I feel happy to see them, but it's not like when I get to be around them, I'm like, oh, this is so cool. I get to spend some time with them in real person. Like I feel something very different and I just feel like there, we'll never be able to replace that experience that people will never replace talking to each other because it just feels different. It's, it's what we've evolved for.

Okay. What else can I do? I'm gonna put a video avatar now to, for my sales agent. Anything else I can do?

I think there are so many ways you can make that agent more effective at coaching the, the user. So, I think a lot around customization and personalization for that particular user where you can really come to learn that user and things like how they speak. So like you could even change your turn detection settings around how they speak, but also you could develop such a good understanding of where that person is in their journey as a, and whatever you're training them on to like develop really good material that gets generated in time for, in real time for that person where you have like, you're, I mean, very similar to like if you, when you were teaching me how to do something, you would see me go off the rails in similar ways and like you could have your You could teach the model like, oh, that person's going off the rails in this way again. And I'm very specifically, I have a specific way of wanting them to go back on target. So I think the next horizon is like user customization and there's just so much neat stuff you could do between our agents framework and the capabilities of these models to build that sort of user personalization.

What about having a group training session or what else I can imagine? Let the coach, the AI coach critique my body language. Are those possible?

Yeah, I've seen those. So people are, I mean, you just have another machine learning model in the, in the mix, right? A friend of mine built, was building a company that was doing that where he was, I'm sure, I would've, I'm sure failed his, his app's coaching test now, but it was about like how you show up to the meeting and how confident you seem and like how many, how, how well you speak without disfluencies and that sort of thing. That like, those are all sorts of neat things that you can build. I mean, you have such an interest in, in depth of experience in machine learning models. So I could see the direction you're going with this without having to worry about the boilerplate stuff that everyone building a voice agent has to figure out, like without having to worry about how to make turn detection fast and how to make it a low latency experience. Instead, it's just something where you can, like, that's our, that's our main value proposition. Our North Star is like, take care of all the boilerplate stuff for developers so they can focus on their own custom app.

I get to do the fun stuff.

Yeah. Yeah. I get to do the fun stuff.

Cool. Well, what are the most interesting use cases that you've seen people building with LiveGoods?

I brought up Portola a moment ago and they just come to mind again because they're, they're an interesting team that has built this great product. And part of what's so interesting about that use case is how deep they've gone on personalization. Like how deep they've gone on really understanding their users and trying to understand what they're— the user is feeling about something and making it so that their, their voice agent that they're talking to, the toll-in, can have a really fulfilling conversation with the user. And that's like one of the interesting ones that comes to mind. I think the other interesting one that comes to mind is more from a business perspective of just like how much growth we're seeing in some of these. So like, and there's a lot of use of, a lot of growth in voice agents for healthcare, for doing things like calling insurance companies to get prior authorization. And, uh, I mean, part of what makes that interesting is I'll be talking to one of our users about some problem they're having and I'm like, well, how many users is this impacting? And they're like, oh, it's impacting like 1% of our calls. So that's like 100,000 minutes yesterday. And I'm like, what? How are you doing that much traffic? Like, how is this stuff?, you know, doing so well. And that's part of what makes it so interesting is like, I feel like I'm living in the future. It's just not evenly distributed. And that like, we're seeing incredible growth that in some of these, from some of our companies in some fields, but they haven't taken over those fields yet. Those fields are still in the process of transforming, but they're changing quickly and the growth is fast. Those are two that come to mind.

If you were to start a voice agent company or video agent, What would you build?

Uh, my last startup, we were trying to build something in the voice agent space and we just wanted it to be generally useful. And it wasn't something that I felt like I had a strong, like what we were building wasn't something where I deeply felt like this needs to exist. I just thought like, oh, this generally seems like it'd be useful and there's a value proposition to it. Let's go out and try to sell it to other businesses. And that was, and that company failed. And I compare that to my first company. My first company was a sensor that I developed in grad school. And to me, very deeply, I felt this thing needs to exist. And it was weird and different. And like a lot of people told me it wasn't a good idea, but I was like so set on this being like my thing. And I think that's why that company succeeded. And I've been, I think that is what would guide me for the next voice agent is I wouldn't want to build something where I'm like, oh, there's clearly a market for that and an ROI for it. It'd have to be something that is like weird and different and like something that I felt would feel like this needs to exist because it's just like, it speaks to me in some way. And I, I don't know what that is, but that's my guiding principle around it. Is it like, it would have to be something that just seemed weird. And my coworker Jesse put together like a, what am I looking for? Like a starter kit for a hackathon we did with Google and YC. And he ended it saying like, okay, there you go. That's the starter kit. I hope you go build something weird. And I just love that he said that, cuz that's exactly kind of how I think about it. It's like, I'd wanna do something just different that, that I really felt like deeply about that needed to exist.

Do you feel it's mostly replacing old systems versus like, you know, creating entirely new experiences?

I suspect that I'm probably, I, I don't have a This is like anecdotal as I search through my mind on how to answer that question, but I suspect a lot of what the, the work— so most of voice agents in production now are telephone-based voice agents, and they're doing things like the healthcare example I gave, or customer support, or lead qualification. And in those cases, they are replacing workflows that humans are doing, but I want to be careful of saying replacing because like one of my customers tells the story about like their, the call center that they're working with. They have 8 people working there and there's one job they don't like doing and there's another job they like doing a lot more. They like hunting for new business. They don't like supporting, doing the customer support side. And so more of the people get to do the hunting for more business now and less of the customer support side because they're augmenting their staff with voice agents. So I, I want to be careful to say that we're just out there replacing humans because I don't think that's actually what's happening out there. Anyway, where was, where was I? So I think there's some amount that's replacing existing workflows, but I also think there's a lot of creative new and different use cases, such as the, the example I gave with Tolands, uh, or Tolo. Like before it never existed that you could have like this companion that you could talk to about your, your stress around your studying for the nursing exams or whatever. Um, and there's, there's other new and creative ones that, I mean, those are my favorite, just dovetailing off of what we were just talking about. The things that people are building where I'm like, oh, that's weird. I like it. You know, that's because it's something new that I've never seen before. Those are the ones I like.

Hmm. The existing ones, like the first thing that comes to your mind is the IVR systems, the ones that you, there's a long menu that you have to go calling the insurance company. It's still everywhere, right? Do you think in the future we can skip that whole thing and just talk with an AI agent with a natural conversation?

Yeah, I think that future is here. It's just unevenly distributed. So I think I love when I encounter voice agents in the wild as a consumer and get to actually see them. And I think it's happened to me like just once or twice, not all that often, but I think we're going to see it more. What's that?

I have never run into one. Which one did you talk to?

I think it was Vuori that had one. I had to call customer support. They're a clothing brand, but I'm not sure. This was before the holidays. And so hopefully it's still up. I don't think— I wish I came up with an example off the top of my head that I've encountered as a consumer that's one of our customers, but it hasn't happened.

I would say hallucination is still a problem, right? For more high-stake companies to completely adopt it. Like the insurance company, healthcare.

Yeah, I, I think hallucinations are still a risk, but we're seeing so much growth in these regulated industries. I think because the way you— we're getting better at contain or preventing hallucinations. And the part of the way you do that is you have shorter prompts, uh, with less that the agent can lose and more you can, you more tightly control that agent. And, and there's lots of, like, the models are getting better at this and the frameworks like ours are getting better at this, where in our framework you can have lots of different sub-agents that are more focused. I suppose the, the point I'm trying to make is that the, the AI industry is getting better at preventing hallucinations by both improving the foundation, foundational models, as well as the frameworks that people are using to build their apps are building better controls into giving developers more tooling to control the agent. And to use the Livekit agent example in particular, if I'm gonna have a 30-minute call with a onboarding call with a new user for something that's higher stakes, I'm not gonna use one giant prompt because that's gonna likely lead to the agent hallucinating something because it, it loses some of that when you give it really big prompts. But Instead, LiveKit gives you the ability to break your agent into lots of small agents that get passed from one— where the user gets passed seamlessly, unwittingly from one agent to the next, so that they— each agent is more focused and less likely to hallucinate because it only has like these 3 things it's allowed to do. And there's lots of guardrails written around that one little agent around what it's allowed to do.

How do developers actually find you? Like what, what, what does a typical customer journey look like?

Uh, a lot of word of mouth. That's like our main growth driver is word of mouth, uh, which is really great to see. And then they also find us through like our, our channels where we post cool demos. So we're on YouTube and X and LinkedIn where we're posting cool demos and they can also come to our website when they're searching voice agents. They, they, they'll find our website livekit.io.

Cool. What about other comparing with other providers like Retail, I think they're more a GUI kind of user interface style, right? You kind of, you put in your prompts, click around, and then set up two calls. Livekit is completely coding. What do you think is the difference and why?

Yeah, I think that's a good way of, that's how I generally describe the voice agent landscape to people who are new to the industry. Like there, you can create two different groups. There's the closed source voice agent platforms and there's the open source voice agent platforms. The open source ones are Livekit and Pipecat. And then the, the closed source ones are RETAIL, VAPI, others. And generally that's like the first important distinction when you're making a decision about which one to build with is which one like shows you what's actually, which one gives you all the code so that you can see what's happening and gives you, I mean, I, I framed that in such a way that wasn't very objective, but like, which I would want more control. I would wanna know which, what exactly is my agent doing and how can I change that agent? Hmm.

And you've raised close to $100 million, right? $80-some million and you are still open source and anyone can fork your repository. What would you say is your moat?

So we just raised our Series C and we are a unicorn as of just recently.

Yeah. Congratulations.

Thank you. Thank you. Very exciting. I think our moat is that we do the full stack. So we have the WebRTC layer and we're really good at WebRTC infrastructure. And it turns out that's really important, like how fast you can move audio around the internet in a very reliable way is really important. And then we also have all this tooling for people to build their agents. And that's mostly what we've been talking about today, the open source framework for how to build agents. But there's another layer above that, which is the running your agent so people can We host people's agents on the cloud for them. And it's really fast to host your agent on us. It's convenient for the developer, but also really fast. And that's one of the ways we make money is your agent is fast if you're hosting on us. And we, we charge money for doing that, just like any sort of cloud would. And so those are like, and then we have the observe part, which is the other part of our platform where we show people what's, what their agent did and how to test their agent. So I think it's owning that the reason I'm bullish about LiveKit and our moat is like, our first moat is just quality. I should have started there. Like, all these companies are migrating their voice agent workload to us, and enterprises are picking us because the quality is there. That's the best platform for building a voice agent. And then the second one is that we own the entire stack.

What if OpenAI or Google, if they decide to just fork your repository and build native voice agent infrastructure into their models, right? And the speech-to-speech gets good enough, and latency something, you know, gets better. Do you think you still have that moat?

Uh, yeah, we understand the problem more deeply and we already have the, the community's trust for it. And I, I don't know, how do I, uh, we work with those companies, so I want to like acknowledge what great products I think they have, but I think that a lot, there's a lot of value that comes from focusing on one company, focusing on one particular thing, and really doing a great job of it.

AI is changing the gap between senior and junior. Some, some, some people said that it's widening and some people might be saying that it's actually shrinking. What are you seeing at LiveKit?

Is that gap shrinking here?

Senior and junior engineers. They used to have years of a gap.

Yeah, I see. So, the first thing I see is that the toothpaste isn't going back in the tube. Like all the engineers I know. Are using coding agents and that's not gonna change. We're not, we're not gonna revert like to people doing everything by hand. And I think that, I don't know, I, my first reaction to that is to think of my own coding projects where like, if I would, I don't under— let me say it this way. I don't understand how people are running like 6 Claude Code sessions simultaneously from their phone in remote control mode and building something useful at the end of the day that actually is actually works as you expect, because I have to babysit my coding agent still. Like I have an idea, I come in with an idea of the architecture that I want from my experiences of building things. I'm, I know what is good architecture and I can keep the agent on the rails. And I feel like that's something that requires human judgment and that's not going to go away. And so, and that judgment often comes from training and experience. And I think you can have those things as a junior engineer. Maybe you don't have as much experience, but you can have the training and kind of the intuition. And I think you have that as senior engineers too. I think we're just kind of going through an interesting stage right now in software development where like you go on Hacker News and like 10% of the posts are fear, uncertainty, and doubt around coding agents, people panicking about it. And I think we're going to get through this and realize that like engineering is always going to exist. We just have some better tooling now.

Different roles to play. Play.

Yeah, different, yeah, different roles that, that, but these, these things are still tools and you still need engineers to know how to build things.

Let's say if one engineer with AI can do what 5 used to be able to do, how does that change, uh, how you think about team size and career growth at your company?

Uh, I should be, uh, I, I'm not on the hiring side of the company, so I actually don't know what our take is on these things. But I do know that just from looking around the company that we hire people from various experience levels. I just, I think that, uh, I think that the, these tools can help us move faster, but they still need people to work the tool. Like the tool doesn't make the human go away, especially in this case. Like there was the, I think before going on earlier this morning, I was on Hacker News and I saw a post. About like Amazon having an all-hands meeting about people needing to, about new policies for using coding agents because they keep having outages from people shipping code that coding agents wrote. And that the thing that needs to be the, I think what the policy they were talking about was that they're, that like every, every code has to be, every PR has to be reviewed by a human. Sounds sane enough, but like, you know, it's like, I still think that like there's going to be a place for humans designing these systems and making sure that they're built right. I just think it's, this is just the latest in tooling around it. But you also need someone to design it, right? Like you can't, someone still has to write the spec and that person who's writing the spec has to understand how the code works.

If there's one thing you can teach me or teach anyone who's trying to build a company, what would it be?

Yeah, I think the weird thing I was saying earlier is definitely true that, uh, cause I, that seems to be a, a through line with a lot of the conversations I have with other founders that are early in their careers. Like, I'm building this cuz it seems like there's a big TAM, but it's like, but do you, do you care about that? The other one, this is maybe on the surface, the opposite advice is like growth just solves everything. It's all just gets, if you can go out and get users that, uh, get paying users, then all the other pieces can fall into place around raising money and getting, finding co-founders, getting employees, all the other pieces work if you can figure out how to grow your product.

Make your product not so weird. It will happen, right? Once the majority of the population think, you know, this is a needed product, then you're not weird anymore.

Yeah. Yeah. And I guess for that second advice, I mean, I think that maybe reflects how I'm thinking about things too, in that like, I don't want to do B2B sales ever again in my career. I think in my first startup, I was, I never got out of the sales role and I never wanted to ever be in a sales role. And so if you really are set on wanting to do a startup, the fast, the best, the tried and true path to like actually getting to do it is to go do sales and get customers and sell your product. And everything else will fall into place. I'm now at a stage where I will have lived my life, the rest of my life well if I never sell something ever again. Not because I have a disrespect for the sales craft. Like I think it's just, it's that I don't like doing it. And, but I have a tremendous respect for how difficult sales is because of all my years doing it. And for the people who are good, I have a lot of respect for them, but I just don't want to do it anymore. So like I'm looking, You know, that's why I'm espousing this, this idea of go do something weird that you don't have to sell. Like, it's just weird enough that like, for whatever reason it catches.

It will sell itself because you are weird enough.

Okay.

I take that as the key advice from this episode. Yeah.

And I think there's probably a reason I'm working at a company now and not building my own startup, right? This is my North Star, building something weird and not building value. It's probably best that I stay employed. My happy place is being deeply absorbed in a problem and not context switching or talking to people all day that about the same thing I've already said to them, said to people 100 times before. That was the part that crushed me at my last company doing sales is like, I've had this com— it was like watching the same movie. Every sales presentation was watching the same movie for the 10,000th time. Like, here's the part where I say this. I felt like I was my own little puppet. It's like, here's the part where I say this about our value proposition. Here's the part where I say this about this objection that the customer, the prospect is having.

I was reading a book about like, uh, written by someone, uh, who's doing hostage negotiation and he was talking about, it's called Never Split a Difference. I don't know if you've heard about it.

Yeah, Chris Voss, I think, right? Yeah. Very famous negotiation coach.

Right. Sometimes I'm thinking, should I treat sales calls as hostage negotiation so it's more fun and then, you know, trying to read the reaction of the other party and try to make a deal, but it's very hard. It's so hard.

Yeah, I think that's probably what's— I mean, the salespeople I've worked with have been so good. I think that they are fascinated by people in the same way that I'm fascinated by technical challenges. Like a technical challenge, I want to understand all about it and how you can make it work. And I think they're instead fascinated by like Who is this person I'm sitting across from and how do I bring them along in the journey I'm trying to bring them, which is just, I like people, but that's not my fascination.

I see. Yeah, that makes sense. It's, it's like different machine you have to understand and read and analyze.

Yeah.

Is there a darkest day that you had building this or being part of Livekit?

Uh, not, not Livekit. I just told, I just confessed my darkest days from my first startup doing sales and just being done with, done with sales, but at Livekit, it's all been good.

Okay. Okay. Life kit is good. Good life then.

Yeah.

Okay.

Yeah. Yeah.

Yeah. When, when, when you feel vulnerable, who do you talk to? Someone? Like, how do you solve it? Who do you talk to?

Adrian. I talked to the CTO of my last company, who was the— of my first company, who was the co-founder of my second company and is now the, the lead engineer on, on our inference product. At Livekit. Okay.

How does he help you? He sounds like another technical mind.

I think he just, I mean, we've been working together for 15 years. He knows the sorts of things that get under my skin and can kind of talk me down from it. Uh, when it comes to work-related stuff, I mean, I talk to my wife too about work-related stuff as well in sort of a different way where it's less in the weeds generally, but it's more, um, and more about kind of the higher level work stuff that's getting under my skin, but I feel like Adrian has been always there looking out for me. Running your, your own business consumes you. It has to because like it only survives if you make it survive. Yeah.

Yeah.

It's not like, it's not like one of those things where one of those choices you make, other people make choices in life where they work, can work a job and then they can leave it at the job. That's not the, the path that you've chosen. But it's also more interesting, like for those hours that you have to be at work, that you're working, it's interesting. It's your journey, right? That I think that's what's so fun about it is it's your own journey and you're making progress on your own journey. It's your own story. Whereas if you're building widgets for whatever company, it's not your journey. There's no, there's no sense of the human journey.

Do you feel better now? Like now that you're working for a company, you have better lines with your, with your work?

No, because LiveKit is a rocket ship. I dream about LiveKit at night, just like you were talking about. And yeah, it's just like, I'm, I was telling our co-founder David this about the other day is like, I'm just trying to not fall off the rocket. There's just so much happening. It's cuz we're growing so fast and doing so many things.

How big is it?

It's all-consuming. It is 100 people, around 100.

Okay. And you have conviction about it?

Oh yeah. I've never been a part of something like LiveKit. I don't think I added this to my list of reasons why I'm bullish about LiveKit, but another reason I'm bullish about it is like, people are going to be talking to computers. They're going to be talking to robots and they're going to be talking to websites and we power that. This is the future.

Okay. You are at the start of it. So, you could retire and never do sales again.

Yeah, hopefully, hopefully. Then I will have lived a good life. Not because like, I want to follow that up with, for my friends in sales that are listening, that not because I don't respect their craft, their craft is just too hard for me. I just can't do it.

I mean, you are a parent with two kids, same age as mine, and what are you teaching your kids? But you know, today with, with this AI hype, everything going on and, and why?

I'm hoping that they find the things that they find interesting and weird to them and that they pursue those. I hope they have the opportunities to pursue those things. I don't like that, that path that you described earlier. I came from, uh, where I went to high school is very similar as everyone was talking about trying to get into the best school. And I think that's the wrong framing. It should be like, what is— get into the right school for you, take the right path for you. I hope that that's more of what the dialogue is for our kids as they start having to figure out what their professional paths are.

I'm just copying people around me and I thought that's the right path.

Oh, isn't that so true? Like, that's what we do is like, I used to think about that in my first startup when we were working out of San Francisco for a while and all anyone seemed to value was growth and money. And I was like, maybe we can value other things. We could choose to value other things, but everyone around you is valuing that everyone else is up to us. Obsessing over their growth and stuff. It, yeah, it takes a lot of guts to value other things.

That's the thing. How do you teach your child to teach your children to have the guts to do what they truly want to do?

You model it like you're doing now. You're modeling it with pursuing your own path. Not, not doing what other people, what seems like the right, what other people think you should be doing.

That's why I'm really poor doing a podcast. Maybe, maybe, maybe, maybe I can pitch to your CEO, consider sponsor my, my, my, my episode.

Yeah.

Do you feel there's some, some type of human judgment that machines will not be able to replace?

Yes, absolutely. Because judgment is actually an emotional thing. You hear about this with the experiments with people that have— there's— how does that work again? There's some people that I think they're stroke victims that some of the emotional centers of their brain get knocked out and they can't make decisions because every decision they don't, they can't make a decision whether or not to pour cold water or hot water into their cup. They can't make the decision whether to like eat an apple or an orange that because every decision is emotional and emotion is embodied. It's how we feel about things. It's not an intellectual thing. It's, it's a feeling. And, and I don't think At least the way we're building intelligence and machines now, it's not any, it doesn't in any way approximate judgment. It, it, it, there is no feeling. I think you would have to, I'm not sure I've thought about that before. How would you build feeling into an LL? But it's, it's just not, it's not how we're building them. You need judgment is emotion comes from emotion. And another concrete way of saying that is that when people, you present people with, Sam Harris did a famous study of this where, not famous study, the famous Sam Harris did a study of this where They took fMRI images of people's brains and they present them with images of things that are disgusting, like a food they don't like, or they present them with images or statements of beliefs that they don't actually believe. And the same parts of their brain would light up. Like disbelief is a disgust. If you tell me something I don't believe, like that, you know, I live on top of a mountain. I'm like, yeah, I don't, I live in a valley. I live in the Bay Area in a flat part. I would like, I don't believe that. I feel disgust around that. And if you close your eyes and present yourself with some ideas you don't believe, you'll see a similar disgust, however small, emerge. I don't know. What do you, what do you think around that? Do you think that machines will ever have judgment?

I feel like it's like, imagine putting a soul into the machine. That's what's missing. The representation is emotion. I think you, you, we can feel, we can we, we describe these, these emotions, but it's a set of data that's never collected and cannot be collected because sometimes we cannot describe it, right? We have our language, but language has a tremendous limitation. Um, that's why, you know, when you, when you— but this is really a big tangent— when I, when I look at like, you know, Zen Buddhism's, you know, scripts, I have no idea what they're talking about. And then they're all saying like, you know, I don't know what they're doing, but they tell you stories and then you ask them, what did you learn from learning Buddhism for so many years, for 50 years? They will tell you nothing. That nothing has a lot of things that cannot be described, and that's not captured in data, and that will not go to LLM. That's my read. That's my interpretation.

Oh, that's so interesting. Yeah, that is something I need more time to think about and digest. I love that idea of what you're getting at there. Idea.

Me too. Me too. I'm done, but I still have the rapid-fire question. Let's go through it because it will still add some value and then it will just 3 seconds each.

Thank you so much.

Yeah. Thank you so much as well. Let's go. Let's go through the rapid-fire questions.

Okay.

One, is Livekit voice agent framework open source?

Yes.

Is Livekit's focus video conferencing or voice agents?

Voice agents.

Okay. What is the easiest way to get started with building a Livekit voice agent?

Go to our— log into Livekit and you can use our— I did not say this in a rapid-fire way. Try again. Log into Livekit, create an account on Livekit and start building a voice agent. Awesome. Okay.

I have a traditional closing question. If you could recommend anything, well, if you could recommend a book to, let's say, yourself when you were a 20-year-old, when you just graduated from college, yourself today after building 2 startups, and maybe yourself 20 years from now, what would they be? No pressure on all 3, just 1 or 2 is great.

A book for my younger self, I've enjoyed coding so much, I didn't start coding until grad school. So 20 years ago, I would've recommend a, a programming book for myself. And then my current book that I recommend for myself is the, what my 20-year-old self would recommend for me, which is to go read Homer's Odyssey again, read the classics again. 'Cause that's what I was really into back then. And I feel like I've swayed so much towards the intellectual technical interests that I've lost some touch with some of the more like emotional artistic interests of mine. And then 20 years from now, I do not know how to recommend a book for myself in 20 years.

Yeah, we'll see. We'll exchange ideas along the way.

Yes. Yeah, that sounds good.

Okay. Thank you, Tom. Like, you know, I know this is such a wonderful conversation. I feel like I learned from you everything I need to know to start building a voice agent. It sounds like LiveKit is the solution. Also, thanks for your generosity of your time and sharing your deep thoughts and wisdom with me as well.

Yes. Well, thank you for sharing your deep thoughts and wisdom with me too. This was, this was wonderful. Thank you.