In this interview, host Erin (Arin) talks with Tsahi Levent-Levi about OpenAI’s recent blog post detailing how they built a low‑latency, scalable voice AI system using WebRTC. Tsahi highlights the significance of OpenAI’s public endorsement of WebRTC, praising the company’s transparency in explaining the technical decisions behind their architecture. He walks through key design choices: avoiding TURN servers, using a single UDP port, placing a dedicated edge router behind a stateless load balancer, and leveraging ICE ufrag/SDP fields to route packets without additional infrastructure. The discussion contrasts this approach with the more common SFU‑based pipelines used by services like LiveKit or Daily, explaining why a direct client‑to‑edge gateway makes sense for voice AI, akin to cloud‑gaming architectures. Tsahi also describes operational optimizations such as running the UDP pipeline in user space, spawning multiple single‑threaded router processes per CPU core, and pinning them to specific CPUs for cache locality. The conversation touches on future transport possibilities, noting that while WebRTC is likely to remain the primary protocol for now, OpenAI may explore alternatives like WebTransport if they offer lower latency. Finally, Tsahi mentions an upcoming LinkedIn “Build with WebRTC” session with Justin Uberti, inviting listeners to follow his LinkedIn for details.
Chapters
Thanks for joining me today, Tsahi Levent-Levi. OpenAI published a blog post this week on how they deliver low-latency voice AI at scale, and it's definitely worth a read. What's your quick takeaway from this?
Well, the first thing, it's great because up until now you've seen voices saying, well, maybe we don't need WebRTC, maybe it's a WebSocket, maybe it's MOC, WebTransport, or something new that we don't know. But now if someone like OpenAI comes and says, well, it's WebRTC, it's at scale, it's not beta, it's not someone playing with it. And this is how we had our own thought processes about how we do that and how we get that into production. So that's great for WebRTC and Voice AI.
That's a big endorsement of WebRTC for sure. Yes.
The second thing is that I don't recall up until this post in the recent 2 or 3 years any company or vendor that took the time to not only explain what he is doing and how great he is, but rather go and say technically why he made the decisions that he made. And that was an eye-opener. It was really interesting to see. So kudos to OpenAI for that. Yeah. Then I think the most interesting things in there is that it goes almost against the best practices that we've seen elsewhere in the industry. And it makes sense. It makes sense specifically, specifically for voice AI.
Right, exactly.
The good thing, or the interesting thing, is that I've already had the Build with WebRTC session on LinkedIn that is scheduled with Justin Uberti, and now I actually have something to talk to him about that is publicly available, and that would be great.
Yeah, yeah, Justin, Justin got a shout out in the blog post from, from the OpenAI team, you know, for his contributions to WebRTC. MTC, as well as Sean Dubois and the Mayan project. So OpenAI disclosed, you know, using that and basing this in Go. So, so also some nice, you know, I mean, this would be a case where you could say OpenAI is living up to their name in terms of being transparent about the decisions they're making, as well as giving some nice kudos to others who worked on it.
So yes, I think on the detail level, the interesting things, at least for me, were first of all, well, no Turing service. It makes sense in many ways, uh, just in coming from Google, Google Meet not using TURN servers, you know, it's like, okay, I can see that happening. It makes sense for their use case as well because it's always client to server and the server is reachable, it's on the public internet. Um, then there is a decision there to use a single port, so there's a single IP port, there is no more ports, it's only one for UDP. They don't say a lot about what they're doing with TCP, but TCP is always a single port. Um, and that's like the, the entrance into their network. Now they write everything of how they did that and why, and it goes— I think the money part is in the end when what they say is, well, we took the complex part, we decided not to put it on the client, not to put it in all the backend infrastructure, just to put it on the load— on the router there that is behind the load balancer. That one piece, and this is the only thing that it does. It does nothing else. So they took the pieces that is hard, the piece that is really hard when they needed to scale, placed it in a single location, made sure that it is single purpose, and then scaled that. So that was really interesting. Just as an explainer, they start by saying, well, all of the voice AI stuff out there today, or most of it, is based on SFUs. You go to LiveKit, you go to PipeCat with Daily, the default is you're going to use our SFU and our media server infrastructure, and now we're going to connect that into Voice AI. It works great if you have a need for that services and if you want that. Gustavo Garcia went and explained why that is not the right way to go from his point of view in his recent post, which makes a lot of sense. It's like another moving part that you don't really need. And VoiceAI simply said, we're not doing that. We just have something that, you know, intercepts the WebRTC stuff from the browser and then translate it to whatever it is that we need internally. And they don't explain what, what that is. It might be WebSocket, it might be something else, we don't know. And here we don't really care, right? Okay, so they connect the client almost directly to the edge that deals with WebRTC. And from there, they— it's a kind of a gateway or interoperability gateway internally to the other things that they have. Interworking, not interoperability. So that's, that's the main piece. Once they've done that, they start looking at, you know, unraveling the thing. So they need a load balancer, they want everything to be stateless. So they can send the packets wherever and that would get resolved and go to the correct router. The way to do that for them is to use the uFrog in the ICE, in the SDP for the ICE connectivity. And it's a nice— I looked at it and it's a really nice approach because it does two things. It says, well, let's see what the protocol and the behavior has there today and reuse that for something that wasn't really meant to do. On the get-go at least, or maybe wasn't meant to do. And then from the point of view of the standard, if you go back to SDP and ICE and all of the things that are happening there, someone at some point said, well, we might need something to identify that for some purpose that we don't know yet. And it was there, and now somebody uses that. It's probably used for other things as well, not only that, and most likely You really don't need to use it most of the time because you're doing one nice negotiation and everything is related to that. But here it's just want to understand where the packet came from when there are different clients on the other side and you don't have a way to differentiate between them. So doing it this way becomes a really nice hack or an approach that is untraditional, but it works. So I like that, that part as well.
And there's so much in this blog post, I'm going to have to go back and reread it a few times myself. We'll definitely include the link with this on YouTube if you haven't seen it. But I want to go back to that choice of not using an SFU because that is perhaps the most like, you know, counter— it's not counterintuitive in their use case, but counterintuitive to common, you know, advice about building WebRTC applications, right? It's important, I think, to note that this is a— this makes sense in their use case, but that doesn't change advice for most WebRTC applications and even some that involve voice AI, right? Like if you have a standard multi-party meeting tool, you're still going to use an SFU. You're not going to go and try and duplicate what OpenAI has done here. And if in that meeting tool you're going to have, say, a voice assistant AI joining you, you're still going to have them join through an SFU, just like that option 1 that was described in the blog.
That's true. That said, the way I see it, their use case, the direct use case that they have, is more akin to cloud gaming in you don't think about having an SFU, you just connect the gaming server to the user. You don't pass it for an SFU.
Exactly.
That, that would be how I would look at it. For me, the important— the interesting piece was the way they did turn, or the way they said, you know what, we're big enough, we need to do something differently.
Yeah.
And I guess I'm going to bug Justin about that if I can, and if he will let me, because something, you know, it starts by saying we love Docker, we use Docker, we need more ports, Docker doesn't do that security, blah blah blah. Now we want a single port, this is how we do it instead. Now if that was the, you know, the option that everyone chooses, then you would see it with the Cloudflare return server, you would see it with the Twilio one, you would see it with, with what Zoom is doing proprietary. And you don't. So it's like, it means that there are other options that work probably just as well, but they are different. And this approach works just as well, but it is different. So that is something that I know, I just want to touch that one. Yeah, another interesting, very interesting design choice that they've made is that the router runs on a Linux machine. The whole UDP pipeline runs in user mode, not in kernel space. So they decided not to go and optimize that, although they're saying, you know, scale and size and, and stuff. And they explain why, which I found really interesting and really efficient. They said the following: they didn't need it, it was running well enough in user space. The only two optimizations that they have done is to say, first of all, well, we're going to put more than a single router server on the same machine. So if I've got a multi-threaded machine, which all machines are today, I can run a single-threaded process to handle UDP packets and route them, but now stand up multiple of them so I can deal with multiple threads and multiple cores and stuff like that on the machine. So they all siphon data from that single port, single socket that comes in— single, sorry, single port. Each one of them will have a different socket on the machine. but each can— my— each of them might even receive packets from the same user on different processes on the same machine. It still works because it routes it to something else that is going to manage that, so they don't care. So that's, you know, that's an optimization. It is easy to do, and it's an OS-level thing. It's not code. And the second thing was, well, you know what, we're going to make sure that this process runs on the same CPU all the time, so we enjoy you know, the caching and thread locality and stuff like that. So this is like ops that I would do because it just makes sense. And it made sense 10 and 20 years ago, it makes sense today as well. And again, you don't read a lot of these decisions in other blog posts out there. So seeing that after so many years, that was refreshing and fun.
Let's close on one other topic around this, because I want to go back to something that you mentioned in the beginning with this being, you know, like a nice endorsement of WebRTC. Because last month in our conversation, we talked about WebTransport, and one of the use cases that we talked about that could be good for WebTransport is like AI-to-client communications, voice AI scenarios. But that's not part of what OpenAI is doing. They don't mention the word transport in the blog post. Do you think that's something they would move to in the future, or is this a kind of a permanent endorsement, let's say, for using WebRTC? In that scenario?
Nothing is permanent. I started by doing H.323, right? Thought it was stupid. I still think it's stupid, and I love H.323. Moved to doing 3G 324M, which died off because circuit switching moved to IP. And WebRTC will stay with us for many years to come, but it's— nothing is permanent, right? This post was technical, and it specifically focused on what they're doing with WebRTC. I wouldn't expect— like, they didn't say why not WebSockets. They didn't even mention WebSockets. So why would they mention WebTransport? It doesn't work there. Sure, at some point they probably will investigate that if they're not doing that already. Yeah, because again, if you look— if you read between the lines, what they're looking for is something that connects as quickly as possible scales well and has low latency. So my belief is that anything with the characteristics of 20 or 30 milliseconds less in any of these parameters will be a good enough, um, motive to go and try it out for them, right? Especially, you know, with Justin coming from Google and looking at these types of things at that kind of scale. Um, So the answer to that is nobody knows, right? But nobody has done anything on WebTransport that is really voice AI. It's something other than a cute demo online.
It may be coming, as you said. Yeah, yeah. Nothing is permanent, certainly with, you know, I think they said that 900 million weekly active users in the stack and OpenAI, that's enough users that if you wanted to support WebRTC and WebTransport, you certainly could. And and certainly still having people using WebSockets for more text-based interchange instead of going with the Realtime API 2.0, perhaps to reduce costs or for other reasons depending on the use case. With that user base, there's going to be, I'm sure, plenty of options available, but also because of the size of that user base, that's what makes it so interesting how much they shared in this blog post. Again, I encourage everybody to go check that out. So Tsahi, you've mentioned upcoming conversation with Justin Uberti, and you're going to have plenty to discuss with him now because of this blog post. When is that conversation going to be, or how can we find out about it, how we can join it?
So this is upcoming. We've decided we're doing it in May but don't have a final date yet. The easiest way would be to just look at LinkedIn for my account and see there for the events page. It should be there, I guess, within a week or so from now.
Perfect. Excellent. All right. Well, as always, thank you for sharing your insights with us today, Tsahi. To learn more about Tsahi's work, check him out at bloggeek.me and follow him on LinkedIn to find out about that upcoming event and all of his writings. To learn more about future episodes of WebRTC Live, check us out at WebRTC Ventures and give us a follow on LinkedIn too. Thank you, Tsahi.
