High CPU Load Is Silently Destroying Calls

Thanks for joining me today, Tsahi. Call quality is always a big topic in building WebRTC applications, and oftentimes when something goes wrong in a video call, we tend to blame the network first, or maybe the user's Wi-Fi. But today you're going to explain why we should pay attention to CPU usage also. So tell us why that is.

Yeah, so thank you, Aaron. I think if you go to almost any company and ask them, okay, What kind of data are you collecting for your crawl in WebRTC? The answer will be either nothing, we're not collecting, we're running blind. These are great because, you know, we can send them stuff. Then the other one would be we collect specific metrics that give us everything that we want. And then what they collect is going to be jitter, packet loss, and oh, round trip time. Almost that's it. Maybe bitrate or, you know, some other esoteric thing like frame rate. Uh, which is nice, but it's not enough. And the third part is all those that collect everything and trying to make do or make sense out of that information. And the thing is that the category in the middle, the one that are doing the network, they are capable probably of finding something like 90% of the issues that relate to media quality. Not all the issues about media, but most of the issues about media you can see there. The problem is, well, you know that something is broken, but you don't know why, right? And that's why we need all of the other metrics. And a lot of them are going to revolve around CPU and not around network, because once you get to the point that you know that there is a problem, what your goal here is to find the root cause. And root causes are to be found by, well, figuring out from the data what the— where the problem stems from. So I can have an issue that shows up as packet loss, and that issue might be because there is packet loss on the network, obvious, okay? But it might also be due to CPU load on the sender side that causes throttling that makes so that some of the packets that he's sending don't go even into the network. His network drive simply throws that away because it's UDP and nobody needs that. Okay, so there are a lot of these types of issues that you say, well, usually the classic approach would be packet loss equals network problem. Not all the time. And what we want to do is to figure out these times and try to distinguish what the root cause of the problem is versus just saying, well, it's crappy video or crappy audio because it's bad. That's it. We know that it's bad, dear user. We saw that too. Great. What does that mean? How do you help me with that? Yeah, um, so I dedicated this one, for example, for looking at the CPU. Now I think that what makes the CPU more interesting than other types of, let's call it, issues that you might bump into with WebRTC is, first of all, it's not under your control at all, right? It's someone else's machine that you don't know, and even if you do, something else might be running at the same time on the same machine. I had a client that had an issue that at some time during the day, at a specific hour, Usually there were network issues with doing the stuff that he needed to do with WebRTC. When he looked at that, that was a time of day when they did the backup of the machines towards their data center, and that just clogged the network or clogged the CPU or whatever. It caused an issue. Okay, so now the difference between CPU and other things is that You can't just go and ask the machine through the web browser, what's the CPU like? What's the percentage there? Are you 20% full, 80% of capacity? Are you fully loaded at 100% and choking to death? You can't ask it. And the reason you can't is because privacy. Browsers decided that you cannot access that information because it is too private. Why? Go figure. Okay, usually they do that when you need to deal with fingerprinting, finding out what the user is doing, or trying even to hog his CPU, right? So let's say that what I'm doing is I'm creating a browser extension that does something useless, but along that something useless that I do, I'm going to go and try to generate crypto keys out of that. I'm going to harvest crypto. That requires a lot of CPU. I don't want the user to complain or to figure out that there's an issue in the CPU, so I'm going to check his CPU all the time and understand where the— when the CPU is available and use that time to run my extension. So that's the reason, for example, why not to give users or developers access to the CPU information. That said, with WebRTC, actually need that. Okay, the classic case when you're choking the CPU is when you try to run a group call. And the first naive solution for group call would be to say, well, we want the highest possible quality. So obviously, I mean, what we're going to do is I'm going to send you 1080p resolution at 30 frames per second, or everybody in that group call 2026, we're going to do 4K at 60 frames per second because that's what we do with games, right?

Right.

And everybody is going to send that and receive that because we want the highest quality possible right now. Each substream of 1080p, 30 frames per second is what, 4 megabits of data? If I need to do a call with just 4 participants, not many, I'm going to decode 3 such incoming bitstreams. Even if I have a hardware decoder, it might not be able to decode so many streams in parallel.

Yeah.

Okay, so I'm going to kill the CPU. And the thing is that what we're doing, it's like feeding ducks to get their liver at the end. You just stuff things into the browser and tell him, go decode this video. And there is nothing you can do with that besides trying to decode. What's the end result going to be? That if, you know, if you've got an SFU and it's too stupid and this is what it does— and there are such cases all the time, I get them all the time— so what happened is the poor WebRTC implementation would look at it and would say, well, I've got an incoming video, I've got a frame, it's too big, I don't have the CPU, I'll throw that away. Next frame comes in, well, This is a P-frame. It depends, it depends on the previous main frame, but I threw that one, so I will try throw this as well. So you end up throwing a lot of video frames on the decoder side so you can actually do other things. And what you'll have is the network is clean, everything is fine, but the frames decoded is different from the frames displayed, that is different from the frames received, that is different from whatever. And the CPU is choked full. Okay, so, and then you're stuck. And so we need to figure out on the client side what to do about that. The way to do that is to get hints from WebRTC that things are bad. And today there are 3 ways of doing that. The most classic and straightforward approach is this API called Quality Limitation Reason— not API, but part of WebRTC part of WebRTC statistics. There is a quality limitation reason in there. The video encoder is going to say, well, in the last second, minute, up until now, whatever it is, um, 10% of the frames were reduced in their bitrate because of CPU issue or because of a network issue. So you can say, well, I'm going to check and wait on the encoder look at that quality reason, quality indication reason. If it's CPU, then I know that there's a problem with the CPU and I need to figure out elsewhere what to do. WebRTC will be intelligent enough to reduce the bitrate for the outgoing video, but I might want to balance or rebalance other things in the system, or at least know about the problem so I can figure out a way to fix that. So that's one, which is great. Until you remember that there is receive-only cases. So what do I do if I've— let's do it here just for— we've never done that before. I'm going to mute my video.

Ah, okay, there's something wrong with your Wi-Fi, Tsahi Levent-Levi.

No, no, no, I mean it's something that I did on the video call, an actual video call. I muted my video because I wanted to manage something without anyone seeing me. Okay, or, you know, take a sip of my water right now. During that time, there is no encoder, or if it's just a, you know, receive-only use case, doesn't really matter. So if there is no encoder, there is no quality limitation reason to look at to find out that there is a CPU issue. But I still want to know that the CPU is clogged even if I'm doing only decoding. And then I start looking at other types of statistics. And the one that is the classic approach there would be the total encoded— well, I wrote that down for me, always forget the actual names of these things— the total encode time. That would be the time from the beginning of the call that the encoder worked in milliseconds. Okay, so I'm going to take that value. And divide it by frames encoded. And that would give me the, um, average time to encode a single frame or to decode a single frame, depending on what I want to look at— total decode time, total encode time, and then frames encoded or frames decoded. So I can glean out of that data information that is either the time that it took to encode and decode a frame from the beginning of the call on average, or in the last minute or second or whatever time frame I want. I can play with that and do that calculation. What is it good for? In a video call, we usually do 30 frames per second. We said that before. But if I know that I'm running 30 frames per second for this call, that means that I need roughly— I've got like a window of 33 milliseconds per frame. So if I cannot encode within 33 milliseconds or decode within that time frame, I'm screwed anyway, right? Because let's say it takes me 40 milliseconds to encode a frame. I can't do 30 frames per second. Okay, so you might say, well, okay, so if it's 30, we're okay. And the answer is no. What we want is a rule of thumb for the generic use case would be to use no more than a third of the budget that we have for the encoder and a fourth of the budget that we have for the decoder. So that would be roughly 11, 12 milliseconds for the encoder per frame on average, and roughly 8 to 9 milliseconds per frame for the decoder. Okay, why? Because we have other things to run on that CPU, not only the video encoder or the video decoder. And again, there might be specific use cases that are different, but that would be a starting point.

Yeah, okay.

So in a way, what I'm trying to say is this: we've got the statistics in WebRTC, there is a lot of data there, a ton of data. If you want to build something intelligent that works in all scenarios and cases, you need to look at the CPU as well. In order to do that, you need to dive and dig into the WebRTC statistics, run a bit of math in there, and then from there deduce what is the current CPU state. Because you cannot just ask, well, What's the percentage of CPUs at this point in time?

Yeah. I think this is really helpful insight because we do tend to blame the network first and problems may exhibit in network behavior like packet loss, as you said. We can't control the CPU, as you said. When you detect that there are problems, that there's CPU starvation going on or whatever may be the case, How do you react to it? What decisions should you make in the code? And are you doing— do you recommend looking at this as kind of like a, you know, a postmortem analysis of a call that, you know, went wrong, looking at the data, or looking at this in some constrained time window during a call and taking action during the call to try and improve things?

First of all, that's a great question, and the answer is yes.

Always do it all.

Okay. Yeah, the first thing is there are things that we need to do, and that would depend on the exact use case and scenario that we have. Yeah, so for example, if this is running on an SFU call and this is a group call and I've got 10 incoming video streams or 16 or whatever it is, a number, because I wanted to do the, you know, 24 squares or whatever of a lot of small tiles on my screen. So the first thing that they will do is say, we go from 16 down to 9, go from 9 down to 4, go from 4 down to 1 or 2. Just reduce the number of tiles. That's easy. I can say, well, you know what, I'm going to limit the encoder. Instead of 3 simulcast layers, I'm going to send only 2 because I can't, I need other things that require my attention. Okay, before all that though, I would work on balancing my application. So I don't even try to send 4 streams at 720p into a single user because that's useless. In most use cases, that's just useless. Okay, so the first thing would be count pixels. I think we talked about that one. When I count the pixels, I'm going to balance the system because of that. So that would reduce a lot of the CPU issues that might arise. And then I try to figure out what would be my fallback when there is not enough CPU. The same as you would do with network, you know, at some point you just say no video or reduce the bitrates and resolution. So kind of the same to do that with CPU, and that means that we need to do things dynamically. I would like to have that in postmortem to understand my application does that well and then optimize and fine-tune, because these types of things are related not to things that are deterministic in nature. It's not that I can give you like a recipe how to do that, and you'll follow the exact rules and it will work perfectly. It's more like a heuristic. I need to figure out and to play with that until it fits my use case and scenario exactly as I need it. Perfect.

Well, Tsahi Levent-Levi, this has been a lot of great insights here. I've got a couple big takeaways, uh, from this. Uh, the first is never mine crypto while hosting a large group video call. Definitely. If you learn nothing else today, take that away from it. But also that we do need to pay attention to that CPU and that there are some indirect ways we can measure that through the getStats API in WebRTC. Unfortunately, we're out of time, but I know that there's more that people can do too that you've written about also, like quality limitation reason and then the emerging compute pressure API as well. I'll refer our viewers to follow you on LinkedIn if they don't already, where you've been talking about this, and to check out bloggeek.me for more information on this topic and everything else around WebRTC. So Tsahi, thank you very much for sharing your insights with us today, as always. And as I said, to learn more about Tsahi's work, check him out at bloggeek.me. To learn more about future episodes of WebRTC Live, check us out at webrtc.ventures. Thanks, Tsahi.

Thank you.