Your Device is Overheating? Fix WebRTC CPU Now

Hi, my name is Sachel Evan Levi. I'm the owner of Bloggic Me and WebRTC courses and I'm here to break down real time communications so you don't have to. This time we're going to look at five different action items for CPU optimization on the device. Before we start, I want to go and explain why we have an issue around CPU optimization. Let's talk about quality for a second. If we think about quality, the best video quality that we can achieve today probably or likely would be 4K at 60fps. That would be 4K resolution with 60 frames per second. You've got TVs that do that today, so why not have that in your video conferencing using WebRTC? Let's do the calculation. When look at 4k, 4k is actually 3840 pixels on the width on 2160 pixels on the height. That would be the number of pixels per frame that we're going to need to process. That means encode and decode. If this is a one on one call, not even going to group calls here. If we multiply these two, we've got about maybe 8 million pixels, give or take that we need to process per frame. On encoding, 8 million pixels, on decoding, 8 million pixels. Now how many pixels are these going to be per second? Let's multiply it by 60 and what we reach is around half a billion pixels per second that we need to process in each direction. So about a billion pixels per second. And now I haven't even gotten into the RGB part that each pixel is two or three bytes and all of the other complexities that we have. So if we want to reach high quality by saying, well, the higher the resolution, the higher the quality, then we're in big trouble. So this is likely not the way to optimize media quality for the CPU of the device. So let's go and look at the five ways that we can actually use in order to think about CPU on devices when it comes to quality and improving video quality. The first one is we need to understand if there is a problem and that we can do using GET stats. Why are looking at GET Stats as a way to do that? Because inside the browser there is no good way today to know what is the CPU performance of the device. If this were a native application, that's not an issue. You can ask from the native application towards the os, what's the current CPU status? And then you know, we can't have that on a browser. So we need to estimate how do we have that estimations? Well, we can check the encode time per frame. We take the total encode time from get stats and then divide that by the frames encoded and then we reach the average of time that it takes to encode each frame. We can do that second by second, for example, and get the numbers that we need. Now what would we would number here? Likely below a third of the time that we have per frame. So let's assume that we're doing 30 frames per second, which is usually what we do in WebRTC and in video conferencing. With 30 frames per second, there are 33 milliseconds per frame within a second. I took 1,000 milliseconds, divided that by 30. So a good encode time per frame would be below 11 milliseconds, for example. Why? That's 33 divided by 3. We want to have that at that place so we will have more room for other things that we need to do like decoding and other things. We can also look at decode time. Sometimes we don't encode or we decode multiple different streams. So we can take the total decode time, divide that by frames decoded and reads the time that it takes us to decode each frame again with the same kind of an approach. Calculation and understanding of what it means to be too high or too low. There is another one in WebRTC. You can go and check the encoder limits if you are actually encoding video. If you are an active participant in the video. There is a metric called quality limitation Reason which tells you why the encoder decided to limit the bitrate that it is sending out and to encode less than it could or that the network allows. If that reason is going to be cpu, then we know that there is not enough CPU to encode. Which means that might be a problem there with how we've balanced the system. So we need to work on that. A new API that is coming out likely is Compute Pressure API. That's an experimental API that is available today in Chrome. Go check it out. That might be the solution long term. Another thing that we need to look at is the video codec that we are using today. There are multiple video codecs in WebRTC that we can choose from and which one we choose is going to dictate the quality that we're going to have. Now in this diagram I've placed the codecs within their generations. We've started with VP8 and H264. These are I Wouldn't say old codecs, but they are old codecs and they are very popular and common today. The next generation after that was VP8 and HEVC. These codecs are better performant than H6.4 and VP8 because they are newer. They compress better, but they take more CPU to do that. Now Hevc is in lighter color here because it doesn't exist across all browsers and platforms, just the Safari ones. And even there I'm not sure that I'd recommend using that. Then there is AV1. This is the current day technology, the best codec available today. It's rather new in the Market and in WebRTC. It has more CPU, compress CPU, it requires more CPU and memory, but it compresses a lot better than all of the other codecs. Now which one should we use? If we go with the AV one that's going to be great. It's going to compress better. The bitrate is going to be lower, the quality is going to be higher. But it takes more CPU. And there is no hardware acceleration available for AV1 today. At least not across the devices that you have out there. My laptop won't have that today and it's rather new, it's like a year old. My smartphone doesn't either. I've got a Pixel 6 switching to a Pixel 9 so AV1 likely not that a good idea. If we're going to do the 4k 60 frames per second or even 720p encoding. I'm not sure that I would try to do that on a device today. You can check and see if that works for you. Next we can take VP8. VP8 is great. It works really well. Doesn't take a lot of CPU comparing to the other codecs. But it won't have a lot of hardware acceleration available. I would still prefer using that over H264 in most scenarios. H264 has a lot of hydro acceleration available. Or it's there, it's great, or it's not, there might be an issue. So if you want to go for H264 that's because you want to use that hardware acceleration. You know the devices that are being used and you know that they have that available for them in WebRTC. HEVC got hardware acceleration on the devices that it exists on. Apple devices. Another thing that we need to look at is simulcast and SVC scalable video coding. These can improve the quality without adding too much CPU into the system. There are places where I wouldn't use Them there are places that I would, in large group calls, I would definitely use simulcast or svc. Simulcast for example, means that I'm going to send multiple video streams towards the sfu, the media server. I'm going to do that at different bitrate and qualities. And the SFU is going to decide which bitrate to send to which user, which quality to serve each user based on whatever criteria makes sense. And the CPU power of the devices that are viewing is one of these criteria that we need to look at. Up next, you need to know your generative AI. Generative AI in WebRTC means doing things like background blurring, background replacement or noise cancellation. These are great in order to improve the quality of the media of a call. You know, I don't want all of that crappy background. I'd rather have that clean or blurried or maybe replace that to a different location. I might need to add noise case cancellations. You won't hear that dog barking or the people outside lowing the moan where I actually need to wait for them to go before I record these sessions in a video call. I can't do that. I can't wait. So noise cancellation works very well at that area. The problem with these algorithms is that they take up a lot of CPU and that CPU is going to hurt our video encoding and decoding and then the media quality itself. So we're trying to improve media quality and we're hurting it. What do we need to do here? Well, same as always. The first thing is to measure it, to understand what the problem is. Once we do, we can decide one of three things. If there is an issue, if there is not enough cpu, for example, one would be to disable it. You know what? This device doesn't have enough CPU to run background replacement. I might as well disable that capability for this call for that user at this point in time. The second option is to offload that algorithm into the cloud. You see that done for noise cancel cancellation, for example, in some cases also for Burgberg corn replacement in noise cancellation. Google Meet does that in the cloud or on device, depending on whatever they decide. So this can be done. You can decide where to do these things sometimes. And when you can, you can offload that into the cloud and reduce the CPU load on the client side. The third approach, which I highly suggest, is to use a tiered algorithm, an algorithm that will decide to improve its quality based on the amount of CPU that it has available, so that I will do better background replacement. More accurate if I've got The CPU for that. I will do less of that if I can't. Or I'll disable it, depending on the cpu. And the algorithm is aware of that and knows how to deal with that. So when you choose the algorithm that you are going to use on the browser, be sure to use something that is tiered and not static in the CPU load that it has. Last but not least, the fifth suggestion that I have is think about the 10% of the users. You've got a lot of users in your system. 10% of them are going to be on low end devices or low end networks. Low end capabilities. You need to think about them as well. What does that mean? The first thing is that you need to stop thinking about 4k at 60fps. That's your high end. Your 10% of the low end would need the minimal viable experience. What would be the worst experience that you want to give users and still say you can still use the system? What does that mean to you? Okay, does that mean that the video goes down to 360p or even 180p in terms of the resolutions? Do you reduce frame rates to 15 frames per second to 10, maybe less, just to make sure the video works? You need to define that minimal variable experience in order to be able to implement towards that as well. Go for the high end, but be able to go down to the low level. Now, these experiences that you have from the top high end to the low end, they need to be dynamically cleared, which means that I can move from one experience to another dynamically within the session because someone started using the cpu, someone used the network, and I need to understand that and change my behavior accordingly. If the antivirus virus decided to run and do something, or there is an OS update, I need to be able to work with that and reduce the needs and requirements of the call that I'm in. Another thing to think about is to do some graceful failover toward audio only solutions. Okay, can do video, not enough CPU for that. We're going to do voice. I'm going to alert the users that this is the case and we're going to do voice in order to maintain the connection and the call itself. So we actually have a meeting without video, because we can't really have video. It won't be a good enough experience for us. Okay, so we talked about the cpu, the fact that we actually need to optimize for the CPU and not only the network. If you want to 10x the speed of rate in which you do these things and run these optimizations for your WebRTC application. I'm here. Reach out to me, below, above, wherever you find the way to find me. And if not, go and follow me at blogic me. Thank you.