A short tutorial by Tsahi Levent-Levi on improving WebRTC/media quality by reducing resource usage through a technique called "counting pixels." He argues that sending lower-resolution streams matched to display size (e.g., scale at source instead of encoding full HD then downscaling) saves CPU, memory and bandwidth while preserving user experience. A client case study contrasts five 720p streams (~4.6 megapixels per frame at 30 fps) with a configuration of 1 VGA + 4 QVGA (0.62 megapixels per frame), reducing pixels by ~10×. He gives an action plan: map streams (resolution, framerate, bitrate), prioritize importance, count/evaluate pixel load (mentions ~100 million pixels/second as a practical threshold), and re-evaluate periodically; finishes with a consulting CTA.
Questions this video answers
Roughly 4.6 megapixels per frame.
Approximately 0.62 megapixels per frame.
Usually 100 million pixels per second.
HD examples: ~2–3 Mbps; scaled WVGA example: ~600–800 kbps.
Chapters
Hi, my name is Sakhile Vent Levi. I'm the founder of Blockchickme and WebRTC courses. I'm here to break down real time communications so you don't have to. And today we're going to count pixels. What does that mean exactly? We'll see in a moment. Is your robotics application struggling with media quality issues? Many believe that the solution is is going to be throwing more resources at it. We're going to use more cpu, more memory on one network, and essentially send streams at higher bitrates to the other end. Why? Because the higher the bitrate, the higher the video quality is correct, of course, but often this is counterproductive and the opposite is going to be true. To improve the quality and the user experience for your users, you're going to need to use less resources at lower bitrates in order to gain the advantage of having applications that actually perform really, really well. In order to do that, what I use is a technique that is called counting pixels. Okay, now before we start, let's understand why less is often going to be more when it comes to WebRTC and to video sessions. Okay, here's an example. Let's say that I have a content that is being sent from one user to another. The first user is going to look at the camera. There is 1080p and HD video in there at 30 frames per second. We're going to take that video and encode it. Once we encode it, we're going to send it over the Network. We encoded 2 Mbps or 2.5 Mbps or 3 Mbps, because that's an HD video at 30 frames per second. So we sent 2 or 3 Mbps over the network. The receiver is now looking at it saying, well, I've got a bit stream here at 2.5 or 2 Mbps. It's 1080p resolution at 30 frames per second. Let's decode it, decompress it so I can display that. After decoding, it goes and looks and says, well, the user needs to put it on this smallish window on the screen. This smallish window has a resolution of WVGA, okay, which is not 1080p. It's roughly a quarter of that or even less and at 30 frames per second. So I'm going to take the big video that I had, scale it down and display that. Everything's great, right? We had high quality video and we show it as best as we can. Now let's look, the other approach. Now Viewser here is smart. He says, well, the guy over there needs it at wvga, so I'll just take the content, ignore the fact that it's hd. I'll scale it down at the source, encode a WVGA at 30 frames per second instead of 1080p. I don't need 2 or 3 Mbps. Now when I encode it, I can do that at 600 kilobits per second at a good enough quality or 800, but definitely not 2 or 3 megabits per second. So now I send over the network only 600 kilobits per second instead of 2 or 3 megabits per Second, which is a huge saving both in the encoder and in sending the receiver. He now receives only 600 kilobits per second instead of two or three megabits. So he decodes that, which takes less CPU and less memory to do. So now that it's got it decoded, he needs to display that. And it doesn't need to scale up or down because it's the right size already. So look what happened here. We had the same video or the same experience for the receiver, the viewer that needs to watch something at wvga. We can send it at a high resolution and then scale it down at the destination, or we can decide at the source to actually scale it down and process it in a way that everyone, including the network, uses a lot less resources. So we're now using less things to do more. Why more? Because, well, one of the CPUs might be burdened already. So we can do this high scaling or encoding or decoding and the network might not be able to do the two or three megabits per second and we might be putting too much strain on it. So the less we use in this case to get to the same exact solution is going to mean less resources using along the network. And we can do more with the resources that we have. So less is more when we do these things. Now let's see what counting picksql exactly means. And I'm going to do it by giving you an example. And the example is going to be a real life example that I actually had with a client of mine a few years back. The client came and said, well, I've got an application. This application is an application, a social application, okay? There are multiple streams in it. There is this live stream that goes on in WebRTC that is being sent to everyone, and then there are different participants in that application, and they can talk to each other at the same time that they see that live stream. So we have five streams in total, the one big one, and the four people that talk to each other when that stream takes place. And he said, well, you know, it's high quality, the application, and it's mobile. First, we're going to use high resolutions. At that point, it was 720p HD ready. Okay? Not that big, especially if you're looking at TVs and 4K, but quite a lot here. And we have five such streams, okay? One outgoing, three incoming, and then there's a live stream as well. If we take and calculate it all, there is roughly 4.6 megapixels per frame. And with 30 frames per second, that's a lot of pixels that we need to process every second. Okay, now it was too much. And guess what? A smartphone, even today cannot handle these types of numbers. So the first thing that we did was say, well, if we count the pixels, 4.6 is too much per frame. Let's try to lower this down. Let's take the big live stream that you want, the live stream feed, the important one, and send that at VGA resolutions, not 720p, something, you know, a tad lower. And the other four, let's do them at quarter VGA, even smaller ones, because they're less important to us. What happens at the end of the day that they're all watching the same video, and that video is an important one, and there's a bit less important than they even show on the screen. It's smaller boxes, okay, Smaller windows. So if we take 1 VGA and 4 QVGA streams, we get 0.62 megapixels. So instead of having 4.6 million pixels, we now have 620,000 pixels per frame. So we went down by a factor of almost 10 just by doing that. And now it's even manageable on the device that we have. So here's your action plan for counting pixels. And this is what I do with all of my clients as the first step. The first thing that we do is map the application. We're going to look at all incoming outgoing bit streams, the video ones, the voice ones. But for the video ones, it's quite easy. You go, you check the resolution, the frame rate, and the bitrate for each and every one of these video streams. You map them out to write them down. What is the purpose of this video? What is the resolution that is being sent, received, displayed? Okay? Then we're going to look at all of these video streams and prioritize them. Which ones are the most important, which ones are less important. I'll give you another example. If this is a large group call and someone is not speaking or is muted or hasn't spoken for a long time, their video is going to be less important. And if it's less important, I can reduce the frame rate or reduce the resolution for them. And that means reducing the bitrate, for example, and having less pixels to process. Okay, so we're going to prioritize these streams to understand what's important and what's less important. Then we're going to evaluate the efficiency. We're going to do the actual process of counting the pixels that we need to send and receive. Usually 100 million a second is an amount that we can deal with. More than that, we might have a problem and we need to change the approach that we have as we counted these pixels. Okay, so check the efficiency. And the last thing that is important is that this is like a recurrent process. Every three or six months, you should go map it again, prioritize it again, evaluate efficiency and, and see if there is room for optimization yet again. Now we need to do it not on the application level, but on the scenario and layout level. For each different layout, you need a different strategy to optimize for that. And again, remember that less is more when you count pixels. Now, one last thing before we go. If you want to 10x your speed and your ability to succeed here, come to me for consulting services for these things. Specifically how to connect more calls and to have higher quality for these calls. I deal with connectivity and quality issues in WebRTC, so reach out to me and even if not, follow me@grogging me.com and make sure to subscribe for these videos and also like this video. See you in the next video.
