8 ways to optimize WebRTC performance

Hi, I'm Tafil vanjlevi, founder of Blogecme and WebRTC courses. I'm here to break down real time communication technologies so you don't have to. Today we're going to discuss eight different ways to optimize WebRTC performance in your application. What's optimized and what's performance? We'll check in a second. Before we start, look at the description below. There's a three step plan to successfully launch WebRTC services there. Download it and read it. So what exactly do we mean by optimized and by performance? Optimized means improving something for the better. In our case, it's going to be the performance of the application. We're not going to look at the CPU or the cost that we're going to incur because of the service, but we're going to look at the end result, what is going to be the user experience and how we improve that. This is a performance gain that we are after. Now we're going to start with the first four items and these are going to be around doing more with this. How can we cram more out of WebRTC, you know, squeeze the performance out of it, but we're going to do less in order to get there. We're going to be lazy in order to do that. Why does that work? Because the more effort we put into things and the more we're trying to improve performance and resolution and frame rate and the bitrates, the harder it is going to be on the devices and on the network. And if you just alleviate the pain a little bit and make sure that we fine tune whatever it is that we're sending to what actually our scenario needs, things are going to be a lot better. First things first, we're going to do something ty call count pixels. What we want to do is to lower the bytes that we're receiving and sending over the network. Take video for example. To lower the bitrate means the following. I'm going to send a lot of bitrate, but then I'm going to send a bit less. The less I send, the lower the resolution is going to be in order to be able to send that resolution. Now if I lower the resolution, actually lower the CPU that is needed to send that and the memory that the machine needs in order to do that. And also what goes on the network, the lower the resources that I need, the better the performance of the machines are going to be. And if the resolution they end up with is within the quality limitations that I actually need for my use case, that's even better. So go count your pixels. Check what are you sending and receiving versus what you're displaying to the user and what you have from the camera. Now the second item is about video codecs. Video in the past WebRTC was simple. We had a single video codec or two, VP8 and H264. Today we also have VP9, AV1 and HEVC in some cases. Which one should you use? What is the better codec? You need to pick the best codec for the situation that you have. So let's look here. For example, let's say that we're using VP8 and we want to switch to use VP9. Okay? So the amount of bits that we're sending for the same quality with VP9, which is a better codec, we can send less bits to have the same quality. Exactly. Okay, so we lower and reduce the network use to gain the same quality. I can also increase network use back to what it were with VP8 when I used it. But now I have a better quality for the same bitrate. So if I just choose my codec right, I can improve the performance or the quality of the end result. Now, switching from one codec generation to another has its own costs. For example, CPU codecs might not be available everywhere. So we need to decide when to use which codec. Okay, bear that in mind. Let's talk about audio. Don't receive too many audio channels. Here's what happens. A few years back RoboticsI had an optimization. Nobody told you about it, but it was there. If you had a multiparty call with a lot of participants in it, and as user received multiple audio streams incoming to the device, WebRTC would look at these incoming streams, pick the top three streams that have the highest volume in them, and mix only these. Throwing everything else into the trash. That was the end result of it. Okay, now that was great because then the CPUs of the device would go lower because you don't need to decode all the rest of the streams and you don't need to mix them in when you play that back to the user. What happened is that at some point Google decided that they are going to change that and everything that is incoming will be mixed. Why? Because it gives more flexibility to you as a developer. So now if there are 10 different audio streams coming in, are going to be mixed together. Now guess what? When people speak and in a conversation, which is most of the sessions out there, you don't need all of these audio streams. Because at most you'll have one or two people speaking on top of each other. Three, maybe more than that, you don't really need. So mixing only the top three makes a lot of sense. What we should have done in the past and definitely today, is not to send too many other strings to the user. So instead of having the device pick and choose which ones to mix, you need to do it on the server side. The sfu, the media server needs to look at the incoming streams that he receives. He needs to then look at the volume levels in them without decoding the packets. Okay, this is not something that he needs to do. And then get that data and send only the top three that he wants to the participants of the call. This way the participants get only the top three and they will mix only the top three. So we are going to optimize the performance here to use less network and less CPU at the same time. And while we are talking about Media servers and SFUs, here's one more thing. This is a mistake that I see happening all the time. Companies using simulcast when they shouldn't. What is Single cast? We have a user. The user is going to send two or three different layers, two or three different bit streams of the same video. Exactly. Just in different bitrates. All three goes into the sfu, our media server. And the media server is going to look at them and say, well, you know, this bitrate that is the mid level or the lower or the higher, this one fits that user because of his network, the layout on the screen or whatever the reason is. So the SFU can now selectively pick and choose out of these three bitrates that you got, which one to send where? It's great. It works like a charm in large group calls. But what happens if we're in a one on one session and frankly, 80 to 90% of all the calls out there, of all of the clients that I've seen, are one on one. In one on one sessions, we've got the user sending three different streams. In order to create and generate these three different streams, I need to use three times the same encoder, low bitrate, medium level and high level, and then send all of that data over the network, which is a lot of data. So I'm going to take CPU memory and again network. But that's in order to talk to another user, a single user that might be on a smartphone with low resolution. So he's going to get only the lowest bit rate that I have. The rest that's total waste. So when using SFU and Simulcast. Make sure that what is being sent to the server is actually being used by the midaserver. If there are certain layers that are not used, don't send them from the client to the server and you form a one on one call. Maybe even just disable Simulcast for that specific call. Let's continue. We talk about the first four items. These were around more is less. How we can do more by doing actually less. The next four are going to be about know your scenario. The more you know about the scenario that you have, the better you can do to optimize for it. OVERTC includes a lot of different tools. It's like a huge toolbox with a lot of different Lego pieces that you can pick and choose for what you want to build, but you need to understand what it is that you're building now. One thing to remember, this is all dynamic. You can be on a one on one call and then a third person joins or then he leaves. Or it might be that the network is good and then something fluctuates for one of the users. So things happen and you need to dynamically fit whatever you're doing to that specific scenario. Let's start. So we have multiple scenarios. We need to understand them better. First thing to ask is what's our frequency? If there is a scenario that happens a lot, we might want to invest time in optimizing that before others. If there's a scenario that is very important to us because these were the users are or this is what happens when they onboard, then we want to optimize for that first, even if it's not that frequent. When we're going to optimize, we need to decide what to do exactly. Some of the optimizations might have a lot of effort in them or low effort. We're going to pick the low hanging fruit first. Then there's a matter of complexity. The more complex they are doing that optimization, the riskier it is to put that optimization in place and the better experience you need to have in WebRTC in order to do that. The last one, which we don't talk about much, is smoothness. How smooth is it going to be to move from one scenario to another? Remember we talked about two users and then switching to three and switching back to two? We need to remove simulcast, add simulcast. How smooth is that transition going to be? That's very important for the user experience. It all boils down to the ROI of the optimization, the return on investment. Make sure that you know your scenario so well that when the Time comes to optimize, you understand what you're going to gain out of it, so you can extrapolate what the ROI is going to be. Another thing that happens a lot is media servers. Common sense says that the more we have, the better the performance of the application is going to be for the user and the more expensive it's going to be to us because we have more servers. How many do you need in which regions exactly? That depends. Look at that graph. Or there. That graph simply indicates what different C vendors the video API platforms have in terms of regions, how many regions across the globe they have to serve their users, which are global users. And you can see that any number between 1 to 300 is fine. Most of them get stuck in 10 or 20 different regions, which is fine. Okay, where do you need to place your servers and in which regions? That depends on your user base. So again, you need to know your users, understand their scenarios, and place your media servers as close as possible to your users. Now that we have the servers close to the users, how to allocate servers to the users? The easiest answer? Well, I already said that. As close as possible to the user. But what happens if this is a conference call with three participants, two in the us, one in Europe? Where do you allocate the server? Exactly. If we allocate the server in Europe, there is one user there and two users across the ocean that need to connect to that server. So they are going to have very poor interaction because, well, they need to go back and forth through the ocean in order to talk to each other. That's a lot of latency. Why did we allocate that in Europe? Because the first one to join was in Europe. First user joined the call. Who could have said, yes, you know, let's allocate it in the US because we know the call and who is there. If you do that, it's great. These two have a great connection, but the European guy, not that nice. The longer we go over the open Internet, the higher latency, the jitter, and the packet losses that we're going to incur without being able to affect that or even pay to improve that, which is a problem. Another solution is to cascade things. We're going to have a server here and the server here. Every user connects to his own server and the servers talk directly between each other. That's going to be on our lines, our network, on our supervision. So we can improve that over time if we want. The problem with that is that it's more expensive and it's more complicated to actually go and implement it that way. Which option is good for you? That depends on the use cases that you have and the scenarios and the ROI involved in it. And that leads me to the eighth way in which you need to optimize your overtc performance, and that's by collecting, measuring and monitoring your overtc metrics at all times. You need to look at the KPIs, the key performance indicators. Collect them from the end users, the clients, not from the servers. The servers know a little, the actual user devices know exactly what goes on on the device. So you need to collect this data from everyone at all times, measure the results, look at what happens there, and then monitor it over time. Okay? Doing that means that you're going to optimize in a continuous way throughout the deployment of your application. Things are improving with WebRTC. Your competition is getting better at the same time and you need to be on. Okay, remember roboticsy is a marathon, not a sprint and you need to be there continuously. So if there is something you need to remember from this is to collect the metrics that you have for WebRTC to understand the quality that your users are actually gaining. Then measure the results and understand what they mean and monitor them over time. If you haven't done so yet, check down below. There is a link for my three step plan for launching successful TC applications. Other than that, there's a subscribe button. Be nice and click that as well. To continue following me in the videos that I put out to follow my work, go to Blogg Me and check out things that I write there. Thank you.