Performance Testing Audio and Video with Domeniks Bisofs and Nikolajs Varlamov

Welcome to the Test Guild Performance and Site Reliability Podcast where we all get together to learn more about performance testing with your host, Joe Calantonio.

Hey, it's Joe. And welcome to another episode of the Test Guild Performance and Site Reliability Podcast. Today I have a special treat for you. It's on a topic I rarely see any information on on performance testing, audio and video, and we have some awesome experts with us. Today we'll be talking with Dominic and Nikol all about performance testing, audio and video. Dominic is a QA engineer at Test Dev Lab. He works on audio video quality testing projects and keeps an eye on new developments in testing. Nikolay is a audio video Testing Manager, R and D Lead and Engineering Manager at Test Dev Labs. So he has a lot of responsibilities there. As I said, I think this is a topic that as especially as Covid has shown, audio and video is really important. I think a lot of teams haven't been on top of it and this is the episode for you to get on the ball. Now to get ready to start doing more audio and video performance testing and testing in general. You don't want to miss this episode. Check it out. This episode is brought to you by SmartBear. Listen. Load testing is tough. Investing in the right tools to automate tests, identify bottlenecks, and resolve issues quickly could save your organization time and money. SmartBear offers a suite of performance tools like Load Ninja, which is a SAS UI load testing tool, and LoadUI Pro, an API load testing tool to help teams get full visibility into UI and API performance so you can release and recover faster than ever. Give it a shot. It's free and easy to try. Head on over to smartbear.com solutions performancetesting to learn more.

Hey guys. Welcome to the show.

Hey, hey, hey. Pleasure to be here.

Yeah, really great to be here. Thank you for noticing our work and we are really happy to be here and be part of part of it.

Awesome. Absolutely. I saw something on LinkedIn and it brought me down a rabbit hole of a topic I haven't seen much on. So I was really excited about seeing that blog post. And Also there's a YouTube video on the science of audio and performance testing. So we're going to touch on a bunch of these topics before we get into it. Is it that I missed neither of your bios that you want the Guild to know more about? I know they were kind of brief.

Yeah. So my name is Dominic and ever since I've been in Test Lab, I've been involved in large audio video testing projects, working on apps that are used by a lot of people and that need extensive investigations and testing and constant things that I need to delve into and learn just to understand the processes that are going behind the scenes.

Awesome stuff.

Yeah. And I think I guess I'll mention something about myself, so thank you for introduction, Joe. So, yeah, I'm looking after many different projects regarding audio video and as you mentioned, audio and video is a hot topic right now. Covid is still carrying on and it doesn't look like we're gonna quickly recover from that. So there is a lot of things going on in audio video. And regarding my bio, I was always like, through all my life, passionate, passionate about audio, was making music. And also in the end I ended up working as a sound engineer, doing live concerts, theater. And also a lot of that work was around technical troubleshooting and organizing my work. So. And then life brought me into the information technology where I quickly reutilized my skills.

Yeah, that's awesome. It seems like a lot of people in testing have a music background. I started off as a music major and I started getting into digital audio, which got me into the computer. And then all of a sudden I got into testing. So it's funny how that leads us to where we are now. So I'm just curious to know though, why audio and video at your company? Is that something your company specializes in or is it like something you just found yourself doing a lot of testing and then you specialized in it?

I think. Dominic, let take this one. So yeah, Test the Flap. One of the specialties is audio video testing and we started with that long time ago before COVID started. So Test the flap overall is 10 years old as a company and around five years we are doing audio and video quality testing. So actually when Covid came in, we were ready to scale and we had already processes in so we could offer to our clients to test audio and video quality products. Because also yeah, for many companies also started to work on the new functionality. It could be video video conferencing or it could be like streaming products. So everything around the audio video just plummeted, Dominic.

It seems like almost audio video to a lot of people seems like a black box. So you need to have a specialized set of skills in order to test it. What skills do you bring in order to be able to do that? Is this something you had to learn on the job or is there anything you recommend people knowing?

I would say that's something that most people have in their personalities. When I see my colleagues, for example, there's a big attention to detail definitely. And a lot of patience since also testing involves finding the smallest details. All of these apps that are in the market are in the main scale, awesome product, but they lack in those small areas which are extremely crucial. So anyone who has the attention to detail is just someone who excels in these things. And we at TestDaleUp just like to look at those things very carefully and get the most information possible out of them.

So how does it normally work? A vendor comes to you and so say we have a mobile app that we want a performance test and that's very heavy on audio and video or is it like Alexa? Like what kind of audio video applications are we talking about?

So I would say it will be. It could be divided to three, I guess main parts. So first part will be very popular right now, video conferencing solutions. And it won't be just something very similar to Zoom. There will be some niche products, something like medical applications where we even have like emergency vehicles so technicians can call to the doctor. And there will be very important decisions made over the audio and video services. So very like important. Even like life saving. So very important. And also solutions such as simultaneous interpreters where also you can choose the language and listening to the conversation in different languages. So that will be first batch of like popular products. And second one will be streaming. It could be video streaming. Like think of services as YouTube Live. Then it's very important for viewers to have a great experience, to have low latency, to have good quality, to have also network consumption not so high. So it all should be balanced together. Like you should not have any gaps in your performance. And third part I would say is audio video calling applications. Yeah, where it's utmost important is audio quality and you don't want to spend too much data in the call. So it also should be kept too low because as we know like half of population in the world lives in countries like Africa, India and China and ice like Internet plants, quite expensive and salaries are not so high. So this should be taken in mind.

You brought up a good point. A lot of people are like just audio and video on YouTube. Why do I care about performance? But I know especially with COVID to visit a doctor, I personally had to use an audio video app to speak with the doctor doctor. So you'd want that to be performant. So it's a good thing to make sure that you know we're not just talking about trivial things like a YouTube video. So I would think you would need a big lab or A big setup in order to actually do this correctly. How do you work that? Do you have a special setup? And like I would think like you said, everyone's different. So how do you manage device setup and all that?

So we have laboratories at each of our locations where we have physical devices. So although emulators can give you the idea, we think that physical devices are something that's important for the real world scenario because it gives the actual result that the real users will see. We recently switched to a new network dedicated network setup with access points that have virtual routers. So that gives us much better stability and coverage. We can put limitations to the mind's desire to mimic almost anything in the entire world.

Nice. Yeah, I was going to say, I would think it would be hard. How do you mimic like data centers in certain locations? Because sometimes I'll be running a live stream and people in the US are fine, but then you'll have people saying oh it's awful for me and you find out they're in a different time zone or a different location. Is that something you need to be aware of as well with performance?

Yes, of course. Just as I said, we can mimic the network conditions all over the world. So we've done, we can do and we've done research and if a client wants to know how the app will react to network conditions in, let's just say India where there could be just 2 megabits of available bandwidth and maybe at random times 50% of packet loss, we can do exactly that and see how they react, report the findings.

Yeah. Another point is we have internally built tool called Loadero and it's also used quite extensively for load and performance testing. And we actually via using Amazon web instances we can bring additional participants from different places in the world to the call or to the stream. And it sort of utilizes automation. So we can specify what this particular user, what actions he does. So he can connect to screen share or he can. We also can specify what sort of media he will broadcast. So be it like active video or. Because there can be different types of videos like as we right now talking, it's. We can specify that as live chat video where you have certain background and you just have these head movements which doesn't take too much bit rate. But there are different content like esports where you have all everything moving around and colors changing which will also dictate how will network will be consumed.

So Dominic, earlier you were talking about some top issues you ran into with audio and video quality. I thought maybe we'd Go into each one. I don't know where I I got this either from your blog post or for the video. The first one was you had like top five performance issues listed. First one was low bandwidth handling. How do you handle that? And like how do you find these issues for low bandwidth handling?

So usually there are very smart solutions to that. So there can be adaptive bit rate or it can be set to a certain value. The key is to find the right balance and we can help see what the balance is. Right now is it negative or does it have some. I'll just say some give. So if a standard from the client once their let's say video streamed at 1 Mbps without stalls, we can check at lower bit rates. And the issue usually lies between finding the balance between video quality and fluidity of the video, the playback, rebuffering and so on. So we do some extensive testing like.

Bitrate network problems and what you mentioned is actually from our webinar. So first I would quickly tell how we actually get this data. So one approach is to sniff packets from the device itself or use wireshark on the device itself if it's for example desktop. But in our solution we can grab traffic on the router itself and save it as a trace and later analyze it and look into over time data and look into averages. So the issues we sometimes see with a product is incorrect behavior of bandwidth estimator. So application it will have bandwidth estimator because if for example video resolutions encoded as in video stream streaming or as in video conference like when you're using WebRTC application adapts to network available. And bandwidth estimator is a part of application which is responsible to understand how much bandwidth is available for a user. And there are many things to check around bandwidth estimator good approach is to conduct changing bandwidth test. So for example, in beginning for user, user is in very good network 100Mbps when his network drops to 1 then even maybe drops to 100kbps second connection. And we need to be able to check that. Firstly application will adapt and it won't take a very long time. So user would not abandon the application or maybe switch off video completely. And second part is also to check how application adapts back if bandwidth is available again. And in some cases it can take up to 1, 2 minutes to adapt back to good quality even if bandwidth is available. So it means that bandwidth estimator is not so proactive.

Let's say so Dominic, in your blog posts on Visual quality you acknowledge network conditions are really tightly connected with the visual impression you get from the app when it's under load. You talk about subjective testing. What does that mean? Is that a real person looking at it when it's under load to say, oh, it looks good to me or it doesn't look good to me?

Yeah, subjective testing as the words implies? Yes. That means an actual person looking at the video. Of course, depending on the test, we can look for different details. We can either look as a user and just see whether it is watchable, or actually give scores for each video, or look for artifacts manually. And although on smaller devices a lower resolution might be quite critical from objective algorithm standpoint, possibly for the user it might be barely noticeable. The typical user might just glance over it. So having again a balance of objective and subjective values when testing is great to give you a complete view of a system, especially for a video streaming service.

So a lot of questions I get from people as a How do I even test a video from a user's perspective? Can you automate that? Is there any like visual validation you can take like an image and then compare the image to what it's actually looking like? Or is it all you need a real person to look at it?

General as Dominic already mentioned, there is subjective approach which usually is extra added to objective. And with objective approach the most common what we use also in our company is full reference and no reference approach. So for full reference for when you are evaluating like video files or you you're evalu broadcast where stuff you get you can fully correlate to the original file. So you have reference file and you have degraded image which is processed through the environment and industry standard metrics are vmaf, SSIM and PSNR. And VMUF is one from used in Netflix and proposed in Netflix and we use this one also extensively. There are still important development tasks to be able to utilize that full reference metric is you need to come up with a solution to align frames. Because during your video you might have some freezes or dropouts and reference video it will be full video with over 30fps or 60 whatever it was the originally but then reference video might have some freezes, stalls, drops. Then you would you would need to create another copy of a reference video which will be exact same as degraded. So it was about full reference and second one is no reference. And we internally use brisk. There are different models, algorithms, but we decided to go with BRISK and it doesn't need reference image to compare with. It generates scores based on machine learning algorithm and you can train that model. So for example, if you always test with the same content, for example, you go through the different applications, you compare products with the same content, then you, you're able to train the model for this particular video. So you get as close as possible to subjective actually evaluation. And underneath Brisk has a lot of like advanced scientific video processes. But I think it's a bit out of scope here for this podcast.

Yeah, nice. So besides video, obviously people always ask me how do I test audio? Once again in your blog post, Dominic, you talk about audio quality and you talk about a tool I haven't heard before. I don't know if it's the main tool you use. Poll qa. I guess my question is how do you test audio and what tools do you use to do that?

Yeah, so with audio we go with objective only approach mostly. And there are two most common algorithms to evaluate audio. First one is Polka and we need the license for that. So any company which would like to use Polka needs a license. And Second one is VSCO, is actually from Google I think developed around 10 years ago. And it's free to use, but you need to train the model as well yourself in order to produce like results, expected results. And in our experience, Polker corresponds a little bit better. I won't even say a little. Polka corresponds better to human perception. So for example, if you will have, if you have better audio quality, for example, the codec, the sampling rate of Audio may be 8kHz or 16kHz or codec consumes not so much bandwidth, which might be very good in some cases where you don't want a lot of bandwidth to be consumed. But Polka will register that. Polka also will register if you have a jitter in audio. So sample will speed up or it will slow down at some point or there will be drop and the amplitude will be quite relating to the subjective evaluation. And also inside Polka there are two modes. There is a wideband mode is for recent technology where we have sampling rates of up to 32 and 48 kHz, which means that audio frequency will be up until 20 kilohertz. So high quality audio. And there is also narrowband which is sort of for more telephony products where you have 16 kilohertz sampling rate. So quite narrow band. So yeah, so two different products also different little approach with audio. And also on top of that it's about quality. But also Polka is great because it returns to the tester's latency results as well.

Oh nice. So now we touched On a few of these already top video quality performance issues a tester needs to be aware of. You talked about the low bandwidth handling, packet loss handling, anything else that seems to be a critical piece when it comes to what you need to test. When you're testing video, audio like does device performance issues ever come into play and I guess does your lab handle all types of configurations? How do you even test for that?

Yeah, there are many objective things that you can just gather through various metrics like video quality metrics like VMAF we were talking about. But there are many things that one could glance over. For example, your app might perform in a great way on the newest greatest device, but it could just fall flat on an older phone with an older Android version or maybe just one that's made for low income markets or developing markets. So we have a lot of those physical devices and it's not just a few. We have over 1500 devices that we do active testing on and we can not only simulate any network needs, we can also do actual testing on actual devices that the users would use in real life. Not just the newest S21s or 12 Pro Maxs, but also anything that an everyday user would have.

Yeah, I would like to add also to that that it's actually very important because in many markets of like mentioned before, India and Africa, very popular are low budget phones and actually their performance will be also low and application needs to account for that. So we still always want adequate performance on even budget devices.

So either Nicolay Dominik Just curious to know when I was starting off as a performance engineer, you know, just finding out how to test a simple system with concurrent users to me was really difficult and getting it correct was hard. I assume with do you have all these different clients, all different audio, video quality needs, different apps. How do you even plan something that difficult? When devices come into play, regions come into play. Just curious to know how do you handle different scenarios when you're testing? Do you have a framework you follow or anything like that?

Yeah, thanks for question. I think it's quite important one. So first of all, I think in our company communication style is communication culture is very good. So knowledge is shared across the colleagues and even upper management is really helpful. Like if you have doubt, if you have question, like things are brainstormed and like from the question to solution there is not like a long way. So we all feel like everything is possible to like to solve. So I would say this is the first one and the second one is we do have a framework and we do have as mentioned Dominic, we have laboratory so called and this is environment where we have dedicated network, we have audio evaluation set up in and we have hardware ready to evaluate video. Of course it changes from video customer from a product to product but there is a certain baseline. Yeah from where we work around to meet the needs of a client.

Dominick, I don't know how much I touched on your blog post but so.

What I really touch on is these five, these few big aspects and five basic things that we really need to experience and really delve into deeper and research and find out and just disassemble metric by metric. But also I would like to add that sometimes in this kind of testing some simple things should not be missed. Like sure, how fast an app loads, how fast a video loads is really good to get in a performance aspect, but even things like content and fulfillment need to be taken into account. And sure there are many, many, many challenges to that and it's hard to give the accessibility to your content for every content kind of human. But it's a topic that's even more important daily and something that the developers should look into. Especially now that regulations are coming in that require certain level of accessibility in the apps.

Now that's curious. Do you notice like different countries then have different requirements that you have to test against as well? That must be difficult to know and keep on top of.

Yes, our experts heads are definitely currently pulsating with each new change.

Like probably the most important also part is when you get all the metrics you need to make the proper analysis and draw the conclusions that everyone will understand, not just engineers, developers, but also product managers. So it should be both like low level and high level and all of those metrics, they should be connected together because if you see bad performance in one metric here, you might be able to see good performance in another metric here. And then you need to understand with a team is that the behavior we would like to have, maybe we can can lose some quality here, but we need to make sure audio quality stays on top.

I mean I was looking at some of your graphs, I didn't know like with so many metrics, how do you know what correlations are? What happens when you see this metric bad? How's it correlate with another metric being bad? And looking at these graphs it seems really difficult to even dive in to know what's causing maybe an issue like that. So how do you express that to your teams then so that you're not getting a lot of false positives like hey look, this metric's bad, so I assume that it's caused by by this, this other thing. How do you know what the correlation is between what's causing the issue?

Yeah, we have a process called validation. So before actually putting the presentation together, putting the final results, we go through the tests. We have a team for that which actually checks if results we have, if they comply with what we really see in real life and what the user will experience. And by this we can feel that we are secured with the data we have.

Awesome. Okay guys, before we go, is there one piece of actionable advice you can give to someone to help them with their performance testing of audio, video and what's the best way to find or contact you?

Yeah, I would say my best advice would be find information on the web and do not hesitate to contact the guys who made this blog, who made this webinar. Just write to them, just ask them like it wouldn't hurt. Like usually in my experience, people like quite often and they will share the knowledge because that's what all about. Making the world better.

Absolutely. Dominiks. Anything Parting Words of Wisdom Also, it's.

Good to criticize all these apps, but we should also remember there's a human behind everything. So that's my word of wisdom.

That's a great point for sure. And the best way to contact you all or learn more about TestDev Lab.

The best way is to go to audiovideotestlab.com we have audio and video information there, our blogs, our link to the webinar and also you can grab us on LinkedIn and write in private.

Thanks again for your performance testing on awesomeness. Links to everything of value we covered in this episode. Head on over to testguild.com P70 and while you're there, make sure to click on the Try them both Today link under the exclusive Sponsor section to learn all about SmartBear's two awesome performance test tool solutions, Load Ninja and Load UI Pro. And if the show has helped you in any way, why not rate and review it in itunes? Reviews really do matter in the rankings of the show and I read each and every one of them. So that's it for this episode of the Test Guild but Performance and Site Reliability Podcast I'm Joe. My mission is to help you succeed with creating end to end full stack performance testing awesomeness. As always, test everything and keep the good. Cheers.

Thanks for listening to the Test Guild Performance and Site Reliability Podcast. Head on over to testguild.com for full show notes, amazing blog articles and online testing conferences. Don't forget to subscribe to the Guild to continue your testing journey.