Real Browsers Performance Testing with Keshav Vasudevan

Five, four, three, two, one.

Drop that.

Welcome to the Test Guild Performance Podcast where we get together to learn more about performance testing and site reliability with your host, Joe Colantonio. Here it is. Here it is. Here it is.

Hey, it's Joe. And welcome to another episode of the Test Guild Performance and Site Reliability Podcast. As you know, teams are more eager than ever to incorporate testing into their agile and DevOps workflows. But what about performance testing? Today, Keshav Van Schudvin shares why protocol based performance scripting can be time consuming and how using a real browser based protocol performance approach may be better in certain situations. So listen up and discover actionable insights for creating user based performance testing and how to add them earlier in your development lifecycle. Check it out. Let's face it, performance testing is tough. Traditional load testing tools create scripts that are bogged down in data that's hard to read and require a lot of work, even for simple playback. Load Ninja cuts out the dirty work by using AI to inspect and debug your code almost instantly. No coding necessary. No changing of hands or relays between teams either. Give it a shot. It's free and easy to try. Head on over to testguild.com smartbear and click on the link to learn more.

Hey Kesheville, welcome to the Guild.

Thank you so much, Joe. I'm really excited to be on this.

It's great to have you. We had you at Perf Guild as one of our speakers last year and it was a great experience. Got a lot of input, especially around the shift left type of activity that's going on in performance, especially with browser performance testing. And also I think it'd be great to have the audience know a little bit more about Load Ninja. But before we get into it, could you just tell us a bit more about yourself?

Absolutely. My name is Keshav, so I've been in software development for about five years now. I'm currently leading the product management of Load Ninja performance testing platform on the cloud. I was in functional testing and performance testing before and also of course developing applications. So it's really good to actually help build a product that solves some of the problems that I had encountered while I was building applications, especially around performance testing, because it's something which everyone believes that they need to have, but no one necessarily knows how to get started with it or even becoming successful at it. So absolutely, I'm really grateful for getting that opportunity with SmartBear to do it.

So that's interesting. You do have a background in automation and you have struggled with this yourself. So is there any particular feature in any of the products that SmartBear has released that you had input to say, look, this is what people are struggling with and I think this feature really would help them?

Yeah, I think it's. I definitely think there's.

It's probably, I would say in terms of like just the scripting aspect to it. Right? Because many times, you know, specifically on performance testing, you know, you start by first gathering the application requirements, gathering the end user transactions that you want to start testing, but then immediately right after the next thing is like, how do you start writing that into a test script? Right. How do you get started with that? Do you do that using like a. Do you manually code that in Selenium maybe, or do you have a recorder in place? Is there anything else that you can do that's typically where you have to have some level of deep expertise through trial and error to a lot of learning through online resources in order to become successful? I think definitely there's one aspect to one strategy in SmartBear as a whole is from a product perspective, we try to improve on the outcome that our end users want to achieve. So focusing on the job.

That they're trying to. So in this case, if it is successfully setting.

Is to create a test. So anything that we can do from a product perspective to maximize on the outcome, in this case it's minimizing the time it takes to set it up. That's something which we want to focus on as a product. So specifically in load ninja, one of the things that we have is the ability to create scripts from the browser's perspective. That essentially means that you can, one, as a result of this, you create transactions and capture transactions from the end user's perspective directly because it's the browser. But two, as a result.

You'Re spending less time doing any of the other manual editing and correlation and programming that you'd otherwise have to do in a protocol based recording tool. So this is something which we wanted to focus on primarily because it maximized on a key outcome which most testers wanted to do.

Awesome. Because it's using a real browser as well and because people are more familiar with interacting with the real browser rather than a protocol. Does that also help shift left the performance activities that more teams probably should get involved with?

I think so. I think it's a chicken and egg situation. Right. Because shifting left is something which we've been talking about for a couple of years and I definitely foresee more and more people doing that in 2020, because the demand for application testing services is continuously and over the last decade itself there's been a shift from waterfall to Agile and DevOps methods. Speed to market is such a critical thing for many organizations. And so a lot of these forward thinking IT departments have essentially recognized and accepted that you have to test early and test often. So in order to do so, typically what happens is you have more and more people, as you know in shift left, who's been responsible for say having a center of excellence or like a specific job functionality, you know, essentially collaborating more often and creating a more central, a more, I would say open approach to it, where everyone is really in charge of quality and you're having a quality mindset much early. And so as a result of that, you know, you have, because of going through the browser based approach, you do have more developers. We see this so often. We have so many such, so many of our customers who are just like regular, who are just developers who have no performance testing background or we have like a lot of functional testers who've been tasked with doing performance testing and now have to do unit testing, performance testing, functional testing, regression testing. They have to do all of these things that they're responsible for and they just need a solution where you can do this in a seamless way, in a seamless manner, without necessarily spending weeks on end just creating a specific script for functional testing, then a specific script for performance testing and so on. So not working in silos.

Awesome. So for the people that don't know though, traditionally what I would do is back in the day I used a tool protocol based and then in order to get a user experience type, I would have to write what they called a GUI VUSER that ran a real browser and then did a loop and did timings. What do you mean by real browser based performance testing?

Great question. Traditionally, I would say for 20, 25 years performance testing was typically done by capturing the end user transactions on a protocol level. This is all the chit chatter, all the interactions between the server and the client, the request response cycle. So if you capture say a transactions, you're going to be capturing all those transactions as protocol traffic on the script. The challenge in this approach is anytime there's any sort of dynamic components that may be part of say this interactions. This could be for example cookies, session IDs, tokens, anything that's dynamic. You'd have to essentially identify all those dynamic parameters and go through a process called correlation or dynamic correlation or programming in order to essentially successfully carry this forward across all requests and responses. So identify what is changing and then do this process of correlation and programming in order to make sure that this propagates correctly. Typically this takes anywhere from several hours to a few weeks, right, because you first need to identify what's dynamic. So typically when we used to do this, you would have like five different.

We would capture those transactions in the first iteration, then we'd shut down the browser, then we'd capture it again, then we'd see what's changing and we do this four to five times to see what's really dynamic across all the requests and responses and then try to identify the first time that was dynamic and then go through that process of correlation. Let's add to this the fact that most modern web applications right now there's a lot of single page apps that are coming out, so there's a lot of client side business logic that's being employed. React and Angular are leading this charge over here. So in such cases there is no interaction between the server. So for such a tool you'd have to do additional scripting and programming in order to capture those kinds of transactions as well. So really the process of setting up a script becomes increasingly hard.

There's been a notion of browser based load testing for, I would say it's something which people have become educated on over the last five to 10 years, I would think. But making it a reality was relatively hard. Right. In browser based load testing, instead of going through the protocol level, you're driving the browser to capture those transactions and then spinning up multitudes of many browsers to actually generate the load as part of this. So the pros of this is you're abstracting that, you're insulating the tester from going through the protocol because you're driving the browser, the client, to go through those transactions. And as a byproduct, and when you're spinning up the load, you're actually using real browsers to generate this. And so this notion has been in the market for a while, I would say 10 years maybe. And maybe, Joe, you may have more insights saying maybe it's more or less. But to achieve this in reality is hard, primarily because many tools would say spin up to like 100 or 200 browsers. But if you want to scale up to anything larger than that, we're going to be consuming a lot of resources, so we would recommend not doing that. So with Load Ninja, essentially, we've been able to perfect this mechanism to essentially ephemerally spin up tens of thousands of real Browsers on the fly while still being cost effective. Right. So this is this way we're able to truly generate a browser based script and scale that up to tens of thousands of real browsers. So that way, you know the problem you mentioned, Joe, where you know, you as a tester had to create two different scripts, one for doing the regular performance test. And then if you want to do a GUI based load test, you'd have to create a separate script, run that up to say 50 browsers. But you can go beyond that now. You can just run one script, generate the load up to tens of thousands, you know, whatever the requirement you have, and get the results based on the scripts that you have.

Okay, interesting. My next question was one of the concerns we had back in the day, like you said, all the load was on the back end, but now it's on the front end. We worried about if you're limited to the client side resources to get proper performance testing metrics. So how do you generate a load where the performance isn't impacted by the spec of a local machine?

Yeah, so we have essentially, we have our cloud infrastructure where we have a bunch of these dedicated machines that we've allocated of.

Varying capacity in order to generate the load, and we employ lambda functions in order to ephemerally spin them up as needed. This is essentially how we've been able to do that at scale because we have a huge pool of IPs that we essentially leverage as part of this. We also show you, for example, in our test results, we also give you access to the load test engine metrics just to make sure that you're, you're able to monitor the health of the load generators and making sure that they are actually performing as intended and they're not the bottleneck for some of the results that you see as well.

Very cool. So are there any approaches people need to think of when they're doing performance testing from a browser perspective rather than a protocol level when they're developing their scripts or their plan in general, their strategy?

Absolutely. I think the strategy is overall the same in terms of how you're from a high level. If you look at the jobs to be done on actually going through the performance testing method, still follow the same approaches, which is first you start by gathering the requirements, then understanding what the end user transactions are, setting up the test plan, creating the scripts, making sure it plays back successfully, setting up, configuring your load test, running the load, and then analyzing the results. I think it's important to just have a different mindset when you actually come in and analyze the results. Because the metrics that you may be getting from in say a browser based approach may be traditionally different than some of the metrics you're used to in say a protocol based approach. So I think that's where I think testers need to be thinking through a little more and seeing what's the best approach to what they're trying to do. So in say, for example, when a KPI that we recommend is always starting from looking at the results from the end user perspective, you'll start by looking at how long it's taking for all of those those browsers at any given point in the load test to go through all the actions that you've defined your script. Let's say, for example, at 50 seconds in your load test, you realize that it takes about, for one virtual user, it's taking about 10 seconds to go through a specific transaction. This one user, because it's a real browser.

Is just like a real user coming in. So you can start by identifying certain trends and saying, okay, if this is how my end users, if this is how my real users are actually using the application, where is it that I'm finding some anomalous trends in terms of whether there's a spike in the time it's taking to execute those transactions, identify the pages that may be causing some degradation. From there, we typically recommend going to the second level which is the navigational timings. This is another advantage of going through the browser based load testing approach because you essentially get browser based metrics that typically you may not find in a traditional protocol based tool. So things like, for example, you may realize that you may try to identify the DNS time, the DOM load time, the time it's taking to process all those images, the fonts and everything else across the network and rendering it on the dom, any event time that may be associated with the browser trying to execute a JavaScript function, the response time and the time to first byte and other metrics as part of your overall navigation timings. You may find out and see if there's any interesting trends that you're identifying.

And correlate that with the anomalous trends you identified in the durations it's taking for the virtual users to go through the transactions. So for example, if, say for example, example the DNS time is really high. So maybe there's an issue in the DNS lookup over there. Maybe the download time is taking a really long time. So does that mean that there's certain resources that's taking a while for the browser to process at any given instant. And if you identify certain key things, maybe the next level right after is to see how long it's taking for the browser itself to process all those resources and essentially see if there's any anomalous trends right over there as well. So that's the third level that we typically recommend. And once you go through all three levels, you may be able to identify certain key trends as to what may be going wrong. How do we overcome this? And so on. The good thing over here is the results that you get back as well in the browser based approach is typically the results which developers are used to using when actually developing and building applications. So it's easier to diagnose, it's easier to essentially have that conversation with the developer to say, hey, this is where I think the issue is, why don't we work together to solve this? There's that translation, there's nothing lost in translation or the chances of getting lost in translation is lower.

I guess another problem is you have all this data now and I found back in the day that the analysis piece was the hardest part. So now that you have these real browsers, all these metrics being generated.

Is any sort of intelligent piece of technology on the back end that analyzes it and points you to the places where there are potential issues?

That's a great question. I think, I think that's something which is a huge opportunity for us as an industry to try to overcome together.

There's a lot of improvements that are happening in the machine learning space. I think observability is something which overall people are becoming more acquainted to, which is essentially understanding the overall applications, functional performance.

Functional correctness and performance across the entire stack all the way from the application infrastructure, from the front end all the the way to the back end. You're collecting all of these metrics from different tests, different logs, different monitoring solutions, et cetera.

There are solutions that allow you to identify the sources of all of those data and plug them in and visualize all of them together. We have the open source elastic solution, for example, that allows you to source in all of this data. This is where I think maybe improvements in machine learning can really help us look through all of this voluminous amount of data data and assist us in essentially correlating and identifying potential sources of issues. Right. And many times you may notice like, you know, while doing application performance monitoring, you'll identify something that is erroneous and that's when you tip that typically informs how you go through your performance testing or maybe your functional testing as well at some point and vice versa. So this way you can essentially really, by having maybe an overview of, you know, your pre production test as well as your post production tests and having them in a centralized location and being assisted with machine learning and artificial and eventually artificial intelligence to identify the sources of issues. I think it could be, it's definitely an open opportunity for the QA market, QA companies as a whole to come together and identify how we can solve that. And I don't know Joe, if you're seeing any interesting trends in that as well.

I mean, absolutely, it's, I think it's the ideal use for machine learning because you actually have logs and data and the correlation, it could do that for you automatically. So I think it's a perfect fit. I think people force AI on functional automation. I think for performance testing it's a no brainer.

Yeah, absolutely. We have a feature right now that we're doing which is essentially a sensei capability in Load Ninja that essentially will analyze all of your network based information like your network timings and the time it's taking for your resources to download and process on the browser, find that across different, at different points during the entire load test, analyze that and essentially using a machine learning algorithm identifies certain key trends. It's still, we just released it, it's still in kind of its early stages right now. And what we aim to do is make it self learning and allow testers to give it input to say, you know, whether this was helpful or not and essentially improve on its algorithm. That's the vision for us.

Very nice. So how does this work now? Is it just taking a Selenium script and just like running it in a service, for example, in the cloud and then you're just running it on multiple machines and then is that what we mean by Load Ninja? Is Load Ninja a proprietary type protocol that's rendering the browser.

No. So Smart as a whole has been a pretty.

Vocal support of open standards. And so yes, our scripts are actually built on top of Selenium, which means.

Relatively straightforward for us to like port across different other frameworks as well, because Selenium is pretty friendly in that regard. At least that's my opinion. And so yeah, so fundamentally from a high level it is built on top of Selenium, but the way we're actually using that Selenium script and then generating load across real browsers is a proprietary mechanism that we developed within specifically for Load Ninja.

So I know what some of These other browser based solutions where you can run in the cloud, you have the ability to watch a video playback. So with load ninjas, the way to say, okay, let me see my application when there's 100 users using it and 200 just so I can visually validate it, is that an option?

Yes, absolutely. I mean this is something which is so fascinating. Like when we were developing this, we showed this as part of our beta to many testers whom smartware is a big testing organization. So we do have some valued loyal customers whom we shared the product to before we launched it. And every time we showed this capability of allowing them to visualize active views inside in a running load test, it always valved them because it was giving them full understanding of how their application is behaving. And if there's any sort of degradation in performance, they were able to see that live. That's when we realized that's such a key powerful capability. We do have that, that ability right now. So you can essentially we pick four random virtual users for you to actually visualize as part of your load test. We do aim to add more improvements to this capability, allowing people to maybe at some point be able to define what are the specific virtual users as well they want to visualize. The challenge is one thing we have to overcome is we have to stream all of those images as part of the load test.

So that you, the tester can visualize them. So doing that at scale for maybe 100 to 205,000, 10,000 virtual users is something we have to overcome.

I guess another issue people have with functional automation is they try to automate everything. So I'm just worried now that you can do it through a browser. Have you been seeing people try to write a performance test for everything? I believe you had like a framework, an 8020 framework you showed at Perf Guild to try to help identify the right transactions. Any thoughts around that?

Yeah, I mean I. It's something which I employ, I think I employ and when I was, when I used to build applications, I employed. It's something which I follow to this day is, you know, the 8020 rule. Because if you think about it, you know, most, if you know, if you have your QA manager or maybe your product and lead or your engineering manager, when they come into a QA organization or to a QA tester, they typically the go to thing will be just like automate everything, right? Find, find everything and just, just automate all the tests. It's a hard thing to achieve in reality, which is why you Know the industry exists and there's a great community that is supportive and helping each other. You can't automate everything, at least not immediately. It takes time to build up your entire automation, automated tests. And the framework which I've typically followed and which is, which I've seen being successfully employed is kind of like the 8020 rule, which is, you know, at the end of the day, 80% of all of your erroneous results or your key capabilities of your application is essentially accessed by 20% of the transactions that exist within the product.

It's the same thing. There could be like 80% of the impact could be essentially brought in by 20% of the actual overall functionality of your application. Most people don't use every single widget that exists on your application application. Most people don't click on every single button. So you know, just identify looking at it from that perspective and then using this framework to one, identify key transactions based on usage data. And also simultaneously looking at the business value of that of those transactions as well could inform you into prioritizing what are the transactions you may need to be creating automated tests for as well. And eventually of course the goal for any test team I think is to have a full fledged automation. But as teams are getting more agile, as developers are building more functionality and releasing it faster, it's up to the QA teams as well to keep up with them, but also have a prioritized way of identifying the transactions that they need to be starting to automate as well.

Great advice. A thought popped in my head. Another thing I think people would be worried about with UI automation and browser based automation is they say it's flaky. So I'm just thinking when you get to 100, 200, 1000 browser based performance tests, is there any functionality built in to make the test more stable?

Yeah, I mean it's definitely a valid point and it's something which definitely we've seen people as well ask us about it and there are certain best practices that we would recommend. First is when you record your transactions, play it back five to seven times, four to five times, just to make sure it's like everything's captured correctly. We do have capabilities specifically within Load Ninja to help the tester in this process. One is we have an engine that's built in that essentially, you know, there's multiple ways you can identify an element on the UI Specifically for Load Ninja. We use multiple heuristics like the shape and size of the object, the XPath CSS, the position of the object, on the page, the DOM and the DOM link path. So it could be multiple any of these heuristics we have an engine that essentially will look through all of these different ways to identify an object and essentially scan the right way to do it. And whatever the first positive signal is, pick that as part of the script.

You can enable this AI capability on or off. And this essentially if you have enabled this capability, load ninja will insulate you from essentially insulate the tester from having to go through identify what is the best way to recognize the object as part of the script and do this for every single object record that exists on the script and allow you to just focus on recording the transaction. So.

This capability in addition with some key best practices like having the tester, once you record the script, just play it back two to five times just to make sure everything's working and then it typically works for running this at scale across tens of thousands of browsers.

Before we go, is there one piece of actual advice you can give someone to help them with the performance testing efforts? You've already given a few, but any other ones that you would recommend and let us know the best way to find, contact you or learn more about your awesome product load Ninja.

So the one advice I would give in addition to some of the things that we discussed, Joe, is to have an open minded approach towards performance driven development. That's a fancy word of just saying have an emphasis focus on performance as you build applications in today's day and age. At the end of the day you want your app, your web applications driving business, or your APIs are driving business or whatever you're building at the end of the day is driving business and consumers are becoming more and more impatient. Right? We what typically in the past 10 years ago we would be happy with waiting for a video to load in 10 seconds. Now if your video doesn't load in 2 seconds you're going to go to the next page, next video you're going to go to the next year. The switching costs are getting lower, right? So it's important for software teams in general to have an approach, to have an open minded approach to performance driven development. So start thinking about how you can do this right from I would say the code optimization layer all the way till when you're actually defining and building the UI and then launching it and employing all those different frameworks that may or may not add up to the overall loading load load time of your application and have an open mind towards, you know, what are the best practices, what are the best tooling that exists.

UI based load testing doesn't solve everything. Protocol based load testing doesn't solve everything. There's always going to be a give and take and it's really important for you as the testing team, as a testing organization to define what is of most importance to you. Which is why the operators principle always has helped me to identify what is of key business value to me and help me prioritize my tasks accordingly and then work towards automation as you go through it. But one Performance testing should be accessible to everyone. That is Load Ninja's mission. You know, it should not become be a centralized. It should not be something which is in the hands of few. Everyone should be able to do performance testing to at least some degree and be successful with it and get the right diagnostics and as part and hopefully achieve a full motion of end to end performance driven development in the application lifecycle. To learn more about Load Ninja please do visit loadninja.com you can always add me on LinkedIn. You can find me as keshavasudevan. You can also follow me on Twitter. My Twitter link Twitter handle is kaishinpoint K E S H I n P O I n t and I'd be happy to have a conversation on testing in general or software development in general. I love talking about all things code and testing and product management and development and so on.

Thank you Keshav for your performance testing awesomeness. Thanks to everything value we covered in this episode. Head on over to testguild.com plakeautomationp06 and while you're there, make sure to click on that Try it Today link under the exclusive Sponsor section to learn all about SmartBear's awesome performance testing solutions like their new Load Ninja to learn about some of those features that Keshe have actually shared with us in this episode. So that's it for this episode of the Test Guild Performance and Site Reliability Podcast. I'm Joe and my mission is to help you succeed with creating ant and automation awesomeness. As always, test everything and keep the good good. Cheers.

Thanks for listening to the Test Guild Performance Podcast. Head on over to testguild. Com for full show notes, amazing blog articles and online testing conferences. Make sure to subscribe to join the Guild and continue your testing journey. This has been a Joe Calantonio production.