Performance KPIs with Keshav Vasudevan

Welcome to the Test Guild Performance and Site Reliability Podcast where we all get together to learn more about performance testing with your host, Joe Calantonio.

Hey it's Joe. And welcome to another episode of the Test Guild Performance and Site Reliability Podcast. In a few days I'm actually turning the big 5.0 so I'm taking the rest of April off. Because of that I'm going to my back catalog of all my awesome perf Guild sessions we we had from previous conferences and making those some podcast episodes to share with you some performance awesomeness without me having to interview people because I am away for the month of April. So this session is on performance KPIs by Keshav, who was at that time the product manager at SmartBear. He's moved on since then, but he shared a lot with the Guild on what are the critical KPIs you should measure in regards to your web application and how it appears in the eyes of your customers experience. So this talk is going to go over a bunch of real world examples to understand different protocols and browser based metrics to measure how well your application's performance is doing across the UI and API layer. This episode is brought to you by SmartBear. Listen load testing is tough. Investing in the right tools to automate tests, identify bottlenecks and resolve issues quickly could save your organization time and money. SmartBear offers a suite of performance tools like Load Ninja, which is a SaaS UI load testing tool and Load UI Pro, an API load testing tool to help teams get full visibility into UI and API performance so you can release and recover faster than ever. Give it a shot. It's free and easy to try. Head on over to smartbeer.com solutions performancetesting to learn more.

Thanks for tuning into my talk on how to paint a cohesive picture of your web application's performance through measuring the right KPIs. I'm Keshavasudevan and I work as a Product Manager for all things performance testing at SmartBear Software. In my spare time, I love writing. You can check out my blog guru to school and you can also find me on Twitter. I love hearing from people about how they're building their applications or delivering great quality for their customers, so feel free to reach out anytime. Now. I want to preface this talk by mentioning that this is in no way an exhaustive list of all the metrics. There could be so many different metrics you could measure in order to understand your application's performance. Performance testing is an art.

There's a Science and there's a tactic to it, but there's also a creativity. There's a creativity about it. In order to get to the right data and to the right understanding of your application performance.

I want to start this talk by presenting a case for better performance. Performance testing is more important than ever before. We have so many more businesses going online, especially during this times of the pandemic, and customers given so many options that we have, we're spoiled for choice. Which means the switching costs are lower than ever. For right. There's a 7% potential drop in sales for every 100 second delay in app performance people leave. There's about 40% customers abandoned a website when it takes over 3 seconds to load. Excuse me. And there's a potential abandonment rate of 87% if an actual sales transactions takes 2 seconds longer. Right? So that's how impatient we are. Which means as people delivering great software, we have to account for performance testing. We have to make sure we're delivering the best quality. So really, when we are advocates of the end users and to their experience itself, before we even get into the metrics, it's very important to understand what are our testing goals itself. Why are we even doing this? It's one thing to say, hey, I want to just deliver the best experience. But as performance testing, you need to have some clear goals. And some common goals could just be determine the responsiveness and the reliability of the system. Maybe you want to make sure that your application is meeting the SLAs that customers have come to expect from other competitors or other players within the industry. Maybe you just want to determine the system's capacity, which is the maximum amount of load it can take before it doesn't meet the SLAs itself. Or it could also be that maybe you want to handle growing workloads, right? Maybe your traffic team is showing you that you're steadily increasing the number of users logging in or signing up for your service and you want ensure that your system can handle that. Or finally, you just want to diagnose some performance issues or identify bottlenecks. So just start with identifying those goals and then figure out what are the metrics you need in order to measure how close or how far you are from those goals. So performance metrics are just that. They're used to calculate the performance parameters and help you identify leading and lagging indicators to measure against your performance goals. Now leading and lagging indicators, you know, there's like economic terms, but really leading indicators are just metrics that tell you the actual that are more predictive of the system's success or failure, right? Whereas lagging indicators, they are not necessarily predictive, but.

They are the result of systems failing. For example, if say your DNS load time, your DNS time was really high, right? That could be a lagging indicator because it tells you that the DNS is high, but that is not necessarily the cause. The cause could be maybe there was poor network connectivity, or maybe the DNS server and the client who's accessing this page are really far away, for example. So it could be that.

The leading indicator, the cause of this, is different than the actual result of this. By definition, leading indicators are harder to measure, which is why we use lagging indicators to get a sense of what's actually happening in the system.

Before you actually get started with the metrics as well, you need to first understand how is your system performing when there's no load? What is the ideal case when your system is meeting the sla? So always prefer starting your test with one virtual user or just you, you know, just accessing your application and measuring and noting down what are the baseline metrics. So then when you actually do the load test to.

In, in service of your goals, you can compare the results with the actual baseline itself to determine what's breaking, if there's any degradation, and to diagnose the actual issue itself.

Now let's actually dive into the metrics for us to identify the right metrics. We start from the end user experience. How is the end user experience affected when your system is under load? And then working our way towards identifying what's causing a potential degradation when the system is under load. So let's say we are the end users over here when we're accessing all of these web applications across the browser. So let's say you are clicking on a link and that's taking you and that's sending a request across the wire. There's going to be a DNS server somewhere in between and that's going to then identify the right IP and it's going to send that request to the actual app server.

From the app server is where the in the app server is where the actual processing of the request will take place. And finally the code gets, finishes execution and then is sending back a request response across the wire and is sent back to the client. Once it's sent back to the client, the DOM or the browser receives this HTML content and begins processing it and constructs the actual dom. It constructs the skeleton structure of the page. Finally, once the HTML is finished Processing and it then goes on to render the actual page. And across all of this, there's different events that are being captured by the browser and sent back for people to measure and for the browser to know when to go to the next step itself. So this is the typical transactional flow. Now, when we determine that something is broken, that the end user experience has degraded, we can now start diving into some of these different metrics and see how they are when compared to the baseline. So let's start with the initial metric when the initial request is sent itself. So some network metrics across this would be, for example, the DNS time, right? This is the time spent performing a DNS lookup, which is obtaining the IP address of the website from the actual DNS server. Now, if the value is high, it could indicate that there's problems with reaching the DNS server and retrieving its response. Or maybe there's some poor network configuration. Or maybe you want to just contact your DNS provider, right? There's a ton of them available right now. And whoever your DNS provider is may be able to assist as well to figure out what's happening.

The connect time is the time spent establishing a connection to the web server. After the DNS lookup, it's the TCP handshake phase. Again, if this value is high, which is, you know, the potential lagging indicator, it could indicate that there's some possible routing problems, maybe there's bad configuration or low efficiency of the server bandwidth itself, right? So that there could be those kinds of issues over here.

You can then go on to the second phase, which is the actual application processing itself. So application metrics, this could be, for example, a very famous metric, which is the time to first bytes. This is the time spent waiting for the first byte of the response from the server. This includes, for example, processing the request. Maybe they're accessing the database and then finally you're selecting and generating the response itself.

Then there's the response time, which is the time to actually send back that transaction.

Since when the client sends the composed request until when the response is received by the client or gets downloaded by the client. If these values are high, it could indicate that there's a high server load with the database queries. Maybe there's some bloated web pages, or maybe there's some memory leaks. It could also indicate some caching issues or maybe a lack of caching itself. Caching is useful to deliver content faster to customers because, you know, CDN providers help help with this. So, you know, if Maybe your if your response times are really high, maybe it could also be a case where you want to talk to your CDN or see if there's some issues in your cdn. If you are, maybe it's the time to think of purchasing a CDN or getting a cdn, right? There could be some CDN issues as well. Over here.

Application server metrics like throughput could also be useful. This is again, it's a potential lagging indicator. You know, it helps you determine the simultaneous requests or the transactions per second your application can handle. The TPS can correlate with response times if you are sending your requests sequentially or.

Consequent cycles, right? In such a case, longer response times equals longer term transactions per second, meaning your system is not able to handle.

As many transactions. So longer response times equals lower tps.

It's not necessarily this correlation. If the requests are not sequential, and if they're all happening simultaneously or in parallel, then you won't be able to establish this correlation as easily. This example on the right over here, for example, shows you how the TPS clearly breaks when the system's capacity is reached. One thing to remember is that a lower TPS doesn't necessarily mean that there's some problem with the application server. It really depends on how you compare that with the baseline, which is why it's an art.

Because maybe there's again, there's some caching that's going on, which means automatically the server is doing less work, which is a good thing in this case. In the example over here, it's not necessarily the case because you're seeing the TPS build up to a certain point and then drastically drop.

Error rates are also useful and really gives you a quick sense of understanding if there's any 400s or 500s or unnecessary 300s that are happening for some of these requests. I typically think that there's never been a case where there's not some percentage of errors that happen, especially under strenuous amounts of load. But again, you have to figure out, as a company or as a team, what is the threshold of risk that you want to allow for in your response layer as well. So it helps tell you how many failed requests are occurring at a particular point in time of your load test itself.

There could be some health metrics as well that could be useful, right? Maybe you'll want to at some point realize that given how much traffic that's coming in, maybe you want to increase the physical memory available to process some of these requests. Maybe there's some Processor usage optimization, that can happen. Or also some disk time as well. You know, if there's, there's a lot of time that the disk is busy executing read or write requests. So there could be some health metrics as well that may be useful in order to understand what's causing any performance.

Once it's sent across the wire. Then we can look at some of the DOM processing or page rendering events as well, or metrics as well. The DOM load time is useful, and again, this is something which the browser automatically collects and many load testing tools help identify these and capture them as well. This is the total time it's taking to load and construct the dom. The DOM is considered completed when there's a specific event, the DOM content loaded event, and that's when it starts.

This is when it constructs the dom. But it's important to remember it doesn't mean that the CSS and any sort of, you know, events or JavaScript events have finished. They may still be processing at this time. It's only when the onload time is completed or the event time is completed. That's when you know that the HTML is fully parsed, processed and rendered and includes all sort of scribe scripts and style sheets as well that may be a part of this. If these values are high, it could indicate maybe there's some heavy client side resources. You know, maybe your CSS scripts are kind of, your CSS are kind of bloated, maybe you have some big images or maybe you're using some sort of image repository or other sort of file asset repository that's not able, or some dependencies on third party services that are taking a while to process those requests. Right? So there could be some useful identifiers to figuring out what the actual issue is itself. Now, of course, if you want to go in full detail into some of the other client side metrics, there's obviously the W3 resources over here, which is the official World Wide Web Consortium, and this is where you can take a deep dive into some of these navigation timings itself.

Now, we can look at some real world examples over here and I've taken a few examples of some of our customers who are using Smart Bears load testing tools like Load Ninja. This is an example of one case where.

The chart over here that you're seeing over here tells you that for a specific step, which is a collection of actions on a page, or it could be the page itself, how long does it take for different browsers to go through and execute on those actions? So, and this is, we've done 150 virtual user load tests for this example. Now over here you can see that as there's more and more virtual users coming in at a certain points in time, the time it's taking for the browser to go through all those actions. And remember, these are all real browsers coming in and going through those actions, we're seeing some spikes. Like you can look at it at 1000 second time mark or at the 1500 second time. Now when we look at the navigation timings over here, we're able to identify some correlation as soon as those spikes occur. We're also seeing that the response times are really high over here.

When we dug in a little more for this specific page and looked at the application's code, what we found out was that there's functions with inefficient query logic. If you recall, if the response time is high, it could mean that there are some poor database queries that could be happening, for example. So this was an example where such a thing really did happen for our customer. Another example is this case over here for a different page.

And over here what we did was we looked at a specific step or a specific page that contains these actions. And what we saw was again, when there's low virtual users, when there's about less than about or just about 50 to 70 virtual users, the system was behaving as expected. It was taking just about five to seven seconds to execute all the actions on that page, which was meeting the sla. But as soon as we went a little higher, right when we reached up to that 150 VU mark, the system started taking a lot more time to go from the first action or the first event all the way to the final action. So again, what we're doing over here is we're looking at the end user experience, which is what this chart shows us, and then seeing how that translates into identifying what's the issue that's causing this. And right over here, we notice that.

As there's more virtual users coming in, there was a high DOM load time and the time the first byte was starting to peak or get higher and higher as the system was under load. When we dug in a little more using the network Visualizer within the load testing tool, what we found out was.

When the system was behaving smoothly, when there's just about less than 10 virtual users, when it was meeting the SLA. You can see over here how the different resources on the DOM, which are constructed by the browser for the dom, how they were behaving and you're seeing how the response times and how that waterfall is. However, when the system starts to have more virtual users, these metrics started to fluctuate and take a long time to process. There are many services and resources that took way longer to process under load, which helped us identify these services and try to fix them and optimize the response times for some of them as well. So this is how you can work your way from the end user experience, which is what the script durations chart was showing you, and then work your way towards identifying what could be causing this issue.

And ultimately it's all about using the right tool to gather these metrics. And really the right tool depends on what you're trying to do. There could be the protocol level load testing tool to gather these metrics we shared, or it could be the browser based load testing tools itself. The browser based load testing tools are great if you have like, you know, a lot of client side logic, right? So if there's like a lot of React or Angular.

Applications, for example, where there's heavy client side asynchronous calls happening, and in such a case, in the browser based approach, there's very little programming because all you're doing is driving the browser in order to capture those actions. And as a consequence of that, you're also generating really accurate load on your systems because it's really the truest form of load available and it helps you capture some complex user flows as well. It's great for teams, especially teams who are more agile and who have a lot of developers or functional testers, for example, doing a lot of load testing. Or maybe there's a very agile performance testing team as well that's keeping up with the team that's delivering a lot of new updates to their web applications. So some examples are like we have the open source flood element, there's smartware load ninja tools that helps you do this at scale without having to maintain the servers. There's of course also true client which is from Micro Focus that allows you to do this as well. There's a protocol based load testing tool. So and this is where this would be better, this would be great for applications that have less client side logic and are more traditional. And this is useful for testing teams that are more methodical and centralized because even though it would take a lot longer to create the tests.

If the application that is being developed and updated is not necessarily being updated every week, for example, then these tests are easier to maintain as well. And it's great for really capturing request response, traffic level metrics. Some examples of these tools are obviously LoadRunner and JMeter, the popular open source load testing provider as well. So yeah, so this is the end of the talk. I hope you found this talk useful. This again, as I mentioned, is in no way exhaustive of the different metrics you could use. It's really Performance testing is an art and you know everyone in this community is an artist and I would love to hear from you. If you have any feedback or additional questions or just, you know, have a different way of doing performance testing or understanding your metrics, please feel free to reach out to me.

Thank you Kashev for your performance testing awesomeness. For links to everything of value we covered in this episode, head on over to testguild.com P63. And while you're there, make sure to click on the Try Them Bolt Today link under the Exclusive Sponsor section to learn all about SmartBear's two awesome performance test tool solutions, Load Ninja and Load UiPro. So that's it for this episode of the TestKill Performance and Site Reliability Podcast. I'm Joe and my mission is to help you succeed with creating end to end full Stack Automation awesomeness, which includes performance and site reliability. As always, test everything and keep the good. Cheers.

Thanks for listening to the Test Guild Performance and Site Reliability Podcast. Head on over to testguild.com for full show notes, amazing blog articles, and online testing conferences. Don't forget to subscribe to the Guild to continue your testing journey.