How to Automate Your Performance Test Results with Joey Hendricks

Welcome to the Test Guild Performance and Site Reliability Podcast where we all get together to learn more about performance testing with your host, Joe Calantonio.

Hey, it's Joe. And welcome to another episode of the Test Guild Performance and Site Reliability Podcast. Today we'll be talking with Joey all about how to automate your performance test results analysis. Joey is fueled by a staggering amount of caffeine and dumplings and he has over three years of experience helping teams on the quest to protect software applications from those pesky performance problems. I know you all have experienced yourself and you don't want to miss this episode because if you want to discover a better way to take all the raw results of your performance test efforts and really make sense out of them, this episode is really going to help you. We're going to take some talking points we got from his session at this year's I think it's Tricentis now. It used to be Niotis pack and we're going to dive in a little bit more on how to help you actually analyze your results better. So you don't want to miss this episode. Check it out.

This episode is brought to you by SmartBear. Listen, load testing is tough. Investing in the right tools to automate tests, identify bottlenecks and resolve issues quickly could save your organization time and and money. SmartBear offers a suite of performance tools like Load Ninja, which is a SAS UI load testing tool, and LoadUI Pro, an API load testing tool to help teams get full visibility into UI and API performance so you can release and recover faster than ever. Give it a shot. It's free and easy to try. Head on over to smartbear.com solutions performancetesting to learn more.

Hey Joey, welcome back to the guild.

Thank you Joe. Glad to be here. I do, I do love dumplings and coffee and a bit more higher caffeinated cola. So.

What kind of dumplings? Chicken dumplings. What is it?

I love chicken dumplings. Whenever you're in the in the neighborhood of Maastricht, hit me up. I know a very good dumpling restaurant around here so I am a foodie.

So I'm definitely pick you up on that for sure. That's awesome. So Joey, it's been a while since we spoke. I think was last time was episode 56 and it was about your open source project Quick Potato. Anything new going on with that project?

It's actually going great for performance testing Library for Python. When we talk about testing it's kind of grown in popularity a bit. I got some good feedback on it, it's grown, become technically a bit more better, which I like. So it's growing at a steady pace. And when I have time, I edit and continue working on it, which is a great little repository. So if you have any. If you ever want to look for a profiling tool for your Python project, don't forget to hit up Quick Potato and find it on my GitHub.

All right, so at this JS pack event, I think you spoke about the topic of how to analyze your performance test results automatically almost. And you have some really cool things going on. Just curious to know what made you want to submit that topic.

To be honest, at apg, we've been doing this for a very long time already. We religiously store all of our test results. We do that because we want to see trend lines and we want to do analysis on our data. So having a wealth of information available to you to analyze all of multiple releases and look back and see what changed is amazing. And one of the things that really triggered me was to start thinking about how can I automate my. My result analysis was mainly that I was thinking about my robotic vacuum cleaner, because I really love my. So my vacuum cleaner broke and I bought this new vacuum cleaner on the recommendations of a friend and I was like, yeah, these automated vacuum cleaners are great. And I was very skeptical because if you work with it, you know that it's never going to work the way you want, but it works absolutely great. I love the thing, it cleans my apartment, it keeps everything nice and tidy, and it only triggers me when I want to be notified. And the same is with my testing. I only want to be triggered when I'm notified. If there's no change and there's nothing wrong with the application, why should I? Look, just let me go and let me do my other stuff, which I have plenty of other stuff to do, but I just want to be triggered when it's relevant. So that's kind of where this topic kind of came from. It came from a robotic vacuum cleaner, which I bought because my vacuum cleaner broke. So it was like a strike of, ah, let's go.

It's awesome. So I guess I always say this on the show when I was doing mainly performance testing really, really like 20, 20 odd years ago, I always found creating the scripts were a lot easier than actually analyzing the data. And so how do you get from that to actually utilizing some methods to help you automatically analyze it so you're only, like you said, looking at it when you need to?

Well, first you know, everybody has, I think most people that are already doing continuous performance testing, they have that testing process in place where the test get automatically kicked off. It's in the pipeline. So it's part of that quality assurance process. And if you're already at that step and you need to export your raw results into a database or on a file share or however you want to do that, you already are at that point where you can continuously do that. So you can be triggered. After a release that goes through the pipeline, the performance test kicks off and then export raw results and compare them to the previous one because you've stored everything. So if you're at that point, you can start doing that, which is amazing. And if you then want to continue and start analyzing it, because you can just go back, you can just look at the previous test that you've done and you have the new test. So if the previous situation against the current situation, and you compare them and you can use a lot of statistical tools that are available to us to start doing this. And what I typically see while people do this is they compare means or they compare averages, or they just run SLAs over them like, can't be higher than five seconds. If it's that, just kick off the test, stop the test, that kind of stuff. Which is okay because obviously SLAs are important, but it's not actually something that we always want to be looking at. We are more looking at change. And the difference, if there's something relevant for me to look at on the second note, is sometimes that SLAs are not always that well defined that you can really fail a performance test on them. Usually they're much higher than what the actual response time really is. For example, if your application generally responds within a second and you have an SLA of 5 seconds, just high balling some numbers.

So I'll be honest, I probably was a terrible performance engineer after seeing your session because all we really did was look at the averages and we even did like 95% average. And then we'd run it like three times. And if we had one that was really an outlier, we'd say, oh, let's just ignore it because it's outside the scope. But you made me rethink that. I think you had a quote from one of your mentors is averages hide the ugliness of an application. Can you talk a little bit more what that means? Because I think a lot of people, maybe they're like me, old school where they always use averages for performance.

I don't dislike averages. Let's get this off the butt. Averages are great because if I show an average line to a business owner, that makes more sense to them that clicks. But when I, you know, we're technical people, so we want to see everything. And I have a great visualization of how that the impact of an average, it's on my GitHub, it's something that's also shared online on LinkedIn a lot. And there's a test that I've done where I plotted the average line and then I plot over it the exact same data I plot over the scatter plot. Like every single measurement, I see a way more interesting pattern starting to fold out. So the averages are great for reporting to stakeholders, but they are very difficult to just base your entire analysis on because if you have all the data available, because generally we have everything available, especially if you're running a load test on it, your load test tool will record everything. You can just export that and you can see the entire raw data. So I would recommend you to check out my GitHub and just see the actual visualization because I think that visualization really speaks about the importance of raw data. And Stein Schepos, he always says like that it hides the ugliness of the application because of the averages. It starts to going. Patterns start to become smaller, less visible, or even gone completely. And you can see that perfectly in that example that I have. And because of that, it hides a bit of how ugly the application really is. It makes it a bit more prettier than it really is. So for a business owner, it's okay to share it with, but to solely base your entire analysis on just an average line, that's quite false. To just go on a limb on that because you're not looking at the complete picture, which you should be.

So I, I guess the challenges there, it could be if you have one outlier or a few outliers, it's hard to determine what actually caused there. If it's a real issue, how do you know you're not going down a rabbit hole at that point?

I think it's also a bit about context. So if you know, if you know what you're doing right and you know how the system is behaving, if you generally get like insane outliers, then there is a problem. But like also if I take APG for example, we have customers that have just way more complicated calculations going on than your average Joe. And because of that we have complicated customers. You have normal customers or you have very small customers and very complicated customers with a lot of history. And a lot of problems inside are not problems, but more like complexity around them. Because of that, it just takes longer to calculate them. So most load testing tools will also allow you to see which customer number you maybe have used in a request, and then you can also verify if that outlier is a correct outlier. It could just be a very high outlier because you're just having complicated data behind that. And that is just doing a lot more work.

Absolutely. So, you know, you did talk a lot about raw data. What is the raw data? Does it also have things like metrics from the different monitors that you're monitoring? You're able to do correlation or is it just raw data? What is usually included in the raw data that you're sending to your database that you then use to create these graphs?

Well, raw data is a general concept. It's basically that you take every measurement, so it depends on what you take measurements of. So usually for me, the most important thing is user experience. I really care about how people are experiencing our application and how it's performing. You know, I really care about that. I care less about CPU statistics. I care about them, but they're more a secondary measurement to me because if the CPU is blazing at 80%, but I'm delivering a stellar user experience with under the load that I want, then everything is fine for me. I do care about them. You know, if you move to the cloud, we have to pay it as like a gas bill. The more you're going to use in CPU usage, then the higher your bill is going to be. So from that point of view, once you're on that kind of system, you also want to try to reduce that, but for just giving a good digital experience, which is why we built these applications. We built them so we can provide our customers with an online service. We do want to give them the best experience possible. And if that is with a higher CPU consumption, then I can live with that as long as we deliver a good experience to them.

So you brought up two points. When you're doing analysis, it all depends on the quality of your data in your testing environment. So how do you make sure you have good raw data? I don't know if that makes sense, but rather than just throw everything, is there certain.

I think the question that you're trying to say here is more like, how can you trust your raw data? Because.

Yes, right, right.

Raw data can be raw data. You can have raw data from an unstable test environment for A stable. But generally what you want to be testing is not how unstable your test environment is, but the change that you've introduced to it. So if you have an unstable test environment that goes like haywire every time you start a test and you get like, you change nothing, nothing has changed. And you see suddenly the average, for example, go for five seconds to seven, eight seconds and you've done nothing, then there's probably something wrong with your testing environment. So before you can even start. And also that's also, in a sense, start with continuous performance testing. Make sure that you have a stable test environment and that it reacts always the same way and only changes when you introduce a difference in them. I think that's mainly the point that you want to establish. You want to make sure that whatever you're doing and whatever change you're introducing to your test environment, that that is what you're testing and that your environment will always remain stable. A good thing to test this out with is just run 10 tests against your test environment and see if they look completely alike. Because if they look completely alike, the test results, and they're exactly the same, so the numbers are within the same ranges, then you're going to have a stable test environment. And that is kind of the cue to start continuous performance testing and to start also trusting the raw data that flows out of that. It's the same with water. You know, you don't want to drink dirty water, so you make sure that's clean. And it's the same with your test environment. You want to make sure that it produces clean test data, which you can trust.

Cool. So I guess that would then create a baseline. And then was that what you would use as your baseline, that you would measure going forward, that anything that runs after it, if it's off the norm, you'd be alerted. Or how does that work?

So the thing is that I care most about is change and positive change. Right. We're going faster, can be bad. It can be that it's very good and we wanted a positive change. So aiming towards changing the system a lot by performance improvements that we've introduced, then we know when we are going to release that we are suspecting a performance increase, but we are expecting a change. But it also could be that we are expecting a negative change. So negative and positive are both changes, and I care about looking at them. So even if the results of difference between two of my tests are positive, then I still want to do something with that. I still want to hold the pipeline and look at it Even though that's good, even though it's positive, and the same goes with negative, I only don't want to look at it if it's the same, if it's the same as always, then I don't care about it. Then just go, there's no change, so there's nothing to really investigate there. So for a positive and both for a negative change, I want to hold that pipeline and I want to look at it manually. And when I look at it manually, I'm only looking at it when it matters. Because it's either that we have introduced a performance increase, we have bettered the system, or the system unexpectedly became better and we have no idea why, because that's also a problem. And if it slows down, obviously I also want to look at it because then we also have a defect. So I really care about change. And if I can spot change and through these kinds of ways, that's great because there's a very lot of ways where you can achieve that.

So to spot the change. So it seems like you have this embedded into your pipeline. So you run a test, it's automatically gone when it does analysis. And then it only triggers alerts or stops the pipeline based on criteria you put in, like what's the technology or tooling you're using to get that?

Statistics.

Oh.

So it's like, to give a bit of backstory on that, I started playing around with these. I don't have a statistical background, so when I was kind of starting with it, kind of was a nitwit in it. I didn't really know what I was. I knew what I was doing generally because I knew math, but not on that level. So I started researching a bit about how in statistics do people find change? So I started looking at them and they generally talk about distance in statistics. So they talk about what's the amount of distance between one distribution, basically a test against another distribution. A test. Right. So if we have two performance tests, there are two distributions of data. So if I want to say how many distance is then between those two performance tests? Well, there are generally a lot of distances that are rooted deeply into the statistics, and they are excellent for finding out.

How much distance is there. And there are two, which I really like. And I'm probably going to mispronounce this for the 15th time, but the Kolagamov Smirnov distance is one of them. It comes from a hypothesis test. So if you, if you remember that from high school math flashbacks, helicopters flying over, it's not fun but they're very powerful distance measures. They're metrics in sense. And you also have the Wasserstein distance, which is also a very powerful metric. So these two metrics I kind of used to see, okay, how much distance is there between two performance tests? And the Kolagomov is one interesting one because it depicts the maximum, or basically the absolute maximum distance between two distributions, AKA performance test. And the Wasserstein kind of gives you the surface area between two distributions. So it gives you the general distance between two performance tests. And they will always be around the same number, so you can classify them. So you can say, if I have my baseline, which is this a line, and then the benchmark needs to hug that line, they need to be as close as possible. And for some visual representation, I would really recommend to check out my GitHub because I have these all in visual animations where they make a little bit more sense to understand them. But because those lines have to be very close to each other, because we don't want distance, we want it to stay the same, we don't want any change. And because of the Wasserstein and the Kolagomov, they stay generally in the same numbers if it's very low, because they're going to stay near zero. And then you can actually classify them. So you can say 0 to 10 is low, 10 to 20 is medium, 30 to 40 is high, and so on. And you could put a letter rank next to that to say, okay, if those numbers are those distance metrics are around these numbers, then that's an S. Or that's an. Like if you take the Japanese letter system where S is super A and you then take the A is then good and so forth, right? And you can give them a rank to them, and then you can steer on a rank saying how much positive or negative change have been introduced. Those numbers, those statistical distances, they can be used for that kind of stuff to just automatically verify is there an interesting change between two tests. So this can become very technical because I'm skipping some parts of it, which I recommend to also read on my readme because there's a lot of other very complicated stuff like cumulative distribution functions, the normalization of the data, all very important factors to consider. There's obviously a lot of Wikipedia articles behind that, where the calculations, they look completely elvish and difficult. But there are a wealth of information, especially if you're trying to automate and make smart decisions around your data.

Nice. So I guess, how much statistics does a performance engineer need to know, though? Are these codified in some sort of function that someone could just use, or do they actually have to program it up in that pipeline to actually use these different statistic methods that you, you just mentioned?

Well, normally if you're in a pipeline, you already have some ability to program in that pipeline. So you could use Python and Python has like, you have the scipy package for Python, which is basically every statistical calculation that you could possibly want in there. So there's no need to actually understand these from a math point of view. So you don't really need to understand how does this equation actually work, but you do have to understand what does this number represent, you know, what does it mean? And for the Wasserstein, that is then the general area between two tests. And the Kolagomo Smirnoff is then the, or the KS value to make it a bit easier on the, to make it a bit easier on me to pronounce it. So the KS value is more the absolute distance between two. So what's the absolute maximum distance between the two tests? So they're great metrics to tell you is this change big, small? Is it concentrated in one space? And you can make smart decisions on that. And you can supplement those decisions with stuff like just checking if the average is slower or faster. So you can determine if your test is running faster than the previous one or slower than the previous one, and making decisions based on that. You can start checking the throughput levels. If you're still taking in the same amount of throughput, you can check the errors maybe some, and you can start also do like a secondary layer where you put your SLAs over it. So if there's difference and it doesn't fit the SLAs, then we have a giant problem. And you can start supplementing those metrics like with other stuff and you can rank those. And I called in science, they have this word for this, and I love this word. And it's called just a heuristic. So it's basically a, a set of criteria that if it really meets those criteria, then it's good. And a heuristic is just a way of making very complicated data like your performance results, right? Or your entire application to make very complicated stuff like that understandable with just a rule of thumbs. So if we know that the Wasserstein is this big, we know that that is an interesting change for application. If the error rates are then out of these number ranges, we have a problem as well. If the throughput goes below a certain level, then we also Have a problem if the average is lower or higher than this, then you can take some percentiles and put boundaries around that as well, or the percentage change between 2 percentiles. So you can really create what you really want. You can really tune this to how you want, which decisions you want to take around your data. So it depends a bit on the context of the application as well, how well this works. For me, this is working very well because I have a very stable test environment. My numbers are kind of in the same area and that allows me to actually rank them from S to F and then tell a business owner, okay, the change that the team has introduced was a F, but a positive one. So they made a massive improvement to the system. Or it's a B. So I would recommend not to release because it has a low impact on the system and it produces more errors, Something like that. I can say that now. And for a business owner, a B, an S, an F, a C is more understandable than here are all these numbers and they'll be looking and it's like information overload for these people. They just want typically an easy, understanding, understandable metric. And we can also, in a pipeline, we can also make decisions with those ranks. If I have an F, I do this, I create this defect, or this team, or I do this. So you could also focus this kind of analysis on every single API or action that you're doing in your test to really verify and say like, okay, if this API breaks, we know this. Okay, this API is broken, it has more errors than normally it does an interesting change in it, performance wise. Let's create a ticket to that team. Just a JIRA defect. You'll look at it and hope the pipeline, the release stops there until that team looks at it and continues to release, either with a fix or that they accept the risk of going to production with a slightly slower or slower transaction or application. Interesting.

So when I think of F, I think bad, but you just said F could be good because it's a big change. But like you said, it's a positive change. So do you do automatic alerting like your vacuum when it was stuck in the rug, do you say, okay, the check in worked, you get an F and with the high level analysis automatically. Or do you actually analyze it first before you alert people? Like, at what point does a human get involved with this?

As soon as it same with my vacuum cleaners. If it kills itself on the rug and it's standing there and it's trying to drag the rug around the Room, it usually takes a minute or two and then it's going to start complaining and I get a notification on my phone to come check out what's wrong with it. And then I see it standing there, so then I know, okay, yeah, I should have been here. So the same is also with my tests. If it really goes wrong, I want to be there. So I'm usually the first contact that goes out and some other teams that get a notification, hey, it's wrong. And also when I look at this from the APG context, we also have teams looking at these numbers. So when they're doing the release, they're looking at these ranks. We generally use a score. So they can also look at the score and see, ah, okay, it's bad. And then this kind of gets fed up to the command chain and then it kind of ends up, they analyze it and they see, okay, okay, we can accept this, this is okay. The benefits of releasing the software now outweigh the performance problems that we might get. And now we're taking a calculated risk and we're no longer taking an unknown risk because we know exactly what is wrong and we can fix it in production later. Or if the risk is too high, we can just decide not to do it right, just halt the release, fix it, work for it on a week longer, and then go to production with it. It really generally comes down to what your organization wants to do and how they want to tackle it.

You know, with automation testing, there's usually a lot of false positives where you're getting alerted, but it's really not an issue. How, I guess, how stable is this approach? And is there ever a time when someone maybe updates your environment without you knowing that makes a difference? That is negative, but you didn't know that a change was made.

All right, that's a very good question.

So generally speaking, if your testing environment is stable and it detects something, right? So if you have a stable testing environment and a change introduced, it will catch that because it's just numbers that change. If there is something interesting and the numbers change, it will happen. Do keep in mind you need a lot of data to do this. So you can't do this on a very low throughput test. Same goes for high throughput tests. If you're like a telco that pushes out a couple million transactions and they have huge raw data sets yet. And those calculations are also going to take longer because they're more data, more numbers to crunch. For me, I'm kind of in the low end when it comes to throughput. So for me the calculations don't take long at all. I can calculate them fairly quickly.

And that's the point which I'm trying to come to is that I kind of guard my test environment so nobody touches it, nobody does anything on it without my permission. So I can safeguard it a bit from weird changes that happened because somebody decided to, you know, let's update all the test data. And now there are different types of. Now some customers are gone, some are there or the kind of information that these customers have changed which impacts the performance for me, for example, because the context of the customer really impacts my performance. Because if I have a database Stan from April this year and the customers that I use in my data set in my test, they for example, become way more complicated and the reaction time goes up because my data has changed. So that's why generally we have rules in place around the acceptation environments for performance that do not touch them unless without permission. Because if you do so, you can impact the test environment. So we create a lot of awareness around that. That's also something generally that you should do. We treat them a little bit as a more special test environment with limited access where people request access to be done to start testing on them if they want to test something against a more complicated test environment than they usually have. It all runs through me then and some other people that do the analysis and say, okay, yeah, you can do that. Nobody's using the test environment at that time. You're not in the way of any performance tests go at it and changes. They also have to get approved by then my team to just make sure that it doesn't impact the tests. And if we do that, we generally test it and usually all the changes that we do in the environment are going to be done a week or two later in production anyways. So we have to make them anyways and we see what the impact is then great.

Also in your session you have really good looking graphs. What are you using to create those graphs?

I use Tableau. I think Tableau is. And I think I am the least powerful Tableau user and the least sophisticated because what I literally do, I either connect to my databases, an Oracle or a MySQL database and I visualize the graphs. This is just raw data stored in there. I visualize them, I get the nice graphs out of it or I just connect to a CSV file and I create the graphs. All the graphs that you always see from me there are either plotted with tableau or I use plotly for Python, if you have to generate for a GIF a couple hundred images, I'm not going to do that by hand in Tableau because that's a lot of work. So I generally use Tableau for graphing if I quickly need to graph something or I need to make dashboards. And I use Plotly for anything that I have to do Automated. So if I want to send something to a team in an automated fashion, I either use Plotly or maybe an APM tool like Appdynamics or Grafana or something like that.

Okay Joey, before we go, is there one piece of actual advice you can give to someone to help them with their performance testing efforts and what's the best way to find or contact you?

Yeah, you can always contact me on LinkedIn, shoot me a message, connect with me, and let's talk performance. Generally, all of the stuff that I've spoken about is very well documented on my GitHub, so you can find all the code needed to start doing this yourself. So it's all available in Python. All of the stuff is way better explained on my readme because it is a very difficult subject. It's not an easy, it's not, it's not an easy talk, it's not easy to understand in one go. It's all in there. If you want to test it out, there's some example data you can run it on so you can see it for yourself that it works and that it works in the way that I describe. You can look back at my top topic at the pack for more information. And generally you can find all of the stuff on my GitHub profile and if you want to commit and try to make this better, if you know way more about statistics than me, please help me out here. Let's make a better project out of this and let's try to make a one size solution for all performance engineers that they can just download a Python script, embed that into their pipeline and start automatic result analysis on their own and use it for their own needs.

Thanks again for your performance testing awesomeness Links to everything of value we covered in this episode. Head on over to testguild.com P80 and while you're there, make sure to click on the Try them both Today link under the Exclusive Sponsor section to learn all about SmartBear's two awesome performance test tool solutions, Load Ninja and Load UI Pro. And if the show has helped you in any way, why not rate and review it in itunes? Reviews really do matter in the rankings of the show and I read each and every one of them. So that's it for this episode of the Test Guild Performance and Site Reliability Podcast. I'm Joe. My mission is to help you succeed with creating end to end full stack performance testing awesomeness. As always, test everything and keep the good. Cheers.

Thanks for listening to the Test Guild Performance and Site Reliability Podcast. Head on over to testguild.com for full show notes, amazing blog articles, and online testing conferences. Don't forget to subscribe to the Guild to continue your testing journey.