Machine Learning in Performance with Gopal Brugalette

Get ready to discover some of the most actionable performance engineering, testing and site reliability device for some of the world's smartest engineers. Hey, I'm Joe Calantonio, host of the Test Guild and SRE podcast and my goal is to help you succeed in creating application performance awesomeness.

Hey, it's Joe and welcome to another episode of the Test Guild Performance and Site Reliability Podcast. Today I want to share with you a session we did at the Perf Guild way back in 2019 on machine learning, application and performance analysis. So managing the performance of complex systems requires more than simply running load tests. You need to perform a careful analysis of test results and production metrics, and the sheer amount of data generated makes analysis a huge challenge and but with machine learning and the application of data science techniques, you can derive valuable and actionable information from big data. So in this episode I'm going to be sharing with you the session that Goople Brugaletti did at the 2019 PERF Guild. Goople is a Senior Performance Architect, Manager, Software engineer and he's going to share the basic concepts behind machine learning covering clustering, predictive analysis and neural networks. He's also going to show how to implement algorithms using open source tools and languages like Python R and AWS cloud services. He's also going to show some real world examples that really illustrate performance engineering problems like performance monitoring, test results comparison, error message analysis and useful insights. So you don't want to miss this episode to find out how you can use machine learning to actually help you start solving your quality and performance engineering challenges.

This episode is brought to you by the awesome folks at SmartBear. Listen, we know load testing is tough but necessary. So investing in the right tools to automate tests, identify bottlenecks and resolve issues fast saves your organization both time and money. And that's why SmartBear created Load Ninja, a SaaS load testing tool to help teams get full visibility into performance so you can release quality software faster than ever. Make testing effortless and give it a shot. It's free and easy to try. Head over to loadninja.com and learn. I think performance testing is a critical skill that's becoming increasingly important for more and more organizations. Still, unfortunately, it's not a skill many testers possess. Having an application that is not performant and not meeting your customers expectations can be dangerous to your company's reputation and bottom line. And sometimes it could even lead to layoffs as organizations try to cut costs. As a tester you might think, hey, I don't need to know performance testing. But trust me, you do. My philosophy is that anything that could be automated in the software development lifecycle, even if it's a non functional test like a performance test or reliability or anything to do with chaos engineering or security, it should be handled by testers and automation engineers. Plus a tester with multiple testing skills in the toolbox is much more valuable to their company. Almost making yourself recession proof. And that's why I created the last two days of Automation Guild to include many sessions on performance testing. So if you really want to accelerate your performance efforts and your full and automation testing efforts in 2023, definitely head on over to Automation Guild and register for this event. So I'm going to share with you a sneak peek of three of the performance sessions we'll have at this year's event. We have four of them. I think you can get a lot of value out of it. So listen in and then we'll go into Goople's session that he did for the 2019 event to give you a taste for what a Automation Guild Performance Guild session is actually like.

Hi everyone, my name is Marie Cruz and currently I work as a developer advocate for K6IO. I'm really excited to share that I'll be speaking at Automation Guild 2023. My topic is going to be about browser automation and front end performance with XK6 browser. Now historically when you think of doing back end performance or front end performance testing, you tend to use separate tools. But what if there is a single tool that can do both? So in this talk I'll be introducing you to XK6 Browser and how you can use it to adopt a hybrid approach to performance testing again. Come join me at Automation Guild which will be held from the 6th to the 10th of February 2023. I hope to see you there.

Bye bye.

Hello everyone, My name is Kat Kmiotek. I work as a quality engineer at Zoopla and I will be one of the speakers of this year Automation Guild. My talk will be about load testing with Artillery and Playwright. I will show you how to write test scenarios in YAML that is not that difficult. What sort of Artillery plugins you can use to write your tests and how to use Playwright to test your front end service performance. Hope you can join me.

Hello, welcome to these green fields. I am your amigo Leandro, also known as Senor Performo, bringing you here to this openness to tell you about the Automation Guild 2023 from February 6 to February 10 and on it I am gonna be also presenting Where I'm gonna be telling you about how to performance and load test your production systems. Yeah, testing in production, that big taboo topic that we have avoided. Well, now we should be testing in production. I'm gonna tell you why, how tips and tricks and many more things that will let you start touching those productive waters, those productive fields. So amigos, see you soon on the Automation Guild 2023 from February 6 to February 10. Don't miss it, subscribe, check the links below and see you soon. Adios.

I'm Kopal. I've been doing performance engineering for over 15 years. That includes working with e commerce, financial institutions and SaaS providers. Now when I think about performance engineering, it's all about the data analysis where we collect so many different metrics like cpu, memory and response time. And we summarize that data into information and from the information we get insights about what is the customer experience or what's the health of our system. And from those insights, we take the appropriate action either to improve the customer experience or to resolve system issues.

And it's that process of going from.

Data to information to insight to action.

That that really makes me think of.

Performance engineering as a data science. Now, the typical way we approach to analyze our data is by creating dashboards. But it just doesn't scale. We can't go on creating hundreds and hundreds of dashboards. We need a better way to approach our analysis. Now machine learning gives us a way to do just that. It's a really common tool within data science. And machine learning is really just the math that allows computers to do the type of analysis that that we as performance engineers do and do it in an automated fashion. In this presentation, I'm going to show you three different common machine learning algorithms. Linear regression, clustering and neural networks. I'll show you how they work and the use cases in which you can apply them to performance data analysis.

It's going to be a really great.

Starting point for you to go and explore machine learning on your own.

So in our first use case, we want to build a machine learning model to better understand the customer experience. We start with what's called feature engineering. We first have to understand what data we have and what it tells us about our goal or what we want to predict in performance. We have a lot of metrics like cpu, memory utilization, response time, user load or counts and we need to understand how are they related. And one of the best metrics to understand customer experience is going to be response time, of course. Now what is the independent variable or feature that tends to drive that response time of our system. Well, very often it's going to be the user load or the number of users on the system. And in fact, it tends to drive all the other metrics as well. So it's a really good starting point when looking for a feature.

So the first thing we're going to do is graph the response time versus the user load. Now, this is actually real data from a system I worked on. And here we can see that I can pretty much draw a line right through the data points on this graph. This means that there is a linear relationship between response time and load. Now, in other use cases, this might not be so obvious, but you can use mathematical techniques to evaluate how well the data fits a line. But we're not going to go into those now. But here we can clearly make use of a linear model.

So now we're going to do a little math. Mathematically, we can express our model as saying response time is a function of load. And that function, as we saw earlier, is a line. The way to mathematically express a line is here. Y equals a plus bx, Y is our response time and X is our load. A is a constant and B is the coefficient of X. Now, here we can see various examples of how varying those values affects what the line looks like. We're basically just changing the intercept A and slope B. And so this is the equation of a line, and that's what our model is going to look like.

So now what we have to do is solve for A and B. Remember, X is our load and Y is our response time. Each point of our data set or our training set is a pair of these values. For any value of a load, we have a response time that we measured. Now, we solve for A and B using this formula listed here. And this process is called linear regression. Now, this formula might look a little complicated, but you don't have to actually do this on your own, because the great thing is many languages have this calculation built right into them. In this example, I'm using a data science language called r, but you might also do it in a programming language like Python or even a spreadsheet like Microsoft Excel.

So in line 202, you can see I'm calling the function LM with some parameters. LM is linear model. So it does all those calculations we saw for you. I'm basically telling it to build a linear model for response time using count and also what data set I want to use to calculate A and B. Now, in line 199 to 2,03. I'm actually looping through to create or train multiple models for the different response time percentiles that I want to predict, like P25, P50, P95 and I mean like 20th percentile, 50th percentile, 95th percentile, etc. And we'll see how we use that more a little bit later. Then once I have my model, I want to use it to make a prediction or given a particular count or load, what would my model or equation tell me the response time should be?

So now what we've essentially done is built a machine learning model to predict response times using linear regression. And we'll see how to put this to use in a few slides. So that's the math and code. What does it actually look like in process of actually being used?

So here's what the actual workflow looks like to use this machine learning model. First I need to identify training data. So I'm going to go into my APM or monitoring tool, look at the logs and make sure I'm not using a day that has an outage or something weird happening. I want good, clean data to train my models and then I'm going to extract that performance data via an API or log parsing tool. And I'm going to do this for a number of endpoints because I want to monitor the performance of all my key endpoints measure. I want to monitor all the business critical transactions that really affect the customer experience. I can start to see one of the benefits or the power of machine learning as I can do this automated analysis for any number of endpoints. So once I've built those models, I'm going to deploy them maybe to AWS, to an EC2 instance or so on, and then I'm going to start running the monitoring. So what does that look like? Well, my APM tool or my monitoring tool of course is going to be recording what the actual loads are at any instance as well as measuring what the response time is. So I'm going to take those loads as measured by the monitoring tool and I'm going to feed that into my model and based on that, it is going to make a prediction of what the response time should be. Now I have the predicted response time as well as the actual response time that is measured by the monitoring tool and I am going to now send those out to elasticsearch just for visualization processes or just for visualization purposes. And I can of course compare them. So now we'll see what it looks like. Here you see a visualization of how I use my model to analyze production performance. While the model does produce an actual single value for any given input of load, it also gives an upper and lower bound of the predicted value. There's always variance in the data, and giving these bounds helps account for it. It makes the model more accurate and useful. In these graphs we see four models, one for each of the important percentiles of the response time distribution. In red, I'm plotting the bounds of the predicted response time value for a given load. And in blue, I'm plotting the actual value of the response time for that given load. The way this works is I read in the real time production load via a monitoring tool and then I make a prediction on what the response time should be based on that load. I do this, of course, using my trained model. I'm plotting the upper and lower bounds of that prediction in red. I'm also reading in the actual production response time as measured by the monitoring tool and plotting that in blue. Then I could compare the actual values in blue to see if it's within the bound predicted by the model in red. When the blue line or the actual value is within the red lines, I know that the system is performing in accordance with how my model predicts it should. That means everything's okay when it goes out of bounds, something has happened. Either there's a problem with my system or there's a problem with my model. In either case, I need to investigate what has happened. Now, the really important point to understand here is that this whole process is dynamic. As the load on the system changes throughout the normal course of the business day, the model automatically adapts and it makes the correct predictions for that changing load. That's what makes it machine learning. If all of a sudden a lot of people come onto the site, maybe there's a big sale or an ad campaign, or the business naturally has a peak every evening at 6:00pm well, the model can still make an accurate prediction and determine if the system is performing properly. That's very powerful and that lends itself to a number of use cases. The first use case is to evaluate production changes. A common situation is a team will deploy code to production during off hours and then they go and test it and everything works and performs well. But the site is under low load, it's off hours. Then the next day or later in the week the peak load happens and under this high load the system fails. The performance is terrible. This scenario occurs because it can be very difficult to detect performance changes when the site is under low load. Everything looked fine. It only became obvious when the site is under heavy load that there is a problem. The type of machine learning model we showed can be used in these situations. It dynamically compares the actual current performance to the predicted performance. If you train the model on the previous version of the code, it can compare the new code's performance in production to the previous version. And it can do it no matter what the load. And so it can detect if a system is performing worse on the new version of the code than it would have been performing on the old version of the code. Given the same load, this could potentially help you predict and detect major issues before they occur. The next use case is to detect less than optimal performance. I was running the model to monitor a production site when I noticed that every day at 6pm the site was slower than the model predicted. It wasn't very much, just 50 or 100 milliseconds or so. And this type of deviation wouldn't typically be noticed by the normal type of production monitoring, which is usually based on SLA thresholds. It's just too small. It's not a major issue. But the model had a much greater precision and could detect small changes. And this increase could impact the user experience. When I did some investigation, I found out there was a batch job kicking off every day at 6pm and it was causing a small slowdown. Now, it may not seem like much, but in E commerce, milliseconds are money. And this detected that the system was performing less than optimal. The third use case is trending over time for different load profiles. For systems that have performance dependent on load, it can be difficult to determine if the observed change in performance is due to a change in load or something else like code or infrastructure change. I've had many discussions with developers over this issue. The question is, is the code getting slower or is it just the increasing load, using more resources and causing things to slow down? Well, using the model, we can compare the performance of systems under different loads and determine if it is still performing to the predicted values or not. Essentially, we can normalize the performance for different loads and compare them to each other. And the final use case is to use the model to detect outages or severe problems in the system. If there is a very large variance from the predicted values, there's probably something going on, but there are better, easier ways to detect outages. Now we're going to talk about our next machine learning use case. We usually have a large number of servers running to support an application or service. There could be hundreds of servers. And in theory, all these servers should be performing the same. But sometimes they aren't. And it can be hard to find the few that are acting differently. It can be in any number of metrics or symptoms that we would have to look at to detect an issue. And our typical statistics would usually show that things are normal. But customers are on those errant servers and they're not going to get the best experience. So in this case, where we aren't sure what to look for and don't have a great set of historical data to compare with, we can use what's called unsupervised learning. Here we're going to use an unsupervised machine learning algorithm called K means clustering. The way it works is it segments data into k number of groups. And based on feature similarity, it starts by selecting a center for each of the k groups or centroids. Then each point is assigned to the nearest group. Then a new center is calculated as the average or mean of all the points in the group. Then it iterates reassigning the points to the new nearest center and then iterates until the groups are all identified. Now we'll look at an example of how to apply it in practice. So here I analyze the response time data for each server in the tier by using K means clustering. I took the median response time for each server in 5 minute intervals and clustered them. You can see in the left diagram that the algorithm divided the data into two different groups. In the middle is a visualization of the data in that time slice. And on the right you can see what the data looked like graphed in a monitoring tool. It's very clear that two of the servers are performing one to two seconds slower than the other 80 or so. Now remember, I could apply this machine learning analysis to any number of server groups and any number of metrics. And all I'm doing is saying look for servers that are different. I don't have to go and set a lot of arbitrary thresholds and I don't have to look at 100 different dashboards. So this is very dynamic and scalable. That's the benefit of applying machine learning to this problem. I can find a lot of things other than just failures and outages. In this case, working with operations, we found out that those two servers had a BIOS setting for energy saver on. They were using less electricity but running slower. So that means that about 2% of our customers were getting a slower than optimal experience.

Now here's some basic code in R To do this analysis, it just reads in the data. In this case it's a CSV file, but it could easily be a real time API call. It cleans it up a little bit and then performs the clustering analysis and prints out the clusters. There's always a challenge with using clustering, and that's determining what the optimal number of clusters should be. There's a few ways to arrive at that number. You can do it manually using hierarchical clustering, where you draw a horizontal line through the vertical ones in the diagram. There's other ways as well, using libraries to calculate it automatically. As an example, when I used K3 to look for three clusters, the algorithm did in fact find three clusters. And it turned out that there were different server groups that had slightly different hardware and configurations. And the clustering algorithm could see those differences. So we looked at an example of supervised machine learning using linear regression, where we trained the model with a set of data. And we looked at unsupervised learning in clustering, where there was no training of a model. There are a lot of different machine learning algorithms and one of the hardest parts is picking the right one for your problem. Then you have to determine which are the best features or metrics to use. And that can be an iterative process. Data wrangling or getting the data, cleaning it up and getting it into the right format so you can use it is also non trivial. Then you have to make sure you have a good set of training data for your model. And then finally you have to get it to run in production. And that's more software engineering than data science, but it's an important aspect as well. Let's look at our next machine learning use case. In doing performance data analysis, we spend a lot of time looking at dashboards. We might have to look at so many different dashboards and do it over and over and over again, maybe for each iteration of a test, or to diagnose a problem, or maybe to monitor a big event. So I'm thinking maybe there's a better way. Maybe we can automate it somehow. And I don't mean setting up 100 different alerts to fill up our inboxes. Maybe we can apply machine learning to do this. So how do we go about doing this? Well, if I think about how I do it as a performance engineer, I. I pretty much just look at the graphs and based on my experience, I know when the graphs look fine and when something looks interesting or something is wrong. And so that's a situation that should lend itself to using Machine learning, I should be able to train a model based on those examples to know what's good and what's bad. And specifically, neural networks are very good at identifying and classifying visual data or working with images. And so I started looking at neural networks to analyze dashboard type data. I wanted to start my POC with a narrow use case, so I began by looking at response time distributions. Like, in this histogram, we have the distribution of origin response times from a cdn. The typical distribution for performance data is what's called lognormal, meaning it has one mode or peak and then a long tail out to the higher values. And you'll see this for just about any performance data. And so anytime the distribution looks different than the log normal, it means there's something interesting happening. So in this example, there was a situation in which some of the users were complaining that they saw slow response times sometimes. And we can confirm this in the data because generally the response times are good with mostly one second or less. But occasionally we see some out in the 16 second range. And that indicates that, yes, sometimes something goes wrong on the server end. Now, this type of data is really useful to be able to find. We can either confirm customer complaints, or better yet, we can proactively detect it and take action. But it's really hard to see through statistical analysis, and it takes a long time of sorting through histograms. It's very slow work. There are a lot of different types of neural networks, but really diving into how they actually work is outside the scope of this talk. But at a super high level, an image comes in like a duck, and it's essentially a matrix of the pixels with numbers representing color values. Those pixels are fed into each neuron or node, starting at the input layer. Each layer then successively processes the outputs from the previous layer with each neuron using weighted sums, applying a bias and an activation function to determine if it's going to fire. And then it sends its results to the neurons in the next layer or triggers them. You see from the picture that the communication from neuron to neuron forms a network. So if you have properly trained a neural network, it can recognize the picture of a duck or cat or dog or so on. Now, neural networks perform best when they have a lot of training data, but finding training data in actual dashboards is very time consuming. So for this initial investigation, I went ahead and created synthetic data and then generated JPEG images of different types of distributions. I separated out the generated data into a training set and a test set. So I could validate how well the neural network would perform. For this PoC, I actually used Amazon SageMaker to build a neural network. It's a pretty cool service in aws and they have templates for common machine learning algorithms for you to modify for your own use. It makes it really easy to train your model and get it up and running as an endpoint. And you also don't need to really know the details of how a neural network or your machine learning algorithm actually works. In this POC phase, the neural network is performing well. It's been able to identify single versus multimodal distributions about 85 to 90% of the time that's in the validation set. In the example here, it has identified the middle image as having multiple modes with 98% probability and the next step in this POC is to start working with real data and then I'll probably need to tune the neural network parameters or most likely modify the training set. And of course eventually we'll want to work with real dashboards from a monitoring tool. Now there's still a lot of work to be done, but I think it shows promise.

In this presentation I introduced you to three different machine learning algorithms, linear regression, clustering, and neural networks. We use linear regression to detect the performance impacts of code changes, clustering to find servers which were performing differently than their peers, and we saw how neural networks can be used to visually interpret images of performance graphs.

I hope that this is a good starting point for you to begin your own explorations of using machine learning for performance analysis.

Thank you Google for your performance testing awesomeness. For links of everything of value we covered in this episode, head on over to testguild.com P108 and while you're there, make sure to click on the Try for Free Today link under the Exclusive Sponsor section to learn all about SmartBear's awesome performance testing solution. Load Ninja and if you haven't already also head on over to automationguild.com and register today to get your five day ticket to experience full end to end Pipeline Automation Awesomeness sessions. We have over 40 sessions that it's really going to help you and your testing team in the new year. So that's it for this episode of the Test Skill Performance and Site Reliability Podcast. I'm Joe. My mission is help you succeed in creating end to end full stack performant automation awesomeness. As always, test everything and keep the good. Cheers. Hey, thanks again for listening. If you're not already part of our awesome community of 27,000 of the smartest testers, DevOps and automation professionals in the world. We'd love to have you join the fam@testguild.com and if you're in the DevOps automation software testing space or your test tool provider and want to offer real world value that can improve the skills or solve a problem for the Guild community, I'd love to hear from you. Head on over to test guild.info and let's make it happen.