Beginner Guide to Load Testing

Get ready to discover some of the most actionable DevOps techniques and tooling, including performance and reliability for some of the world's smartest engineers. Hey, I'm Joe Colantonio, host of the DevOps Toolchain podcast, and my goal is to help you create DevOps toolchain awesomeness. Hey, Today I have a special episode for you that features Karimi Salomon, one of our top rated Automation Guild speakers who presented at the 2024 event in February. In this episode, Kremie explains why load testing has shifted from a niche practice to an industry standard that everyone in DevOps needs to know, especially after 2020, when the world pivoted to more remote work. This session is packed with practical steps, from understanding your environment and data considerations to setting realistic testing baselines and analyzing key performance metrics like response time, error rates, resource utilization, and scalability. And by the end of the episode, you'll feel more equipped with the necessary knowledge to implement effective load testing. Also, if you miss Automation Guild 2024, no worries. You could still get instant access to all the awesomeness that took place from this year's event. Head on over to automationguild.com and register now. So join me as we embark on this load testing expedition and your quest for practical application knowledge and load testing starts now. Check it out. Hey, if your app is slow, it could be worse than an error. It could be frustrating. And one thing I've Learned over my 25 years in the industry is that frustrated users don't last long. But since slow performance isn't sudden, it's hard for standard error monitoring tools to catch. That's why I think you should check out bugsnag, an all in one observability solution that has a way to automatically watch for these issues. Real User monitoring. It checks and reports real user performance data in real time so you can quickly identify lags. Plus you can get the context of where the lags are and how to fix them. Don't rely on frustrated user feedback. Find out for yourself. Go to bugsnag.com and try it for free. No credit card required. Check it out, let me know what you think.

Hi everyone, I will be presenting the Load Testing A Beginner's Quest for Practical Application. But first of all, I'm going to introduce myself. I am Karim Salomon. I am from Bolivia and I had more than 12 years of experience in technology. I initially started as a web developer and then I had a great opportunity to be part of the QA team. I started as a manual tester then I moved to work with automation. Then I had a great opportunity to be a tech lead. I was working with different teams with functional non functional tests and then everything for low test and performance came in for me. And then I had the opportunity to to work as a QA manager and then as a QA director handling multiple line of business. So I'm really happy to be here and I hope this is going to be really useful for you. So let's start with it. Okay, so let's start the expedition. First of all I'm going to, you know, everybody's going to ask the question why do I need low test? Or is this really important for me? And maybe this is a question that many of us was having in the past because whenever I was working even as a tech lead or a QA engineer, I remember that there were just a few opportunities to work with load test. It was something really specific and according some specific requests from the team or from the application. But something happened, right? Suddenly we start with the pandemic on 2020 and everything has changed. And it means that the impact in our industry was huge because everybody started with remote work. And it means that it was not only work but it was also, you know, schools, university work and everything, meetings, everything was going online. So this will represent, of course these services overloaded, right? And of course with it, because we were not ready for it. We were, we never even imagined that the world is going to stop and suddenly nobody can go out, right? So we start having outages. So as a result for it then we had a lotus mandatory. So the thing has changed, right? In the past it was kind of an opportunity thing, but right now it's mandatory. And when I say it's mandatory, it's everything is online and everything is going to be part of it. This huge consume of these services and the applications and Internet is going to be part of our day to day. So okay, we say it's mandatory, we say we had to do it, but then where and how should I start? So I'm going to tell you a story. Whenever we are defining this kind of scenarios, you know, in the past, as I said, it was not something mandatory, but we were doing it for just some specific cases. Initially we were using JMeter. We had like an expert person defining the scenarios, creating the jobs in Jenkins and having everything that we need.

In order to have the low test that we need for it. But suddenly we had some questions from the development team. They were interested on it, they wanted to know how the things are Working how they can actually use it and where they can go and review the code and doing some changes to make it work for them. And there were multiple things that at that point some years ago, we couldn't have in the way that we want to have a shared tool between dev and QA. And then suddenly we moved to K6. K6 is a really powerful tool that worked pretty well for us. So we decided to move to K6. And whenever we had this scenario of the Pand, and then whenever everything needs to be part of load test because we saw that we had some problems in production and we need to make sure that this is not going to happen in the future. Okay. So we moved to K6. We had some experts that were using it on my team. They were creating documentation, they were creating all the steps that everybody needs and they were helping them to define or to understand how they can define the test that they need. Right. But as I said at the beginning, this is not about tools. All right? Because we had the tools. But suddenly what happened is that whenever I was discussing about some specific results with my guys, then they were having some kind of questions that for me were something that I realized that we need to talk about this topic because even if they had the tools and even if they had the knowledge about the application, they were not sure about the real points or the critical points that we need to consider for low tests. Right. So it's basically where this conversation is starting. I decided to do this presentation because I. I think that in the same way that I saw some of my people having these questions about what to expect or how can I define a good Lotus, or. Or when my results are going to be good enough and why then I think most people around the world maybe will have the same question. So that's mainly the reason for this presentation. And I hope this is going to be useful for you too. So before you start with your low test, basically you need to consider your scenario. Okay. You need to understand. Sorry, you need to consider your environment. So you need to understand where you are according the amount of services, according the infrastructure, the size of your database. Right. Your real environment where you will run your load test. So this is going to be critical to define if it's going to pass or fail. And I'm going to say this is an ideal environment. I say it as an ideal because I know that it's not so easy to get. And it's going to be ideal because whenever you had it, everything is kind of easier. But what represents to have A load environment. A load environment should be. Whenever I'm talking about the load environment, I'm talking about a mirror from production. Okay? So you had some specific environment that will replicate the same that you had in prod in terms of the servers, the services, the database, right? And then you will be able to mimic some specific behavior, some load or some traffic that you have. But whenever I say that this is ideal is because it's great to think about having it, but it's going to represent a lot of effort. Remember that if you're trying to build something that is exactly a sprout, it will need some effort for an infrastructure. It will had some effort from dba, it will have some effort from other development teams. And of course it's going to be expensive, right? Because it's expensive in effort, in resources and expensive in money. Because, for example, let's say you are in AWS and you are trying to replicate your environment, then you will need to pay for both, right? So that's when I say it's expensive and I say it's ideal because not everybody can have it. But I had the opportunity to have or to create the load environment for some specific line of business where I was working on. And this was really useful for, you know, to have the results for Lotus to reproduce some bugs. So it's really ideal because it's going to make your life easier, but it's kind of expensive and not everybody will be able to have it anyway. What happen if you don't have the load environment? So mainly what happened is that you need to have your load test.

In the environment that you have. So you don't have the load environment, you don't have the resources, you don't have the time. Because that's also another thing. It's going to take a lot of time to build that environment or to maintain that environment too. So you don't have all of that, but then you have an environment where you can run it. So you need to identify these three main things. The first one is identify the real difference in terms of data. This is really important because whenever you are doing your Lotus, remember that the behavior of your service is not going to be the same. For example, in production you have like a million of rows for something and then in qa you have 1000. So the behavior of the service that is connected to a million of rows is not going to be the same that is connected to 1000, right? So in terms of data, it's really important that you know this difference. Then the Limitations in the environment. Remember, if you had just one server in QA on foreign prod and your server in QA is tiny compared with the powerful machine that you have in producing, then that's another thing that you need to understand. Otherwise probably your results look like too low, but actually they are according your environment. So you need to know it. And of course you need to understand the portion of load that could be considered as a baseline. I am all of this presentation is going to be to show you how you can actually define that baseline and understand what needs to be done as part of your load test. But this specific point is regarding if you understand your limitations on the real data, then you will understand what's the portion of the load that you will need to use. Right? So this is going to be like your. Your. Your starting point for your. For your expedition and load testing. Okay, then what to expect whenever you're running low test. You need. In general, the people used to tell me that you were trying to see what's going on with the response time, right? Everything is around the response time. And that's true, but it's not. All right. So whenever you consider the response time, that's correct, you need to consider your response time. But you need to consider also the response time in an average or the response time in a peak. Okay. In a peak traffic that you're receiving for your service. Then resource utilization is extremely important to understand or to see what's the CPU or the memory utilization that you have. Your scaling policy, of course, it needs to be part of the results that you will analyze whenever you are doing load test. And one of my guys, at some point he told me, hey Karime, why do I need to test the scaling policy? If the scaling policy is coming from aws, we are consuming it. And of course AWS has their own test team, so I shouldn't test the scaling policy. And basically you're not testing, of course the scaling policy itself, but you are testing how it's configured for your application, right? You need to test that if you put a threshold, the threshold is working. If you say, or the developers say that whenever we are, let's say over 70% of the CPU or 65% of the CPU, suddenly I need to scale up for more machines. I need to see that this is working. And if they say four, it's four. And in some cases, for example, we saw that we were exceeding the threshold and nothing happened. And it was because the scaling policy was not configured properly. Right? So that's the point that you also need to consider here and of course the percentage of errors. Whenever you see an error, you know that you need to report the bug that maybe is not functional, but it's going to be part of the specific load that you have. So because of this we need to go back to the basics, right? Whenever we are defining the Lotus, we need to consider what are going to be my baseline. It's going to be part of my rps, or depending on how I am monitoring my service, it could be from the request per second, the request per minute, or it could be even the baseline. Right? This is going to give me what I am expecting to define as the load that I'm going to be using.

And what about data? Okay, we're saying that we understand that we will see the response time, that we'll see the review the resource utilization, scaling policy and percentage of errors. This is about the whole result that I'm going to be expecting from load test. And then what about the data that I'm going to be using? I put this slide here because in the past I had some experience when we were trying to reproduce an issue in producing, it was produced after some specific load was there. So before trying to put the fixes on database and the fixes in code, we decided to reproduce the issue so that we can actually say, yeah, this fix is working as expected. Whenever we were running, let's say that we had like 20,000 requests per minute on the moment of the issue in production and then we were trying to reproduce it. But when we were putting like 20,000, nothing happened. 30,000, nothing happened. And we started like 2x3x, you know, duplicating the, the peak amount where we saw the, the issue in production, nothing happened. We were not able to see it, you know, and the reason for it was because of data. Whenever we, we pick up some specific data, we were going through a path, right? To a single path and we were not actually touching the different options that were actually part of the problem in production. So that's what I say, that data is extremely important and you need to use random data. When I say random, I'm not talking about using anything, I'm talking about creating some specific pool, but a huge pool of data. And then randomly you can take the data from it for your test so that you are touching like different paths whenever you are generating this load. In that case or in the same case that I just tell you that work pretty well for us. And we were able to reproduce the issue. Consider static data only for non critical Fields. This is also something that is going to help you to don't spend too much time defining the data because some specific fields are not so critical and that you can just reuse the data that is going to be there only if this data is not going to be part of a transaction or is not going to represent some specific actions. That can take some time and will be important to consider and then understand what need to change during the load. That's going to help you a lot because in some specific cases probably you will need to have some specific data that need to be generated, that need to be verified, you know, before sending it. If you are isolating some service and then the service is integrated with something else and you need to understand which kind of things I am getting I am generating and which kind of things will need to be created or part of the task that I have. Okay.

So the next point is how do I know all of these things? Right? I say I need to understand my data. I need to understand or define the basics for my low test. And the way to do it is monitoring your services. And as I said before, whenever you are monitoring or reviewing how it's behaving, you need to consider the average and the peak times. So here you will see for example in the top you will see that 3,788 are the milliseconds, 37 milliseconds are the average response time for this specific period of time. And as, as you can see, the traffic is kind of a standard is going to a top of 4,000 requests per minute and with an average of 0% on error rate. And then what happened? In a peak? In a peak I know that I had like a 10,000 requests per minute. So as you can see my response time is a little bit higher, but not too much. It's 37.99 instead of 37.88. Okay, so I need to consider all of these parameters and I need to monitor and see what's going on. And of course selecting the highest peak that you have according your application, you will have some specific period of time when you know that your application is or the traffic on your application is going really high. So you need to monitor that point because that's going to be part of your baseline. As you can see here. Also, the percentage of error is still zero even if you see that there are some 0.0025% errors. Those of course, if you see the same, you need to report it and you need to make sure that all the team is aware that even if it's kind of a tiny percentage of error, this is happening, and if this is growing, it was something that was already reported by the QA team. Then you also need to monitor your resources. Remember that you need to monitor, as I said before, the CPU and memory, because it doesn't matter if you're running your load test and you're monitoring your services in general, your load test will be running for, I don't know, like one hour, two hours, maybe three. We had like a maximum of four hours, you know, consuming the services and doing these executions for a long period of time. But then you will not continue running. And in production in general, whenever you have a peak, then everything goes kind of down and then they continue. Right? So it's never going really down to zero in the same way that you have in your testing environment. But it's extremely important that you monitor your resources. Because if, for example, your memory is not freeing up and you run your load test for a period of time, then probably in the reality in production, this is never going down and it's just going up and up and up. And at some point, maybe not now, but at some point it's going to break and then suddenly your service is going to behave differently or you will have some degradation in the service. So it's extremely important to have the ability to monitor your resources. And then you also need to do something really important that is the warm up. Okay, so until here we saw for example, what need to be understand in terms of how do I define my, my load. I understand my environment, I understand my limitations, I know which data I'm going to be using and then I know what I'm going to be verifying or what's my expectation for my load test execution. And then I am ready to go. Right, but you need to to be careful with this point because warm up is really important and it will depends on the business where you're working on. But there are two main scenarios that I consider to mention here. So the first one is when the service has low traffic and suddenly receive a peak. This is kind of not too usual a scenario. But for example, if I don't know.

Let'S think about some cases. Let's say that it's Mother's Day and then you had a business, you are selling in an application some cards, virtual cards for Mother Earth. So maybe if you are in February or March.

Your traffic is really low because nobody's actually getting cards for Mother's Day. But suddenly, whenever it's May, right. The traffic over these cells for these cars is going to be really high, right? So it's like you are kind of in zero or kind of zero and then suddenly you're going up now. Okay, so this scenario is something that could happen for some specific line of business or some specific cases, but not all the time. But if that's the case that you're testing, then you can start from zero and then go to a expected load for the service and then going from 0 to a higher low, right? Let's say 2x or 3x for the load that you are using. And with that you will be okay because you. That's actually the way that your service is going to react. But the most common case is going to be the second when the service has constant load and should increase progressively. And this is something important to consider because from my experience, whenever, for example, we were doing some specific load test, we were reporting some issues because we were going from 0 to 10,000 and then the service was not responding fine. We're having like a 500 errors and a lot of things was kind of, you know, broken. But whenever we reported this to the team and we were reviewing it, it was not like a real scenario because the application is never on zero, right? So the database need to warm up, the service need to warm up and we need to start sending some load and then we need to start growing up, right? That's going to be like the real way and you need to do your test in a more realistic way. Of course, if you are trying to do stress test, maybe you can do different things, but I think that's another topic. But right now, regarding low tests and regarding what you need to consider, these kind of cases for warm up is going to be really useful for you to avoid having false negative, right? Because maybe you're reporting a bug, a huge bug that at the end is not a bug. Okay? So warm up. It's extremely important. And just to finish this.

What is an acceptable result for you? So as I said before, we're going to just have an overview on what's going to be part of your analysis. So remember, response time, it's of course the first thing that you will review, but it's not the only one, right? So whenever you review response time or you are reporting your results in terms of the response time, that response time will depend on multiple factors, will depend on the data, will depend on the average traffic, will depend on the peak traffic, and will depend also on the environment where you're running your load. Test. So all of these considerations need to be part of the results that you're giving regarding your response time. The second point is going to be your error rate. If there is some specific error rate, for example, like this one that I just show you here, it's showing me 0% because my error rate is 0.0025. It's really low. But your team needs to know that this is real and this is happening, right? I am having this specific error rate I am reporting, I am looking at the specific transactions that are failing or the errors that I'm having. And if somebody decides that this is too low priority or is not going to be part of the fix, or any decision is going to be like a team decision. But your task as QA is to report it properly so that we know in the future that this is going to be our baseline also, right? The next one is going to be the resource utilization. As I said before, if you don't monitor properly, your CPU or your memory maybe is going to be just a matter of time whenever something is going to fail. So that should be part of your analysis. Also, if during the peaks or during the specific time of the load, I am having what I need to have from my resource utilization perspective, scalability, again, you're not testing the functionality directly, but you're testing the configuration over this. And you need to have the ability to go and see that this is actually working. There are multiple tools in the examples that I gave you is new relic, but you can also use, I don't know, Grafana. And there are different monitoring tools where you can actually see that when I am hitting this threshold, suddenly I had like four, five or ten more machines and I know that this is going to work well whenever this is in production. So scalability is really important and that need to be part also of it. You cannot ignore that part because maybe you think that it's going to work, but maybe it's not. So remember, we never assume and we always prove, right? And the last but not least, you always need to have a comparison with your baseline or with your historical data. Because even if you're having different results compared with what you have before, you need to understand which changes. Maybe we had an infrastructure change, maybe we had.

A huge code change, or I don't know, an upgrade on the, on the system or you know, something that can really affect it. But we need to have this historical data and that's going to give you actually the, the real result to understand if something is acceptable or not. And with that. I just want to thank you everybody to stay until now and to listen my presentation. I'm here also to answer questions that probably you can have. But first of all I just want to give you my Twitter on my LinkedIn so that we can stay in touch whenever you need or you had any questions, you can always reach out to me. I will be happy to answer that. And now I'm happy to answer any questions that you can have. Thank you very much.

Thank you Creamy for your performance Testing Awesomeness the links of everything of value we covered in this episode. Head over to test guild.com P153 Remember, latency is the silent killer of your app. Don't rely on frustrated user feedback. You can know exactly what's happening and how to fix it with Bugsnag from SmartBear. See it for yourself. Go to Bugsnag.com and try for free. No credit card required. Check it out, let me know what you think. So that's it for this episode of the Test Guild DevOps Toolchain podcast. I'm Joe. My mission is help you succeed in creating end to end full stack pipeline DevOps awesomeness. As always, test everything and keep the good. Cheers. Hey, thanks again for listening. If you're not already part of our incredible community of over 40,000 of the Shoppists, testers, DevOps and automation professionals, we'd love for you to join us@jointestguild.com or if you're a software company or test tool provider looking to deliver real world value to our testing community, let's collaborate. Head on over to TestGuild Co to book a call and explore how we can work together.