9 Load Testing Best Practices with Joe Colantonio

Welcome to the Test Guild Performance and Site Reliability Podcast, where we all get together to learn more about performance testing with your host, Joe Calantonio.

Hey, it's Joe. And welcome to another episode of the Test Guild Performance and Site Reliability Podcast. Today it's just going to be you and me, and we're going to be talking about the nine Load Testing Best Practices. You need to know what common mistakes you should be avoiding when planning a load test. You want to miss this episode? Check it out. Let's face it, performance testing is tough. Traditional load testing tools create scripts that are bogged down in data that's hard to read and require a lot of work, even for simple playback. Load Ninja cuts out the dirty work by using AI to inspect and debug your code almost instantly. No coding necessary. No changing of hands or relays between teams either. Give it a shot. It's free and easy to try. Head on over to testguild.com smartbear and click on the link to learn more.

In my experience, one of the most misunderstood parts of performance testing is load testing. Most folks think all performance testing is load testing, but that is extremely incorrect. There are many types of tests that make up performance testing. Load testing just being one of the many options you should be incorporating in your test suites. Before we get into things to consider before performing a load test, I just want to go over some basic info on what a load test actually is. So one of my all time favorite books that I go to is I have an old Mercury Interactive Load Runner workbook and it has a definition for load testing that reads a load test is many concurrent users running the same program to see whether a system handles the load without compromising functionality or performance. I really enjoy Senior Performo's Leandro Melendez's definition how he explains load testing. And basically he says a load test is a specific sub family of a performance test. Sometimes it you know, he likes to say that a performance test would be an equivalent of say, Microsoft Office. So performance testing is Microsoft Office and a load test is say Word. Word is just a component of Office, but it is not Microsoft Office. So even though Word is heavily used along with Excel, they're probably the two most used components of Office. They still are not Office. They are components that make up Microsoft Office. So the same thing with load testing, load testing is just a component that makes up the bigger, higher level performance testing. So now that we have that out of the way, here are the nine strategies to think about when preparing to do a load test. Now I'll Be honest, most of this info was gleaned from my 2017 PERF Guild session with Shlomi Nissim who was he's no longer there, but he was a R and D manager at AHP's Performance Testing Tools at the time. And if you don't know, I know the rage now is all online events with the coronavirus going on. But I've been doing online conferences since 2017, just online conferences. I will never do an on site conference because our whole mission is to provide quality tester education all online without the waste of traveling airplanes, having to worry about you getting sick, taking time out of work, all the headaches that go with traveling. And as we've seen, the type of health risk you actually open yourself up to when traveling. Even before coronavirus, the chances of picking up an illness from your fellow travelers is huge. When you're going on an airplane because your immune system is already kind of down because you're in a pressurized airplane for hours. We really believe in bringing the on site conference to the world online. So we dedicated ourselves from day one to optimize to give you the best experience possible from an online community driven event. So we're not exploiting a virus, we're actually just continuing what we've been doing all along and that's giving you the best online experience for learning, automation testing, security testing and performance testing. So we'll be doing a Perf Guild in 2020 as well. It's going to be happening June 1st to the 2nd. To check that out, just head on over to GuildConferences.com and check out PERF Guild 2020. So I don't know why I went off that rant. That's probably why I stick to interview style formats. But on with the show to so the first thing you need to consider when creating a load test is targeting the right workload. Don't test your application on a large scale without actually needing to do so. I see a lot of people follow oh Google does this or Facebook does it. So therefore my company needs to do this. Your company is unique. Your needs, your team needs are unique. So never look to another company as you have to do exactly what they do is number one. So if there's no need to stress test your platform beyond what is actually expected, then don't do it. It's a big effort and take you a lot of time and effort. So focus your time on the things of the risk areas. You know, as a tester your application is exposed to sure it's always a great idea to produce higher traffic loads than expected to stay on the safe side. But to be realistic and productive, you really should focus in on evaluating the right workload that your app will experience in production. Of course, your website or app may experience demand during more than one peak or spike load time, depending on seasonal events. I used to work for an insurance company and every April we'd have agriculture farms that would renew their policy so we'd see a spike there and then another month we'd see other spikes. So remember, keep that in mind as well. But it is recommended that you begin load testing by emulating a normal day, so not focusing on those load and spike events, but a normal day's throughput first before you focus on testing your peak load times. And also the keyword here is throughput, which is another commonly misunderstood performance testing topic. You want to make sure that not only is the amount of users concurrent users you're putting on your system is accurate, but the amount of activity that they're creating and amount of transactions is also realistic as well. So when coming up with your workload transaction testing profile, think about number one, on average, how many operations occur in a typical hour? How about at peak? What is the goal of the test? What does your server or database activity typically look like? When choosing tasks to emulate, are you focusing on the ones that are the biggest risk to your business? How heavy of a load do you really need and what metrics will you track or are really required? Spend some time getting the step right before continuing, since you know it really is a fundamental cornerstone to creating a proper load test. So that was number one. Number two is load generator readiness. Make sure your load generators are actually ready to sustain a big workload. If you're not already aware, load generators are the computers that actually run your virtual users, and a virtual user is a script that acts like a real user would when making requests to your application and systems under test. This is usually done by emulating browsers communicating with the back end of your system, and basically you're capturing the communication going back and forth, and when you replay it in your performance tool, your backend system doesn't know if it's a real user or a browser that's actually making the requests. So here's some things to consider when doing that Remove any software that's not needed from your machine. May sound like a no brainer, but this is something a lot of people forget to do. You want to verify your generators actually connected to the network and have enough bandwidth. You want to verify your generators are connected to the network and have enough network bandwidth, things like that. If you're going to run many virtual users, you're going to need many load generators. So reliable connectivity between the different components of the testing tool you're using is mandatory for success with your test. And when it comes to large scale tests, it's recommended to allow load generators to have some sleep time between iterations. Shlomi recommends allowing as much as possible because you have a thousand virtual users running on the same load generator, it gets pretty intensive. So to freed up space, free up memory, things like that, it's usually good to leave some breathing room between different scenarios on your different generators. All this is important to understand because it's always a risk. If you overutilize your load generator itself, you really can damage the reliability of your test. Number three is scripting. It's really not enough just to create a test that will mimic real world scenarios. You also want to make sure that your scripts are not overloading the test tool itself. So make sure you've optimized the settings for your test scenario. In loadrunner, there's a bunch of different options you can use. You know, time, duration of the run, monitors selected. A biggie is the amount of data that's being logged. Any of these can really put stress on your tool. Take any hard coded or dynamic data that needs to be parameterized into consideration. Because if you're running a test and you're getting a lot of failures because you didn't actually script it correctly, that could put undo load on the back end and cause issues that really aren't realistic. So you don't want to code a test that generates unrealistic failures because of poorly written script or improper correlation. And you want to make sure you're generating the right amount of data which will enable successful tests at the end of the day. Number four is Think Time. Think Time really is essential part of any script logic. All tools I know of really should have a logic on how to add Think time by specifying how many seconds you want the virtual users to wait between either a transaction or between different runs. Think Time really is useful to mimicking the right workload according to your real user behaviors. But just like a functional Selenium test, it's also advised to use Think time rather than a sleep function. So Think times you could do like Think Time five and then it will vary between one and five seconds of Think time. So it adds some sort of. It just makes it so. It's just not a Hard wait. And it doesn't make a test run longer than needed as well. So now utilizing think time correctly is another common performance testing mistakes. People either forget to add them or else they just use 2 milliseconds. You know, something arbitrary. You really need to create a realistic performance test scenario that emulates how a real user is going to interact with your application. You should look at the tool and ask yourself, okay, is this going to take five seconds to make a decision on this page? Is it going to take a minute to perform this task? And baking that logic into your script with your code? This really can cause people to create unrealistic results if they're not aware of this. And the results are going to be invalid, which will in turn cause teams to panic for no reasons. So avoid this mistake at all cost. Another may seem like a no brainer, but it's type of things you need to think about. And that is number five, ramp up and ramp down time. Now, typically you're not going to throw in a load of say 10,000 virtual users hitting your application all at one time. Usually what happens in a real day is users log on, they do work, they log off. It's really artificial to say we have 10,000 logging on the same exact time, unless that's the exact scenario you're going for. But it probably is really more unnatural for an application to get so many users logged in at the same time and out at the same time. So if you can afford to make the ramp up time and ramp down time longer, then you should do so. It's a good practice to have ramp up time when running a load test, as it will take some time before your tests actually reach what I call a steady state. And so the steady state is when everyone's logged in doing their normal workflows. That's when you measure the time. So response time and other measurements should only be taken between the ramp up and ramp down time frame during that steady state time. Number six is a biggie. Monitoring and diagnostics so monitoring and diagnostics also needs to be taken into consideration when it comes to really doing a large scale load test. Capturing these deep diagnostic type information during a normal test means there's going to be a lot of data that's going to be captured, and that's during a normal load test. But for an extended test with many virtual users, the amount of data captured can be staggering. And the side effect can be that your test tool might have a challenge trying to cope with all this information. And the same goes for performance monitoring and actually Shlomy mentioned that he's seen many customers and users. As I mentioned, Shlomi worked for HP and I think he used to work when they used to be Mercury as well. So they've seen and talked to a lot of customers and he saw many of users selecting seemingly random measurements without even knowing what each of the counter does that they're adding or even if it actually needed to be monitored at all. So generally speaking, it's really a bad idea to capture as much monitor data as possible. You really want to minimize the amount of test data being collected and really focus in on the crucial ones to choose. Really only the essential performance counters is what you should be focusing in on when performing a load test. Save yourself a bunch of headaches and stick to using counters you know how to interpret and use later on to troubleshoot and isolate your performance issues. Having a bunch of data about a bunch of things, you have no idea what they are really isn't going to help you. It's going to cause a lot of noise. It's just like having a lot of flaky tests. You really want to focus in on actionable counters, performance measures that you know what they are and you know how to improve on and then go back to and see if the adjustments you've made actually has improved the code itself of the performance of your application. Next tip or next consideration you should have is if you're using GUI virtual users. So back in the day when I used to be doing a lot of performance testing, in early 2000 a GUI virtual user was a Windrunner. So what you would do is you put a load on your backend systems using the HTTP protocol. So you could easily scale up to a lot of users. But to get the client side type of time, you would use a GUI virtual user. So if you are doing this type of testing as well, so you know a big trend now is to actually do browser based performance testing. Scaling up browsers against an infinite amount of Docker instances, that's great. But you need to remember that browser based GUI virtual users as opposed to HTTP virtual users consume a lot more resources like your memory and GDI resources. So GUI users are really heavier virtual users and their footprint is way more substantial. So it's critical to know how many GUI View users you can run on a load generator. So you just can't have one machine and think that you're going to be able to run a full blown load test. You need to have multiple load generators spread out to spread out that lo and if you're doing GUI based vusers, you need to even have more load generators. This is definitely something you want to plan for, so you really should be ready in advance to avoid redundant failures and to prepare the right amount of load generators, balancing the workload across all of them. And the last tip is analyzing test data. Just something to consider. Be prepared to analyze your load test data set. As I mentioned earlier, with the load test you're going to have a lot of test data available to you. Remember, the longer the test, the more considerable the number of captured events during the test and the more challenging it's going to be to analyze it. Regardless of the tool you're using, there's a lot of tools out there that can make it easier, yet it still takes a lot of time. Load testing generates a lot of data and it's not easy to dig into the test results and find everything you need. Even if you have an easy way to analyze the data, say if it's automated, it's still pretty challenging process. So you need to plan on how to handle this. Just keep it in mind. Make sure you have a team in place. Just make sure when you're planning a test that you give a time frame at the end to give you time to analyze the data and not just say the performance testing is done and just hand it over to management because you're going to start a lot of fault issues. Because you really do need to take the time to not only go through all the data, but also analyze the data. Make sure that what you're thinking is correct. Do little experience before publishing results because publishing results too early is going to get you in a lot of problems, is going to give you a lot of problems. So keep in mind that faulty analysis of your software data will produce faulty results. Don't jump to conclusions without properly analyzing the data. It takes a lot of experience and skill to extract relevant conclusions from the data produced by a load test. So don't take this step lightly. I highly recommend you take your time to analyze your data. Do not release results before you're comfortable. A lot of times you would have managers, people asking, how did it go? How did it go? How did it go? I mean, you really need to take a stand and say, hold up, we have data here. I want to run the test maybe three more times, Compare the data, make sure that everything matches up, look at all your log files before making or generating a report for management. So to really wrap it up, some key takeaways from this episode is before you start a load test, you really need to target the right workload traffic. You want to identify your virtual user's footprint in order to prepare the right amount of load generators and the distribution of the virtual users among them. You need to create the data that the virtual users are going to use. You need to prepare the environments, the hardware, the servers, whatever it is. And it doesn't matter what tool you're using. If it's JMeter, K6, Load Ninja, LoadRunner, it's all the same. These steps would apply to them as well. And also remember, since large scale load test is not a trivial exercise, you should have all the right stakeholders available to help support and analyze things as the test is running to determine optimized actions. This is something Mark Tomlinson would say is you know, performance testing is really a team sport as I mentioned in our last episode and one bonus tip is I really believe performance testing is not something that should only be accessible to a few, but rather it should be accessible to everyone. It should not be an activity that's done at the end. We're shifting left everything. Performance testing is no different. You should not have to rely on a performance testing team. Of course you may need experts in performance, but you really should. If you have tester in your title or you see yourself as a tester, I think you should do all type of testing related activities, or at least have an understanding of them, performance testing being one of them. And as I mentioned earlier, that's why I created the online conference Perf Guild over four years ago. So if you really want to take your performance testing to the next level, check out PERF Guild 2020. Head on over to Guild conferences.com check out PERF Guild 2020 right. I think this is actually the first episode I've done solo for Perf Guild. If you don't know, I actually have another podcast called Test Guild Automation Podcast and I'm almost up to episode 300 so I've done a few solo episodes there as you can imagine, but this is the first one for performance. Hopefully you enjoyed it and I'm always open for feedback and for guests. So if you feel like you have a tip, tool, technique or best practice you'd like to share with Test Guild Performance and site Reliability Podcast listeners, you can always send me an email@joetestguild.com with your suggestion. The links to everything of value we covered in this episode. All you need to do is head on over to testguild.com P26 and while you're there make sure to click on the Try It Today link under the exclusive Sponsor section to learn all about SmartBear's awesome load ninja Performance Testing solution. Definitely check that out. So that's it for this episode of the Test Guild Performance and Site Reliability Podcast. I'm Joe and my mission is to help you succeed with creating end to end full stack automation continuous testing awesomeness. As always, test everything and keep the good. Cheers.

Thanks for listening to the Test Guild Performance and Site Reliability Podcast. Head on over to testguild.com for full show notes, amazing blog articles, and online testing conferences. Don't forget to subscribe to the Guild to continue your testing journey.