Building Performance Testing Pipelines with Viktoriia Kuznetcova

Foreign.

Get ready to discover some of the most actionable performance engineering testing insight reliability device for some of the world's smartest engineers. Hey, I'm Joe Calantonio, host of the Test Field and SRE Podcast and my goal is to help you succeed in creating application performance awesomeness.

Hey, it's Joe and welcome to another episode of the Test Guild Performance and SRE Podcast and today we're going to take a blast tests from the past. I want to share with you a session we had at the Performance Guild back in 2019 with Victoria on building performance testing pipelines for CICD. If you don't know, we're in the early stages of early bird specials for the Automation Guild Online conference taking place February 6th to the 10th. It's a five day event that includes not only functional automation but also performance and SRE sessions. So to find out more about Automation Guild, you definitely want to register now. Head on over to automationguild.com and register today. But in today's session, I'm sharing with you a recording that Victoria did for once again the 2019 PERF Guild. So as you know, nowadays it's often the case that performance testing needs to be executed repeatedly, fast and with minimal manual work for a fast paced program to go from development to production. This, as you know, raises challenges of creating realistic test environments, maintaining test data and analyzing performance test results automatically or semi automatically. So in this session I took the audio from the recording Victoria did where she shares her experience in building frameworks pipelines to do just that in an AWS enabled environment using open source tools and a bit of scripting. So you want to listen all the way to the end to understand how to approach building an automated performance testing pipeline for your projects and how to pick the right tools and how to get the best values out of it. Hope you enjoyed this session. If you want more of this session, as I mentioned, definitely head on over to automationguild.com and register for the 2023 Automation Extravaganza event. Hope to see you there.

This episode is brought to you by the awesome folks at SmartBear. Listen, we know load testing is tough but necessary. So investing in the right tools to automate tests, identify bottlenecks and resolve issues fast saves your organization both time and money. And that's why SmartBear created Load Ninja, a SaaS load testing tool to help teams get full visibility into performance so you can release quality software faster than ever, make testing effortless and give it a shot. It's free and easy to try and Head on over to loadninja.com and learn more.

Hello there. My name is Victoria and I've been in software testing for 15 years. For the last five years of it I have been dedicating my time specifically to performance testing and in that time I have built a few performance testing pipelines for automated testing in the modern world. Development teams are doing a lot of continuous integration and continuous delivery kind of processes and they release software a few times a month and they release software for testing sometimes every day or even more often to keep up the quality of software in such tight you need automated testing and performance testing is no exception here the company is mature enough to do performance testing at all. This is a big topic and we only have 25 minutes or so, so I had to limit my material sample. I have decided so that the most important part for me is to give you a high level understanding of how to build a performance testing pipeline for automated testing to name a few problems that you might run on your way there and to point you to solution. My hope is that this talk will be a starting point for you to build your own performance testing pipeline. So without further ado, let's dig in. You all surely know how challenging performance testing can be. Continuous performance testing elevates most of those challenges on a new level because you need not only to solve those, but do it in a repeatable fashion and without human intervention. You need not only to set up a realistic test environment and prepare a large volume of realistic test data, you need to do it on demand and do it fast, in a matter of minutes or at most hours. You need to ensure your tests are trustworthy and any significant results variation points to an application change. You need to automate test results analysis and come up with a way to present your test results in an easy to read format for your target audience, most commonly PMs and dev teams. We will talk about solving these challenges in more details in the next slides. One more thing that is I believe important is storing the whole pipeline, including test scripts as a code. If you rely on anything that is not code that you cannot commit to a source control system and easily review when changes are made, your performance testing pipeline will start falling apart when maintenance is needed, especially if a hand over to another person is involved just gets so messy so fast. I will talk about it a bit more further on. Now here's a high level view on a typical performance testing pipeline. You will go through these steps each time you execute the pipeline. First test environment needs to be set up. This includes not only installing a new version of the application, but also ensuring that everything around that application is ready for a performance test. Then test data needs to be taken care of. After that you set up your load generation tool and monitoring and you run the tests afterwards. You need to gather up and analyze test results and you need to display them in an easy to read way. And lastly, you would need to clean up. Specifics of that would depend on your project and whether or not you are creating a test environment from scratch for each new test or not, but you will need to clean up. Now. Let's go through the stages one by one and talk about specific considerations and challenges you might encounter. First of all, you need to figure out your test environment. One major question here. Do you use an existing test environment that is probably managed by someone else and is likely shared between projects and test teams? Or do you have an option of provisioning a test environment each time you need to run a test? Dev shops are increasingly using clouds for development and testing, but other companies such as banks, hospitals, Internet providers you call them, they are likely to have on premise test environment that cannot be shifted into the cloud easily. If you are bound to an on premise environment, your setup would mean ensuring it is in the same state each time you need to run a test. You will need to validate that state and stop the pipeline if environment is not ready. Think about it this way. Is there anything in the test environment that can influence the results of your test resource utilization, other active users in the system, changing latency between tests, database caches and so on and so forth? These are all the things that you need to be aware of if you're setting up performance testing pipeline in a shared existing environment. You would also need to validate that the application was successfully deployed before you go any further. But you don't need to worry about third party systems because should come as a given that in a maintained test environment they are good. If you work in a company that uses a cloud for test environments, you will need to set up a process where your test environment is provisioned in the same way each time. You will usually piggyback on whatever process dev team is already using for their deploys, but you will need to plug in at least your own database server and maybe some other servers as well. And you will need to ensure that environment is scaled up either to a production size or to a smaller size, but it should still be enough for meaningful performance testing. Dev teams usually use simplified deployments with certain components stopped or shared between projects and that won't do for performance testing, any component that you are putting load on needs to be in a dedicated test environment. Otherwise not only your test results will be skewed, but you also risk bringing down the whole testing infrastructure for the company. Been there, done that. Not fun for anyone. Lastly, when you are provisioning test environment from scratch, remember to validate that not only the application you are testing but also all of the third party apps it is integrating with have been successfully deployed. Most applications have a health check built in that you can use for that after you took care of the test environment. Next big task is sorting out test data. Let's split it into three categories. First there is reusable test data. This is data that your test does not affect in any way. All you need to do here is prepare it once and then ensure no one is messing around with it. Next, there is a non reusable data that isn't destroyed when application is redeployed. This is data that your test is modifying, deleting or creating creating when it runs its course. This can be managed by the test itself. For example, you might first run scenarios that create data, then use that data for test cases that modify them, and finally run tests that would delete it. But it is quite rare you can do that, especially in big highly integrated systems with multiple levels of caching. Another option is restoring the database to the same database snapshot each time you run the pipeline. In the cloud you would automatically use that option because you probably will be recreating the database server as part of the test environment setup. But if if you're bound to a shared reusable test environment, then you probably can't restore database to a snapshot. So you will need to find a way to replenish your non reusable data supply. And also you will need to ensure that whatever process you employ for it maintains the statistical test data distributions. So something like how many encounters you have per patient on maximum, on minimum, and for an average patient, things like this. Otherwise your test results might vary from test to test due to the test data variations and that kind of renders the whole exercise meaningless because even if performance changes, you don't know why. The last category of test data is non reusable test data that is also destroyed when an application is redeployed. This would greatly depend on your application and on the deployment process, for example whether it's a clean deploy or whether you upgrade from a previous version. Let's say history of user activity would usually be destroyed if you do a clean deploy and in some cases you would also need to import the whole user base each time you do a clean deploy as well. Depending on your situation you might not have this problem at all, but if you do you will need to introduce some process to generate such test data to a production like volume before you start the testing and import it into the system. Lastly, be aware of the caching on all the architecture layers starting from the disk caches on the database server through to the database caches and all the way to the caches in the application under test. You will need to make a decision on whether you always test with cold caches or whether you always test with warmed up caches and ensure that whatever decision you made it is being implemented. Next Stage Setting up a load generation tool is a fairly easy part. On prem you would probably have a load generator that is always online in the cloud you would either have a pre configured instance that you turn off and on as you need, or you might have a VM image and you will spin up a new instance from that image. But still it's going to be a preconfigured instance. In the cloud you would also probably need to take care of firewall rules and other network configuration because you need to point your load generator to the right place and you need to allow it access to that place. Regardless of how you spin out the load generator, remember to upload the latest test scripts there so you're not running out of date tests. Now for monitoring, you can pretty much use whatever monitoring you used when you tested the same application outside of the automated pipeline, as long as your process allows for either API based access to the monitoring or involves an automatically maintained dashboard with access to historic results, you're all set. For example, if you happen to own and use a fancy resource monitoring system such as New Relic or Datadog or something else, you're probably good to go in terms of resource utilization monitoring. Alternatively, you can use cloud provided monitoring tools in the cloud such as AWS cloudwatch for a high level picture or either on PREM or in the cloud. You can use systet for Linux and Perfmon for Windows to get a detailed picture of your resource utilization during the test. Aside from the resource utilization monitoring, remember to configure logging on all relevant layers. Think Haproxy logs, Apache logs, application logs, AWR reports for Oracle and so forth. And of course remember to monitor the load rate generator. You need to be able to automatically invalidate and stop the test if your load generator cannot handle the load itself. Especially in the cloud, you might save yourselves hundreds of dollars and hours of time. And on prem you would just save time, which is also pretty good if you manage to stop the invalid test early. Running the tests is probably the easiest part. Once you've got everything else handled, A few notes here. Remember to monitor for error rates for automated test and validation. That's the same as with load generator, not handling the load in JMeter there is a special plugin for that. Pretty sure in Gatling you can program it in fairly easy and I don't know about other tools but they should all have this kind of functionality. Another thing is before letting anyone see and use your performance testing pipeline, make sure that the test results are stable. Run the pipeline a few times on the same release over a few days and measure the variation in test results. It should be minimal and if it's more than you accept, figure out where the variation comes from and fix it. If you cannot trust that when your tests show performance degradation it's due to application changes, then you cannot use the pipeline. Just imagine dev team jumping on the performance degradation bug and then discovering that the degradation is in fact an artifact of your tool not providing stable results. The trust is lost and it's quite hard to restore it again. Otherwise you just run the test as usual. Whatever test scripts you would run, you run them. So you've run the test and now it's time to deal with the results. First of all, gather and store test results alongside the test. Both Bamboo and Jenkins allow for that. Just zip it up and you have it there for future references. If the test fails, some kind of investigation will need to be done into the reasons of it, and having as much data as possible can save a lot of time. Secondly, you must define a programmable criteria upon which the pipeline will mark the test as passed or failed. You can use any number of metrics for it. There is no correct answer here. It's just a decision that you will need to make and it will have to work for your specific project. For analysis itself, you can use automatically generated HTML reports for the basic analysis stuff like JMeter HTML report or Gatling HTML report, provide that kind of thing. Or you can use R to write your own smart custom analysis R is a language specifically designed for data analysis. I've used it quite extensively in my performance testing to correlate a few logs from different system layers to measure processing times for asynchronous operations for aggregating data with the filters applied to it, and so on and so forth. It's an open source tool and it's free to use and you can generate neat HTML reports with it, so all in all, it's very useful. One last remark here is choose your main percentile wisely. What you want is a repeatable test result that ignores random outliers but describes performance for most users most of the time well. So for example, I wouldn't use 99th percentile for an automated pipeline, but I would consider anything between 90th percentile and 97th percentile depending on how stable is the application and how much it depends on the performance of third party systems out of my control. Once the analysis is done, you want to display the test results so they are actually useful with continuous performance testing. Test results would usually be looked at by a dev team on daily standups, by PMs and release managers before releases to production, and by other performance engineers or even yourself when the test has failed. So to support these use cases you need a few levels of reporting. First, you need a clear pass fail indicator that can be reported upon on a daily standup. Next, it's good to have a default dashboard showing the trends in performance from build to build, and ideally someone who's not very technical should be able to understand it. So for this you need to select a few important aggregated metrics and graph them with one data point per test per metric and then you'll get some trend line. So something like 90th percentile response time of the most important operation would do. And finally, it is very useful to have a second dashboard report or HTML report with detailed test results. So this is not raw data. This is just one layer above raw data so you can easily review some kind of aggregated results if you want to dig deeper into them. The last stage is cleanup. In the cloud it is fairly easy. Just turn off or destroy your test environment and you're done. On premise, especially in a shared environment, ensure you return everything to the state it was in before you start it. For example, if you stop certain services, start them again. If you started something extra, stop it and clean up the database when necessary. The goal is to avoid spoiling testing for someone else. Also, it is a good practice to archive application logs and clean up any unwanted artifacts of performance testing, be it a bunch of extra rows in the database or JVM thread profiles or even heap dumps. Especially heap dumps. The idea behind is sustainability. You don't want to run out of disk space or skew the reusable test data because you generated so much extra during the test that it actually changed the Statistical data distributions now that we went through the whole process of running automated performance testing, I want to touch on the tools that you can use to actually implement all this. We touched on it before on a few cases, but let's do it again. Of course it would depend on what you're already using and on your project, but here's a few tools that I personally found very useful in the context of testing web applications to run the pipeline you would like a user whatever build and deploy server your dev team is already using to drive their deploy automation. I am myself familiar with Bamboo and Jenkins, and between them I prefer Jenkins. But basically if you can run a bash script, you can build your pipeline. My first automated performance testing framework was in fact a number of bash scripts that tied together a whole bunch of tools that were required for each stage and that framework is still in use, although it's quite a pain to maintain. To set up test environment, you would normally piggyback again on the tools that development team uses to deploy the application. You might need to run extra tasks though, and for that I would recommend using Ansible or Puppet for application installations and AWS QLI CLI or a similar tool for another cloud for the provisioning of the instances themselves. For example, I would use AWS CLI to provision a database to handle that pesky part of test data that just doesn't survive a new deploy. You can use a bulk load tool or you can use SQL script just so you can generate and upload test data directly into the system in the fastest way possible right before the test. If there is no way to bulk load some data that you need, do not hesitate to demand it from the development team. They need it as much as you do and it's often a matter of adding a small extra piece of code for them and it will greatly improve the stability for load generation. In this particular context I recommend using either Gatling or JMeter, but if you're using JMeter, use it through the Ruby JMETER library. The point here is to keep your test scripts as a code to enable easy code reviews and maintenance. Using JMETER as is brought me and my team a lot of pain and any change required to view of the whole JMETER script in the graphical interface to ensure nothing got broken and JMETER has a tendency to break the scripts when you save them if they use any kind of modules or something like this Ruby jmeter solves this problem. It's just a code in Ruby that is used to generate a JMETER script for test results analysis. I would recommend R It is awesome. Check it out. I've talked about it before. I'm just going to say it again. It's a tool for analyzing data. So there you have it. If you have data to analyze user for test results presentation if you want a dashboard and you already have a Splunk license, just use Splunk. It's powerful, it's pretty. It's already there. If you don't have a Splunk license, it's not worth it to pay for it. It's too expensive. I recommend looking into the Elk Stack elasticsearch Logstash Kibana. It's free and it provides a very similar experience to Splunk. Also, Jenkins Performance Plugin has a simple dashboard you can use if you're using Jenkins and you can combine it with HTML HTML reports and have detailed test results for you like this and yeah, that's pretty much it. So let's wrap it up. I hope this talk gave you a clear understanding of what is involved into building a performance testing pipeline in the CI CD setting. We talked about different stages that you need to go through to get it going and I pointed you to some of the solutions. If you have any questions, please save them for the live Q and A or you can also reach out to me directly through an email or a LinkedIn account. Also, I am planning to run a Udemy course on performance testing later this year and if you just happen to bump into it when it's released and you're at all interested, give me a shout and I'll happily share Discount coupon as a thank you for you listening to me talk to slides for almost half an hour in this bad Russian accent. So thank you.

Thank you Victoria for your performance testing awesomeness. For links to everything of value we covered in this episode, head on over to testguild.com P100. And while you're there, make sure to click on the Try for Free Today link under the Exclusive Sponsor section to learn all about SmartBear's awesome performance testing tool Load Ninja. So that's it for this episode of the Test Guild Performance and Site Reliability Podcast. I'm Joe. My mission is to help you succeed at creating end and full stack Performance automation awesomeness. As always, test everything and keep the good. Cheers. Hey, thanks again for listening. If you're not already pleased part of our awesome community of 27,000 of the smartest testers, DevOps and automation professionals in the world, we'd love to have you join the fam@test guild.com and if you're in the DevOps automation software testing space, or you're a test tool provider and want to offer real world value that can improve the skills or solve a problem for the Guild community, I'd love to hear from you. Head on over to test guild.info and let's make it happen.