Creating a Performance Testing Framework with Stephen Townshend

Welcome to the Test Guild Performance and Site Reliability Podcast where we all get together to learn more about performance testing with your host, Joe Calantonio.

Hey, it's Joe. And welcome to another episode of the Test Guild Performance and Site Reliability Podcast. Do you want to know what the components are of a performance testing framework and why you may want to build one? Open source performance testing tools are awesome, but for serious performance testing you need a framework of tools to provide all the features you need for your performance testing efforts. In this episode, Stephen Townsend and SRE IAG shares why you should build your own performance testing framework, the components that make up a performance test framework and suggestions for tools and scripting languages, and also some examples of frameworks implemented in the real world. This session was actually taken from the 2021 Automation Guild. If you don't know, Automation Guild is one of the first ever online events dedicated 100% to automation testing. It's a five day event with the first three days dedicated to UI functional automation and the last two days dedicated to all other DevOps automation type activities like performance Security, chaos engineering. So this was taken from Steven's performance session at the 2021 event. If you haven't already, make sure to head on over to automationguild.com and register today. We have some awesome performance and reliability sessions for that event as well. You don't want to miss it. Check it out.

This episode is brought to you by SmartBear. Listen. Load testing is tough. Investing in the right tools to automate tests, identify bottlenecks and resolve issues quickly could save your organization time and money. SmartBear offers a suite of performance tools like Load Ninja, which is a SAS UI load testing tool, and loaduipro, an API load testing tool to help teams get full visibility into UI and API performance so you can release and recover faster than ever. Give it a shot. It's free and easy to try. Head on over to smartbear.com solutions performancetesting to learn more.

Hi there, my name is Steven Townsend. I am a performance specialist based in New Zealand. Today I'm going to talk about the benefits of building your own load testing framework with some examples of what that might look like things that you could build. I also want to explore some of the guiding principles around frameworks and how to do them the most effectively.

Now, historically at IAG there has been multiple performance testing teams. Recently three of those teams merged together into one from different cities and that's the team that I'm in now. Now my colleagues in Melbourne, which is in Australia, built An open source load testing framework called Mark5.9, which is available on GitHub. This is a fully featured load testing framework in New Zealand. We hadn't built anything quite as mature, but we had built some tools and some processes and some sort of mini frameworks along the way. And when we compared Mark 5.9 and the tools and frameworks that we had built, I found that we were often solving the same kinds of problems, sometimes even covering exactly the same functionality. And that was interesting for me. It got me thinking about the role of frameworks in the work of a performance tester. So one question I have is, in the modern era, has building a load test framework become a core part of the performance tester's job? If it has, what are the universal things that every framework should have? What needs to cover, no matter where you are in the world or what team you work in? But conversely, what are those unique or specific things that you need to include in your framework which other teams won't necessarily put into theirs? I think it's those unique or specific needs that drive the need for each team to build their own load testing framework and not just download something from the Internet. Because you need to build by knitting together a collection of tools or custom built tools that you built yourself to meet the specific combination of people, technology, process and organizational structure that you and your team has. In this talk, I'm going to talk first about why you would build a load testing framework and then about how to understand your process in your team as a performance tester, which helps inform how to build a framework. Then what are the specific goals that you set for your frameworks? Because that will change the way that they get shaped. I then want to talk about a specific examples for different parts of the performance testers process and finish with some takeaways. Okay, why build a load testing framework? With the framework, we can use multiple tools and chain them together into one smooth process. And that's a powerful thing because it means we get to use the best tool for the job at each stage of our work. And that's a powerful thing. In the past, I think performance testers have tended to compromise because they didn't want to have to deal with too many different tools. So we just stick to one tool and deal with what it provides us, but that's no longer the case. Frameworks can enable us to enable other people to do more performance testing and engineering work for themselves. And that's a win win for everyone. And it's a necessity in the modern era where there's just too much happening and too much change for a centralized performance testing team to do all the work.

Frameworks can provide more enjoyment for our work as performance testers through putting automation on top of repetitive and tedious tasks. Frameworks can allow us to be more efficient in the way that we work through automation by but also through things such as scheduling tests out of hours. Suddenly those times of the day where normally nothing productive would be happening, we're now running a bunch of tests. That's great. Frameworks can also provide consistency at multiple layers, consistency between different projects or platforms or teams. They can provide us a consistent language to talk about performance or performance testing. It can even allow us to have more consistent results when we run load tests by doing things such as running a test at the same time of the day every day. Because as we know in an environment, the performance can change wildly depending on the time of the day. And the last one is very subjective. But my observation is that high performing performance or load testing teams implicitly build frameworks. So I think it has to be a good thing. Okay. Before we can build a framework, we first need to understand our working process.

So when my talk in 2019, I did a virtual near Tus Pactalk and it was about preparing yourself to automate performance testing. But I also talked about something called a value stream map. So value stream mapping is a way of understanding your process or your team's process and visualizing that in an easy to understand way. So in that process you define and explore things such as what are the stages of your work? So that's what we see on the screen. Here is some example stages of a load testing process.

We also in our value stream map cover things such as what are the individual activities required to complete each stage of the work, what are the inputs into each stage and the outputs that we produce from each stage of the work. If you want to understand the way that you and your team work and your process, I highly recommend looking up value stream mapping and looking at that further.

So here I've added on top of the different stages, individual activities or groups of activities that we might complete. And as a performance engineer, I know that there are dozens, if not hundreds of other stages of work that you might go through. But here I've tried to keep to the stages of work, the basic load testing process. And I'm assuming that these stages would be pretty universal for almost every team in the world and every organization.

I've also tried to stick to stages and activities that we are either at IAG Already using a framework for or considering building a framework for. The point of this picture, and there's a lot of detail going on here, is that when we talk about a framework for load testing, it's easy to get fixated on test suite execution. But we really need to consider the whole picture, the whole life cycle, and think about how we can apply frameworks at any stage, not just test execution. So when you go about building a framework, you need some goals. You don't just want to build a framework for the sake of a framework. Each team might have a different set of goals or maybe just some slight tweaks on those goals. At iag, some of our goals are being able to enable others. So delivery teams throughout the organization to do more performance testing themselves. That's good for everyone because it gives them accountability for the performance of their own platform. But also importantly is that in modern delivery, when there is so much change and so much happening throughout the organization, a centralized team of performance testers just doesn't scale, it just doesn't work. So when we build frameworks, we build with self sufficiency and user friendly tools and try and build things in a way where developers and testers or bas don't have to learn load testing tools or performance testing process in detail. We also build our frameworks in a way that where we try and automate repetitive and tedious tasks as much as possible. We also try and build tools within our frameworks, if we build our own that can be used independently. So if a team wants to pick up part of our wider framework, but not the whole thing, they can do that. That helps with adoption because sometimes teams have their own partial frameworks and series sets of tools and they don't necessarily want to adopt everything that you have to offer. Another principle I have is that let's try and use the best tools and process for each task and utilize frameworks to enable us to do that. Everything we build and everything I talk about today serves one or more of these goals. Okay, now I'm going to go through these different stages that I talked about earlier. And for each one of these stages, I want to introduce what a framework might look like.

If you don't know what workload modeling is, it is a process of describing the load on a system at a given point in time. In other words, it tells us how much load to apply during a load test.

Workload in itself has different aspects. Throughput is how much stuff are we throwing at a system. Concurrency is about how we apply that load. So you might do 10,000 requests an hour. But how many users, concurrent users are going to be applying there? Or how many concurrent API calls are going to happen within any given second. For example, the nature of the load is more about what kind of business transactions or processes are occurring or which APIs are called more than others.

So where do we get information about workload? If it's an existing system or if there's a legacy system that we can look at production logs or events stored somewhere, that's the best way to go. We can get data from the real world, use it to do a calculation so that we understand what the historic load on the system has been, to help predict what it will be like in the future.

Sometimes we don't have solid numbers or logs, so then we fall back on projections. So we might talk to a marketing team or a data team and get some indication of what load we're going to expect in the real world. But failing that, sometimes there's no numbers at all. So then we fall back on assumptions. And assumptions might not sound very scientific, but they can be incredibly useful and can often be very accurate because people have a pretty good gut feeling about what's going to happen a lot of the time. Now, how would you go about building a framework for workload modeling? I think the goal for workload modeling would be to take the numbers from the real world and somehow insert them into our load testing automatically so the right load is applied. Now doing projections, coming up with assumptions, these are creative tasks. They require human beings to make ad hoc decisions and have lots of conversations and to be perfect constantly, it's difficult to put a framework around that. But taking production logs, putting some calculations on top of them, turning it into some kind of configuration that we can pass to a load testing tool, that's something we can build a framework around. So to get the logs themselves, you might use a tool like Splunk or Azure Log Analytics. If it's public facing, you might use Google Analytics. And all of These tools provide APIs, and so we can automate that process. So once you've got the logs, then we need to do some calculations ourselves. That's up to us. But once we've done that, we get some numbers and we need to convert that to a configuration that our load testing tool understands. I've only mentioned JMETER here because that's the tool we're using the most at the moment. And When I use JMeter, there are two common ways that I pass configuration into JMeter. The first one is you can pass properties or Parameters from the command line individually. Another way that you can do it is you can have a properties file, which is just a text file with a whole bunch of parameters or properties that you want to include. And you point jmeter at that properties file and they all become properties or variables that you can use as part of your test. That's a very powerful thing.

Now, for the framework, we're going to need some kind of glue to hold it all together, some orchestration layer. Now, you could use a CI tool such as Jenkins or Azure DevOps, but you don't need that, especially when you're starting out. You could just use a scripting language of some kind to define some basic steps in process and automation. I often use PowerShell because pretty much every developer or tester in our organization has access to a Windows development machine or laptop. You could also use Unix shell scripts, you could use Python. Whatever scripting language works for you, you can use it.

Two things. Whatever your load testing tool, whether it's JMeter, Gatling, Loadrunner, NeoLoad, whatever it is, will have a way to pass in configuration from the outside world. If you are using a lesser known tool and it does not provide you any way of passing in configuration, that's going to hinder your ability or prevent you from being able to automate your performance testing. The other point is that the difference between a framework and automating things. So at the top of this diagram, where we've got our production logs, our configuration for our load test, and then we pump that into a load testing tool, that's the framework. Everything below is how we are implementing the framework. And in theory we should be able to swap in and out different tools as required. The second stage I mentioned is strategy and approach. And you might be thinking, how is it possible to build a framework around developing a load test strategy? How can you do that?

My perspective is that you can't define a strategy or approach until you understand the risk. I call that a risk assessment, a performance risk assessment. And a big part of that risk assessment is understanding the workload on the system. The workload model. Now, defining an approach, that's another creative task. It requires a person to make ad hoc decisions. It might be custom built every time. It's not something we can easily build a framework around. We've already talked about workload modeling. One of my colleagues is working on a framework around the performance risk assessment. So what they've done is build essentially a questionnaire. It's a web application that you go to. There's a set of Very simple questions which asks you about the business context of your project or your platform, the technology that you're using, and at the end it provides you some very basic recommendations which could be you don't need to do any performance testing you might need to, or you definitely need to. And also all the answers to the questionnaire get automatically sent to the performance engineering team so that we can proactively go and reach out to the team who filled out the form and ask them if they need support or help. So essentially what we're doing here is a self service performance risk assessment and it's something nice and simple and pragmatic. And we are currently building a proof of concept for this.

Test suite. Development and the maintenance of it is historically an extremely costly and complex thing to do. We don't get any value out of building tests only when we run them, which is something to keep in mind. Now, like I said before, a centralized team of performance testers isn't scalable for modern delivery. There's just too much change, too much complexity. So for me, the big challenge that frameworks might be able to help us solve is how can we get delivery teams to the point that they can build or at least maintain their own load test assets. It needs to be intuitive. I don't think it's fair to ask developers and testers to learn a load testing tool.

I want to be able to put together something that utilizes skills they already have. So writing code in languages they know or filling out JSON or YAML config that they're familiar with. Now, I've got quite a few concepts around how I want to try and tackle this problem, but my colleagues in Melbourne have already tackled it using Selenium Automation scripts. So mark 5.9 is a fully fledged load testing framework. There's a lot to it, but part of it is that you can utilize out of the box Java Selenium scripts as your load testing scripts. So the basic process is a developer or tester writes their normal Java Selenium and most teams at IAG are using Java Selenium. So it's not a, it's a skill most teams have. So they just write a test. They might already have a test script already built that we can reuse.

We do need to add some timers to that, which is just a few extra lines of code. This gets built into a jar file which is imported into JMETER and used in a Java Request Sampler and then that is what is used. Then JMETER goes ahead and runs a performance test the performance tests that are run this way are using.

Selenium webdriver, Chrome driver, and lots of instances of Chrome browser in headless mode. And you might be thinking, oh, how can you possibly produce enough load? Well, we have a lot of infrastructure. We can produce many hundreds of concurrent browsers or users at a given point in time, and they can produce a lot of load because it's not just about concurrency all the time. But yes, there are some situations where this can't produce enough load, and in those contexts we fall back to using the web protocol. So this is a nice, simple, pragmatic way of solving a problem. Most teams know Selenium, how to use Selenium, let's utilize that and turn it into a load test. Test execution is probably the most widely automated part of the process, and there are frameworks galore all over the world. So I don't want to spend too much time here. Here I'm only focusing on the test execution, not the post processing that happens afterwards. We'll talk about that next. So we've got two different test execution processes in our team, and they are almost identical. They both involve Jenkins, which either schedules a test or there is we invoke a job or pipeline ourselves manually that goes ahead and runs either Windows command lines commands or PowerShell scripts, or a combination of both of them, which then those are what pass in the configuration to JMeter and then JMeter goes ahead and runs whatever tests that we want to run against the system under test. The only difference here is that one team primarily passes in configuration via the command line, and then the other team. In New Zealand we tend to use properties files, but other than that, almost the same process. Okay, so here we go. Load testing tools are really, really good at doing one thing, generating load. But in terms of results analysis, there are much better tools out there, tools designed for exploring and interrogating and understanding data. So I've built a framework to solve that particular problem because I find that the results analysis built into JMETER isn't sufficient for my needs. So when you run a performance test in JMeter, it produces a CSV result file with the raw data. By raw data, I mean every single transaction or every single request that we make.

Is logged in the CSV file as a row. Now, the process is this CSV is automatically put into a special network folder. And then I've built a node JS application, which I called the stash, which is listening to that folder. When a new set of results is put in there, it picks it up and Processes puts the Data into a PostgreSQL database. There's three tables that I produce. One is the raw results, which is essentially just importing what's in the CSV straight into the database.

Each result file that we upload, I also produce something I call the metadata table. It's like a test log. So there's one line for each test that we run as well. I also have something an aggregated results table where I do some processing some calculations to calculate things such as what is the median and what is the 90th percentile for each transaction we ran during the test. And the reason that I do that is it provides a much smaller table which is much faster to work with for basic reporting. This database with these three tables I connect to using a tool called Tableau, which I've spoken about in the past. Tableau is a bi analytics tool which is very good at connecting to and interrogating data. And that's what I use for my results analysis.

Here's just a screenshot of a dashboard that I've built in Tableau, which is connecting to that PostgreSQL database. I just talked about every part of this. I've custom built the way that I want. And that kind of flexibility isn't something that I don't have in any low testing tool that I'm aware of. Okay. The last step of the process that I've mentioned before is investigating system behavior. Not all performance testers do this most often. When we do get down to this level of investigation, we are looking into an issue. Probably the most common issue is there is a slow end to end transaction and we want to find out where is the time being taken in the different components of the system isolate where the time is and when that kind of problem comes up. I want to see a timeline, I want to see a gantt chart of events and when they occurred.

So one of my colleagues has built a tool for this purpose. So we have this middleware platform in IOG and pretty much every system that wants to talk to another system goes through this one API or middleware layer, which means that the logs from this middleware layer are a treasure trove of timing information. So in this particular example, we're taking those API logs from Splunk. There's actually the team that manages the APIs has built something called a diagnostics API which is in front of that and calls Splunk to get the logs that we need. Something else with mentioning is that many, I see this throughout the industry now is using correlation IDs or trace IDs as headers, and passing them throughout the different components of your system so that you can map a transaction end to end. So in this particular case, we pass in a particular correlation id, give it to the Diagnostics API and it will give us all the logs for that transaction. Now, the Node JS application which my colleague has built, it uses the Diagnostic API for that purpose. It presents a web front end to the user and all the user has to do is paste in a Trace ID hit go and a timeline of all the different API calls triggered for that transaction is produced and that is presented to them on the screen on the web browser. Now, my manual process to do this used to be, here's what the timeline looks like in the web front end right now. So this whole thing takes about 10 seconds. You go to a website, paste in a Trace id, hit go, you get this. The way that I used to do this is I would go into a JMeter project where I would call the Diagnostic API with the Trace id, get the logs and the response, paste them into a file, reformat them, then I'd go in, I run a PowerShell script that I built to parse the logs and to pull out all the timings, put them in a CSV file, then I would go into tableau and I would open that CSV file and either build or, if I already had a workbook, render my timeline. So that manual process used to take about, I don't know, four or five minutes, which might not sound like very much, but if I'm doing that 20 times a day, it adds up and it's tedious, repetitive work. Now, because of this awesome framework, I go to a website, paste something in and get my result in 10 seconds. But the best bit about this is that I other people in the organization can use the same tool, which is giving them independence to go and look into issues themselves, which is a fantastic outcome. So that's load testing frameworks and how we are using them at IAG and why I think that they're useful for every load testing team around the world. My takeaways would be use frameworks to be more efficient, to get higher velocity, to deal with this modern era when we're delivering at such a rapid pace and there's just so much change and so many people and teams involved use frameworks to enable others to do more for themselves so that they feel involved and engaged in performance. It's not a thing that a team in the corner does. And so that you can scale performance testing in your organization. And lastly, use frameworks to improve your work experience and the experience of your colleagues by automating away tedious tasks and getting everyone more engaged and doing more valuable work.

Thanks again for your Performance Testing Awesomeness. Links to everything of value we covered in this episode. Head on over to testguild.com P83 and while you're there, make sure to click on the Try them both Today link under the Exclusive Sponsor section to learn all about SmartBear's two awesome performance test tool solutions, Load Ninja and Load UI Pro. And if the show has helped you in any way, why not rate and review it in itunes? Reviews really do matter in the rankings of the show and I read each and every one of them. So that's it for this episode of the Test Guild Performance and Site Reliability Podcast. I'm Joe. My mission is to help you succeed with creating end to end full stack performance Testing Awesomeness. As always, test everything and keep the good. Cheers.

Thanks for listening to the Test Guild Performance and Site Reliability Podcast. Head on over to testguild.com for full show notes, amazing blog articles, and online testing conferences. Don't forget to subscribe to the Guild to continue your testing journey.