Effortless Load Testing Using Open-Source Tools with Simon Aronsson

Welcome to the Test Guild Performance and Site Reliability Podcast where we all get together to learn more about performance testing with your host, Joe Calantonio.

Hey, it's Joe. And welcome to another episode of the Test Guild Performance and Site Reliability Podcast. Today we'll be talking with Simon all about effortless load testing using open source tools, a code first approach to performance testing, monitoring, performance testing and much more. Simon is a developer, advocate, open source maintainer and GDG organizer for Sweden. He has more than 10 years of experience working in the tech field as a full stack developer, an ops engineer, a tech lead and architect. During the last couple of years he's put a lot of time and effort into DevOps practices like cloud development, automation and creating highly effective self organizing teams. And he also was a speaker at this year's Perf Guild. He had a really well received session there. So really a lot of great stuff there. If you do anything with performance testing, you don't want to miss this episode. Check it out.

This episode is brought to you by SmartBear. Listen, load testing is tough. Investing in the right tools to automate tests, identify bottlenecks and resolve issues quickly could save your organization time and money. SmartBear offers a suite of performance tools like Load Ninja, which is a SaaS UI load testing tool and Load UI Pro, an API load testing tool to help teams get full visibility into UI and API performance so you can release and recover faster than ever. Give it a shot. It's free and easy to try to head on over to smartbear.com solutions performancetesting to learn more.

Hey Simon, welcome to the Guild.

Hi Joe, thanks for having me.

Awesome to have you on the show. Simon, before we get into it, is there anything I missed in your bio that you want the Guild to know more about?

No, I think it sounded great.

Cool. Hey, what's a GDG organizer?

Well, GDG is actually Google's like developer groups that they sponsor for doing different kind of events related to Google technology basically. So it's a meetup group or a meetup group concept that is available all over the world.

Very nice. So I know you're a hardcore performance engineer, but also as I mentioned in your bio, you are into DevOps. What got you into DevOps and are you a believer in this site reliability engineering? Is that one of your interests as well?

Yeah, for sure, for sure.

And why do you think that is? Is that a growing trend you've seen as more companies? I think it may even started with Google segment. Reliability engineering is a growing trend you see more and more performance engineers getting involved with. Yeah.

You know, testing in an environment that is actually not your production environment. That usually comes with a lot of drawbacks. Right. For instance, you might not have the right sizing, or you might not really know that the results you get are actually equivalent to what you would get when running them in your production environment. So automating things and making sure to actually test in your production environment makes a lot of sense for that reason, I think.

Absolutely. But another trend that I've been seeing is, and I don't know how many companies you see are doing this, is shooting performance testing more like software development. So more and more companies are bracing using tools that people can actually develop their own performance tests in the same language that the developers are using. Is that also a trend you've been seeing as well?

Yeah, for sure. And that's one of the key things with the tool that I work with, K6 as well. We really think that giving developers the tools they need to be able to do performance testing and site reliability engineering, using code rather than having to do it through manual steps, really smoothens the process and makes it deliver a lot more value to the organization.

Is there anything you see developers struggling with though, to embrace this type of approach or maybe that they get tripped up on because maybe they may be familiar with development, but not necessarily performance testing in general?

Yeah. One of the things that I think developers are really struggling with is to sometimes understand the importance of testing. Right. Because most of the time that's not been something that you've been doing. Maybe you've done unit tests, you might have done integration tests and end to end test, but doing performance test or usability testing or reliability testing might just not be something that they're used to. So it's really important when you do this kind of shifting right or shifting left, I'm not really sure what to call it at this point, but shifting left. Maybe it's really important that you have experienced people around you that can help you and mentor you into thinking like a tester. Things like traffic shaping or common scenarios where you would usually get problems, for instance, during a user induced retry storm or things like that just might not be that common for developers to even think about.

Yeah. What I've seen also is performance testing is such a niche and a lot of times it includes a whole team. Like you need a network guy, you need a mainframe guy, you need everyone coming together after the test to say, okay, here's what happened. So as people are shifting left and they're getting developers more involved with this. Is there still a need for some sort of higher level performance person that's able to shepherd the team and be able to maybe interpret the results in a way that everyone can understand better?

Yeah, I definitely think so. I'd say that it's probably more a shift in terms of that person being some kind of mentor and guidance to the rest of the team, rather than being obsolete. Right. So they still have a very important role to play on the team. In coaching, people think about performance testing in a valuable way as well as knowing what, as you say, what to do with the results that they get from testing. Because having visualization and being able to watch a diagram and all that, that's really good. But unless you understand what you're looking at, you really don't have that much value from it.

Great. So as people are producing, they're creating more and more performance tests, the results themselves. Back in the day with something like loadrunner, it had a pretty nice report. It would be able to gather everything for you. But I think nowadays more people using open source tools or tools like Elk Stack and Grafana. Is that something you see as well? And if so, like any suggestions to help people to visualize their data more like you mentioned?

Yeah, I think Grafana is a great start for most people. It's easy to set up and you'll be up and running in like 10 minutes, maybe using Docker or something like that and depending on what tool you use. For instance, if you use K6, you get a lot of dashboards and things like that for free just to get you up and started. But for sure, I really think that is one of the necessities to really gain value from your performance testing. For sure, sitting and analyzing RAW data points or raw metrics, that's really not something that I would recommend to anyone.

Yeah, I know you all do a lot there at K6. So any lessons learned from maybe building your own tools and solutions?

Yeah, it's still a bit immature when it comes to the open source part of performance testing. In my opinion, people use their own standards. We really have not agreed on any common standard to use to talk to each other between these different products or tools. I know OpenMetric and OpenTelemetry are great initiatives to try to solve this, but it's still a bit messy when it comes to integrating between tools. For instance, the format that we might provide from K6 might not be compatible with Datadog or for that instance, New Relic or something like that might expect so that's definitely still an issue.

So besides that, I think a lot of teams struggle with the visualizing data, coming up with some meaningful monitoring and data visualization. So is there anything you recommend folks use or set up for being able to get really meaningful data visualizations out of their performance results?

I think one of the key necessities to do meaningful visualization is to first really think about what you want to what is critical for your company or your organization. Do you have SLAs in place? Have you ever thought about your SLOs? If not, maybe there's where you should start, because your visualization should probably be some kind of reflection of whether you've succeeded in meeting those SLOs or SLAs rather than just being like arbitrary numbers on a graph, right? So having your SLOs and SLAs, I.e. service level objectives and service level agreements in place before you get started with building your visualization is probably a good investment in terms of time.

So I guess along that kind of line, as well as how do you transform your output into actual insights that maybe help your SLOs and SLAs?

I think it might make sense in that case to make really easily digestible metrics. For instance, we often talk about using a pass fail criteria, right? So you make it kind of like a Boolean visualization. So either you fail it or you pass your slo. And if you integrate those into your load test, even at the testing stage, then you'll get really nice actionable insights of whether you still fulfill your SLOs or whether some kind of regression has hit that makes you fail it. But at the same time, to make that actionable, to really make it actionable, you need to provide some metadata together with that. For instance, you might want to visualize the, or rather render the commit hash for the commit that introduced this regression in case it's that clear cut. Or you might want to provide a set of commits since you started to see the trend going downwards, or if you will.

So I know you mentioned metadata earlier and one of our previous conferences you had a proposal for a session and you mentioned there's a significant difference between visualizing data and having data describe something meaningful. So is this metadata one of those ways? And if so, are there other ways people can make that data more meaningful?

Yeah, I think so. The metadata is actually really important to get some kind of sense for what you are describing and what other context there might be that could help the user to narrow down the possible sources of the regression that you're seeing.

At the same time, it's also important that you try to create visualization that really shows something that you could action just as you say. I mean, it's a huge difference between having your graph display all the metrics that you've done during a test, which might be hundreds of thousands, or rather showing the average duration of a call that's in the 95th percentile of your requests. And I really think those subtle differences makes a huge difference when it comes to whether you'll be able to get any actionable insights out of your visualization.

Yeah, so I know we jump right into visualization. I guess the visualization is only good as the data you're capturing. So I guess another thing that folks struggle with is how to create a performance test to create data that is going to give you visualizations that give them proper information. So do you have any suggestions on how to create like a goal orientated approach to performance testing that would help create more meaningful data once the test is done?

Yeah, for sure. We usually recommend that you rather than just hammering on endpoints, which might have been the classical way of doing load testing that you actually try to think about what the user is trying to achieve. For instance, if you want to test a site that's used for ordering tickets to a Metallica concert, then it might not make sense to hammer the start page for 10,000 requests per second because that's not really how your users would behave. But rather than doing that, that you actually think about what they try to achieve, maybe they try to go to your start page, log in, search for the Metallica concert, pick their seats, order them and then download the receipt. Then that's probably what your load test should try to mimic. Right. Or try to implement. Because that's when you'll know whether your, whether your web page or whether your app will actually be able to withstand a. Yeah, a real user scenario.

I think that's great advice. I think a lot of people don't focus in on user journeys. I think another place that they fail to leverage performance testing is in everyday test automation. Do you have any or have you seen people incorporating the load testing into what we traditionally call functional test automation or everyday type of test automation that runs maybe in CI cd?

Yeah, I think there's still a lot of way to go when it comes to incorporating your tests into cei. I know some users have started doing it, but many users still run their performance tests as maybe a one off or something they schedule once a week to do during night. I really think it makes Sense to try to incorporate it into your CI and have it run on if not every commit. So at least once an hour just to make sure that you're actually able to pinpoint the problem before it reaches the user. And now with all the shifting left and trying to incorporate your or trying to deliver on even every commit or multiple times a day, it might be necessary to actually get that kind of reassurance early on in the process. So I would definitely recommend that you try to incorporate it into your CEI and actually run it continuously.

So to actually do that, I guess, well, without causing a lot of issues is planning. So any best practices on how to effectively plan maybe a load testing strategy or performance testing strategy before, you know, as a shifting left before it goes throughout the full pipeline? Yeah.

One thing to think about, which is common for all kind of testing, especially when you test against production or where there might be actual user data, is to make sure that you actually segment your actions right. So you'll be able to separate what you actually are creating as part of the test from what's being created by an actual user. And the closer you get to production, the more important that usually is. For instance, it wouldn't be that good if a test actually went in and placed orders on tickets for a real user event. Or that would quickly be pretty messy. So making sure that you actually segment your data generated by your test into something that's easy to roll back and easy to separate from the actual user data, I think is key. The same goes for when to do these performance testing. If you are in an organization that might be used to running in production and really testing there, you might have the scalability or the ability to to run these tests. Meanwhile, users are actually using the site. But for most of us that's not really the case. So we'd have to think about when we might have higher or lower usage of our application and try to plan the tests around that. And trying to keep those reoccurring continuous tests in some environment that won't really affect the user. So trying to run continuously but planned in a production environment I think is a really good practice. But at the same time running even more frequently in an environment that doesn't affect your users.

Also makes a lot of sense. So it's not really black and white. I think you have to mix and match a bit.

So we talked about actually, you know, as people are shifting left, they're planning their performance tests or their load tests, but once they start actually implementing it, is there anything you see? Maybe Common things people are doing wrong that still, do you think they should be avoiding maybe some common pitfalls while they're scripting their load tests that you see over and over again. If they just avoided it, they'd be in a better situation.

Yeah, for sure. Especially with K6, where we've really focused on allowing the user to ramp up quite a massive virtual user count during the test. So having this kind of concurrency really creates a different set of problems that you might not have thought of when you're doing maybe 100 virtual users at the same time. One of those is how to treat the responses of your actual requests. I see a lot of people almost on a weekly basis that write their test and don't make sure to, for instance, discard the response that they get from the server. And if you do 10,000 requests per second, this quickly adds up and becomes quite a huge amount of data that you need to keep in memory. So, for instance, trimming small, subtle things like that to alleviate the load on the machine, generating the actual load against your system makes a ton of difference. So being mindful of what data you get in response from the server and making sure to clean that up so you don't get any memory leaks or ballooning memory, as well as trying to keep it as simple as humanly possible. Because a lot of users might think that, okay, let's sit down and plan this actual test in detail up front and visualize the end goal and try to build towards that instantly. But that's usually a great recipe for never being able to finish your project or finish your testing, but rather trying to keep it as small as possible. Find a really, really tiny part of what you want to test and start testing that. Make sure you get that down really solid and that it gives you the value that you want to get out of your test before expanding your scope. So, for instance, in the previous example that I mentioned with the Metallica concert and the tickets, instead of trying that whole user journey instantly, maybe just start with trying the login functionality and making sure that you get that down solidly. And once you get that add the next step. You won't get as much value out of it as you would from your final, like your end goal or your end game. But at the same time, you'd get value much quicker and you'll be able to evaluate whether your assumptions were correct or not. So I guess to make some kind of summary, make sure that you are cautious about memory usage on the load generator by cleaning up the data you receive from the server or whatever else you might create and to start small and iterate.

Great advice. So you know, you did mention K6 a few times. I'm not sure how many people are actually familiar with K6 and how it helps them to shift their performance testing left more so I just want to dive in a little bit. I know I had previous episodes on K6, but the folks that don't know what is K6 and what do we mean by it, it's more developer friendly.

K6 is an open source load testing tool. It's been around for a couple of years now. I think the first version was released in 2017. It's from a company called Load Impact has been in the performance testing industry for about 12 years now. So we've really been doing performance testing for a while and we develop a scriptable performance testing tool and a scriptable performance testing tool that is built in Go, but also provides a JavaScript API or a JavaScript layer for the user to to build their own logic around their performance test. So you'd be able to, for instance, in comparison to many other tools, write quite advanced logic or take conditional paths depending on the outcome of your previous action that just wasn't possible in the previous tools. And together with being able to script it and also not having to use a GUI to create your performance tests, we really believe that we provide a experience that closer matches what developer might.

Be used to Cool. So as they're developing it's just like a command line that they could type in to do certain activity to do quick performance tests.

Yeah, for sure. If you compare it to for instance, node and the NPM package manager that you use for node development, you usually have a package JSON file where you're able to set up your scripts, install your packages and things like that. K6 works similarly in that you'd be able to create your test, have it sitting in a folder somewhere or in your source code repository and then you just be able to from your command line or from the CI just run it instantly and have your results output in the terminal or to whatever source you might whatever data source you might want to send it to.

Can they also bring it up in their own ides so they don't have to constantly be switching back and forth between different user interfaces?

Yeah, as it's just JavaScript or it's pretty much vanilla JavaScript with some tweaks they'd be able to use whatever editor they use when they're doing their daily work and even be able to check in their code together with the actual server code. So they'd be able to have that in the same repository if they like to working with it side by side, maybe even reutilizing different functions or class shapes or things like that that they might have from their actual application in their test code as well.

That's pretty cool. Is there anything in K6 you think people don't know about, like functionality it has that you don't think enough people are using?

One such thing is the recently created ability or feature around scenarios. With the last version of K6 0.28, I think it was, you're actually able to set up different kind of quite powerful executors that simulate or that create different types of traffic and have them run different test scripts at the same time, maybe with some kind of delay in between them or in parallel. Or you get a lot of flexibility regarding how to shape your traffic and setting up quite advanced traffic scenarios. And to my knowledge, at least what I know, I haven't seen it in any other performance testing product yet, but might be that someone else has it as well. But I really think this is a really powerful feature that you should make sure to check out if you're into performance testing.

Very cool. Okay, Simon, before we go, is there one piece of actual advice you can give to someone to help them with their performance testing efforts? And what's the best way to find or contact you or learn more about Case 6?

The most actionable advice that I could probably provide is that which I already mentioned earlier in this. In this conversation, you should start by looking at your goals. What is it that you're trying to achieve? You really need to have a vision, a realistic vision of what you're trying to achieve before you get started. For instance, it might not be feasible to. Or rather, if you start by writing the tests and then formulate your goals, then you're already biased by the outcomes of your tests. Or you might have unrealistic expectations due to just picking some value at random. Right? So to me it really makes sense to first think about, okay, what is it that I'm trying to achieve? For instance, I need to be able to have 2,000 concurrent users using my application or my system under test. Okay, then I should formulate goals around that and have those provide the path of how I implement my performance tests rather than the other way around. So I really think it makes sense to start with the goals and then build your tests around those goals. And that's something I see quite a lot that people do the opposite way around or the other way around.

Great advice. And Simon, the best way to find or contact you.

You can find me either on Twitter where I'm 0x12B or on GitHub where I'm S I m s k I g.

Or through email at sime6io.

Thank you Simon for your performance Testing awesomeness. For links to everything of value we covered in this episode, head on over to testguild.com P49. And while you're there, make sure to click on the Try Them Bolt Today link under the Exclusive Sponsor section to learn all about SmartBear's two awesome performance testing tool solutions, Load Ninja and Load UI Pro. So that's it for this episode of the Test Guild Performance and Site Reliability Podcast. I'm Joe and my mission is to help you succeed with creating end to end full stack performance Testing awesomeness. As always, test everything and keep the good. Cheers.

Thanks for listening to the Test Guild Performance and Site Reliability Podcast. Head on over to testguild.com for full show notes, amazing blog articles and online testing conferences. Don't forget to subscribe to the Guild to continue your testing journey.