Performance Testing Using Artillery with Hassy Veldstra

Get ready to discover some of the most actionable DevOps techniques and tooling, including.

Performance and reliability with some of the world's smartest engineers.

Hey, I'm Joe Colantonio, host of the DevOps Toolchain podcast, and my goal is to help you create DevOps toolchain awesomeness.

Hey, it's Joe.

And welcome to another episode of the.

Test Skill DevOps toolchain. Today we'll be talking with Hasee all about performance testing using Artillery.

There's been a lot of new features.

Coming out with Artillery. Really excited about the solution.

We featured them on a few times.

On our performance guild. So really excited to have the actual creator of Artillery joining us, Hassi, who is an open source developer, former sre, and the creator of Artillery IO. If you want to upskill your performance testing efforts in DevOps and you don't know what to do or where to get started, this is the place to go. Listen all the way to the end to hear how Artillery can help you make better, more performance software. You don't want to miss this episode. Check it out.

Hey, if your app is slow, it could be worse than an error. It could be frustrating, and in my experience, frustrated users don't last long. But since slow performance is a sudden, it's hard for standard error monitoring tools to catch. That's why bugsnag, one of the best production visibility solutions in the industry, has a way to automatically watch for these issues. Real user monitoring. It detects and reports real user performance data in real time so you can quickly identify lags. Plus get the context of where the lags are and how to fix them. Don't rely on frustrated user feedback. Find out for yourself. Go to Bugsnag.com and click on the free trial button. No credit card required. Support the show and check them out.

Hey Hastie, welcome to the guild.

Very excited to be here, Joel. Big fan of what you're doing.

I appreciate it.

I've been following you for a while.

And it's great to finally have you on the show. Like I said, you seem to be releasing more and more really cool new features, so I really want to dive in. But before we get into it, I'm just curious to know, as someone that probably is busy, why did you decide to create a solution like Artillery IO?

Personal pain. That's what it was.

Artillery goes back quite a few years now. I think the original release on GitHub is either 2015 or 2016. So we're talking, you know, seven or eight years. And the origin story, I mean it's the same for a lot of open source tools. I think I was on a team that was working on a real time chat app and we needed to load test it because we had reasonable reasons to believe that would get 50 to 100,000 users on day one. And nothing that existed at the time could do socket I o load testing. So I put together just a bunch of scripts really. It wasn't even like a tool that someone else could use. And then what happened? I gave a talk about the actual architecture of the application that we're building at a conference. I just mentioned kind of as a, by the way, you know, we built these scripts to load test it and then most of the questions that I got afterwards were about the load testing script and not about the architecture of the app itself. So I kind of thought, okay, you know, I'm not the only one who's maybe struggling with the load testing tools that you know, existed at the time. And if we go back to 2015, 2016, the landscape was quite, quite different. So Artillery ended up being, you know, a fresh take at the time of load testing. And you know, we're doing it again now with some of the features that were released recently and I'm happy to talk about those as well.

Awesome. Yeah. So, you know, probably people around then, JMeter probably was the biggest one. And how did you design Artillery then to be different than maybe some of those other tools that may have been around at that time?

Yeah. So JMeter was, I think it still is by far the biggest load testing tool. Right. It's pretty much the standard that everybody knows. I remember when I tried using JMeter, you know, I was really kind of turned off by the installation process. You had to install Java, you had to get all of that set up. It was still is mostly a GUI first tool as well. And I kind of wanted something a lot more lightweight, something that was easy to dockerize. You know, Docker was up and coming at the time already. I wanted something that was really easy to run in CI CD pipelines and I really wanted something that would be source control friendly, which JMeter wasn't at the time. Like there are some solutions that, you know, you can use today, but that was the main motivation, a really lightweight, source control friendly tool. And I also wanted something that I could extend in JavaScript using Node JS specifically. Just because the ecosystem is so big and you have a module to do pretty much, you know, anything. And you know, anyone who's done kind of production grade serious load testing will know that you always end up writing some custom logic for complex apps. So I wanted something that I could use JavaScript with as well to make it easier.

Nice. So how is it to get started with. I know some tools have come out recently that cater towards more developers where it actually plugs into the ecosystem, uses the same language. I'm trying to think of Artillery. We had some sessions at my conferences over the years. I could have sworn to use it as a YAML file, but I'm not sure if that's true. So like, how easy is it to get started with Artillery?

I guess the question it's pretty easy and you know, we try to make it as easy as possible. So you have a few options to write your first script with Artillery. One of them is that YAML, the main specific language, which ends up being a really readable representation of say HTTP request that you want Artillery to make. So you know you're literally writing get URL headers, whatever else that you want to customize. You also have the option since fairly recently, to write Playwright based code and then run that as load tests. And that's amazing for testing really dynamic web apps. And with that it means that you don't even need to learn anything or too much. That's Artillery specific. You write your Playwright script and then you run it as a load test with Artillery. And then Artillery also lets you drop down to just raw, JavaScript and Node so you can write your load test definitions that way as well. And also these days with, you know, Copilot and other AI helpers that we get as developers, what we found just internal testing on the team is that GitHub, Copilot and ChatGPT with GTP4 especially is actually really good at writing Artillery scripts. It's kind of become my just go to, you know, first my go to way to get a first draft of a low testing script with Artillery these days.

So can you explain that process then if someone goes, what? So what is your flow then if someone wants to try it for themselves?

So if you have a ChatGPT account, especially with GPT4 enabled, GPT 3.5 is pretty good, but not as good. You can ask it to write an Artillery script for you and you can say, for example, I have a JSON, I have an HTTP endpoint that returns some JSON and I want to send a get request to this endpoint and I expect, you know, this type of JSON to come back and it just writes a script for you that just works 90% of the time. And then if you're using Copilot in VS code, it will autocomplete a lot of Artillery specific, you know, settings for you if you're working inside an existing Artillery test script.

Nice. That's very cool. So you also mentioned Playwright, which has been really gaining speed and functional automation.

Yeah.

So how hard? It sounded like it was pretty easy. So I have an existing Playwright test. I want to make a performance test out. I guess we need to go over what do we mean by performance testing when we're using Playwright compared to just straight artillery. But say someone has a Playwright test, how do they instrument it to just run an artillery then?

Yeah. So maybe the best way to frame it is that you can use the Playwright API. So the Playwright API that you can use to work with page objects or pages or, you know, the browser window in the browser context. And you can write scripts using that playwright API using JavaScript and then you can use the artillery CLI to say, hey, artillery, run, you know, 10,000 copies of this for me. Which means 10,000 browsers. Right. So one thing that we released recently is support for running load test using serverless technologies on aws. So Lambda and Fargate and our Playwright support plugs directly into that. So what it means is that let's say you're a QA engineer and you want to run a production grade load test on a web app that you know you're looking after. You can write a short Playwright script that goes, you know, go to the homepage, click on this button, fill in this form, field the entire user journey, and then you can spin up, you know, 5 or 10 or 20,000 browsers on AWS to run that for you with Artillery. And you don't need any DevOps knowledge. Artillery just does everything for you. And it's free, it's open source, it's out there for anyone to use.

Wow. Okay. That's another question I'm going to ask after that. But so when would you use this approach? I guess a lot of times people say, oh, because you're bypassing the browser normally. Well, when I was starting off with performance testing, we had tools like loadrunner that kind of bypassed. It caught the traffic and just bypassed the browser. This way you're leveraging the real browser. So I assume you would probably catch. Is this More for like JavaScript front end applications? Really heavy front ends. That gives you a more realistic idea of what the performance is going to be for the end user on the desktop rather than the backend server?

Yes, that's pretty much exactly it so I've done load testing on complex web apps, you know, using traditional tools. So the two common approaches that you kind of would reach for today as a developer QA engineer is you either completely bypass the front end and you try to replicate what the UI does using direct calls to backend APIs, right? And it's really, really time consuming to test a complex web app that way. And it's also never really representative of the actual traffic patterns to the backend because the UI does its own thing. You know, a single action on the web page can result in 10 calls to backend APIs. And it's pretty much impossible to do that in a, you know, in a way that's truly representative of what's going on. And, you know, you'll spend a lot of time trying to track down, you know, what's going on through the network tab and Chrome dev tools. The tests end up being really brittle. They go out of date really, really quickly. So that's the first approach. Nobody enjoys doing that. The other one is you can save HAR files in a browser, so a complete kind of web archive format of everything on the page, and then you can use load testing tools to do a replay using those HAR files. So that's a bit less time consuming. But you run into the same issue where your load test actually ends up being not really representative of what's happening in the real world in a real app. So what we saw is, hey, it's 2023, right? We all have access to the cloud, we all have access to serverless compute on demand. Why not just run headless browsers? This is not something that was possible even, I don't know, let's say five years ago, but it just makes sense to do it today. So that's the motivation. And by using Playwright. Playwright is absolutely amazing tech. It makes running headless browsers so, so Easy and the APIs are so nice and they're always releasing new features. So what you can do now is write a Playwright script from the point of view of an end user and then run that as a load test. And that ends up basically exercising all of the backend systems in the exact same way that a real user would. So you get the best of both worlds. You've got your say, you've got your backend APIs hooked up to a monitoring system like Datadog or New Relic or whatever. So you can track the performance of that in that system. But then because you're running real browsers, you also get user experience metrics. So Core web vitals. So you can actually track how things like largest content paint or first input delay are affected by increasing load or by high load.

So maybe I'm an old greedy capitalist, but why is this open source?

Like you spend a lot of time.

On, Are you doing it just for fun? Like how is there a model where someone could pay for extra consulting or is there extra features that are more paid based on like a dashboard or something? I don't know.

Yeah, I'm a capitalist. You know, I'd wear a T shirt. Capitalist club. So the way that we look, by the way, Artillery is an open source project. All of the stuff that I've talked about so far is free, it's open source. But there's also a company behind Artillery that's two years old at this point. There's an entire team. You know, we obviously want to build a sustainable business around it. So the way that we look at load testing is that load testing is one of the most valuable and high ROI tools that you can have in your toolbox when it comes to performance, when it comes to reliability, when it comes to making sure that, you know, your services can scale and can withstand peak traffic. Right. But it's really severely underused compared to other types of testing, you know, and that mostly comes down to how clunky or difficult the tools are that we have today and how, you know, how limiting they are in many ways. So pretty much every other tool that's out there today, other than Artillery, is stuck in that pre cloud native world, you know, where you can't just scale out and run your test from your own AWS account. But that should be possible, right? It should be something that just is available out of the box today. So what we want to do is we want to make load testing as easy to to do as unit testing or integration testing. And we want everybody to load test and we want everybody to do it continuously from, you know, on schedules or from their CI CD pipelines. The way that we look at it is by doing that, we help increase the reliability of all of the software that runs in the world that, you know, we all depend on. And then where the commercial side of it comes into is that once a team runs those load tests on the schedule, they run them from a CI CD pipeline. What you kind of end up needing is a way to collaborate on all of the performance testing data that is generated by those tests. So we offer a completely opt in optional add on, which is a cloud dashboard that you can configure the cli to send all of those metrics and reports to and it turns that dense numeric data into actionable reports and it allows you to collaborate on those reports across different teams or even different business units. So that's the business model behind artillery the company. But the, you know, to answer your question, the reason that we're given all of this stuff away for free as open source is because we want more people to run low tests and ultimately have more reliable and more resilient systems.

Absolutely. So I started load testing performance sucks in 1999 and it was very difficult and been trying to get people more on board seems like still a struggle, but it seems like it is picking up. And I'm wondering why. Is it because people are going cloud native and that just the ability of the technology now to really be able to get real, real data back in the day like we tested in a staging environment and then you go to production. Completely different nowadays. I guess with the cloud it gets more realistic. So I don't know. Is there a reason why you feel this way? Is there an uptick in why performance testing is so important nowadays?

Absolutely, yes. I think what we're realizing more and more as an industry is that the stuff that we work on is just incredibly complex and it keeps getting more and more complex every single year. You know, things like, things like the cloud, things like microservice architectures, things like external third party dependencies that we use more and more for all sorts of, you know, non core things to the product that we're building. All of that stuff means higher development velocity usually. But with that comes increased rates of change and also just increased complexity. And with that complexity come things like emergent behaviors, unforeseen interactions between components. So I think that what we're realizing or what the industry is realizing is that load testing is a great tool to kind of tame that complexity and to maybe understand it a bit better. And to use load testing as almost a fitness function and kind of evolutionary terms to help shape your system as it grows, as it develops, and to stay continuously ready for max traffic. And it kind of, you know, it kind of tracks with a couple of other trends that we're seeing in the past few years. Chaos testing is one of them, like exactly the same motivations. You know, we've got this thing, it works, but there's no single person that can kind of understand the top to bottom anymore. So we need to actually induce failures and see how it reacts. Another trend is testing and production. And again, kind of the same motivation, production Systems are so complex we cannot hope to replicate them in staging. So we need to test directly in production as well. So that's what I've been seeing.

Absolutely. And also observability, it sounds like, would play well into that also. So you've been speaking about testing in production for a bit. I think I saw a YouTube video from 2020. So back in the day, once again, this was very difficult to do. People would think you lost your mind if you said, we're going to load test in production. So is it because we're able to scale up environments that are just like production that we're able to do this? What do you mean by load testing of production?

So a lot of people still have that same reaction these days when you suggest low testing production. It's like, whoa, no, we can't do this, we can't do this. You know, it's almost a reflexive reaction because it. Well, it sounds dangerous. I've. So, you know, you mentioned my background is in SRE, it's in DevOps. I started out as a software engineer and gradually transitioned over to more, you know, operational backend type of work. And I've done a lot of production load testing over the years. And what I would say is it's probably the most value that you can get out of load testing is to do it in production. And 90% of the time, it's actually really safe to do. There's a few things that you need to do to make it safe. But load testing in production does not mean bringing production down with your low test. And the main barriers to it are not actually technological. They're more around people's perception and organizing and aligning different teams and different stakeholders just around the idea of being able to do it. But every single time, every single team that I've been on, every single customer or user of Artillery who are doing production load testing, it can be a long road to get to that point. But once you've done it, it's almost like it's a magic moment. All of a sudden you realize, oh, hey, we're low testing in production. So we know that our actual system can, you know, handle plus X amount of traffic or, you know, no matter how many times we deploy tomorrow, we know that it will stay performance that, you know, our SLOs are going to continue to be met. And yeah, it's. Everybody's an instant convert once you make it happen. That's what I've seen.

Absolutely. So, once again, nowadays we used to have a dedicated performance team that own Performance. But now I guess you're trying to make this easy enough for everyone to use. And this name of the podcast is DevOps Toolchain. So who does performance nowadays? When you talk to companies or other people, is it SREs, is it still a performance group? Who's responsible for it?

Typically it's a real mix and I would say that. So there's kind of the reality on the ground today, but there's also a trend that we're seeing that's emerging. And the reality on the ground today is that most of the time load testing is done by some sort of SRE or SRE adjacent team, or maybe the back end team, maybe the operations team. Some companies do have dedicated performance and reliability testing teams, but you know, I wasn't necessarily. So I can say that a lot of the times the SRE team is adjusting giant bottleneck and often an impediment to product velocity. So there's always this tension between product teams and SRE teams and a lot of it is inherent. It's just because of how things are. If you're an sre, you are thinking in terms of reliability and that is often fundamentally at odds with the rate to change and throwing things into production. But a lot of it is also kind of accidental and SRE teams are always overworked on their staff like at the same time. So anything that you can do to make SRE things self service for developers, that's usually a big win and that's usually something that you want to do. So what we're seeing is that more and more companies that operate at scale that really want to invest into performance, they build out internal load testing platforms for their developers. But if you do that, every single one of those teams, they run into kind of the same set of issues. So the first one is that if you want to run production grade load tests across the company, you need a way to run distributed load tests. So that means you need to come up with some sort of an infrastructure based solution where developers can create those distributed load tests. You know, it could be dedicated Kubernetes clusters or maybe fleets of, you know, EC2 instances, whatever that is. And that's a lot of work. The next problem that you need to solve is let's say you're using JMeter and you can run multiple instances of it on a Kubernetes cluster. That's great. But how would you aggregate all of the metrics report that JMETER produces for you and all of the logs? Right. So you have to build something in house to do that. Now, the next step is if you want developers to actually use this platform, you have to offer good developer experience. So that means building some sort of an internal cli, some sort of an internal dashboard to do it. And you know, none of this is really rocket science, but it takes months and years of person time to do that. And every single company that, you know, we're looking at is reinventing, you know, the same set of wheels over and over and over again. And coincidentally, not really. Right. Artillery takes care of all those things. So if you're on that SRE team or a platform team, you can grab Artillery open source and it gives all that stuff for you out of the box. So it can really cut down the time for you to build your internal developer load testing solution. And then to go back to the trend that we're observing as well, we're seeing more and more activities that are traditionally associated with SRE teams or production engineering teams moving more and more into product teams. And by product teams, by product teams, I mean front end engineers, full stack engineers, and QA engineers as well. So things like load testing, things like active monitoring, we're seeing more and more of that being picked up by those teams because they tend to be the people who are the closest to the user visible surface of the product. So there's this inherent kind of understanding of, let's say, why performance is really important. The problem is a lot of the time those teams don't have the tools that work for them today. So we want to make a big bet at Artillery at making things easy for full stack and front end and QA engineers with load testing. Which also loops back to why we added support for Playwright and why we want to make it really easy to run distributed load tests with Playwright.

Love it. So as I mentioned, it seems like maybe just because I've been following you more because I have a new show, it seems like you've been releasing more and more features. Do you have anything on the roadmap that you're excited about or any other features that have been released that we didn't cover? I think you did some with GitHub actions as well, if I remember correct.

Yeah, we just released an official GitHub action for artillery. So it's really easy now to integrate with GitHub and trigger production grade load tests from your GitHub actions that run, you know, in your AWS account, which is something that's really difficult to do with more traditional tools. We have a VS code extension on the way, which will make it even easier to write new load test scripts if you use VS code. There's a bunch of other stuff in the works. So today we're supporting Lambda and Fargate on AWS as options to scale out and distribute your load test. We're looking at adding more platforms, not just on AWS as well. A lot of our users have asked us about running Artillery on Azure, so we're hoping to add that before too long. And we're also looking into production load testing and supporting it as a native load testing type of in Artillery. And that's something that we have some really early POCs of and we're really excited about. There's no, you know, formal release date set for it, but hopefully as soon as possible.

Awesome.

So I guess one last question is what are the main things people look at now when they're doing performance testing? Once again, I'm talking about back in the day, I used to have CPU disk space, you know, throughput, you know, response time, even know if that's even valid now with microservices and everything. So when they use a solution like Artillery, I know some other ones like you have to come up with your own crazy, like you said, backend to kind of collect all this data. How does it work with Artillery and what are the main things that you usually would look at when you're doing a performance test?

We try to surface as many metrics and as many measurements that are directly relevant to user experience as possible. So, you know, I mentioned core web vitals that we expose if using Playwright. So that's if you have a web app, that's probably the best set of metrics for seeing performance from the user's perspective. If you're load testing APIs, let's say, you know, GraphQL APIs or JSON APIs, there's a few other. There's a couple of other things that you can do. So we have built in support for something called aptx, which, you know, some of your listeners may have come across. It's an industry standard for kind of compressing and describing the performance of an API from a user's perspective as a single number. It's an open standard. There's a formula that lets you calculate it. So an Artillery, you get that reported out of the box. So, you know, a lot of the time you can just look at that single APTX score for your low test and just be like, okay, this is good. This is within the threshold. You know, I don't need to look at anything else. And then we also Have a feature in Artillery which lets you set up arbitrary checks and success conditions based on any combination of metrics in your load test. That also gives you that, you know, green, red, pass fail score at the end of your load test run. And the motivation there is the same. You know, you run your test unless something is broken. You should need to really look at very detailed reports and you should be able to see that single score. So that feature gives you a lot of flexibility to define what that means for your application. Exactly. And then it gives you that pass fail at the end.

So I guess having a score then, like 95, I guess it makes it easier to run continuously in your sprint. Every two weeks you could say, did it get worse or didn't get better? Easy thing someone could see right away. No, it dropped by 10. Let's look at. Or it's about the same. We don't have to worry about it. Sounds like using the score.

Yeah. What happens a lot of the time, right, is there's rarely in the real world, there's rarely, you know, a single commit or a single configuration change that introduces a massive drop in performance. Usually what you have to do is you have to look at, you know, three or four or six months of data, and then you start seeing, you know, latencies creeping up over time, that kind of thing. Yeah. So trends, like. Absolutely. And the easier it is to see those trends, the more likely it is that someone will look at them. And that's why we try to compress, you know, as much as possible into those really digestible kind of single number metrics as possible. That is not to say that that's all we report. Artillery gives you access, literally to everything down to, you know, the time it takes to resolve a DNS name, for example. So it's all there if you want to dig into it. But by default, I don't think, you know, you should need to look at it every single time.

Awesome. Okay, Hasi, before we go, is there one piece of actual advice you can give to someone to help them with their DevOps performance efforts, and what's the best way to find or contact you?

So the best way to find me is probably Twitter. H Velstra or Artillery's website is on Artillery IO. If you haven't seen it, go check it out. Best piece of actionable advice, I would say, on a DevOps journey is implement SLOs. If you haven't, it's one of the best, one of the easiest, not easiest, one of the best wins that you can make to align the entire company around kind of the same goals. And then once you've done that, start running load test to use as a Guardrail around those SLOs. It's one of the best next steps that you can take.

And for links of everything of value we covered in this DevOps toolchain show.

Head on over to testguild.com p124 Remember.

Latency is the silent killer of your app. Don't rely on frustrated user feedback. You can know exactly what's happening and and how to fix it with Bugsnag from SmartBear. See it for yourself. Go to Bugsnag.com and click the free Trial button. No credit card required.

So that's it for this episode of the DevOps Toolchain Show.

I'm Joe.

My mission is to help you succeed with creating end to end full stack DevOps toolchain awesomeness. As always, test everything and keep the good. Cheers.

Hey, thanks again for listening. If you're not already part of our awesome community of 27,000 of the smartest testers, DevOps and automation professionals in the world, we'd love to have you join the fam@testguild.com and if you're in the DevOps automation software testing space or you're a test tool provider and want to offer real world value that can improve the skills or solve a problem for the Guild community, I'd love to hear from you. Head ON over to testguild.info and let's make it happen.