Locust Performance Testing with AI and Observability

Most teams still treat load testing like a black box. Today we're going to open it up. Hey everyone. I'm Joe. And performance testing has changed fast, especially since when I started AI generated code Python first Workflows, observability and CI CD have completely rewritten what modern load testing should look like. Today we're going to dive deep into Locus load testing, why Python has become the secret weapon for realistic performance test how teams are replacing heavyweight tools like JMeter and how you can trace slowdowns all the way from your load generator to your database. And I'm joined today by Laws Holmberg, founder and CTO of Locus, to break down what most teams get wrong about load testing and how to fix it before it costs you in production. You don't want to miss it. Let's get started. Hey Lars, welcome to the guild.

Thank you. Great to be here.

I've been hearing a lot about Python, especially for Locus. Locus seems to be a Python first performance testing tool. What do you see maybe as the advantage of using Python vs JavaScript for load testing?

So I would say the main benefit is if you're already working with Python, it's a great place to be and you don't have to think about different languages. And I guess, look, Python is slightly more like expressive, like gets you straight to the point and JavaScript has a little bit more boilerplate getting started and stuff.

Awesome. What made you create Locus? Why an open source tool?

Yeah, all right, all right. Yeah, so let's go back in time. Yeah, so I didn't actually create Locus from the start. It was created by another suite called Yunatan Heyman and he was using it at company called Dice, makers of Battlefield. So basically for testing web services, websites like a forum and whatever they had going back then. And yeah, it was basically just started out just doing HTTP testing, but already from the very start it was created to be very flexible. At the time there was already other competing tools like JMeter, but JMeter just isn't. It's not built with that sort of flexibility in mind. I've used JMETER a lot. Like that was probably the first serious load testing tool that I used. But it's always slightly limiting when you do want to model some really complex behavior. So that was one of the reasons that he created it. And at the time I was actually just started working in performance testing. This was in I think 2011. So I had some experience working in load testing already. And I would say a couple of years, few years later I got into load testing and Started working as a consultant in load testing. I was working at H and M here in Sweden and later at Klarna. And Klarna is where I really started using Locust and started contributing to Locust and eventually took over as the core maintainer.

Nice. So you know. Yeah. JMeter seems to be the most popular open source tool and you mentioned one reason. Are there any other technical reasons why someone may want to switch to locus in 2026? Is it easier to get up and running? Does it fit better with AI type of works like that?

Yeah, yeah, yeah, yeah. There's a number of benefits, especially if you're using AI and using ChatGPT or whatever to create your load tests. And also because Locus is just Python. So if you want to use some other SDK to make the requests against your system that you're testing, it's very easy. As long as there is a Python SDK for whatever you're doing, you can wrap it and let Locus know what it's doing. And then you can basically test any protocol which makes it like far superior to something like JMeter where you would have to write Java code and then tie that together with JMeter.

Nice. You also said it's flexible, so I guess because it is written in Python, I guess you can use it the way your normal flow is. You don't have to conform to a tooling workflow maybe.

Exactly. There are a number of hooks, event hooks that you can tie into. Let's say you want to do something specific during startup. Let's say create some heavy data or some test data that you will then iterate through when your actual load test is running, for instance. It's very easy to hook that into the execution as well as if you want to do some specific output of the results or validate your results in some way. It's very hackable because it's like I said, it's just Python, so it's very flexible.

So you did mention you start off 2011 using open source tools performance testing, but do you know anyone stories of maybe big misconceptions teams may have about open source load testing tools versus commercial load testing tools. Because sometimes you get in these chats and people like enterprise tools will do because they have the full, full workflow, the analysis piece and everything.

Yeah, I would say so. I used, I used to use Loadrunner. This was maybe 15 years ago and I find that the commercial tools, at least, at least back then, because I haven't used Loadrunner in the last maybe five years, was sort of tailored More maybe to a functional tester, like very much focus on the recording of the flow and then sort of press play testing I guess, versus actually writing code to like specify your test. I mean you could do that in a commercial tool like loadrunner too, but. But it's not like a main way to do things. I would say the commercial tools, maybe this is a common theme of Linux tools and everything like that. It's all for the narrow scope that Locust and other open source tools solve for. They're really good. They're just as. I mean there's no reason JMeter can't be as performant, at least close to as performant as Lode Runner or something. Just because it's. Just because it's a paid product or as stable or whatever. I've crashed loadrunner too, so you know, it can't happen. But I guess what the commercial tools at least back then offered that some open source tools don't is like a nice GUI and maybe slightly more polished for the sort of less technical performance tester I would say.

Absolutely. And you're right, I know recently two open source tools have come out with a more GUI based almost recorders for performance testing. Do you see Locus going in that way as well or is it because it's in Python? Wouldn't make sense.

There is a tool for Locust too called Har2Locusts, which takes browser recordings. It doesn't integrate directly with the browser, but you can record a session and then you can convert that into a Locus script. So I recommend that like it's again not super polished, but it works at least getting a starting point and do like basic correlations of request response data, which is helpful especially if you're not like, if you're new to Locust. I would say like it's a very good way to get started.

Yeah, I don't know if this is still a use case, but back in the day when I started, no, let's say over 25 years ago, we'd have to do distributed load across different regions and we'd use a commercial tool because we were able to kind of emulate that. Can you still do that with Locust? I guess you can now with, you know, these ephemeral environments. Can you like how hot?

Absolutely. Yeah, yeah, yeah. That's, that's, I guess one of the core things about Locus as it's distributed from the start. So it's like you have this master worker setup that sort of takes care of that. Some other load testing tools like K6 for instance opt not to do that. It simplifies the architecture a little bit. But then you have to do some stuff like result aggregation and stuff yourself and you can't coordinate the run from a master instance and then start and stop tests from a single place, basically.

Great. So as I mentioned, have that blog post on the top open source performance testing tools. What would you say? If someone's listening, what is the perfect team to use Locus in your mind? If you say if this team does this, this and this, then man, they're going to really succeed with Locus?

Yeah, I would say again, if you're deep into Python or at least somewhat used to Python, if that's your preferred language, I would say it's one of the best. Another thing is, if you want to do massively parallel tests, Locust isn't the most performant in terms of requests per second because Python isn't the fastest language as everybody knows, but. But it handles concurrency extremely well. Much better than say JMeter where every user is an actual OS thread. So you can do really big concurrency. And yeah, I would say, and now with the added observability features that we created last fall, I would say if you're using otel, that's another great reason to have a look at Locust, because we offer like built in integration so you can get end to end traces and that sort of thing very easily. Just.

That's a good point.

Add a flag.

Yeah, I've been seeing a lot more about you posting about the time with observability. Can you talk a little bit more about that? Maybe people that don't know about observability or why that's a big, big new feature?

Yeah, yeah, yeah, yeah. So I mean a lot of people, serious shops these days have observability implemented, so they would have a way to collect logs, metrics and traces. And traces particularly make it easy to see end to end, where your request is coming from, which systems it passes through, if there were any problems or errors along the way, and where time is being being spent basically. So which, let's say a request takes one second to respond and you can see in detail which systems were responsible for this response time. Right. Let's say this was 500 milliseconds in the database, 200 milliseconds in the load balancer or whatnot. The thing that Locust adds to this is end to end tracing. So you can see all the, all the way from your load generator, which would be in reality would be an end user to your system. So you can see the trace from end to end. And this is great. If you're encountering the very common issue, I would say when you're doing load testing, which is your load testing tool measures poor response times, let's say 10 second response times, but your servers log in, their access log or whatever they log, let's say 200 milliseconds. Oh, it's super fast. What is the problem? Where is it coming from? A number of cases where like, you could have a real issue with your load balancer being misconfigured or something else amiss. Another common one is that because when you're doing load testing, you tend to be coming from one or a few IP addresses. So load balancing might not be properly configured for that use case versus in reality, where people would be coming from all over the world, maybe. So that's, that's very useful.

So, yeah, like break that downwards. If you had a transaction that took 20 seconds, you don't know where normally, but with the observability now, you could say, okay, of this 20 seconds, 15 seconds were taken in the load balancer. That type of. So you can help correlate then that. All right, not only does this transaction take a long time, but most likely has to do with this.

Yeah, yeah, very nice. Exactly. And, and like, and, and there could be a number of other situations as well. Let's say you on average have pretty good response time, but you occasionally have spikes. And with tracing, you can see from your load load test, from your load testing tool from Locust, you can see, oh, this transaction took 20 seconds. All the other ones were fine, but this one took 20 seconds. And then you can see in that trace, assuming you have implemented observability in your own systems, you can see like, oh, what was the difference here? Maybe there was timeout in the network, maybe there was a spike for a specific amount of time when the database was doing some background job or whatnot. Basically, it takes the guesswork out of, out of these.

Yeah. And it helps you not ignore it. I know a lot of teams back in the day would just do like an average of the three. So if you saw an outlier, you're like, I don't know.

Exactly. Exactly. Yeah, yeah, yeah, yeah. This is a, this happens every once in a while. Like you have to explain for people who are not performance testers. And it's like, oh my God. It's like, oh, you have a, a 0.3% error rate. And I'm like, that's freaking awesome. That is great. Whereas in, like in other cases, you would definitely not be happy for sure.

So it seems like you, you are keeping up, it is an open source project, you are keeping up with the new features. How, how heavy, how, how good is the community? A lot of times when people are choosing a tool, they want to make sure the community is there. It's not just one person that if they leave, then a whole project's dead.

Yeah, there's, there is, there's a, a fair bit of community going on. I would say most people are just using it like we have in total, I just checked the pypy stats and according to that we, we have had 130 million downloads of the tool. Which of course isn't all real people. It's probably mostly CI groups downloading this, but we have a, we have a fair amount of, of people around. I would say most people are maybe not contributing but like, I don't know, having some conversations and there's a, there's a Slack channel as well that you can join.

Yeah, we'll definitely have links to all the community stuff down below, I guess. Another question I have, as we mentioned, Python is. Seems to be the language of AI and, and with a lot of AI generated test scenarios become popular, do you foresee Locus adopting to maybe helping teams create more intelligent load tests using AI type of features? Maybe?

Sure, absolutely, yeah. And going back to AI. One of the things you can do with AI is to ask for example ChatGPT, give it your application and say, hey, create a load test using Locust for this application. And it's actually, it's amazing how well that works using Chat GPT or Copilot or anything like that. So you don't actually like. It might not build the perfect load test or perfect Locus file for testing your application, but it's going to get you started. And it's like it removes sort of the intermediate step of maybe recording a test or understanding what kind of requests that should be in your load test and just skips right over that. It works extremely well with like flask applications and FastAPI in Python land, but I'm sure it can be made to work for other languages.

So is that through an MCP server or just because it's written in Python, ChatGPT and all these things know how to write Python.

It's actually just using ChatGPT, so it's pretty impressive. There are some MCP things as well that we've been working on, some of them specifically relating to this conversion from HAR files to Locus Scripts.

That's nice.

There's stuff going on there.

I guess another thing people always look for with the performance tool, especially nowadays, is make it part of the CI CD pipeline. GitHub Actions or Jenkins. How does Locus fit into this type of kind of flow?

Yeah, we have a couple of examples of how to implement for example, GitHub Action to do a load test. So that's pretty easy to do. Basically just install Locust and then run the, like any, like any command line tool. It's, it's fairly easy to do. And in GitHub Actions or whatever.

I know you probably, I don't know if you've seen this. I think a lot of people use performance testing tools sometimes for API testing because it's like headless and you know, maybe they're just doing some quick API testing, validation. Do you see that as a use case or do you see a time when you would do that versus a full blown user journey using load testing type activities?

Yeah, I definitely see the use for both versions. Yeah, yeah. And I think sometimes if you're not an experienced load tester, you might go to the sort of end to end user journey a little bit too soon. Like in reality you'll probably be more productive aiming for like the main flows, the most common ones, the ones that, you know, might have issues. Let's say for instance, if you have like a website with lots of nice pictures on them and I don't know, but you have like a CDN that serves all the static content, you wouldn't performance test that because it's not, it's not, it's going to scale. Basically you don't want to be, I say this all the time. You don't want to be testing Amazon. You want to be testing your application because, well, you can test Amazon if you're Netflix. Go ahead. You might need to load test Amazon to make sure that they can actually deliver on their promises. But if you're not Netflix, if you're not Microsoft, then probably focus on testing what's specific to your application as opposed to infrastructure that you already outsourced.

That's a good point. I've seen people trying to performance test downloading of files where I think that's more of a browser function. So why would you incorporate that into your performance test? It just makes it messy, I think.

Yeah, yeah, exactly, yeah. And it also adds like a lot of overhead to your load testing tool, so it doesn't usually make sense. And the same thing for like rare scenarios, like a functional test. If you're writing functional tests. You want as much coverage as possible, right? Performance tests. You're, you're kind of, you're good if you have coverage of the things that 95% of your customers do. The last 5% is like, maybe, maybe, maybe if, if there is a real. If there. And we should also differentiate between like, load testing and performance testing because you might want to performance test those rare cases too. You don't want the, the user experience to be terrible. That's not, that's not good. But you might not need to load test them because there will never be a hundred thousand people, I don't know, doing some, like, changing their email at the same time. There might be a hundred thousand people buying shorts from H and M, but probably not changing their emails.

Right, Absolutely. And that's a good point. A lot of times people also do like really crazy transactions per second and then start a fire saying, look how bad the performance is, but it's not realistic. So that's another common issue, I would think, for sure.

Yeah, yeah. And also we can talk about like, requirements. Requirements and performance testing is always interesting. Right. A lot of times you start out a project and be like, hey, we should be able to do 100, say 2000 purchases per second or whatever. And then it's like, oh, we tried that and it's terrible, it doesn't work. But then after a while it's like, yeah, but we only actually peak at 100 in production right now. So if we can just do 200, then that's, that's decent. Like, but of course they want to do. We want to do really. And we also want to have like, really fast response times. Right. And that's, I mean, that's fine from a requirement standpoint. Let's say we want 100 millisecond response times, but in reality, maybe we'll accept 2 or 300 millisecond response times, especially in combination with peak load. Maybe scaling the system to do like, we absolutely have to do 100 millisecond response times at peak load. Maybe that's not realistic. Maybe that's like not a good use of people's time.

Absolutely. So, you know, once again, when, when I started, when you started, it sounds like performance testing really was a specialized kind of like, skill. And not a lot of people had almost seemed like there was a morph in 2020 into more like it's now an SRE role or that like, like you don't hear as much as, as you used to. Like, who does performance testing nowadays? Like you actually work on an open source tool. Do you know like who's doing performance testing? Is it more testers now or like who does it?

Yeah, I think there is like there are two basic types of performance testers really. There are the people who, who just focus on the actual creation of scenarios and execution of those scenarios and sort of like they hand off the results to someone else and then they do all the analysis and they do like, I mean they could, they could still do advanced stuff but they stay within the sort of load generation and, and test creation side versus the ones that maybe come more from a developer background and have sort of branched into load testing. Like maybe they're doing, maybe they're actually a part of a functional team, develop developing features, but then someone has to load testing because they realize they need it. And it's sort of more of an ad hoc thing where you're just like, you're 50% developer, 50% load tester. And that's sort of like, yeah, you can come from, it's so, so different. Like, but, but I still see both types of performance testers, load testers. So, so I wouldn't say like one is, is, has, has completely taken over.

Is there maybe one top load testing mistake you've seen over and over again, even with experienced teams that they make that you're like, oh my gosh, I just wish this message could go out.

I mean there's one very common one which is when you run your load testing tool usually, I mean different load testing tools have different ways to approach this. But typically you would say like, hey, give me 500 concurrent users or something. And then when you launch that type of test and then you see your throughput is not very good. The first. And this is like this is load testing 101. Maybe it's too basic, but that's the first thing I come to think of. Like if you see your response times going up as you increase the load, you shouldn't be surprised that your throughput isn't increasing. Right? Because a lot of times people will be like, hey, there must be something wrong with my load testing scenario or there must be something wrong with my tool. When in reality you just have really, really bad response times because you maxed out your system. And it just there, there's a formula for it. I might write a blog post just about that formula for like response times versus concurrency versus throughput. It's really basic, but some people need to hear it. And I mean we've all been beginners once, so it's not. Yeah, that's one thing. I would say there are plenty others as well. For, for instance, the thing I mentioned before about like how load balancers might behave under, under a load testing scenario versus a production scenario. Yeah, those are the things that comes to mind.

So say a team has maybe two hours to set up their first load test. What's the locust quick start, like path. Like, how do you get started with Locus? If someone like heard this, like, okay, I want to get it up and running quick.

Yeah, I would say if you're just, if you just have two hours, then basically all you'll have time for is just to hit some URL. Like you, you're just a basic, basic scenario. You could just copy paste the Locus quick start and then just run it.

And.

Yeah, that's, that's. And if you're gonna go slightly beyond that, like maybe have. Think about the, the sort of core user flows, right? Like if you have a web shop, it would be like, put something in your cart, go to checkout, make a payment, and that's, that's. You should, you should just try to get that working as soon as possible.

Are there any performance testing metrics? Metrics? Should they go completely overrated or underrated?

I would say, like, sort of related to the thing I said before. People need to think about throughput before they think about response times. If you're load testing and you see you have poor response times but good enough throughput, then you're probably good versus if you're like, if you're just testing a single user. If you're doing performance testing as opposed to load testing, of course you'll be more focused on response times.

Cool. All right, Laz, I know you can't commit to this, but is anything on your roadmap that you're really excited about maybe in 2026 that you're working on, that's like almost about to be released.

I'm super excited about adding support for Asyncio for modern Python. This is really cool. And also free threading Python. Basically working around the major performance bottleneck of Python, which is the global interpreter lock. This is going to make Locus not only faster and better for supporting Asyncio, then it's also going to make it simpler because without the global interpreter lock, you won't need one Python process per CPU core. So you'll be able to run a single process and utilize a whole machine, regardless of the number of CPUs.

Very, very cool. Okay, Lars, before we go, is there one piece of actual advice you can give to someone to help them with their performance testing efforts and what's the best way to find or contact you?

So the best, the best way to contact me would be through the locust GitHub page or on the Locust Slack. Yeah, I would say don't forget about like when you run this big ass load test and you start seeing some errors, don't be afraid to go back to run just a single user. Look into detail exactly what is going back and forth over the wire. If you get some error message that you never seem to be able to reproduce in the browser, check to see exactly what is your load testing tool sending. You can do this easily in locus just by adding some flags and yeah, just go back and forth. Basically don't just run big load test after big load test and then you're like oh, I have a million requests and some of them failed and I have no idea why.

Absolutely. And you haven't tried out Locust yet. I highly recommend you do using all the links down below. Thanks again for your automation awesomeness. The links to everything of value we covered in this episode. Head on over to test guild.com a575 and if the show has helped you in any way, why not rate it and review it in itunes? Reviews really help in the rankings of the show and I read each and every one of them. So that's it for this episode of the Test Guild Automation Podcast. I'm Joe and my mission is to help you succeed with creating end to end full stack automation awesomeness. As always, test everything and keep the good. Cheers. Hey, thank you for tuning in. It's incredible to connect with close to 400,000 followers across all our platforms and over 40,000 email subscribers who are at the forefront of automation testing and DevOps. If you haven't yet, join our vibrant community@test guild.com where you become part of our elite circle driving innovation in software testing and automation. And if you're a tool provider or have a service looking to empower our guild with solutions that elevate skills and tackle real world challenges. We're excited to collaborate visit Test Guild to explore how we can create transformative experiences together. Let's push the boundaries of what we can achieve. With lutes and lyres the bards began their song A tune of knowledge, a melody of code through the air it spread like wild fish fire through the land Guiding tester showing the secrets to behold.