Locust and Azure Load Testing: Create and Debug Load Tests with AI

Hey, are you vibe coding your app and assuming that AI has performance covered? Little secret, it doesn't. LLMs are great at shipping features, but they often miss the non-functional stuff that fails over the worst possible time, like Black Friday or payday. And today I'm joined by Lars Holmberg, lead maintainer of Locus, and Abhinav from Microsoft's Azure App Testing team. You're gonna hear and see how to create Locus tests from a Postman collection in VS Code, Run it, distribute it on Azure, and let AI insights do the heavy lifting on your results. Lars also gives you a first look at AIO-Locus, his new async OpenTelemetry native take on Locus. And if you're a functional tester who has always seen performance testing as a barrier, this one is for you. You don't wanna miss it. Check it out. Hey, Lars and Abhinav, welcome to the Guild.

Hey, nice to be here.

Hey Joe, nice to be here.

Great to have you both. So I've spoken with both of you in the past. Abhinav, you did a great session on Automation Guild. Laszlo, I think we had you on here last year going over, uh, Locus. So I guess before we get into it, I just want to do some level set. Uh, first one is high level. I want to get both your definitions of what is performance testing, especially, uh, with the age of AI, if that changes anything.

Yeah, I think the The big changes in load testing and performance testing in the last couple of years has been the, like, how much more observability is available these days. And then at the latest, in the latest years, of course, using AI to aggregate that data and analyze it automatically, making it much easier for someone who isn't a full-time load tester, performance tester, to do that work, basically.

So with AI, first and foremost, performance testing or load testing is more important than ever before, right? We have a lot, a lot of demand for real-world traffic in a managed environment today. And it's become a lot more easier to create your scripts, to generate load tests. And using AI, it's also become a lot more easier for you to analyze the test results and make sense of it much faster. and translate it into actionable insights that you can use to improve your application's performance.

Yeah, and on the system side of things, like, people are building more and more apps, right? And AI is really good at getting features in there, but the non-functional stuff like performance might not be obvious to, like, Copilot or or some other tool, how to optimize for that. So load testing is important for that reason as well.

So I guess in general, why would you do load testing? Why is it important? How to get started? A lot of people sometimes, especially my audience of functional testers, I think they should be more involved in all types of testing. They see it as a barrier. Any thoughts around this?

So I guess the difference between functional testing and load testing or other non-functional testing is that you're not just testing things once. You're testing a lot of users, they're running concurrently. You're focusing more on metrics rather than binary outcomes. Like a functional test usually passes or fails. I mean, it can be flaky, but ideally it passes or fails. Whereas a performance test, There is rarely a 100% fail or 100% pass. You usually say, oh, we could do throughput X. We could do 700 users per second or requests per second or whatever your metric is. So there is a— it's slightly more complex. And the debugging is also harder. We talked about— Sorry, we're gonna, like, we're gonna get into observability and that stuff. Uh, so, so, uh, that's, uh, that's one of the, uh, uh, key things here as well. Like, it, you need additional tools for load testing. And I guess, uh, the, the, the real, the benefit of performance and load testing is, is just to be proactive. Uh, 'cause you could just monitor stuff in production, right? But usually that's not good enough because you want to find the errors before they happen. And the errors that do happen in load tend to be at the worst possible time, like during Black Friday or payday or any, some other high impact events. So you want to, you want to test beforehand for, for peaks in traffic and you want to test before you launch. A new software or a new system, put it into production because rolling it back might take time. And you want to be able to test stuff where things aren't really going the way they should. You want to test like failure, degradation testing, chaos testing. And that's another case where load testing is really important. So systematically, basically being able to do systematic tests rather than just looking in production and seeing how things are.

So Abhinav, I don't know how many people are aware of Microsoft's investment in performance testing or load testing. Is there a solution that you all have to help with this?

Yes, Joe. So, so yeah, I'll give you a sneak peek or an intro into what Azure App Testing is. And Azure App Testing is Microsoft's unified offering for validating the application's quality and performance at scale. There are 2 important testing disciplines that comes under the umbrella of Azure App Testing. The first is the functional testing side. We have a service called Playwright Workspaces, and this is where you validate whether the application is behaving correctly, the different functional components are working, and whether the user journeys are successfully completed. across different browsers and operating systems. Then the second is the performance testing service, which is called Azure Load Testing. This is typically used to answer a simple question. What happens when thousands and thousands of concurrent users are using your application at the same time? With Azure Load Testing, you can simulate real-world traffic without having to manage any kind of infrastructure. And it's for anyone. If you are a complete beginner to performance testing and you do not know how to get started, and even for experienced load testing or performance testing engineers who are already familiar with open-source frameworks like JMeter or Locust, Anyone can quickly get started with Azure Load Testing and detect performance bottlenecks in your applications and do that before your customers do, which is, which is the highest priority for you at the moment when you're building these applications for production scale.

So I'm just curious to know, you said it's easy to get started, but like for people who have never seen it, maybe we do a little show and tell what it looks like.

Yeah, sure. Let me just share my screen. So, so this is the Azure Load Testing homepage or overview page. And as you can see, we have 3 different options to get started. The first option is creating a simple URL-based test. So this is for the absolute beginners. All you have to do is just provide a test URL. Specify the load, the number of virtual users that you want to generate or simulate, and the test duration and the ramp-up time, and you can just quickly get started. Very simple, very easy to get started. Then we have the Run Existing Tests at Scale option. So here you can upload a JMeter or a local script, and you can configure your test for advanced scenarios. You can use authenticated scenarios through managed identity. Again, you can specify your load, you can distribute your load across multiple regions, you can configure your network parameters in case your application is privately hosted and not available in the public domain. You can even configure it for privately hosted applications. You can specify all kinds of parameters that you want to use in your test script. You can set them, you can set secrets for your credentials that you want to use, and you can also specify test criteria. So you can specify what kind, what is your pass/fail criteria for a performance or load test that you want to run. And you can also configure server-side metrics. So this is another interesting feature that we have. Because quite often it becomes important to correlate your client-side metrics with your server-side metrics so you really are able to double-click into where exactly the issue is. Is it on the application or the underlying infra? You can even select— if your application is hosted on Azure, you can link it. We already have all the metrics available in Azure Monitor and you can connect to those server-side metrics. And again, you can set your pass/fail criteria. So the third option that you have is to use AI to generate your scripts. So you can— we have a browser extension. So using the browser extension, you can record a set of actions on your browser screen and it will automatically generate the script for you. So let's take a look at how that works. So I just open up any website that I want to generate my script for, and then I open the Azure App Testing extension, browser extension. I hit the Start button and then I perform a set of actions. So I just search for, for an item. I can click on that item. I can just add it to the cart. So if this is something that I just want to test, I can just save this and I can just review and create my test. So once the test is created, it'll take you to a screen like this. And here you can— again, you can see what are the scenarios. You can review the scenarios. So again, AI generates names for each of these scenarios that you can relate to, and you can identify what are the actions that have taken place, and you can review it. If you want to apply think times and if you want to generate a scenario where there are real-world scenarios where people actually think before they perform an action, you can add that. You can add correlations, you can add parameters, and so on. So, so yeah, these are a few different options that you have to just get started with your load test or performance test creations.

All right, dumb question. Do I need a license or is this per usage? Like, can I just use it per server time? Is that how it works?

For the extension, there is no license. You just need to have an Azure subscription. And once you have an Azure subscription, you can just get started and the payment is completely pay as you go or based on your consumption. So based on what kind of usage you incur, you are charged for that. No other license.

So you did show an option for JMeter and Locus. So I was just curious to know, Lars, as the main open source maintainer for Locus, how did you get involved? How does this help you? Where do you see this partnership going? Like, what are the benefits for both of your sides?

Yeah, so the benefit from my side, I work in my day job as a performance testing consultant, helping teams basically build better, more performant applications and systems. But what my partnership with, or our partnership with Microsoft has given me is more time to spend on maintaining the project and also collaborating with around things like today and talking about Locust and yeah, basically helping more people find their way into Locust and Azure Load Testing.

Yeah.

helping people get better results, basically.

Abhinav, any benefit of using Locust over JMeter?

Oh yes, definitely, Joe. So we've seen JMeter has been around for years and years, right? Locust is quite new, and we are noticing the shift where more and more developers are shifting towards Locust because it's, it's very developer-friendly, right? At the end of the day, it's Python code, so it's very easy to maintain and make edits compared to your JMeter scripts. And also from an AI standpoint, it's very easy for your— the AI applications to really create your Locust scripts as well as edit your Locust scripts. Maintenance, all of those parameters are much more easier with Locust. In fact, we also have an extension, an Azure Load Testing extension, which Lars can talk about, which does exactly that. It uses AI to generate a local script for your existing code, and you can also save it in your source code repository, so it's easy to maintain.

Oh, cool. So Lars, how much input did you have on that? How does that work? How do you trust— you just trust the AI? How good are these scripts?

Uh, so it's, it's really convenient, and I would say like if you, if you trust Copilot, you can trust this stuff. And I'd be happy to demo, uh, briefly how to, for example, create a basic A Locust test. I was thinking we should start with a recording. It's a Postman collection. So, it's basically a set of requests. This one was created from a recording, but you can maybe also, if you have some functional tests or something, some other source for your Postman recordings. And I'll be using the— Azure Load Testing extension for VS Code. So it's super, super simple and it walks you through all the hard stuff basically of setting up your first load tests. So let's just say, hey, I want to create a test, I want to create a Locust test, and I will start with a Postman collection. And just open this file and then select which requests do I want to add to my scenario. Let's, let's do all of them. And then basically it gets working on this one and it will create a load test for me. And this is of course just a starting point. Once you have a starting point like this, you will continue using the AI to, let's say, parameterize your test or modify it somehow. So, let's see what it created. I think, let's go to the core of it, which is basically these transactions that I had in my Postman recording. And this is really good as a starting point. Like, if you've never done load testing before, if you've never touched Locus before, You can get to a point where you can just start running and then work from there. But the cool thing doesn't end there. The cool thing is actually when you want to run this. And then, so let's say I want to run this on Azure Load Testing. I'll just do run load test. You can run it, you can run load tests locally, just start Locust as normal. But you can also just say, hey, I want to ship it. All the way to Azure and do a distributed load test up there. And it will create a configuration file for you. I'll just show it briefly because it's good. So you can see what type of settings could you put in there. You could put in the number of users, the runtime, and stuff like that. Any extra files that your test might be dependent on, like CSV file for users or whatever you might need. Or a separate module, because, you know, Locust is just Python code, so you can just load any modules and structure your tests as you would any other source code. But let's go ahead with this, and that's going to start the test. And now once this is started, I can monitor it in real time and show visualizations and analysis and logs and whatnot from the test running on Azure. So I could go here, but in order to save some time, I actually ran a test earlier. So let me show you that. So here's a test where you can see the number of users ramped up over time and the various response times of different requests. This was a similar scenario, which just logs into a website and adds some stuff to cart, basically. But the core of this is, of course, that you get all these metrics, you can do all this analysis, and you can also get server-side metrics if you configure it using Insights. And you can even, and this is the coolest thing, like Abhinav mentioned, You get AI insights. So in this case, you can see that because the AI has access to all of the data, all the results from the load test, but also from your system if you have it monitored using Insights. So you can see all sorts of clever conclusions, basically. The stuff that you need to spend maybe hours doing manually, uh, can now be just provided by the AI. It can even do recommendations. So let's say it notices that you have too few instances, like the CPU is maxed out on your application servers or whatnot, and it might suggest to increase the number of instances or tweak the network or something else.

Mm-hmm.

So, that's really, really cool. And here you can also see, like, yeah, like everything that happened during the test. You can show the logs from the test, like this regular Locust output. Ah, this is too old. I would have to download the zip file, but you can see the logs here. And you can also see any, the inputs for the test so that you can tell like, what was the test that I actually ran? I can see like exactly what was the test plan that I submitted, any extra data files that I put in there. So it's very traceable. It's very— Yeah, it's very, very convenient for that stuff. And also, like Abhinav said, you could just run the test again from the portal, and you could say, hey, I want twice as many Locust instances. Let's see what that does to the performance, or run distributed across regions, if that makes a difference.

Nice. All right, so this might be a weird question, uh, so Abhinav, and maybe I, I don't understand. So if I was using VS Code with Copilot and hosting my application on Azure, if they found an issue that we showed with the summary and it's not a resource issue, can it actually show, hey, you have a dynamic SQL error that you need to fix and here's how you fix it? Can we get to that detail? like actually the code level, or is it more just you have a resource issue here, you need to up your memory or your server settings?

Yeah, yeah. So, so, um, so like Lars showed, uh, one of the key, uh, features that we have is the server-side metrics, right? So what client-side versus server-side metrics really enables is to pinpoint to what is that specific metric either at your application layer or at your infra layer where the issue is occurring. If it is, for example, and maybe I have a sample scenario that I can run through as well where I can show you exactly what you just asked, Joe. Yeah, I have an existing test run result which I ran some time back and we can do a live debugging to see what went wrong in this test run. And we can really dive deep and see what are the recommendations that AI has provided as well. We start with the test run status. We can see that the test run has failed while the test has completed and the test criteria was not met. This is the reason why the test has failed. You can see the AI summary and insights are present here. But before we go into this, maybe we can just quickly look at the other elements. So there is an overall statistics section which will show you the overall statistics for the entire load test. So you can see the load, the duration, the response time, error percentage, which is high, and the throughput. And you also have the request statistics which show you for individual samplers what are the request status. So what was the error count for individual samplers? So as you can see, the home page sampler did not return any error, whereas adding a new owner has has returned a higher error count. Uh, even finding all owners has returned a fairly higher percentage of errors. And the next is the test criteria section. So we can see that we have set up 3, uh, test criteria, 2 of which has passed. So for the error percentage and the HTTP server errors, we can see that it is, uh, higher than the threshold— fairly higher than the threshold amount that we have set. Whereas for the response time, it is below the threshold, so you can see it is fairly good. Next, we look into the client-side metrics. We can see the client-side metrics as we have increased the number of virtual users. You can see how the response times are varying, how many requests per second, and where the errors started to show up. You can go into specific areas of the number of virtual users and you can see how it is performing or what are the number of errors for that specific, uh, specific, at that specific time. And then we have server-side metrics. So again, like Lars mentioned, if you've configured server-side metrics, you can see that, uh, you can see the infra layer metrics on some of the responses here. And finally, we also have load engine health metrics. So this is, this is just to ensure that, uh, the the infra that is running the load test did its job and there is no errors on the load test engine that is generating. So after this, once we have just gone through all of this, we can just quickly jump into the AI summary. And to your question, Joe, you can see that the summary shows us really where exactly was the error. So you can see here it says that the client-side errors peaked at 53% and response times were consistently high. And with, with the MySQL CPU, which was your server side, was stuck at 100%, and the App Service had 5XX errors. So it showed that it has critical backend bottlenecks. And once you go into the recommendation, you can see. So you can see the specific recommendation, right? It tells you whether you need to scale out your App Service, whether you need to optimize. Yeah, there's another thing here. You can scale up the MySQL server and so on. So it, it really gives you an idea of where exactly the error is, right? Even from a client-side perspective, whether it is in the application. If the error— if, if all the server-side metrics are looking good, then it's more likely an application-side issue. And that is where you can dive deeper into your application and see Various. So you can— these samplers will also give you an idea of which, which section of your test is the one that created more errors. And then you can recheck that portion of the code to see whether there can be any optimizations that can be done.

All right, so Lars, just random question. You know, when I started, I used to have a raised floor with 12 load generators in a rack. that I'd have to maintain, and then I had to get a license for like 1,000 vUsers to be able to ramp up to 1,000. How does this work now with Locus? Is it just like point to Azure and like unlimited as long as you have the budget to run it on the servers?

I mean, pretty much. It's really, I would say it's really, really good pricing. The pricing is basically per vU hour or for Playwright server minutes, but it's no fixed costs. So it's basically a no-brainer if you're, especially if you're already in the Azure ecosystem, it's super, super simple. And Locust itself is, is of course free. So you can do any, like, When you're developing your tests and running them locally, you can do that completely free. And you can still use the Azure Load Testing extension. It works without Azure as well. Like if you want to create tests using that Copilot-backed stuff, that works fine as well. So, so it's, it's, it's very easy to get started and very cheap.

Have you seen anyone say, wow, this AI Insight really helps? Any feedback?

Oh yes, Joe. We have a bunch of big enterprise customers using our service today, Azure Load Testing, and they are relying on AI Insights. We've heard a lot of feedback on how it has really simplified test result analysis. And it has reduced the time to value essentially, right? Because like you mentioned, it's not easy because you see a lot of data, you see a lot of logs, you can download these reports and you can put it through your LLMs or models and then generate those results. It still takes time, right? So here you are able to correlate your metrics to each other and generate really valuable insights. And we've seen many of our customers really enjoy this feature. And yeah, hope we see more customers start using it as well.

Lars, is there like a recurring issue you see people doing with performance testing with Locus that they could optimize better maybe to work in this scenario, scale better?

I think one of the main drivers for good performance testing is analysis of the server side, either using Insights or other tooling. Right now, I'm working almost exclusively with OpenTelemetry stuff, and we have tracing all the way from the load generator through all of our systems, and it reduces the time for me, especially because I cannot be an expert on every system that I load test, right? So, that takes down the time from— for debugging some issue or bottleneck from weeks to maybe hours for me, which is amazing. And I don't have to run around and ask everybody. I can just look at the trace and I can see exactly where the issue is coming from. And then I can talk to that specific team.

And that's the piece I was, I was, I was getting at earlier. So that, that's where it comes in. The OpenTelemetry lets you know that this code you checked in, this is the method maybe that caused a performance regression. I don't know. I assume because it gives you the actual trace to—

Exactly.

Wow.

Let's say a request takes 10 seconds to complete. Now you know what subsystem is responsible for the majority of those seconds. Right. So you can start debugging there instead of having to check logs for every system and try to find your transaction among lots of other ones. And yeah, debugging is basically much, much faster.

All right. So Lars, I keep hearing about AIO Locust. What is that?

AIO Locust is a simplified version of Locust that takes Takes the idea of it's just Python another step forward. So the code is actually even shorter and more to the point, less of plugins that you hook in and more about just writing plain Python code. And it's also modern in that it uses asyncio rather than Gevent, and it uses free-threading Python. So it doesn't, you don't need to scale it up using more processes. A single process can utilize a whole machine regardless of the CPU count, basically. So it's more modern. In a lot of cases, it's faster. And it's also OpenTelemetry native. So all the metrics that it uses are regular OpenTelemetry metrics, which makes it even easier to hook into an observability platform.

So it's a syntax or it's an actual UI, like a recorder?

It's very similar to Locust. I can show you.

Oh yeah, if you ever could see it, absolutely, even better.

Yeah, so basically this is as simple as it gets in AIO Locust. So it's using async syntax, which is slightly different from old school Python, but it's very useful, very explicit. And yeah, it uses, like I said, very modern Python approaches and it makes it a lot simpler. And this is still being developed, so it's not finished. And unlike Locust itself, it doesn't have a GUI, at least not yet, but it should be familiar to anyone who's been in Locust before. And you can download it. It's called AIO Locust.

Very cool. Is there any time you wouldn't use this? I guess if you really want to go in the guts of the code, or does it have all the capabilities, or you said you are still developing it? What do you have?

It doesn't have, it doesn't support all the things that Locust does. I think, uh, the main, one of the main reasons to switch to this is if you're using async libraries, uh, because Locust does not natively support asyncio, uh, whereas this one is built, uh, with that in mind.

So if I was trying to choose, is this a separate tool you would say, or is it just a separate flavor?

It's, it's, uh, it's a separate tool, but heavily influenced.

Okay.

So it has a lot of the same features. Some of the flags are identical, uh, and the approach is similar, uh, but it is a separate tool.

Yes.

And this one relies on, uh, recent versions of Python. So you need to have Python 3.14 or later, whereas, uh, Locust, uh, the regular Locust has, has broad support for Python.

And this is open source as well? Love it. And this would work on Azure also?

Is there— Not yet.

Okay.

It's coming soon to Azure.

All right, guys, before we get going, Abhinav, best way to get started maybe with Azure App Testing or Azure Load Testing?

Yeah, so you can just click on the link that comes up on the screen and you can get started right away. with your free trial Azure subscription, and you can get started with running your first load test today. And yeah, so it's just that it's— the barriers to get started have never been lower, right? So anyone who is probably skeptical or scared to start running or don't know how to start running load tests, I would highly recommend that now is the best time to try it out. All you just need to do is have a user journey that you want to do, and we'll simulate that real-life traffic for you.

Love it. And Lars, best way to get started, do you recommend starting with AIO, or does it depend?

I think if you're interested in the latest and greatest, you can have a look at AIO Locust. It's good in the sense that there is a recording tool plugin for a tool called MITM Proxy, Man-in-the-middle proxy. So you can record traffic and create the script dynamically. But I would say it's like, if you just wanna do a performance test and you don't really care about the finer details, you don't have any asyncio libraries that you need to use, you're probably better off with traditional Locust for the moment.

Awesome. I highly recommend everyone check out all this awesomeness using those links down below.