Performance Testing Using Locust.io with Rahul Solanki

Welcome to the Test Guild Performance and Site Reliability Podcast where we all get together to learn more about performance testing with your host, Joe Calantonio.

Hey, it's Joe. And welcome to another episode of the Test Guild Performance and Site Reliability Podcast. Today we'll be talking with Rahul Salaki all about the benefits of performance testing using Locus IO. Rahul has more than 10 years of experience in performance testing and engineering. His primary focus over the years has been providing end to end testing performance engineering solutions that include performance testing, monitoring and bottleneck analysis. He also has experience testing various enterprise solutions in multiple domains like healthcare, retail, insurance and gaming. And he also has used numerous performance testing tools like Locus IO, JMeter and LoadRunner. So he has a lot of Here we're going to really dive into what is Locus, what the difference is between the different tools. Again, his take on how to be successful with it. If you have anything to do with performance testing, you don't want to miss this episode. Check it out.

This episode is brought to you by SmartBear. Listen, load testing is tough. Investing in the right tools to automate tests, identify bottlenecks and resolve issues quickly could save your organization time and money. SmartBear offers a suite of performance tools like Load Ninja, which is a SaaS UI load testing tool, and Loaduipro, an API load testing tool to help teams get full visibility into UI and API performance so you can release and recover faster than ever. Give it a shot. It's free and easy to try. Head on over to smartbear.com solutions forward/performancetesting to learn more.

Hey Rahul, welcome to the Guild.

Hello Joe. Thank you so much for having me here.

Great to have you.

It's great. Yeah.

Really excited to have you. Especially about this topic. I was really excited when someone from your team contacted me about talking more about locust IO. Before we get into it though, Rahul, is there anything I missed in your bio that you want the Guild to know more about?

I think you covered pretty much everything just that I'm working for blueconch Technologies and at blueconch Technologies we provide vendors with to productize their ideas, build commercial off the shelf software products and transform legacy products. So that's about quick introduction to the company.

Perfect. So I assume work on your company. You do have experience with other performance tools like JMeter and LoadRunner. So how did you come upon or stumble upon Locus IO And I guess maybe we should dive into like what is Locus IO in general?

So yeah, thank you so much for the question, Joe. JMeter was the open source solution that we were using so far so mostly number of clients that prefer the open source solution. So JMeter was good for us until a certain limit of time wherein we had a requirement to execute a crazy amount of load like around 2,000 requests per second. So that's when we use JMETER for that tests. But we had a few challenges like we had to set up an infrastructure wherein we had to have number of load generator machines to create a master slave setup for that. So number of slaves were set for that particular test. So we tried to identify if we could have avoided this situation. So we had to do this because the resources, the test results that we are getting out of that particular configuration, it was a little bit like we were not able to rely on it. Like the test results were inconsistent, we were getting number of timeout issues. And when we observed on the server side at that time, we saw that requests that were coming into the server, they were really low compared to what we were sending over there. So this is how we identified the client is facing some bot lag. Also the resources on the client were we tried to identify what's going wrong over there. So we saw that the CPU memory, those were really high because of which we were getting issues on the client side. So we tried to google a few options wherein we could identify other tools that using which we could test this better. So that's when we stumbled upon Locus IO. So Locus IO essentially is a python based tool which basically executes on G event coroutine, unlike what JMETER is based upon, which is a thread based architecture. So since it's a thread based architecture, it ends up occupying a lot of resources. So when you Compare it with JMeter, the number of resources that that Locust occupies is like 70% less. So were really excited to understand what were the capabilities of lukast. So that's how we started exploring into it. We configured a few of our tests using Locust and observed that it was really easy to execute a distributed test using Locust even on a low number of slaves. So a similar kind of master slave setup is required in Locust, but the resource utilization that is on each slave is comparatively very low when you compare it with other tools like JMeter, other open source tool. So that's the reason why we tried to go forward with Locust since it's a python based application. Python being a very strong scripting language. So we saw number of different advantages scripting wise and we were able to easily integrate it with version control tools like GET and that's how we tried to go ahead with Lucast and created a wrapper over the existing framework of locust to suffice our requirement.

Nice. So I didn't know it was actually based on different architecture. I just thought the benefit of it was Python. So it sounds like JMeter has been around for a while, but you're saying, saying at scale, when you're trying to scale it because it's thread, it uses a thread model that it's harder to scale up with a bigger performance test.

Locus. It basically it has a completely different user simulation model which is based on events and asynchronous approach using gevent coroutine. So what is gevent? Coroutine is a Python based networking library and it uses Greenlits, which are very lightweight coroutine and it enables Locus to execute the performance test using very limited CPU and memory resources.

So that's the reason why there is a completely different architecture to both of the tools and that's why it is very easy to execute high number of user load using Locus.

Nice. So another question I think I would have is, you know, with GMeter, I don't know how much you probably don't need to know a lot of java to use JMeter, but for Locus, do you need to know a lot of Python in order to use the tool itself for performance testing?

Okay, so it depends upon the kind of complexity that you have in your tests. So basically if you are required to only script the test and execute any particular API or any particular HTTP scenario, it would be very quick and easy. So scripting wise, if you Compare it with JMeter, since it has a UI, you can directly record and go ahead with it. But in Locust you need to have some basic knowledge of Python. You don't need to be an expert, but you need to have some basic knowledge of Python to go ahead with scripting. But it can be done quickly as well, based upon your expertise. You can implement whatever scenarios you have expertise in Python.

Nice. I assume you could do a lot with it because it is Python. You have the full power of Python, then all the different libraries you could pull in. There's probably things you could do with it programmatically that you may not be able to do it. JMeter, yes.

Basically JMeter has a wide area, it has been going on since years and it has a wide number of plugins to it and it does almost everything. So Locust, being a new tool in the market, it has certain limitations to it. But if you want to overcome those limitations that were in JMeter, like there is a customizable ramp up and ramp down model that you can implement in JMeter using different plugins, like different type of thread groups. If you want to implement the same in Locust, you can do that. Again, you need to have some expertise in Python and you need to pull in the libraries from Locus and you need to modify those and go ahead with it. So that is possible. But the only thing better that Locus would do, I would say, is that it would be able to save your resources. That's the major advantage that you get.

In Locust, obviously saving resources on your node machine. So I guess it saves you money as well, is what you're saying.

Yes, when you're comparing the two open source tool, like both are open source JMETER and Locus. But to set up test which involves like thousands of users, or let's say scenario wherein you want to execute thousand requests per second at that time you might need to set up a test rig which is of, let's say, I'm just giving a random number, let's say 10 different nodes, slave nodes for JMeter. So if you want to execute the same test using Locust, you would probably be good with like three or four nodes. So that would save your infrastructure cost.

Nice. Now you answered this earlier. You said a lot of your clients prefer open source being older. I have experience with loadrunner, I love Lode Runner. It's an enterprise full stack solution, has everything built in. It's beautiful. So are you using Locus and JMETER just because of the licensing of some vendor tools like loadrunner or mopbearer's tools? Maybe it's what the clients are asking for. Is that the gist of it?

The basic reason is that it being open source and other is that the customizable factor that you get with the open source tool, you get to play around with the libraries, you get to implement whatever custom code you want to build. You can write wrappers on top of that to enable us to have exactly what kind of analytical data we require. So loadrunner, it basically gives you a lot of data, but sometimes there is a requirement wherein the stakeholders, they would request for very specific kind of data so that if you want to pull in that data and create a customized dashboard for that, then that is where open source tools comes into play. So that is also one of the reasons why we would select that. Yeah, it definitely saves the cost, so people would prefer that.

So I love having options of many different tools. So this particular tool was your approach to learning it and like your approach to exploring what it did and then maybe seeing if there are any gaps or if actually was going to work for your situation. Can you tell me a little bit about your approach to maybe trying out new tools or you know, just analyzing different tools and seeing what the probably best for.

So basically we have analyzed this tool along with JMeter and Loadrunner, the max. So Loadrunner, due to his licensing issues was not considered. So other open source tool that was available was JMeter and with JMeter we were good to go. Like it was sufficing most of our requirement. But when you compare, when you want to have a specific test, you want to execute a specific test which has high number of user load. At that time it was not sufficing our requirement. That is where we started to look into other options. So when I googled around I saw that JMETER is mostly compared with Locust these days. So that increased my curiosity in Locust and I tried to identify what advantages Locus brings. So basically the advantage here is that you can write it in Python and Python being a very strong scripting language and we are able to customize the user behavior. Like whatever user behavior you want to implement, you can simply write a code for that. I mean definitely you need some kind of Python expertise for that, but that's how you can go ahead with it.

So I love it's the programmatic piece of it because I think it's able to shift left earlier and get your developers more involved.

Yes.

Did you have a chance to look at any other tools like Gatling or K6 because they use kind of similar approach? I didn't know if you had time to compare this to those. I think they're kind of similar to Locus than locus is to JMeter, if I'm correct.

I did look at K6, which was a JavaScript, which is Java based on JavaScript, but just on the fact that Python was involved in Locust and the advantages of object oriented programming and all the advantages that it brings with Python, we went ahead with Locust.

So you did also mention JMeter has plugins for everything and it has been around forever and ever. And so that's a good thing and a bad thing because it could be over complex install JMeter, it's like what the heck is this? The other thing with Locus I would think that you probably are missing some functions functionality that you alluded to. So as you were learning it, is there any key missing functionality that you think you had to create in order to really benefit from locus?

So in JMeter basically it has a internal built plugin which enables you to write data into any database which is like there is a listener for that, so you can directly go and dump data into any kind of database. So that particular thing was missing for Locus, because Locus as it is does not provide any kind of way to dump data into the database for access in the future or for comparison for Baselining. So that is where I found that need to be implemented there, which we then created a wrapper over it and were able to build that.

I know a common thing with Selenium, people struggle because Selenium is just an API from a functional animation that people build a lot of wrappers on top that take advantage of Selenium under the covers, but it builds all those other stuff into it that makes it easier to use. Does your company or yourself have like a solution or framework on top of Locus that enhances it for maybe performance engineers that maybe don't want to have to develop their own stuff out of the box like that?

Yes. So as a part of the performance engineering vertical at bct, we have developed a framework, we call it the Kratos framework. So it basically provides a dashboard. So on that dashboard you would be able to have all the kind of analytical data like your response time, the request per second, the transactions per second, the error rate, all the baselining data which compares it with the last test result, the underlying tool that you want to select along with that, any open source tool yet that can be changed along with that, either JMETER or Locust. Now we give the clients an option whether you want to go ahead with JMETER or with Locust. So JMETER are generally preferred by client who want to have the test developed real quick, who want quick test results and they want to just have a point requirement. And once that's done, that's done. But when there is a requirement of continuous development cycle, like if you are going for shift replacement approach, wherein there is going to be frequent execution of tests at that time, we generally recommend to go with Locust. Why? Because initially it might take some time to develop those scripts. You might need some Python expertise to understand how that is need to be developed. But when you consider it, during the maintenance phase, when you go ahead, the development, the responses and the APIs, they keep changing very frequently. The test data, it keeps changing very frequently. So in this case you need to modify your scripts. So when you have to modify JMETER script you need to do number of different changes in that UI and it generally has that XML script wherein you need to go and do those modifications. So instead of that, if you have it in a script, in a code, you can do point changes. And especially if you're working in a team, different team members, they can push in their changes into the version control tools and then it basically is very easy when you consider it in shift left approach.

Nice. So I guess when you're working with clients, do you create the initial framework or test and then they take it over or is it just they hire you and you do the performance testing for all the different iterations that they have to do?

Yeah, we do the performance testing ourselves as well. Or we provide this framework which enables them to execute the test by themselves as well.

Nice. So it sounds like for so far for Locus in the plus column is it saves you money if you have a lot of a big load test and a bunch of different generators, nodes up and running. And it also sounds like it's easier to maintain if you're making a lot of changes. Are there any other ROIs that you found from working with Locus from your different projects that you found that using this tool was a big benefit for that particular project?

So yeah, when you consider it from the maintenance perspective, I found it real easy because modifying a piece of code is really easy and it gets reflected everywhere where you're using it. So I would definitely prefer that over UI based, wherein you need to go and make modifications and every point. So that is a key advantage that I have known here, along with the resource utilizations, of course.

So obviously I guess because it's Python based, does it easily integrate with the teams like CICD Jenkins type environment?

Yes, it has like it is very easy to integrate it with Jenkins and get it dockerized and execute the test in containers. So that integration is also provided. There are different plugins for that available and we can write wrappers over it to make this functionality work.

I guess the only, not the only, but one of the cons I think maybe I'm wrong is finding information on how to use Locus could be an issue with JMeter since it's been around for so long, you probably could google it and get a lot of hits. So do you find that as an issue or a barrier to entry because there is not as much information out there as there is for JMeter.

Yes, that definitely is a part of the challenge. However, they have provided a lot of notes on their documentation part on their website, so that pretty much covers most of the things that we require.

I guess my next question before I recommend any tool Open source is the community. So how active is the community? Have you ever had to put in a help desk ticket to try to get some features added or is it easy to contribute to? What are your thoughts on how active they are as a community themselves?

So the Locus community is pretty much active. I mean, I haven't posted any questions on my own, but as far as the way I'm following up on the forums, so I see that the community people, they are very helpful and they respond quite pretty quickly.

So I'll ask two more questions. So as an engineer, you've been working for 10 years as a performance engineer, one thing that I like, it's kind of silly is people liking the tools they use if they're fun. So as a performance engineer, do you find it fun to use Locus? And so like maybe that would engage more people to use it compared to maybe JMeter, which is to me seems more clunky.

Yeah, so for me, if you ask me then I prefer to code so it's easier for me to understand the code and make changes to it. So that's why I would rather prefer Locust.

Is the framework you have, how can people get their hands on it? Or is it part of your, is it open source? Is it just something that your company provides as part of a service when they hire you to do performance testing?

It's something that the company provides when they hire us for performance testing. So if someone wants to get their hands on that, then they can contact.

Asmita and I'll have a link to that in the show notes as well. Okay Raul, before we go, is there one piece of actionable advice you can give to someone to help them with their performance testing efforts and what's the best way to find or contact you?

Okay, so if there is something that you need to change is that you need to identify how you can execute your load on the real time scenario. Like there are the production workload. I mean definitely no tool can simulate the exact production workload. But we need to find ways in which you can identify how you can simulate the spikes that are there in the system. Especially into domains like retail domain wherein there is huge spike at times. Especially in today's Coronavirus situation wherein people are using the online platforms like crazy. Like the user load has been increasing exponentially. So that is when we as performance engineer need to up the game and try to simulate actual user scenarios.

Awesome. And Rahul, the best way to find.

Or contact you it would be by LinkedIn or my Instagram. Yeah, by LinkedIn.

Thanks again for your Performance Testing Awesomeness links to everything of value we covered in this episode. Head on over to testguild.com.

And while you're there, make sure to click on the Try them both Today link under the Exclusive Sponsor section to learn all about SmartBear's two awesome performance test tool solutions, Load Ninja and Load UiPro. And if the show has helped you in any way, why not rate and review it in itunes? Reviews really do matter in the rankings of the show and I read each and every one of them. So that's it for this episode of the Test Guild Performance and Site Reliability Podcast. I'm Joe. My mission is to help you succeed with creating end to end full stack Performance Testing Awesomeness. As always, test everything and keep the good. Cheers.

Thanks for listening to the Test Guild Performance and Site Reliability Podcast. Head on over to testguild.com for full show notes, amazing blog articles and online testing conferences. Don't forget to subscribe to the Guild to continue your testing journey.