Top 15 Open Source Load Testing Tools

Welcome to the Test Guild Performance and Site Reliability Podcast where we all get together to learn more about performance testing with your host, Joe Calantonio.

Hey, it's Joe. And welcome to another episode of the Test Guild Performance and Site Reliability Podcast. Today it's going to be a solo episode, just the the two of us talking all about the top 15 load testing tools for 2021. I recently reposted a blog post I wrote in 2017. I updated it with new information and I reposted it and it's been getting a lot of traffic, a lot of likes, a lot of comments on LinkedIn. So I thought I create a podcast around it as well. So even though I called it the top Load testing tools for 2021, it's really some of the top 15 performance testing tools for 21 and beyond. Hope you like it. Check it out. Let me know what you think. And if you want to be a guest on a future podcast episode and you have something you want to share around performance testing or set reliability, definitely Send me an email@joestguild.com let me know what you're thinking. I'm always looking for a guest.

This episode is brought to you by SmartBear. Listen load testing is tough. Investing in the right tools to automate tests, identify bottlenecks and resolve issues quickly could save your organization, time and money. SmartBear offers a suite of performance tools like Load Ninja, which is a SaaS UI load testing tool, and Load UI Pro, an API load testing tool to help teams get full visibility into UI and API performance so you can release and recover faster than ever. Give it a shot. It's free and easy to try. Head on over to smartbear.com solutions forward/performancetesting to learn more.

So here are my top 15 load testing tools for performance testing in 2021 and beyond. And if you haven't already, why not rate and review this podcast in itunes or whatever podcast player you're using. Give it hopefully a five star review or just be as honest as you want and leave a review ratings really help with the rankings of the show and helps more people discover this podcast to learn more about performance testing awesomeness. So you might be asking, before we get into this, what is load testing? Load Testing is a specific sub family of a performance test that includes many concurrent users. Run the same program at the same time. This is done to see to make sure whether a system's infrastructure can handle the load without compromising functionality or performance. This is usually done to make sure the system infrastructure can handle the load without compromising functionality or performance. But if you're specifically looking for performance testing from a front end user's perspective, definitely check out my guide down below on the complete front end Performance testing guide. I'm a big believer in vendor based performance testing tools for enterprise level performance testing, but this list only contains free and open source options so you can download and and get started right away without spending a dime. Some of the vendor based tools do have free community versions you can use to get started, but they usually include limited functionality so this list is only for open source and free options. First on our list is JMeter. JMeter is the most popular open source tool in the performance space to help measure load time. Apache JMeter describes itself as a Java application that was designed specifically for load testing with the ability to measure application performance and response time. JMeter was built pretty much to be the open source alternative to loadrunner at the time, so you find Has a lot of sophisticated features. It's a thick client Java app that's largely driven by right clicks. It can be a little weird, a little hard to get used to, but it has a lot of power. It also has a lot of features that a professional performance engineer would be interested in. For example, it has the ability to performance test a host of technologies including a range of protocols such as Java Objects, Web HTTP, HTTPs, soap, REST services, FTP databases with JDBC and more. Has a nice IDE that you can use for recording, building and debugging your performance test, even though it does look a little dated to me. Starting with Jamie to 3.1 Groovy is the default programming language, so it's easy scripting language that you can use to create performance testing. It's also one of the more popular performance testing tools, so there's a lot of information out there on the Internet that you can search to get help with if you're struggling with anything, and you can configure it to help test mobile performance of mobile apps. Although JMeter is one of the more popular test tools, it does have some shortcomings. So for instance, JMeter can be very difficult to scale for a large distributed test, especially if you have to set up a bunch of machines since you have to configure them to talk to one another. There's also a host of orchestration issues and problems in executing large JMETER test, so if you're doing large scale, you may want to look elsewhere. Next on the list, While not officially or technically a load testing tool, we have Torus. Torus acts Like a wrapper on top of other solutions that had the complexity of running performance tests. So if you found something like JMeter Complex, Taurus sits on top of it and what it does, the power of it, it allows you to write your performance tests using YAML so you can actually describe a full blown performance test and probably just around 10 lines of text, which gives your whole team the ability to describe your test in YAML or J on files. Since YAML is a human readable, easy to edit approach, enables you to describe a test in a simple text file. This really is a big leap forward for teams from the old days of having to bring up a big heavy vendor based specific record and playback tool for load time measuring. Some benefits of using Torres is that it allows more of your members of your team to contribute to performance testing. Also since your test cases are written in YAML and the tests are more readable, making it easier to perform code reviews on, so that's a big plus. Taurus also helps enable you get your performance tests into your CICD pipelines much more efficiently and it provides a lot of abstract layers on top of JMeter as well as some other tools like Gatling, Locus and Grindr and Selenium and other tools we'll be covering in this list. Third on the list is Locus, which is a simple to use distributed user load testing tool that can help you capture response times. And one of the best features of it is its scripting language which is Python. So it allows you to write straight up Python to create performance tests, which could be a big big plus factor for teams that already use Python in their development ecosystem. So you can use it to help performance test websites or other applications. Locus can help you figure out how many concurrent users a system candle as well. Another plus of Locus is it helps you run large performance tests without eating up a lot of resources. I actually interviewed someone on my POD podcast recently, Rahul Solinki, who is a techno leader at blueconch Technologies, and he mentioned to me that when he compared Locus with JMeter, the number of resources that Locus occupied was around 70% less than JMeter. So a lot of times when running JMeter you may have performance issues, but those performance issues are usually tied to JMETER itself, eating up all your resources. Locus frees you from that kind of constraint, so it uses terminologies like if you're familiar with load generators, Locus uses the term swarm and so you can point a large swarm of Locus to put a load on your website without having to worry about affecting the performance results based on resources. You're eating up with the tool itself so you can define the behavior you want for each of your Locus instances, and it gives you the ability to monitor the swarming processes from a web UI in real time. Some pros once again to use in Locus are the ability to create your test using straight Python. You can easily scale the number of users you need to emulate. Has a real nice Web based GUI UI, it's extensible and it's good for testing APIs. Also one con that some people complain about it doesn't have as many plugins as JMeter, but that's because it's a newer tool. Over time you may see more plugins along the way. So this list as I mentioned, is covering mainly load testing tools, but I also included other tools that help you with your performance engineering efforts. And the reason why I bring this up is that next on our list is Fiddler with Black Widow and Watcher. And you might be thinking, wow, this is a real odd combination to have on a performance tool list. But performance engineering expert Todd DiCapia, in a previous Perf Guild conference session that I hosted, actually recommended using Fiddler with Watcher and Black Widow to create kind of a quick start aut testing performance solution for you. So Fiddler enables you to do a lot of different things, but it's probably best described as a packet capturing tool. While Fiddler may not be considered a load testing solution, it does many things that allow you to debug website issues from a performance perspective, and it has many extensions that you can put together to really accomplish even more. Some things you may want to use Fiddler for is troubleshooting performance issues with your web application. You can actually add add ons like I mentioned earlier for security testing, performance evaluations. Debugging web traffic from most computers and devices has many integration features, it's handy for finding performance bottlenecks. And Fiddler is already a pretty popular tool among your developers, so many use it for debugging to view the HTTP requests that your computer is sending to a service or a website. The Watcher piece is a security add on for Fiddler, which will enable you to get some quick security results really fast. And Black Widow is a web crawler that gives you the functionality to point it towards a web app or web address and then it's able to drill down on those results. Next we have nGrinder and nGrinder's GitHub page describes it as having been really designed as an enterprise level performance engineering solution. It was developed to make stress testing easy and to provide a platform that allows you to create and execute and monitor test all in one place. Some features of ngrinder is you can write your tests using jthon to create test scenarios and create stress against JVM using multiple agents. You can extend tests with custom libraries like dot jar and dot PI files. It allows you to monitor the state of your performance agent load generators. It takes care of automatically collecting test results from all your distributed agents after each test and it's a proven solution that's been used at large organizations to very big load test. Another tool on our list with a similar name is the Grindr and the Grinder is a Java based framework. It provides you with an easy to run and create distributed testing solution using many load generated machines to capture your end user response times so you don't have to worry about virtual user restrictions that you have. Sometimes with other tools where you need to unlock a license to get 1000 10,000 the price just goes up and up. With Grindr you don't have that issue. You can perform load testing on any system that has a Java API, has a nice GUI console. It also has support for allowing you to embed code like Java, Jython and Clojure and automatically handles management of client connections and cookies. So a big trend in software testing is Shift Left which is trying to get testing involved earlier and earlier in your software development lifecycle. This has happened with a lot of automation tools for functional testing, but just now a lot more performance testing tools are being developed this way as well. One of them is Gatling and Gatling is a stress tool built on Scala and uses some modern architecture to allow you to test and measure your applications end to end performance and easily scale up your virtual users. It has a really simple yet powerful dsl. It's easy to extend if you're into Scala and the benefit it brings. This is the load testing tool for you. It also has a scenario recorder and because you can develop your test using straight code, it allows you to take a shift left code approach to performance testing. So next on our list, which is similar to Gatling in its approach to being more Dev Friendly is K6. K6 is a developer centric open source load tool for testing the performance of your backend infrastructure. So you can use K6 to start including performance testing into your continuous integration pipelines. So it treats performance tests just like any other coding artifact. So you can write code for your performance test checking into CI cd. You can keep it in version control so Any type of best practice used for software development is the same type of approach. You can use Gatling and K6 for and K6 is a modern load testing tool built with Go and JavaScript. So as I mentioned integrates really well with most developers workflows already has a clean scripting API and it provides distribution and cloud execution. Has a really cool REST API orchestration ability. So it's definitely one of those newer tools that if you're a developer centric type person, this might be a tool or Gatling might be a tool that you should look into first for your team. Next we have is Sung, which is an open source, multi protocol distributed load testing tool. It can monitor a client's cpu, memory and network traffic and also has an HTTP recorder and includes HTML reports and graphs and supports multiple protocols like HTTP, XMPP and LDAP and more. Another type of tool that's really handy for kind of like really quick brute force type load testing tool is Siege. So Siege is a command line HTTP load testing and benchmarking utility. It was designed to help developers measure the code under stress and because you can run it from the command line, developers should be really able to easily integrate it with their current developing solutions. It supports basic authentication cookies HTTP, HTTPs and FTP prot. It allows its users to hit a server with a configurable number of simulated clients and those clients place the server under siege, hence the name. And like I mentioned, it's great for simple brute force type of testing. Next on the list is Bees with Machine Guns. So developed by the Chicago Tribune, this is probably the coolest sounding performance tool on our list. Bees with Machine Guns describes itself as a utility for creating many bees which are really essentially micro Amazon EC2 instances to load, test or attack targets like web applications and apps in general. And so this tool can easily help you to load test a site that requires or has need to handle high traffic. And it leverages Amazon EC2 instances so it gives you that ability to just scale up for massive types of load tests. Next we have 40o which is a kud load testing library, a command line tool. It's an advanced echo server and web UI in Go or Golang. This tool allows you to create it's like a query type of set per second load against your application and the recorder captures latency, histograms and other useful stats. So some pros of using this approach is it's fast, it has a small footprint, it's reusable, it's an embeddable Go library and can record latency histograms and other useful stats. Also, with most modern applications using JavaScript heavy front ends, the need for client side performance measurement has never been greater. And that's why I definitely recommend down below checking out the link I have to my complete front end performance web testing guide that will get you up to speed on client side performance testing. That link will actually be in the show notes@testguild.com P62 so you don't want to ignore front end performance tests. You can use Puppeteer webperf to collect web performance stats like a trace for a web load. You can grab a devtool trace with screenshots. You can also get runtime performance metrics and a bunch more. And if you like this tool definitely should check out all the other tools in my front end performance guide that you can get down below. Our next tool takes a different approach from some of these other tools on the list and that is Flood Element. So if you want to test real browser load from a real user, actually interacting with a browser Flood element may be something you should check out. Flood created this open source solution to help specifically with mimicking user behaviors using real browsers for load testing. Element is an open source playwright type tool that uses a browser based load testing tool approach. Scripts are created using the Typescript and you can use it against your web apps in the same way that your customers do. Opening a browser and interacting with the web elements, capturing that recording and then just playing back in a real browser so that you can capture like more of a real type of scenario that users are actually going through when they're doing a performance test on your site. And this approach can be really helpful to finding front end user facing performance issues. And it's written like a functional test script, but with the help of Flood you can quickly run load testing scripts on as many nodes as you want in the cloud. And this is one of the few open source tools that allows you to create performance testing using real browsers to really more closely mimic how real users interact with your application. And last on our list is Artillery IO. I first heard about this at Automation Guild 2021. We had a speaker talking about how they implemented this at USA Today, which is a huge news organization to handle all the different load that they would experience during the US Presidential.

Election cycle. So they get a lot of load during that time and they use Artillery, which is an open source application with an opt in for premium services to create load tests to simulate load to emulate thousands of users. It has detailed performance metrics including latency, requests per second, concurrency, captures average response time and throughput, and allows you to capture and run peak performance testing to handle maximum traffic for your backend applications. For stability and reliability, it has the ability to write custom logic, post and pre test scenarios using JavaScript, which has a wide variety of NPM modules that you can use. It supports multiple protocols like HTTP, WebSocket, Socket, IO, KINSAS, and HLS. So those in my opinion are the top performance load testing tools for performance engineering. As I mentioned, some of these tools aren't necessarily load testing tools, but they help you with performance engineering. And as always, when choosing a performance testing tool for your team's stress and load testing needs, it depends on what your team needs. So definitely I highly recommend doing a proof of concept to run any type of tool before you adopt it through your two week type sprint to make sure that everyone involved in your team actually finds the tool helpful and fits into your unique team's type of work ecosystem. So happy performance testing with these stress testing tools and load testing tools. And for links to everything we covered in this episode, head on over to testguild.com P62. And while you're there, make sure to click on the Try Them Bolt Today link under the Exclusive Sponsor section to learn all about SmartBear's 2 performance test tool solutions to automate your UI performance testing. They have Load Ninja and for API performance they have a great solution called Load UI Pro. So that's it for this episode of the Test Guild Performance and Site Reliability Podcast. I'm Joe and my mission is to help you succeed with creating end to end full Stack Automation which includes performance testing awesomeness. As always, test everything and keep the good. Cheers.

Thanks for listening to the Test Guild Performance and Site Reliability Podcast. Head on over to testguild.com for full show notes, amazing blog articles and online testing conferences. Don't forget to subscribe to the Guild to continue your testing journey.