Load Testing Done Right using Gatling with Stephane Landelle

Welcome to the Test Guild Performance and Site Reliability Podcast where we all get together to learn more about performance testing with your host, Joe Calantonio.

Hey, it's Joe. And welcome to another episode of the Test Guild Performance and Site Reliability Podcast. In honor of PERF Guild 2020, we which is happening next week. So if you're listening to this episode, it's only about five days away. You definitely want to check it out. If you haven't registered already, head over to perfguild.com register for the 2020 event. But in honor of the PERF Guild 2020 event, I'm actually opening up the Perf Guild vault. Going back, going way back to 2017 to our first Perf Guild, where we spoke with Stefan Lindell, the creator of Gatling IO, to talk a little bit more about what is Gatling, how it can help you, and how it takes more of a dev approach to performance testing. So it's almost like a mock interview. The first half is I'm taking the recording from his session on Perf Guild that he gave to the guild in 2017. I'm also going to take some of the Q and A we had live with Stefan and kind of stitching it together to create a episode for you today. Now, as I mentioned, this is from 2017, so it's a little bit older, but the content is still relevant. For folks that aren't familiar with Gatlin, I think you're gonna get some value from this. You definitely don't wanna miss it. Check it out.

Let's face it, performance testing is tough. Traditional load testing tools create scripts that are bogged down in data that's hard to read and require a lot of work, even for simple playback. Load Ninja cuts out the dirty work by using AI to inspect and debug your code almost instantly. No coding necessary and no changing of hands or relays between teams either. Give it a shot. It's free and easy to try. Head on over to testguild.com smartbear and click on the link to learn more.

Hey Stefan, welcome to the guild.

Thank you very much for having me here. Thank you very much, Joe. And yeah, I'm waiting for your patience.

Cool. So, Stefan, at a high level, what is Gatling?

Gatling is an open project I started six years ago. It's published under Apache 2 license and you can find the code on GitHub in the Gatling organization.

The open source work has been very steady since the beginning, so you can see that it's really an active project. We release several times a year with new features and bug fixes, et cetera. We plan to release the next major version, Gatling 3, either end of this year or beginning of next year.

So I know the main component of Gatling is the open source, but I believe you also have an enterprise solution. Can you tell us a little bit more what the enterprise solution is?

So our company Gatling Corp. Provides, well, does most of the hard work on the open source version and then we also provide professional services like training, mentoring and we also provide this kind of enterprise version of Gatling which is named Frontline, that provides you with many more features that you don't have in your process version, like distributed tests, archiving, on demand deployments and way more metrics than what you have in your version.

So Stephan, before anyone starts using an open source tool, I always tell them to make sure that is actively worked on and that it's a popular community. So how popular is Gatling? And for folks listening, just so you know, the answer Stefan gives was from 2017. So Gatling has obviously grown. They have way more download numbers than they did four years ago. But here's what he mentioned when I asked him this question.

In 2017, Gatling is quite popular actually it's quite great that. Well, when we released the first version I think we had something like 20 downloads and now the project is very popular thanks to our community. For example, last year there were something like more than 20 conferences in 10 different countries all over the world, in Europe, US, even Japan talking about Gatling. So really huge kudos to our community members who spread the knowledge. The project is quite popular on GitHub.

You can check it there.

Also, if you need more information about what we release, I encourage you to follow the GetLink tool Twitter account.

And also join the community mailing list.

If.

You feel like a gattling expert or feel free to help other community members with their questions on our mailing list.

As I said, it's quite a popular project now. It's been downloaded 700,000 times since the beginning of the project and really the downloads are really skyrocketing those last years. It's being reused all over the world.

We are based in France, but the project is being used a lot in the us, in the UK and all over Europe and also in the rest of the world.

So another thing, I think you say that Gatling is opinionated. So what's the philosophy behind how you think people should be creating performance tests utilizing Gatling?

The thing is, if you consider what tests are about, tests are really about providing quick feedback on potential issues.

So that the team members, the developers or the ops people can quickly fix before it's too late, before the impact to fix is is too large and too expensive to fix, or that it would jeopardize.

The go live. We've been able to achieve this kind of quick feedback with functional tests for quite a long time. So with functional tests we can have maintained tests with the test being pushed on a source control system such as Git. We can do peer reviews so our team members can control and give feedback on the test. We can build dry tests, reusable components. We can also build libraries for our tests. We can also run our tests periodically, several times a day, plug them onto our continuous integration. And really what's interesting with functional test is it's not just the sole task for a QA team. Really all the team members and all the developers are really involved into the test. Question is, how could we achieve that with a functional test? We could do that because functional tests are nowadays mostly text, not something that you craft with a graphical user interface. Just have a look at things like Junit or Selenium or Cucumber et cetera. You really have text with an expressive language and not graphical user interface. So really the question was why cannot we achieve that for load testing to adjust kind of test in the end? So what we really need is things like DSL and or actual programming language, actual code. This piece of code is an excerpt of a Gatling test. And I think even if it's the first time that you see this kind of the first time you see Gatling, you can pretty quickly understand what this test is about. It's a scenario named Basic simulation that would hit the homepage and then perform a five second pause. So really, in the opinion of Gatling, load tests are just another kind of test and you should be dealing with them the way you deal with the other kind of tests, not with a graphical user interface, but actually with code.

Also, I believe you have a different view on performance and threads. Can you talk a little bit about how you or how Gatling treats threads and performance?

Quite often I experience performance issues with load test tool where.

The poor performance I was experiencing was actually coming from the load test and not from the system under test.

On contrary to other tools, Gatling has a kind of different threading model. If you consider other tools, you can see that virtual users are implemented as dedicated threads. The thing is that when you just run a few threads or a few dozens of them.

Your load test will execute smoothly. The thing is that when you start trying to generate thousands or tens of thousands or hundreds of thousands virtual users and you implement them as thread, your load test rule will start experiencing some troubles. That's not how you should be using threads. The thing there, also with some other tools, is that some of them use blocking I os.

So those threads are quite expensive resources. And at some point you realize that they spend a lot of time just blocking, waiting for responses.

It's just like you've sent a letter and you're waiting in front of your mailbox waiting for the response of your, of the other person. It's just a waste of resources. Another waste also is the think time. How do you implement think time when your poses are, when your virtual users are just threats? Well, you use the sleep method. And once again, it's just a waste of resources. So when we design the Gatling architecture, we wanted to come up with a modern architecture and a proper same usage of threads. So what we have with Gatling is a message oriented architecture. All the orchestrations of the virtual users are done with Akka, which is an actor framework running on the JVM.

And what we use for iOS is the Neti framework, which is the the top IO framework on the JVM. If you check the Gatling documentation, you won't see any mention about threads.

With Gatling, virtual users are not threads, they are just messages that are being processed along a workflow. And of course.

We use threads underneath, but we just use them as technical resources to process the virtual users. So typically on an eight core machine, Gatling would run with something like 40 threads. With this kind of architecture, you can achieve pretty good throughput. On this example, we are hitting.

A REST API.

So we had two different machines, one for the Gatling injector and one for the REST API that was returning one JSONK payload. And you could see that we could achieve more than 100,000 requests per second with.

1000 virtual users.

So you can achieve a pretty huge load with Gatling.

Also, how does Gatling handle the metrics that you report?

If you really care about your response times.

You should pick a proper metric. What I mean is that there are quite a few tools that just give you one single, let's say chart and line over time with one single metric called response time. This is actually the average value. The thing is that really the average value is a poor metric. Well, if you have a poor average response time, yes, you have poor response times, but you could perfectly have a good average value and actually miss the fact that 20% of your virtual users are experiencing poor response time. You wouldn't see that with the average value. Some of the tools also provide you with, along with the average value, the standard deviation that's supposed to represent the way that.

Values are close or not from the average value. The thing is that actually this, this metric is only significant for normal distributions of perfect curves and symmetric curves, which is something that you never have on real world systems. So what you really need to know about your system is the response time distribution. How am I faring for 90% of my users? And how bad am I for those 10% remaining?

So with Gatling, what we provide is a series of percentiles.

That give you really proper insight on your distribution. You shouldn't be assuming anything about your distribution, you should be measuring them.

Also, if you have a load test tool that provides you with percentiles.

Should make sure how they are being actually computed. Typically be aware of systems that provide you with averages of percentiles because, for example, they are measuring percentiles every second and when they try to aggregate those percentiles every second to, for example, provide you with the overall value or the overall value on the whole cluster of injectors. Or just because you're trying to display a chart that can, let's say, display 100 plots, but you have 10,000 records in the database, so you would have to reduce those many records into.

A smaller number of plots. How can you do that? The only way would be to compute an average value of those percentiles. And this is just completely wrong math. Typical example, the 100th percentile is maximum. And are you ready? Okay. With a tool that would provide you an average of the maximum, it doesn't really make sense. And when you have this kind of tool, you end up with kind of smooth curves with reassuring results. And when you start zooming in and getting closer to the resolution that you have in the database, you'll end up with start seeing all your hiccups and realizing that, oh my God, no, my distribution is way worse than when I first thought.

So pick a tool that provides you with proper metrics and properly computed distributions.

So I know with more developer centric performance tools, one of the selling points is integrations, because you want to be able to use the same tools that your developers use. So how does Gatling work with other tools?

As Gatling is really code oriented, it really integrates well with.

Developer tools.

So with Gatling, what you use is actually the Scala language. So it runs on JVM.

And so it typically integrates very well with the Java and JVM ecosystem like the IDEs such as integer or Eclipse, different build tools such as Maven or Gradle or sbt. And as those are, in the end as your test will just be source code, you can push them, store them into your source control system like Git.

And also it integrates pretty well with a continuous integration pipeline. Typically we provide Jenkins plugin for the open source version and.

Also provide plugins for bamboo and TeamCity for frontline or enterprise version.

Now in your session you talked a little bit about acceptance criteria. Can you tell us a little bit more what you meant by acceptance criteria?

One interesting thing is.

When you plug Gatling in the continuous integration pipeline, it's very interesting because first you can make sure that even if you just run smoke test, you can make sure that your load tests are always up to date when you're trying to run them at full scale. And you can also run them on a smaller scale system where actually you would have kind of a baseline of your stats and maybe be able to.

Figure out some performance regressions from one build to another. And Gatling will let you define acceptance criteria on your test criteria such as I want the response time 95th percentile of my overall response times to be below a given threshold or I want my error ratio to be below a given threshold, otherwise I want to fail my build, report that into Jenkins and maybe send some notifications from Jenkins. So yeah, as you see, the Gatling system is quite rich and it provides quite a few features that we think fit quite well in the new DevOps movement.

So Stefan went to give a actually hands on demo at the 2017 PERF Guild. If you actually want to see that hands on demo and get all the recordings for the previous perf guild or even for the or even for the current perf guild going on next week. For loyal podcast listeners of the Test Guild Performance and Civil Reliability Podcast, if you use the code PODGRUILD25 for any of the online conferences, current or previous, you get 25% off. All you need to do is go to guildconferences.com, click on the drop down, you'll see an option for Perf guild and in that drop down there'll be all the previous perf guilds. Just click on one of the perf guilds you're interested in and then click on the Register now buttons in the registration pages. Then click on the discount code option at checkout and type in POD Guild 25 to get 25% off. All right, so after Stefan's session, we brought him in for live Q and A to take questions from our guild attendees. So here's some of the questions the guild attendees asked Stefan. This is for perfdrummer. What is the scripting language for Gatling at a high level? What are the setup components needed to be configured to use Gatling to start scripting and analyzing results?

Okay, so so the scripting language is Scala. So Scala runs on JVM.

And sometimes it's a bit scary for newcomers who first hear who have heard a lot about Scala on Twitter or on the blogs, et cetera.

So Gatling is oriented, but what you'll be using most is our dsl. You just need probably to. You need to know about high level programming language, you need to know what method is, what passing parameters is, et cetera. But then you don't need to be a proficient Scala developers to be using gafli. Then just like with other tools, sometimes there are features that are very specific to your use case for your applications, et cetera. And sometimes some features are not covered by built in baked in into the tool. And then you need to.

Code some stuff. What's neat with Gattling is that you can directly do that.

In your scenarios in your test sources instead of copy pasting that into some text box into some graphical user interface. Then one important thing is that as Scala runs on a jvm, you can reuse all your Java libraries, familiar Java libraries that you're used to. And also maybe you use some components that are specific to your organization and then it will just run just fine.

Awesome. Stephan. So I guess how much Scala do you need to know since you can import into Java? If someone just knows Java, would they be able to get up and running with Gatling without any overhead really of learning more?

So you just need to know.

The basic syntax, how to declare a reference, how to sometimes declare a method. What you'll be.

Watching in the video.

Is what you'll be using most. Once you get used to that, you'll cover 99, 9% of your use cases.

Awesome. So Stefan, we have another question they want to know. We are an HP Loadrunner shop but want to push performance testing to an Agile Scrum teams for web apps. Is Gatling a better choice than loadrunner for this user case?

So.

I can't really compare with the loadrunner. What I can tell you is that we are currently helping more and more large organizations migrating from loadrunner to Gatling. And I think one of the reasons is that Gatling is a good fit for propagating load tests to the whole organization and have all the different teams jumping into and bringing load tests into the continuous integration pipeline. So.

What we see more and more is having the project teams perform performance tests in isolation and then a kind of organization teams, specialized load test teams sometimes assemble components coming from different teams to build larger full scale load tests. And what's neat here is that really they can use the same tool for those different kind of load tests all across the organization. And also one thing is it's probably very strongly opinionated, but.

I think having a coordinate oriented solution makes maintaining the.

Test a bit easier.

So if you enjoyed this episode, drop me a note@joetestskill.com I've been trying to get the folks on Gatling back on the show because it's been a long time. I've been going back and forth with emails so hopefully we can have them back on. If there's anything else you want to hear covered about Gatling, let me know and I'll make sure to ask them for an update based on your question and for links to everything of value we covered in this episode, head on over to testguild.com P35. And while you're there, make sure to click on the Try It Today link under the Exclusive Sponsor section to learn all about SmartBearer's awesome load ninja Performance Testing solution. And if you haven't already, you only have 5 days left. Head on over to GuildConferences.com and sign up for PERF Guild 2020. If you use the code PODGuild25 you will get 25% off your ticket price register today. So that's it for this episode of the Test Guild Performance and Site Reliability Podcast. I'm Joe. My mission is to help you succeed with creating end to end full stack testing awesomeness. As always, test everything and keep the good. Cheers.

Thanks for listening to the Test Guild Performance and Site Reliability Podcast. Head on over to testguild.com for full show notes, amazing blog articles and online testing conferences. Don't forget to subscribe to the Guild to continue your testing journey.