ScalaTest Performance Testing Microservices with Zubair Haque

Welcome to the Test Guild Performance and Site Reliability Podcast where we all get together to learn more about performance testing with your host, Joe Calantonio.

Hey, it's Joe. And welcome to another episode of the Test Guild Performance and Site Reliability Podcast. Today we'll be talking with Zabeer Haig all about Scala, Test for Performance, Test Testing along with some things he learned. Performance testing microservices using Gatling this interview actually was one main interview around automation and performance, but what I did is I split into two because I have two different podcasts, one for automation and one for performance. So this is the performance piece of my interview with Zubeer. Zubeer is a detailed oriented software engineer and test with extensive experience. With over 10 years of experience in the product development domain, he's expert. He has expertise in a bunch of topics like solution architecture for test automation, authoring, test strategies, and test tooling in a fast paced CICD environment. If you do anything with Gatling or you want to learn more about Scala tests, you don't want to miss this episode. Check it out. Let's face it, performance testing is tough. Traditional load testing tools create scripts that are bogged down in data that's hard to read and require a lot of work, even for simple playback. Load Ninja cuts out the dirty work by using AI to inspect and debug your code almost instantly. No coding necessary. No changing of hands or relays between teams either. Give it a shot. It's free and easy to try. Head on over to testguild.com smartbear and click on the link to learn more.

Lazybag welcome to the Guild.

Hi Joe, how's it going? I'm so happy to be here and thanks for having me.

Thank you so much for joining us today. Really excited about this topic, as I mentioned in the intro. But before we get into it, is anything I missed in your bio that you want the Guild to know more about?

No, I'll just piggyback off of what you said. I'm Zubair. I'm currently working at the Climate Corporation, which is a leader in the agriculture technology industry. I've been there for about two years now. I have about a decade of experience, a little more than that, and more than half of that time I spent being an SDET with the core focus on automated testing of client server systems in a Linux environment and a Windows environment.

Very cool. So I'm gonna start really high level and I thought we'd dive into it step by step. So first Scala. I've heard of Scala. I know Gatling uses it, but I actually have not done much with it. So I'm not sure how familiar folks are with with Scala. So at a really high level, what is Scala?

So Scala is a functional programming language that's on the JVM and it's gained popularity for two reasons. One is for the stateless programming and the second is for the lazy evaluation. So as you've heard, there's a lot of AWS lambda. So when you call the function, it does a lazy evaluation and then it gives you the output.

Would you choose this language over java then, or JavaScript? Like is it a programming language or is it a language you use to test other languages? If that makes any sense.

That makes a lot of sense. Having a diversified background, I've worked with all kinds of tech stacks all the way from NET to Ruby on Rails to Java Spring and now even Java Spring boot. My interpretation or my best practice that I would give to people is to write the API tests in the same language that the back end APIs are written. So if you're going to write integration tests and your backend services in Scala, you should write those tests in Scala. If your backend services and Golang, you should use Go Test. If your back end service is Flask in Python you should use pytest. So that's my reasoning of using Scala. Right? And the reason for that is right. In this microservices architecture of contract driven testing that we're in right now, what basically happens is there's several changes being made continuously in a CI environment. Let's just say a dev makes a change and he breaks about 10 tests. If the tests are written in the same backend language and it lives in the same repository, it's easy for them to diagnose those failures pre merge or even post merge.

So performance testing I think a lot of people overlook, but it sounds like this might be your approach, it might not be approaches. You have a bunch of functional API tests now written in Scala. But it sounds, I'm thinking because Gatling also uses Scala, you can reuse those functional tests and then make them performance tests. Am I thinking that correctly?

Yes, you're thinking that correctly. Now we are using the free version of Gatling, not the one that has a ui. So I'm scripting my performance test just the way I'm scripting my API test. You're correct about that. And these performance tests, right, they're a key source of information for all of the developers, stakeholders, anyone that's involved, right? And in a microservices environment. We basically have blue green deployments where we deploy a service for a short amount of time and we check the health of that service. And if that service is functioning correctly with all of the other services in our platform, we spin that elastic container in AWS down and spin up the new one. And once we basically do that, it's basically gone through its rigorous amount of checks, whether it's API test, performance tests and security tests, or whatever have you. I'm going to go over the specific details on how to set up your performance test suite with Gatling, as well as writing a test simulation from scratch to test our APIs. But before that, I want to go over what Gatling really is. Gatling is an open source load testing framework that's written in Scala. This tool is used to measure the performance of the API that's being developed by the team. With continuous integration, there's going to be several changes made. Those several changes will be incrementally made throughout a period of time, and then your API will be promoted to production, which will be in use by your users and come in heavy traffic by.

Your users. One of the great things about Gatling is that I can write my performance test just like I write my integration test. As you mentioned before, your Gatling scripts can be integrated with CI tools such as Jenkins and others, if you prefer that. The benefit we have when we do that is we get constant feedback from our performance tests and we immediately can identify like, hey, should we roll this change back? Right, because this change doesn't meet the sla, right? It doesn't meet our pass fail criteria as far as a threshold that we've set. Right. And this is important for our organization because we continuously make changes to our API before we actually promote it to production.

Nice. So do you actually have performance criteria in your pipeline? So if someone does a release of microservice, does it have like, if it doesn't meet a 99%, I don't know, like some sort of transaction timing threshold that it doesn't get in the main build.

Right. So when we're developing and we get our design docs right, before we actually get into the core development of things, we have test skeletons that we write for API. So meaning that we're basically writing tests while the developers are developing. Right. And another thing that we're doing is we're writing performance test plans. Right. And basically sometimes we're writing them before there's any SLA or anything even there. Right. In this particular scenario that I'm going to go over. We were rewriting a microservice, right? And we didn't have an SLA and we didn't have a threshold.

So we actually needed those details right before we released the production. Right. And I remember in your Tech Beacon article, right, and the Tech Beacon article was, if I remember correctly, shift. Right. Testing microservices in the wild to tame DevOps. You spoke about using monitoring tools and setting up alerts, right? So currently we do the same thing. We're using New Relic and Splunk. New Relic is a tool that we use for application performance monitoring. This tool has helped us with various things such as transaction response, error reporting, and much, much more. Right. And transaction response, for those that might not be familiar with that is we use transaction response in New Relic to gauge which requests are taking longer to respond. We use error reporting in New Relic to notify us if a system fails, as well as what the recovery was, how it's acting if when it's recovered, and helping us gauge if the API is stable enough to even make it to production. We also use these tools for like, scalability testing. Right. It's helping us gauge the ability of a system to cope with a higher volume of requests, data and users. So just to reiterate, let's get back to the pass field criteria threshold you were talking about. Right? The pass fail criteria for our test at this point would be to maintain the same SLA as the current services in production, right? So we navigate to the transactions module in New Relic and we look for the throughput and average requests per minute during high traffic hours. So let's just say a high traffic hour for us is 2:15pm, right? We're gonna go to the transaction module, New Relic. We're going to look for the average request per minute during 2:15pm and when I once I checked that it had an average about maybe 612 requests per minute. So we knew that the new service should at least have a threshold of 612 requests per minute without throwing a server error or showing a spike in traffic. For those that don't know what a spike is, we can go to New Relic and during high traffic times, we can check if our containers in the cloud are basically overloaded or not. Now, fun fact is Gatling actually has several conditions that can be chained together to accomplish what you said. The LTE threshold condition lets you assert the value of the metric, and if it's less or equal to your threshold, you can actually check if it's greater than or equal to as well, right, etc. Etc. Now, for each incremental software change that you're making and you want to publish the production, you can tag that git commit and you can run your Gatling test against that. And you can validate if it meets your threshold and your sla, and if it does, you can promote it to production.

So how long did it take you to get to this point? Point? It sounds like it's pretty mature. I love this approach. I've been advocating it for a while. So how hard was it to get buy in from your team to actually do all this?

So basically I brought this up during like a retro, that we should be performance testing all of the APIs to meet a specific SLA or threshold before we take it to production. And when I brought it up in the retro, we talked about how we could bake it into our release plan, and then we talked about how we could test various features of it and we added it to our Sprint plan. It also has to do with an organization's timeline, how fast do they want to release to market and what are the precautions they're taking to get there. So for us, downtime in a service. So we need these services to have high availability, right? We don't want any downtime, especially for farmers in the field. So for us, this was a very important topic. I don't know if other organizations will think that's an important topic or want to invest the time up front to outline these procedures.

Now, how complicated was it for you to get set up and get started? Any lessons learned there for a quick start?

So basically, if you're setting up Gatling on your box, you just need Java 1.8 or Scala version 2.12 or above. If you're using Maven, you just need to add that IO Gatling to your POM xml. And if you want to run the test, it's just as easier as MVM Maven Gatling colon tests. If you're using SVT in a Scala environment, you could just write SVT Gatling tests and then you can run the test straight from the base directory in the SBT documentation. It will just let you know that the SBT plugin needs to be added to the file that should live in your project directory. Other than that, once you're done with that, you basically have to bring in the imports, just like you would bring in the request import if you were doing API testing in pytest, just like you would require a gem to be used. If you're using RSpec just like you would require import to be used for rest assured, in your JNUGA test, you would import the Gatling core and the Gatling HTTP imports. Gatling HTTP will allow you to load test the API. It supports all HTTP and HTTPs traffic and works primarily at the HTTP protocol level, meaning it's back end service driven, not UI event driven. Now if you go for and prompt out to use the paid version of Gatling, you can basically do more of like a heat heat map or like a real scenario from the front end. And it could do more of a record and playback, but I'm actually not doing that because I'm using the open source version and I'm interacting more with APIs.

Cool. So it sounds like initially you're doing almost like a one user performance test. So you have a baseline, you have a microservice that's been running production. You know, historically it takes about 60 milliseconds. So therefore going forward every other release shouldn't get worse than 60 milliseconds. Is that how you initially set it up? And then if so, do you ever do any sort of stress type testing?

Ah, yes, that's exactly how I set it up in the beginning. But I want to demystify that a little bit because that's exactly what I do later on as the API matures, as it goes through rigorous testing. And when we tag that commit to go to production with git, the most important thing I do is I write a class that extends the Gatling simulation. Now, a Gatling simulation is a class that you can extend in your script, which allows you to do HTTP headers, protocol and scenario definition. We want to start off with the HTTP protocol definition, basically defining what your base URL is going to be and any URIs that go with that. Let me be more specific with what a URI is, right? A URI is basically your endpoint. Www testguild.com.

Podcasts so podcast would be your URI and that will be prepended to the relative path in your scenario definition. The base URL will also have some kind of headers, right? Like authorization, bearer, token, HTTP request, id, so on and so forth, which will be added to the request when you ramp up and ramp down your users. Now here's the real meat of your burger. The HTTP scenario definition, scenario structure changed, two function calls, exec, which is describing an action, usually a request, such as sending a get request to a version one of your endpoint and the second one is pause. The pause simulates the think time between consecutive requests being made to the server. Now, those that are familiar with JMETER or Load Runner, various performance testing tools, they all have some way of processing think time or utilizing think time. So to clarify, you invoke the exec function which builds an action, makes your HTTP request, then adds a pause for think time between the next request. So an example of that would be we made a post request. Now, usually when you make a post request, you're writing something inside of a database. Then we invoke the exec function again to make a get request which retrieves what we wrote into the database. We can pause many different ways and we can utilize think time many different ways inside of Gatling. Milliseconds is one of them. Or we can have random wait times, right? And I like random wait times because a real user session will have a user looking at what's on the screen and thinking about what he's going to click next. So I usually have random wait times in between my post and get requests of anything between one and five seconds. Or you could just basically pause for any amount of time, like three seconds, right? So we now have our HTTP header protocol and scenario definitions. We can add in checks like for response code such as 200 or 201 if it's a post request. Gatling as many assertions you can utilize to do various checks to see if your endpoint is behaving correctly. Let's just say if you make 60 requests per minute, the first 55 were two or one because you made a post request and the 56 to 58 request it was 500. Gatling with these built in check assertions will let you know how many of these were 201 or 500. Now let's get a little bit into the ramp up and ramp down. The inject function in Gatling allows you to pass in the ramp user functions, which takes on an integer as an argument. An integer would be a number that you can pass into a function without using quotes. So a quotes would take string like quotes 10, end quote. Now an integer would just be the number 10. You won't have to pass in any quotes with that. So this flexibility allows us to ramp up any number of users we want during a specific number of time. For example, I want to ramp up 20 users every 30 seconds. It's highly unlikely that a hundred people will log into my app at the time same same time. So I want to have ramp up scenarios such as that this sequence of injection steps can be processed sequentially and then you can add in assertions for average response time of whatever your expectation is. So a common expectation for us is we want each request to be returned within 500 milliseconds. So there's basic example of that.

How do you handle dependencies? I know in microservices what's nice about it is, you know, it's so discreet, you know, it's this microservice. Here's the functionality for that microservice. You're testing that microservice, but in the wild, if there's other dependencies when you're doing performance tests, how do you know? Like another call from another microservice, calling into that service is going to not cause issues.

So basically what we do is we build test tooling around that. So a lot of things that have changed since microservices architecture has come out, service mesh architectures come out. A lot of time now teams spend in test tooling writing robust scripts to stage data. We write these robust scripts to stage data on the fly. You can call this somewhat of a test setup before your test tear down. We always have this test data stage and we run the performance test against the test data.

Cool. I think you've already touched on this a little bit, but it sounds like a lot of times you may not have all the information you need in order to create a performance test test plan. So any tips or techniques or tricks how to really curate a performance test plan from scratch.

So I would definitely encourage people to learn how to use new Relic, Splunk, Grafana and those kind of monitoring tools. You can go to a previous day or any previous amount of time and you can check user logs to see who is interacting with which service at what time and what's the average request per minute that it should sustain. That's the first thing I should do you guys should do. Now, let's just say you are at a startup or at a smaller company where you can't afford these tools. You could always run H top or tail the logs to see exactly which services are being interacted with. And you can use that to create your performance test plan. Or I have an even better way of doing this. Launch a Chrome browser, log into your application inspect element and go to the network tab and see the interaction of services.

And then basically create a performance test plan from there. And that will be like a user flow, like a business flow. Right. As far as your requirements for your application go, that would be an easier way to develop a test plan too, right? There's different ways of doing it. Depends on how mature your application is and where you're at in your software development process.

Okay Zubair, before we go, is there one piece of actual advice you can give to someone to help them with their Scala Test Gatling type performance testing efforts and what's the best way to find or contact you?

I would really dig into the Gatling documentation. They have an advanced tutorial link which basically goes in to everything that I've mentioned over here. I'm also going to have a blog that I will send to Joe and he can publish it on his website and you can basically go around how to set your suite up. You can always reach out to me. I would be more than happy to troubleshoot any issues that anyone has as far as setting up their environment and stuff like that. And anyone can reach out to me via email H A Q U S.

Which is my last name.first name@gmail good resources scalatest.org But I have a tutorial which is called Scalatest A Beginner's Guide. Anyone can basically use that to spin up their environment and reach out to me if you have any issues getting acquainted with Scala Test. I'll be more than happy to help.

Thank you to bear for your performance Testing awesomeness. For links to everything of value we covered in this episode, head on over to testguild.com P39. And while you're there, make sure to click on the Try them both Today link under the Exclusive Sponsor section to learn all about SmartBear's awesome performance test tool solutions like Load Ninja and Load UI Pro. So that's it for this episode of the Test Guild Performance and Site Reliability Podcast. I'm Joe and my mission is to help you succeed with creating end to end full stack performance testing awesomeness. As always, test everything and keep the good Cheers.

Thanks for listening to the Test Guild Performance and Site Reliability Podcast. Head on over to testguild.com for full show notes, amazing blog articles and online testing conferences. Don't forget to subscribe to the Guild to continue your testing journey.