API Load Testing Strategies with Harsh Upreti

Welcome to the Test Guild Performance and Site Reliability Podcast where we all get together to learn more about performance testing with your host, Joe Calantonio.

Hey it's Joe. And welcome to another episode of the Test Guild Performance and Site Reliability Podcast. As you probably remember, or if you didn't, I actually took April off because I turned 50 and I just was in mourning. So I took the whole month month off to celebrate kind of mourn type of deal. And then when I came back I was sick for almost all of May. Because of that I didn't have a lot of interviews lined up. So what I'm doing is I'm going through my back catalog of my Perf Guild online conferences, taking the best of best of sessions, chopping them up and making them more of an interview style session. So this is a quasi interview with Harsh Shah Pretty from his Perf Guild 2017 session on API performance testing. Harsh is currently a partner Technology Manager at Google, but previously when he made this recording he was the Product marketing manager at SmartBear. So in this episode you're going to discover the importance of load testing. Your backend APIs/ walks through some rest API examples and shares tips on using distributing load testing. You don't want to miss this episode. Check it out.

This episode is brought to you by SmartBear. Listen. Load testing is tough. Investing in the right tools to automate tests, identify bottlenecks and resolve issues quickly could save your organization time and money. SmartBear offers a suite of performance tools like Load Ninja, which is a SaaS UI load testing tool and Load UI Pro, an API load testing tool to help teams get full visibility into UI and API performance so you can release and recover faster than ever. Give it a shot. It's it's free and easy to try. Head on over to smartbear.com solutions performancetesting to learn more.

Hey Harsh, welcome to the guild. So just to kick it off, why do you think API performance testing is so important?

Well, I think when we look at applications traditionally we will see that performance testing is thought of as a performance testing of an application or a or mobile phone application of a browser based application. But then if you think a little bit more about it, you'll find that that application is not just about, you know, testing and black box. It's composed of multiple tiers, multiple layers primarily. You can categorize those layers into two broad categories that will be the front end and back end. Front end would primarily run on your one set of system that is your client that would be mobile Phone application, or it could be a browser based application, which is a different hardware, different configuration, different resources. And the backend is going to run on a cloud server, maybe on AWS or on Azure or on Google Cloud, or maybe your own data centers or servers. Although it's still an abstraction to say that everything runs on a server because the backend would be composed of multiple things. You would see that backend. First of all, when a request comes from your front end, after you press a button, it's going to hit probably a load balancer, right? And then the load balancer is going to take that request and it's going to direct it to one of the application servers. And the application servers is going to have that API, that web service that's going to respond with some data, but that data doesn't resides on that application server, it resides downstream in maybe a database. And the database may have another cache layer before it. So when you talk about performance of an application, it's all these factors. It's the ui, it's the button, it's the resources on your client side, it's a load balancer time, which is of course very fast. But then it's also the application server resources, it's also your cache performance, it's also your database performance. So there are a bunch of things.

So what's some of the benefits of actually measuring the performance of your back end APIs?

So if you can measure the performance of your APIs, you can very confidently say that you know how your entire backend performs. Because that API is going to in turn call your database, it's going to interact with the cache. You're assuming that your load balancer works perfectly fine because that's what it's supposed to do. Its job is to perform really well in all conditions.

So the point is that we will take a look at API performance testing. We would take a look at.

If you have been testing on APIs, you will probably have your functional tests. We would try to convert your functional tests into load tests very quickly. And then we will see the importance of scenarios. We would see how you can replicate the front end scenario that you would have done. Maybe for example, if you're buying an item on an E commerce website, you can generate a scenario like login, search for an item, buy an item, and then log out on your APIs as well. So we'll try to create that kind of scenarios. We will load test these scenarios on different strategies and then we would see some advanced concepts like distributed load testing and maybe performance monitoring. Of your application servers on which your APIs are hosted.

Cool. So what do you think are some requirements that you need to know about before you start load testing your backend APIs?

I think it's very important for you to first of all take a look at exactly what's the use case behind your APIs. For example, it could be that you are running an API that's primary purpose is to serve a front end. So that's going to be a slightly different use case then an API that's primary, that has a primary purpose of serving another API upstream. An example of that would be something like a payment gateway. So if you are a developer of a payment gateway, then most of the calls coming to your APIs would be from some other system or server.

So there are different scenarios because the way your API would be hit by load is different here. If, in case your APIs are being used by users on a front end, then the users tend to work in a different way. They would interact with the ui, they would click on a button, they would pause and they would move forward. But a machine would be just hitting your API continuously, again and again and again as soon as it gets some load. And often these.

Machines will have bad jobs. So they may have a set of data elements to process that could be in thousands or in millions. So they will just iterate upon those data elements and they will just keep on hitting your APIs again and again and again. So having that understanding, high level understanding of what exactly is the use case for your API is very important.

So one thing I think a lot of people struggle with is how do you actually get the requirements for your APIs?

You can get all the requirements for your load testing of your APIs from a number of sources, right? So a lot of times you have systems engineering teams, they capture their requirements in the specification documents, they will define that how should the API perform? What is the throttling rate of an API? What rate limits do we subscribe to? It will also tell you sometimes the number of users that you expect on the application. So you will have to do some calculations around, like if X number of users hit your front end, how does it translate into the number of API calls to the backend? So it's always good to know the connect between your ui and your APIs. So maybe your submit button on your E commerce website might be hitting three or four different APIs at the same time, or multiple buttons on your application might be hitting the single API at the same time. So knowing that connection between your front end and the back end, if you're primarily a front end tester, is really important. A very good way of getting an idea of an application's load is to look at the Google Analytics. So if you have Google Analytics installed on your website, it's always good to know what is the trend of traffic on your website or your application. You can always look at the graphs and see the seasonality of your traffic. And seasonality is not just about seasons like spring or summer or autumn. It's also about the way your traffic changes every day or every hour. So that will help you craft very specific API load testing scenarios for your APIs and your backends. So definitely keep these things in mind while you're performance testing your APIs.

So previously you mentioned it's important to when we test to test full scenarios. Can you explain that a little bit more?

Because it might be possible that you may have you have to add a pet to your data store on your APIs through a rest call. And if you call it again and again and again, there could be issues in your backend, there could be issues in your code, there could be memory leaks that could impact the performance of your other API calls. So a scenario helps you in determining where the bottlenecks are. Typically, you would try to look at the use cases that your product team has. They have already set up certain use cases and you would try to emulate tests based on those use cases.

So before you actually even jump into a full blown performance load testing, regardless if it's API or anything or web, do you ever recommend doing a one user or a single endpoint type of test?

Of course, it's important for you to have a way to simply load test one single endpoint. Maybe you're responsible for just one single endpoint and you should be able to do that. And any load testing solution should enable you to do that thing. So here you can easily add a load test using a single rest endpoint or using different specifications. There's a difference between having a single endpoint or a URL versus a description file like a Swagger or a vistal in an APIs. So let me pull up an example of Swagger here up on the screen. You can simply get it by typing petstore Swagger IO and this is a specification Swagger spec in front of you, which is essentially a collection of API calls. Post, put, get, delete. So it defines all the possible APIs that you could do with a pet store. And this is a job of your developers and API designers to Kind of come up with this API. And if you're testing these, then you should have an access to this specification and you should be able to simply copy these endpoints where the specifications are stored and you should be able to create tests out of it.

So any other tips we should look out for when we're creating scenarios for our performance test?

There are a bunch of things that you would take care of when you are basically looking at load testing. Well, depending on the scenario that you are going to use, you will have a time for which you would run your load test, right? So it could be hits infinity. You can do that for as long as you want. But for the sake of this talk, we would probably go around with a smaller amount of time. We will do a 15 second load test. There are a couple of other parameters that you have to be aware of when you are setting up a load test on either a single API or a bunch of APIs in a form of a scenario. Number one thing is you would see the kind of load test you want to run. So for this scenario we have this load profile. You may want to do this kind of load profile. For example, there is a fixed load profile and this is like setting the baseline for your APIs. You want to run a specific set of tests on your specific load, specific pressure on your APIs for some amount of time.

For example, here we're going to do.

It for 15 seconds. So 15 seconds, the software would put a consistent load on your APIs. It's a good idea to set the baseline. That's the minimum level of users that your API can support. We'll talk a little bit more about users later. So you can have fixed kind of traffic on your APIs. You can make it like a burst. So every few seconds traffic will rise up to certain level and then it'll come down, ramp up. Your traffic would increase from maybe 0 or a fixed number to a very high number linearly. There is a ramp sequence. It increases, goes to a ramp and then comes down and random. You do not care about a specific kind of pattern on your load testing. We just want to make sure that any random load could be would work on your APIs would probably go with the random profile. And then we also have custom, which is pretty interesting. Just want to show you, because you may have a load profile under which it might increase in the morning and then it might.

Stay flat in the afternoon. It might go down drastically towards the evening. Right. So you can create a custom profile of your own to kind of figure out what the exact pattern of traffic.

Looks like for your website.

And then try different combinations of load testing profiles so you can do that custom profile.

So what are virtual users? When do virtual users come into play?

At this point, the concept in load testing is the number of virtual users. Well, it's the number of threads your software would start and it will keep that consistent pressure on your API. So for example, it's like emulating 10 users and at any moment in time there will be 10 simultaneous users hitting your APIs. Right? And because it's a fixed profile, so this number is going to remain fixed 10 users at all times on your APIs, they're hitting it continuously. Even one thread starts and it does its job, goes through the whole scenario and it ends. The software would keep the number of threads@10. It will again start another thread to keep the number fixed at 10.

What about wait times?

If you want to simulate a scenario wherein you might have a situation in which this scenario is occurring after another scenario and you want to kind of wait for the user inputs, you will probably put a few seconds of wait time out here. So that's another useful feature.

How do you measure the actual performance of your APIs as you're doing a load test?

Assertions are the primary way that you would do it. You will set certain assertions so you can basically assertions, you know, if you've been in testing, that basically are the checks against the responses. So the response, the performance of your API could be benchmarks against all these variables. Number of failures, number of requests that failed, the number of test steps that failed, the number of views that you have been running and the time taken. All these fundamental statistics are available here that you can look at. Then you can filter them based on overall response times or per second, what happened in each second. And then there are certain other parameters like constraints and tolerance that you can set up here.

Before we go, any actual piece of advice you can give someone to help them with their API performance testing efforts.

Okay, so again, going back to the idea that do all of this once you know the requirements for our APIs. Well, of course, number one step that we talked about was that set up the right scenarios, know the right scenarios, know the right use cases, set them correctly as API calls, and then know the kind of traffic and the profile that you're going to get and set them up correctly on the tool.

Thank you. Harsh for performance testing Awesomeness for everything valid we covered in this episode. Head on over to test guild.com forward/p65 as I mentioned this session was taken from a live Perf Guild conference session that hashed it for us in 2017. If you actually want to see this in action, and the full session itself where he actually dives into using REST API and you could see what he's doing, how he's debugging and finding tips and tools around that area as well, I'll have that embedded in the blog post as well. And to get that once again, just head over to testguild.com p.

And while you're there, make sure to click on the Try them both Today link under the Exclusive Sponsor section to learn all about SmartBear's two awesome test tool solutions, Load Ninja and Load UI Pro. So that's it for this episode of the Test Guild Performance and Site Reliability Podcast. I'm Joe and my mission is to help you succeed with creating end to end full stack automation that includes performance testing, insight, reliability testing, awesomeness. As always, test everything and keep the good. Cheers.

Thanks for listening to the Test Guild Performance and Site Reliability Podcast. Head on over to testguild.com for full show notes, amazing blog articles and online testing conferences. Don't forget to subscribe to the Guild to continue your testing journey.