Performance Testing of gRPC services with Vikas Kumar

Welcome to the Test Guild Performance and Site Reliability Podcast where we all get together to learn more about performance testing with your host, Joe Calantonio.

Hey, it's Joe. And welcome to another episode of the Test Guild Automation Podcast. Today we'll be talking with Vikas Kumar all about performance testing of GRPC services, microservice testing and more. Vigas is a full stack product automation test practice QA lead with over 13 years of experience with quality engineering and automation, having a lot of global experience in both the US and the Middle east, has a lot of experience around this area. You don't want to miss this episode. Check it out.

This episode is brought to you by SmartBear. Listen, load testing is tough. Investing in the right tools to automate tests, identify bottlenecks and and resolve issues quickly could save your organization time and money. SmartBear offers a suite of performance tools like Load Ninja, which is a SaaS UI load testing tool, and Load UI Pro, an API load testing tool to help teams get full visibility into UI and API performance so you can release and recover faster than ever. Give it a shot. It's free and easy to try. Head on over to smartbear.com solutions performancetesting to learn more.

Hey guys, welcome to the show.

Hi Joe. Hello.

Great to have you. Before we get into it, is there anything I missed in your bio that you want the Guild to know more about?

No, that's all you covered.

Very cool. So I guess before we get into it, you know what is GRPC?

To start with GRPC GRPC is like a protocol. Like Google introduced a protocol GRPC in 2015 remote procedure call you can say Google GRPC. You can say that has been introduced by. So it is a one way of. Like you might have heard about HTTP. So on top of HTTP there is currently we are coming about grpc. That is the latest one. So the idea behind what I say the GRPC is its idea is like a defining a service which specify the method that can be remotely called with their parameters and the return types. And on the server side the server implements the interface and run a GRPC server and to handle the client call. But on the client side we can have a stuff and which provides the methods which we can call the server. So it's all about on the high level about the grpc. There are a few advantages of GRPC over the rest. That's why people come up with a grpc. So that's all in the overview of around the grpc.

So what are some of those advantages.

Then over Rest the major advantage is just I think of the GRPC is like high performance along with the safety. So GRPC is very high performance with Google proto buff. So the transmission of the communication medium is a proto buff. It's not predominantly we see like XML on is JSON but this is more about on the protobuf it is over HTTP 2 and which is multiplex and single TCP connection. So it is very much high performance and along with safety that is the first advantage. Second one I think of it's a duplex streaming. It allows client side and server side streaming simultaneously and GRPC has built in library feature that has intellectually picked the backend to send traffic to. Like you can say like a first class load balancing. That is an another advantage of the GRPC duplex streaming which is in terms of the streaming the GRPC has a very much advantage because it works on the protofuf and it is very much faster. So that is the major advantage of the grpc.

Nice. So it sounds like it has a lot of performance advantages. So what are you testing then? What's the application then? That's made up of, I guess made up of microservices. What kind of application is it?

So it is the proprietary tool. I'm not supposed to reveal that. But on a high level what I would say it's of a like pronunciation. We are testing the pronunciation from one language, what we pronounce and whether you are pronouncing at a very like what is your pronunciation in that level. That is what we are testing on. So yes, it is a microservice based architecture. Totally. So the overall architecture is more about we get when we pronounce something on your mobile device we record that audio in wave file and that WAV file we used to check what is your pronunciation accuracy around that, how much the pronunciation label and what is your accuracy around that on that language. So that is what in the high level we are testing. So from the testing point, as is a new technology, there are a lot of challenges we faced initially while testing the grpc. First of all, initially it is a new thing. We usually test performance for the HTTP calls or the REST calls which a lot of tools available in the industries in the market. But GRPC is totally new and we faced a lot of challenges while testing this because we have to understand the domain, we have to understand the internals before testing this performance tool.

Nice. So what are some of those challenges? Was it just knowing what tools to use or how to interact with the protocol to get proper Results?

Yeah, that's a very good question, Joe. So when I got this challenge for testing the grpc so the first thing predominantly everyone goes for a VR a small startup. So we don't have proprietary tool or something. We got with an open source tool, JMeter, Gatling and all those things. And as an another performance engineer I also went with the same path with the JMETER and other tools at that point of time. Initially I thought of going with a JMeter but somehow I didn't find the right plugin and somehow I didn't find. And I don't want to reinvent the wheels like writing my plugins and all those things with a small duration for the release cycles. Like in a small startup the release cycle is very short and you don't have time to reinvent the wheels in terms of writing plugins or some services. So I evaluated some of the tools and later on luckily I found a plugin for GRPC in Gatling. That is Gatling grpc, that's the plugin that I have used during the test. So I use that Gatling GRPC is pretty simple and as the Gatling is a Scala based totally we have to write the code to get the performance simulation. So we went with that and able to build the project and we moved. There are other challenges we have how to read the WAV file, how to record the recordings that you will get from the mobile devices. Reading the WAV files and then converting and doing the performance testing. Understanding the internals like the proto because the Proto Buff is, you know, like internally the proto buff is composed of a proto file so you will get some dot proto file around that. So you have to understand the dot proto file and accordingly you will call that method for doing the performance. So these are the challenges that we faced. Other than that, Gatling has some advantages that I found. Like a CI CD integration is pretty easy in terms of Gatling compared to. Compared to JMeter the CI CD integration part was pretty easy. Other advantages I found in terms of CPU and Gatling is little lightweight. That's what I felt over the JMeter. I'm not weighing more on the Gatling, but that's what I found during the test. Like. Like it's a comparatively lightweight and it is not a thread based. So most of the time that test runs pretty smooth and you got the result in even if you run with a high tps. So those are the challenges that we face during the setup of this GRKC performance testing.

So it sounds like you were familiar with JMeter but how hard was it to go to Gatling and how would you compare the two? You just said Gatling is a little more lightweight and maybe easier to pick up.

Yeah, I was in our organization like we are using the JMETER only. So there are the challenges that I faced in JMeter running for a high load like it's a thread based distributed architecture. You need to do when you are running for a JMeter. Let's say if you are running with a very high RPS, like a 5000 RPS, you want to run the performance test with the user base of more than 50k. Then you have to go with a distributed setup. You find like most of the thread get stuck and you need to handle those things using the JMETER tools and all those things. Though the industry supports the open source JMETER like challenges if you face many folks are there to help you out in that. So those are the advantages. But while going with a Gatling one thing was like it is coming up in the market and I also thought of trying something new like Y gate link. And the major advantage that I face that I would say that the major advantage is you can invert the your release cycle when you go with the JMeter it's more like it comes performance at the end of the your release cycle. Like you will go functional, you will go all sort of testing and at the after your final build sign off then you can go for the performance testing. But in terms of a Gatling like Gatling is totally a like a. Not like a tool based. It's more like a code based. So anyone can write the code given they are developing an API like even developers can write the code. It's not limited to only to QA folks as to do or something like that. So anyone can keep adding their performance test at the start of the cycle. So we don't need to wait till the end. And the release cycle will be a little bit pretty fast. And the integration with CICD tool is also simple and easy. So those are because in the JMeter like in the distributor setup from the different you get the reports that from the different clients you need to get it at the main server and then the HTML publisher will publish the reports. So those challenges it's not there in terms of the Gatling cool.

So you actually see or do you develop as use Gatling at all or is that like just a possibility now because it is in code almost.

So I see industry is moving like developers are going through the Gatling part in terms of the performance testing. Most People are shifting towards Gatling in terms of they have started the writing and now we are in the combined engineering mode. Right. Like there is nothing dev qa. Most of the startups and everything sound like more of a combined engineering, like it's a one team, everyone has to do most of their stuff. QA is predominantly on the qa but it's not like something manual or so at the start of the cycle only people can start contributing to the mostly on the making their back end strong. So mostly people are contributing more on the performance testing, REST issue, microservices, API testing and end to end testing. So that are the more focused area currently for a faster release rather than going for a UI test or something which are mostly flaky and also those things are changes happening but definitely it will take time to move towards that.

As I mentioned, I never used grpc, I've used HTTP in rest. So how do you performance test for grpc? Are you testing for a payload? Is it a status code? How does that work?

Okay, in terms of what you test for in terms of the finance. So to understand the protobuf, what the protobuf is when to understand how the GRPC what you test. So you will pass the payload you will call an RPC method GRPC that your proto file will expose some methods like in the in the your proto file. So that method you will call with the right parameters what that that method is accepting and you will extract some the response body out of that and that you will validate. So GRPC has different status code. It's not like the HTTPs status code. So GRPC status code. But in our test we have not done in terms of like status code of grpc rather we are extracting the body and response and we are doing around that.

Cool. So was this your first experience performance testing grpc and how was it different from like HTTP? Like would you have any lessons learned that you would tell someone who's starting a project performance testing grpc?

Yeah, there are a lot of lessons learned. First of all, the whole technology itself is new for me. I am not aware of around anything ABC of grpc. So first of all, most of the time when for understanding what the GRPC is the second I was not aware of the mostly the transmission, I was mostly aware of xml, JSON and all those things. So protobuf was itself for me is a new thing like what the protobuf is, why protobuf is used, how protobuf works and all those things. So in terms of that is that the Binary serialization and all those things, encoding and decoding at the client side, all those things are the new things for me. I was of an assumption like the status code and all those things will be similar to what the HTTP and all. But here it is something altogether different. Like it's like a code like okay, cancel unknown something. It's a totally different status code compared to what we see in HTTP. So it's all together a complete new learning in a new. You can say new avenue of the. In the technology. It's nothing straightforward that you will read something and if you go online also you will not find much information in terms of testing the grpc, currently we don't have that much of online materials also around the grpc. So that's all. Things are the challenges that I face. But there are few good resources are there, which helped me a lot in understanding the whole concept of this and undergoing the performance testing.

So I'll be honest, I don't think I understand the protobuf file. What is it? How's that come into play? I know you've explained it a little bit, but just a little more clarification on the protobuf file.

What it is okay, a simpler language. I will say like you for the transmission what we use we use normally JSON, we use xml. So what the Google decided, okay, let's come up some form of format that will be even faster than the like what the predominantly we are using. So it's more like a file with an extension over the XML which mostly have a field. There are the fields values are there and it's indicated with a number and tag corresponding to that. So you will be having a field and there will be corresponding tag close corresponding to that. So you will be have something like a speech equal to one string text equal to two. So that has some meaning on the server side that you can say more often like a schema. So in that format it will encode on the server side and the client side it will decode. So the major advantage of these things are what I'd say like you can write your your server maybe in any language defined in any language. Let's say I have defined in Java, but I have a protobuf file. So you can compile that in any language. In the client side it may be of a Python, it can be of C of any language and it is fast. They encode the data and compress and then it's a duplex streaming it goes. So it is comparatively very much fast. So mostly in terms of streaming and all. It's quite useful where the streaming comes into the picture. So that's more about the overall of protobuf.

Nice. So you know you did mention JMeter. There's probably a plugin but you weren't able to find it. So use Gatling. Did you have to instrument Gatling any way in order to work with grpc or does it work out of the box?

No, most of the performance tool. I see now there are plugins for JMETER also for to test the Gatling for JMETER to test the grpc. Sorry. And Gatling also have like this plugin developed for testing the grpc. So it's not like one tool or other tool don't support it. Okay, so both of the tools support that the JMETER also supports as well as the Gatling also supports. It's. It's not like JMeter not support. That's why we went for Gatling or something. But it depends what is your learning curve and what you want to attend. So it depends on that also. So what is your performance infrastructure in your organization, why you want to go for this, why you want to give a try. So in my case mostly when I want to go for the try, like I find first some plugins around Gatling so that motivated me as a new performance testing tool also. So that motivated me to go for this tool and try that out. And luckily it worked. So I never went and tried with the JMeter plugin. But there are plugins for JMETER also for testing the GRPC and it works also that I am sure.

Nice. Are there any performance metrics or counters you use that are different than another type of protocol you would be testing against for grpc?

Uh, no. Predominantly we are using the same performance metrics that we use. It's not like it's. It's a grpc, so it's a different performance metrics. So it's mostly the client side and the server side metrics we used to take care and we need to see that it's all the same metrics like in terms of the response time, in terms of the throughput on the client side, on the server side, CPU users, memory users. So it's the same metrics. It is coming out of the box from the performance tool reports. So we are not customizing anything. So the reports gives you the like whether it's a JMETER report or whether you go for the Gatling reports, mostly the same type of performance metrics they give and we use that only for our result.

So when do you get involved with performance testing at your company is it as soon as they create a microservice you're on it or you know what's the process?

Usually not every product we go for the performance testing. Like we worked in lot of domains we work so not for the every product we go for the performance testing some of the product that are like going to the end users and where the number of users are more. So those case like it comes for a product requirement that if the user base is more and architects decide that we have to go for the performance testing then only we will plan for the performance testing for that product coming to the when you get introduced to that cycle. Yeah, it will reduce that the cycle in the very early. Like a very early. Like when we get the documentation of the API and all those things. It's like a more of a combined engineering. So we closely work with the developers, architects and get the understanding about the how the system works, what we need to test. Even we work closely with the DevOps also. So and get our infrastructure getting ready. What type of environment where we are going to do the performance test is it should similar to the cloud. It should be similar to the our production environment. Sometimes the client has on premise deployment. So what type we will do for the performance testing at that point of time. So those are the decisions we make very in the earlier stage of our cycle. Whether we are going for an on premise or whether we are going for a cloud for the performance testing. So this is all like how our the production environment will be. Accordingly we make the similar setup and we go for the performance testing.

So you did mention I think that it is integrated with your CI CD pipeline. How hard was it to integrate with Jenkins?

We integrated with Jenkins. So the basic we pass some of the parameters from the Jenkins URL. What is the URL? Which all service you want to run for the performance test. What all the ramp up you want to introduce during your cycle, during your test. What is the TPS you want to maintain during that test? Sometimes it will be based on the loop. Sometimes you want to run the long test long run test for 8 hour test or 2 days test or 3 days test. So like a duration like how long you want to run that. So these things are customized and parameterized in the Jenkins dropdown and we used to run from there mostly the Docker images for this comes into JMeter, Docker and Gatling Docker comes it will spawn and run that test. And after that test finishes it gives that instance and our reports will Come into one directory and we will just using HTML publisher or something, we publish the results and we'll share that URL to the management for making a final call. Like a call go and no go based on the performance reports.

So do you have performance SLAs that you have to. I know you can't delve into what your application does, but are there certain things like requirements that need to be met from the performance and that's why it's critical?

Yeah, we have performance sls, we have performance sls. We deviate a little bit around that. So some of the SLAs are very critical. There should not be any crash looping. So when, if the service goes down, kubernetes will bring the service up. But you need to know how much time the crash load happened for that microservices. So all those things that needs to be fixed that are like. You can say like a. More often a blocker in terms of throughput and in terms of response time. We just, we may like if a response time is less than 500ms, we are planning for and let's say it went for 600ms. But not, not a crash loop. So on our go, no go, we can decide on that. Okay, that's fine. Like if we can go for that we can optimize and all those things. So those things we have the NSLA for all we have the SLA throughput in terms of response time. We have crash looping, CPU conjunction, memory consumption. So we have, we used to go through all those metrics and then we make the final call after going through.

Is there any other considerations you need to go. You need to test for when you're doing microservices other than grpc, are there any like any dependencies you also need to test also?

Yeah, so dependency. I would not say like more like a workload modeling. That. That is the. That is an important part of any performance test testing. So you need to define like what you are going to test what is your flow, which flow you are going to test whether that flow makes sense in terms of going for the performance testing. It's not like just hitting the endpoints and hitting the service GRPC call and just getting the response and validating around that on the service level. Rather workload modeling is an important part. Like one has to understand how you can define the total workload. Like how many are total user, how many the simultaneous user, how much will be the concurrent user, the person. Like we need to differentiate on all the which flow, the person which Flow the user will go at that point of time. So that needs to be defined before running the test. And that needs to be. It's more like a performance test plan or a performance strategy that we need to share in advance. Like this is the flow that we are going to do in terms of going for this testing. Let's say I am going for attending a class. So in a classroom there might be a section. In a section there might be more 30 students in that section. It cannot be like a section composed of 150 students within a same section. So if our UI or something have a call which takes the data of a section, then there might be a. Our user workload model is not correct. So it may create a problem. Right. So those things we need to define because at the end of the day it matters a lot in terms of the final result if the workload modeling is not proper. So definitely we are not going. The importance of the performance test is definitely. It's not about just hitting with some user and some tag TPS or some quest per second that will not work in that. So that is an important parameter that we define. Other than that the server side like a configurations and all that is also very important. We need to baseline our server requirements. What would be the baseline if this many users are coming, what would be the server requirements? And also our performance environment infrastructure should be up to that mark so ready for running that. So these are the few things that we take care before going for the test.

Okay. Because before we go, is there one piece of actual advice you can give to someone to help them with their performance testing? GRPC or microservice performance testing efforts? What's the best way to find or contact you?

My advice is like it's not about learning the tool and just going for that and making your test, writing that test the most important thing that for a performance tester, what I feel is like it's not about Gatling, it's not about JMeter, it's not about Locust or any of that tool like focusing on that part on learning the tool. It's all more about the concepts like the internals. One has to understand, I would say what is throughput, why concurrency is important, what is response time, what is latency, what is the difference? So the concept is very important. And that second thing, the performance test metrics concept, that is very important. The second important thing one has to understand how that microservices that you are going to test, how it exactly work in real time, which microservices going to call which microservices what is the dependency like it is really a microservices or still it's a little bit of. You don't say like it's 100% microservice or something right which all DB getting impacted what all those things are like sometimes we miss the main concept volume testing whether we pump the data already in the DB first time when you run the performance test it might work well because there was no data everything worked fine once data how much data getting pumped in on a daily basis. So all those things one has to investigate a lot. So that is comes under the part of your modeling like workload modeling and all. So those things are. I think engineers should focus more on that part rather on getting the running the test and getting the report or getting CI CD done and all those things because those part are already proven things. We can find ample materials or ample things to get that things done getting into CICD or anything like it's with a Jenkins or any any CID CCD pipeline. So the focus is the more concept. So that is my mostly people directly jump I found like a tester directly jump on getting the writing the test without knowing like what this test is all about why you are writing that. So that gap has to bridge up that makes the understand the system and you can challenge also like you will know the in depth and and you file some performance issue or something you have a points around that awesome.

Great advice. And Vaikus, what's the best way to.

Contact you best way is through my LinkedIn or through my email. That is the best way to contact.

Thank you Vaikus for your performance testing awesomeness. The links to everything of value we covered in this episode. Head on over to testguild.com P47. And while you're there make sure to click on the Try them Bolt today link under the exclusive sponsor section to learn all about SmartBear's two awesome performance testing solutions Load Ninja and Load UiPro. So that's it for this episode of the Test Guild Performance and Site Reliability Podcast. I'm Joe. My mission is to help you succeed with creating end to end full stack performance testing awesomeness. As always, test everything and keep the good. Cheers.

Thanks for listening to the Test Guild Performance and Site Reliability Podcast. Head on over to testguild.com for full show notes, amazing blog articles and online testing conferences. Don't forget to subscribe to the Guild to continue your testing journey.