Performance Testing Big Data Vishnu Nair and Vijay Gavane

Foreign.

Get ready to discover some of the most actionable performance engineering, testing and site reliability device with some of the world's smartest engineers. Hey, I'm Joe Calantonio, host of the Test Guild and SRE Podcast, and my goal is to help you succeed in creating application performance awesomeness.

Hey, it's Joe and welcome to another episode of the Test Guild Performance and Site Reliability Podcast. And today we'll be talking with Vishnu and Vijay all about performance testing and Big data and all those types of things that you're probably struggling with if you don't know. Vishnu is a QA advocate, currently working as a QA Architect. He has over 8 years of experience in designing, curating and writing test cases across different technologies, including big data, UI and backend services. Also, he has an expertise in performance engineering, resilience testing and security testing. We also have Vijay joining us who is a technology enthusiast GE Global Enterprise Engineer who is currently working as a principal QA engineer as well. He possesses over 20 years of experience in QA tools and frameworks across different functional domains and technology stacks. And he has been a core member of the Test and Core center of Excellence in his past experience where he's not only designed and scaled test automation frameworks, but also has established QA practices and standards. As you can tell, we have two experts here. We're going to learn a lot. You don't want to miss this episode. Check it out.

This episode is brought to you by the awesome folks at SmartBear. Listen, we know load testing is tough but necessary. So investing in the right tools to automate tests, identify bottlenecks and resolve issues fast saves your organization both time and money. And that's why SmartBear created Load Ninja, a SaaS load testing tool to help teams get full visibility into performance so you can release quality software faster than ever, make testing effortless and give it a shot. It's free and easy to try and head on over to loadninja.com and learn more. Hey guys. Welcome to the Guild.

Hey guys.

Thanks Joe.

Great to have you. Before I get into it, is there anything I missed in either of your BIOS that you want the Guild to know more about?

Yeah, I can go first. So I have been in industry for more than eight years now working as mostly on the architect side, designing the frameworks for multiple technocross technologies as well as different testing fields. Actually mostly I have been working with the UI and API. I have been working with Big Data for like for past five years now and doing performance engineering mostly on the big Data side as well.

Great, great Vijay.

Yeah. So apart from what you have covered already, Joe, I have been into a test center of excellence, mostly focusing on the test automation bit. But yes, been with my current organization for last five years. We there was a need for the performance engineering for one of our in house solution and that is where I started looking into it along with Vishnu. So.

Nice. So what I like about both of you, you both have backgrounds in functional automation sounds like in qa but you're, you've expanded almost it sounds like to performance engineering which I think a lot of testers haven't made that shift yet or added that as to one of their core competencies. So how did you make that shift? Because it seems like a lot of people don't always get into performance testing as well.

Yeah, okay, maybe I'll go first. So typically when we talk about performance testing or even non functional testing, we only focus on the load testing or maybe max to max stress testing.

Right.

But the current problem solution that we had was, you know, we had one of the third party product that we are using for conducting the surveillance on communications data which is nothing but your chats and email messages. Right. So we had various different, 16 odd different sources which, wherein you know, we get the data, communication data and it was almost like a 1 1.5 millions of data every day that we used to get. And we need to conduct the surveillance on that data to ensure that there is no, you know, important information is being leaked through that communication. So the product was good. However, we didn't know the breakpoint of that product. Right. So there were unknown failures into the production and we used to jump into the root cause analysis and then fixing the issues. Now that that product is almost sunset and we have got into the new one. We wanted to ensure that we don't store our data into that product so that we don't have dependency on third party product. And for that we have created our in house build solution to store the data, index it and let it become very easy for conducting the searches. Right. So that is where we started looking into the performance aspect of that solution. So we wanted to ensure that it is more reliable for the users not only in terms of the response time but but also we should know the breakpoints of that solution. We should also know how it performs under the heavy load and how does it perform under the continuous load of normal concurrent users across multiple regions. So that is where we started discussing and brainstorming around what different aspects of non functional testing or performance testing. We can think about for this solution and then we started working on it.

Yeah, great. So.

So just curious to know then a lot of times people say I don't have requirements for my performance engineering practice. I don't know what metrics to look at. So it sounds like you're consuming a lot of data, you're storing a lot of data and being able to access that data needs to be quicker. So how do you know what to measure when you're trying to design a test to ensure that you are getting good response times but also you're able to, there is no leakage like you said and the data there is being submitted correctly and it's. You're not missing any pieces to start with.

We had some SLMM actually already going in the organization. We have response time already. Describe how much response time should the UI take to get the response back to the user. So that was the base point as per the Vijay mentioned. We already had some already tool, a third party tool which was working for us and it was quite slow. So that was a base point for us actually. So we know that it is slow, how much slow it is. We ran few test scenarios over it and we get the data how slow it is and then based on that data we try to reform our solutions actually. So we have how much slow already existing solution is and what we have to achieve actually. So based on that we tuned our performance testing tool and the infra as well to get the required data actually and the response time.

So in addition to the SLA requirements at org level we had been you know, closely working with the architect who has been, you know working on this in house developed solution and then beyond that requirement we tried to incorporate what else need to be tested and then included into the test strategy. So as I said rightly, I mean normally we don't get the NFRS non functional requirements for any given application that we need to struggle for that. So we did that like a productively along with.

So it was like a brainstorming along with the architect team, the stakeholders and the QAs. Everyone you know sitting in one room and discussing what should be the outcome and how fast our system should work.

Actually I guess what sounds even more complicated to me though is I think you mentioned in the pre show something about infrastructure is key to software engineering and so you need to then design hardware will meet certain conditions. So that's even more complexity. So are you mixing and matching different hardwares as well to say this is certified as being able to accommodate X amount of Data is that. Are you going to that level as well?

Yes. We had like almost 20 billions of record actually for past 7 to 8 years of data we already have. And daily we have 1.5 millions of data flowing through our system actually. So we know how much data we have actually and how much data flow will flow on the daily basis. So maximum 1.5 records across 20 channels that we have for the communications actually, which is at the. Org level has been designed so that all pointers we already have accumulated, we based on that, we did our performance testing and we tuned our infrastructure on that results actually. So there were some challenges in the initial days when we were starting how to do it, how to customize. Because we are talking about an infrastructure testing, actually the performance of an infrastructure testing. So that was. We don't have any explicit tool for that. We have to do a customization of an existing open source. We have to write the custom codes over the existing performance testing tool that is JMeter. We wrote our own codes based on that and then we tuned it actually.

So using JMeter but you did some customization to accommodate you. How important is test data for these type of tests? Like it sounds like you're moving so much data. Like how do you. For your performance test, how do you know you're using correct data then to make sure that not only are you putting the right load, but the data set they're using is. Is accurate to what your customers would be using or you would be getting.

Yeah, so that's actually the heart of the JMeter. You have to pass the data. So it is more close to user experience actually. So it is like the tool that we are using is a solar actually. So it's a search index actually. So we are searching some data in the index. It's like more similar to that. We have a Google search. So you search something, you get the result. So for that what we did is that we had an existing tool which we did for us and we analyzed the patterns of the users, how much use, what all parameters they are using. On the more frequent level. We get all that. We accumulate that, you know, on the files and then we wrote our custom generator actually. So a random generator was written on top of it, which mimics the user experience actually. So let's say if I am searching for. You can take an example of an Outlook where you are searching for a message. You can search it from. From address or to address and subjects and the text itself, the body, you can search in similar fashion. We have accumulated, we have wrote the queries in similar fashion on a random generator which creates the data for us actually. So it creates the test data for us at the runtime and this run data is feeded into the generator to execute for the solar Actually those are.

Like real time scenarios in terms of the search queries consist considering all the possible parameters that we that user would use for searches and obviously in terms of volumes we kind of matched our test data volume to the scenarios.

So not only the test data has to be synthesized, the outcome of that result has to be also monitored actually. So the query generator that we are it should not give you a blank record actually. So the test case will fail. So the test queries has to be in such a way synthesized so that you at least get one record on as an output to measure the response. So you can get one record, you can get thousands of records. So in this fashion the queries has been designed actually.

So you actually validating the record or is it just at least you have a record like how are you doing the validation part?

So we have wrote a validation part as well. So we check the indexed data actually what we are retrieving and then based on that the response time has been measured.

Is anything unique to Apache that you need to be that people need to be aware of when they're testing it is like Apache Solar search engine. Is that an Apache product? And if so were there any anything you learned about it after going oh wow, you know, we need to tune this with this to actually have it be optimal for what we're trying to do?

Yeah. So Apache is Solar is actually a Lucent based search engine actually which for it's more likely a solution for an enterprise searching actually so you can ingest the documents in indexed format. So what it does is that the data that we are trying to search is through the index based actually. So you can, let's say you can take a message ID which can be indexed and based on that the index data, the search are written. So the main benefit of using Solar is that the nodes that it's a cluster actually. So the nodes that you are using, the nodes itself are search engines for the solar actually. So it returns the fastest way of returning the data.

So I'm sure there's other components involved. So you have the solar engine. Do you have like any APIs, UI components or other databases that you're using on top of that to work with Solar?

Definitely. So we have APIs uis, we have Solars, we have Spark Jobs as well for data ingestion part. So this solar is like a hard code database which is storing a 20 billions of record actually. UI is there on top of it actually and it can search the data for us. And the daily ingestion part is like the daily email chats that are going can be ingested actually.

Now what kind of data are we talking about? Do you do like a batch job where people are sending you data that you then process in your software? Like what is the data where talking about?

So this data that we were talking about is all the communications data that I mean the communication channel when you talk about it. It's all like chats and emails from various different sources. Be it like Bloomberg or any other, you know, different sources. You know the communication happens between within the so called as monitor users within the bank or you know, outside the bank.

So. So you can understand it's like a surveillance actually gatekeeper actually that records all the transactions of the mails and what is ingoing. So if we give a small background like so to monitor all what all the. Because it's a bank, right? So it has its security compliances already. So we have to monitor certain data, the ingoing and outgoing chats across the organization as well as the outside. So in that case we have to monitor the entire data in terms of format.

It could be a plain text file, it could be a xml, it could be a CSV, it could be a JSON file. So there is different file formats.

So if you take an example, you can take an example of a Gmail, how we can transfer the chats, we can transfer emails, attachments, we have, we can have attachments of different formats as well. We have PDF, xls, CSV, even the plain text. So all this record has to be monitored.

When you're performance tested, does it matter if you're using PDFs or text or like the different formats? Is it like how do you test that piece as well to make sure oh we can handle PDFs or does it not matter? It's just data is data once it's in the system.

It's not like that actually it definitely matter if you are reading one more than. Because we can get attachments in GB size as well. So it will make the performance slow. So the queries which we were talking, the synthesized query that we designed actually it takes an average of all these types of data actually. So it searches the data through different attachments as well. So even it searches data through your XML files which can have you know, 1 lakh or 2 lakhs of records, millions of records. And it can return back you the Data. Even the PDF files, you can have multiple PDF files, you can have 200 pages of a PDF file in 800 MB sizes also. So it definitely searches through that records as well. So yes, definitely that matters for us as well. That's why we had the infrastructure testing actually designed for this particular because we have to get the data as soon as possible to the UI actually. So our API has to be faster, our search engine has to be faster actually.

So the test data has taken care of all of these variations to emit what users would interact with, including multiple number of attachment, multiple types of attachments and multiple sizes of attachments.

So we don't have a specified test case for particular scenarios searching for PDFs or like that. We had a one single scenarios which can mimic the user experience actually. So it can. You can search through PDFs or XLS or any fight.

So what tends to cause performance issues? Is it the query that your application, your developers wrote to sift through the data? Is it the hardware behind it? Like what are you looking at?

So we are actually looking to everything actually. It's like it's not just a. We are doing a performance testing, we are actually certifying the infrastructure for particular, you know, scenario, all the scenarios that we have. So we have scenarios for, you know, peak, the peak load analysis, we have stress testing for soak testing, we have breakpoint analysis for our infra as well. So based on all this data, we are actually certifying the infrastructure will work. So initial days we all had these challenges actually. The codes were breaking, the nodes were breaking down, the zookeeper was going down actually. So every infrastructure that we can believe, we have gone through all this and after, you know, by each phase we tried to, you know, fix the issue and then we were able to reach this current point.

Actually it does, it has helped us to fine tune the infrastructure requirements in the test environment. We knew for this set of infrastructure how the system is performing so that we could guess in the production environment. If we improve on the infrastructure arrangements, then how the, you know, testing would perform and that would have.

That.

That has helped us to fine tune the infrastructure as well.

So first we started with the lower environments and then we moved to the higher and to the production environments.

How often do you run these performance tests? Is it continuously? Because they're constant updates to the system in production, like, oh, we need to test this, verify it on all the systems before we say okay.

For infrastructure testing, it is a once actually test which is like before going, you know, to live, we have to certify the infrastructure. So that was an ongoing process. That is a one time investment actually. But the APIs that we are using, they are monthly based on the release basis we are running it.

So I guess the next question is. So you mentioned JMeter. I know a lot of times people, the problem with JMeter, you could put load on the system but it's not like a enterprise solution where you can then have all reports and be able to monitor all the things in one place. So did you have to, what are you doing to actually monitor things within JP and JMeter? Then do you have a separate solution to monitor? Do you care about CPU usage and disk space and all that? Or is that something old school?

So we have Prometheus and Grafana for that for us actually. So they are the observatory tools for us for monitoring our ecosystem. So that has also been fine tuned with the JMeter. So when we are triggering our executions we can monitor the data flows and how much heap size, heap memory is being used, how many, how much CPU is being used used. So we, when we are talking about a tremendous ecosystem here actually. So we have around 1670 nodes actually across and we are talking about 500 plus of shreds and we are using Grafana to monitor them individually actually.

So how do you get it so you know you're doing it. Like when I used to do performance testing, we used to test in the lab and say oh here's what it's going to perform. Like when the production was never the same and this is on tiny scale like this. Back in the day we had a web server, right? A database and a web browser. That's it. But you have all these pieces. So how do you know the performance testing you're doing? Maybe in staging is going to match production.

So that is like a trial and miss actually that we did from the starting actually. So we build the solution for smaller ecosystem, then we move to a higher ecosystem and then we move to the product. So once we have all the cso it's not like a performance engineer is behind all, it's like a entire team has been behind all this and we have architectures to provide the solutions for us. So we have a lot of brainstorming sessions. Like sometimes the JMETER fails, sometimes the ecosystem fails. So in this way trial and miss errors, we achieve the performance tuning actually.

So you mentioned three different types of testing. I just want to go into a little more detail. Maybe you give an example for each. So people get an idea. The first one I think you mentioned was peak load time performance stability. So give an example of what that means and like how, like how you do that type of testing in your system.

Yeah, definitely. So let's assume a smaller ecosystem where we are doing some, you know, a login scenario. If you take and we analyze that particular peak time, the maximum load can come for maximum users can log in actually. So you can take, you know, noon time, it would be the maximum. So when we are considering the noon actually we are considering the region wise ecosystem as well. So you can have Asia specific actually then you can have Euro region. So mostly they, you know, have a overlapping time period. So that would be the maximum period where you have maximum users on one system actually. So the peak time analysis is like that you have a maximum users as one particular day and you can assume for a time period of, you know, hour or two. And then we try to mimicate that users on that particular system actually.

So number of concurrent users plus the volume of queries that they would fire within that period of maybe one, two hours.

So actually we got this backend all the analysis data from our stakeholders and all we have analyzed how many users we have on particular peak time and then based on that we analyze and we synthesize the load up for ourselves.

Actually how about for our soak load analysis?

Soak load is like an higher version of the peak time. So let's assuming that this load that we have for the peak time can have a stress for longer duration. So when we do a peak time answer, it's like a spike for the system actually at that spike how the system is going to behave. So maybe it can be like a heap memory can go, you know, to a certain level and then automatically as soon as the garbage collection will trigger, it will come down actually. But when the soak testing is ongoing it is like a, you have a, you are providing a tremendous stress on the system for a longer period of time. So the, you know, the heap size regaining is much more, you know, has to be quick actually. So that will tell the system how it is working in a, you know, a stressed environment. So you can get a spike on the peak analysis which suffice the requirement. But it does not provide us the detail how the system will behave when it's tremendous stress for a longer period of duration will have. So what we did is that we synthesized the same peak time load, which we know is a tremendous load already actually and then we ran it for a Longer duration, let's say for 72 or 48 hours. And then we see so the behavior of our system actually how it is behaving, whether the heap is going above our and if the nodes are coming down. So what happens is that this, the today's ecosystem are much more resilient actually. So once the nodes go down, it can automatically come up. But how many of these cycles are continuing? Is this going a data issue? Are we facing any infrastructure issue? So on the stress level we can analyze that which we won't get on the peak level, actually.

Cool. And how about, I think you mentioned breakpoint performance analysis as well.

So as I mentioned initially, right? So we had earlier product wherein there were uncertain breakages into the live environment in the production environment, right? And then we were curious in terms of what went wrong really. So we didn't know the breakpoint of the system. That was the main issue, main problem statement with the old system and that we wanted to avoid for this new system infrastructure. Right? So that is where we, we're trying to, I mean like we have done the load testing, right? We have done the SOAP testing, but we wanted to understand what is the breakpoint. So we kept on increasing the load, we kept on running the test for longer duration just to understand at what point the system breaks. If we, If I add 100 concurrent users instead of 50 normal users, then the system breaks or maybe if I increase the volume of search queries, volume of data, does the system break, right? So I should know in advance when my system is going to break. So that was the intention.

So if we give an example, we should know if the volume increasing on the volume does the system breaks or increasing the user. The system is going to train. So we have to do a, you know, testing on that perspective as well. So what will happen if we increase the volumes of the data? What will happen if we only increase the users that are trying to access the data and what will happen if we increase on both the levels? So the entire infrastructure testing is like we are trying to understand the system itself actually how it is going to behave on certain conditions. We should also know that how when the system is going to break actually. So that is also the part to certify an infrastructure that it is going to behave for this particular load as an ideal condition, the stress out condition as well as it is going to break at that particular point actually.

So this break point rest helped us to ensure that we fix the system in terms of handling those errors so that the system will break gracefully. Okay. Not suddenly for the users, that is definitely a corner case. It is not going to happen very often, but that is how it has helped us to fix the system in terms of error handling and then other test results has helped us to fine tune the infrastructure that we are reliable.

In the so all these type of.

Tests always done on a complete system or as we mentioned, you have APIs, UI components, they have it tested in isolation, like let's just test our APIs, let's just test the UI components.

So we are actually using both the strategies actually they are tested in isolations also they are in go live also they are actually tested actually. So in isolation we know how so we have test results for infrastructure, how infra will perform, how API is going to perform and how UI is going to perform. And above all that we have a common scenarios for UI where all these E3 ecosystems are in place and working in sync and then we analyze the load scenarios.

So you mentioned a lot about JMeter and you did a lot of customization for people that are listening like oh, JMeter, I can't get it to use it for my test case. Can you talk a little bit more about that customization, what it entails, how hard it was and maybe give people maybe some steps that they could use maybe to customize it for their needs.

Yeah, so JMeter we generally use as an UI or API testing scenario. It's very rarely used as infrastructure testing. So but at the back end it's all jars that are working so we can write our own custom codes over top of the existing samplers. We have lot of samplers actually, so we can write our own custom samplers and we can test any ecosystem actually. So that's exactly what we did in our case as well. We were not able to find any tool in the market which is going to test our infra actually. So we come back, going back and we wrote our own customized samplers and then we saw how the JMETER is going to behave. So we had initial challenges. The heap memory is the best of one actually the JMeter is famous for. So when we are testing an infrastructure of this huge size, two things are definitely going to break. Either the infrastructure is going to break or the testing tool itself is going to break. So we find the solutions. Like we can have a master slave configuration as well in JMeter where you can have slave machines. Because when we are talking about big data ecosystem, it's tremendous in size and when we are increasing the load, definitely the testing environment is also going to fail also. So we have to tune in that manner. So since we know the JMeter behavior of JMeter, we tuned it accordingly. We used master Slave configurations to have different JMETers running in parallel for testing a load of that level as well. So for Breakpoint also we are using the JMETER itself, so it is like a huge amount of data testing against a huge data itself.

And while we are talking about the solution, we did talk about the random query generator, right? So in the application, application basically caches the results right? So that you can give response quickly for the similar set of user queries. So this random query generator was designed in such a way that it generates all the queries randomly and every query is going to be unique so we don't get the false positive results right. So if you are testing using the same queries then we would get the fast response times right. That is what we wanted to avoid and that is why we have created this random query generator which will eliminate that part of, you know, cool.

Any of these open source or would you be just so specific to your needs that no one would be able to reuse it for their own?

We can use. We have not modernized it yet, but definitely it has been designed in such a fashion that if we see a geometry UI it is most more like common, only the parameters you can pass. But it is very specific to that kind of infra actually. So if the similar infra is being used, yeah, definitely.

Okay guys, before we go, is there one piece of actual advice you can give to someone to help them with their big data performance testing efforts? And what's the best way to find or contact either one of you?

I would like to say is don't just look at the non functional departments, right? You need to look at the application holistically. Find out what different tests are required from non functional testing point of view. Not just the load testing or the stress testing, but you need to look at application as a whole. You need to use your experience from the previous products applications or the tooling solutions and then you apply that knowledge while you're building and putting together the strategy for testing for that application.

Yeah.

Can be reached at the LinkedIn thanks.

Again for your performance testing awesomeness. Links to everything of value we covered in this episode. Head on over to testguild.com P110 and while you're there make sure to click on the Try them both Today link under the Exclusive Sponsor section to learn all about SmartBear's two awesome performance test tool solutions, Load Ninja and Load UI Pro and If the show has helped you in any way, why not rate and review it in itunes? Reviews really do matter in the rankings of the show and I read each and every one of them. So that's it for this episode of the Test Guild Performance and Site Reliability Podcast. I'm Joe. My mission is to help you succeed with creating end to end full stack performance testing awesomeness. As always, test everything and keep the good. Cheers. Hey, thanks again for listening. If you're not already part of our awesome community of 27,000 of the smartest testers, DevOps and automation professionals in the world, we'd love to have you join the fam@testguild.com and if you're in the DevOps automation software testing space or you're a test tool provider and want to offer real world value that can improve the skills or solve a problem for the Guild community, I'd love to hear from you. Head ON over to testguild.info and let's make it happen.