Performance Testing for Massive Scale with Anvesh Malhotra

Welcome to the Test Guild Performance and Site Reliability Podcast where we all get together to learn more about performance testing with your host, Joe Calantonio.

Hey it's Joe. And welcome to another episode of the Test Guild Performance and Site Reliability Podcast. Today we'll be talking with Anvesh Malhotra all about performance testing at massive scale utilizing Artillery IO and I'll be upfront with you. This is like a quasi interview because Ambesh actually presented at this year's Automation Guild. At this year's Automation Guild, what I did is I had three days dedicated to functional automation. The last two were dedicated to non functional testing topics like performance testing and security testing. So what I did is I took his session, a little bit of his session and I kind of chopped it up to make it more of an interview style type episode. Cause I think there's a lot of great content here that I also wanted to share with you with the folks that listen to the Test Guild Performance Podcast. If you actually want to get your hands on this session or all the sessions that we did at the shares event, you still can actually purchase a ticket just by heading over to automationguild.com and you get instant access to everything when you buy a ticket today. But this is just a small sample of what you're going to get. Amfest shares a lot of interesting innovation that they're doing there at USA Today and I think you can get a lot of value out of it. So you don't want to miss this episode. Check it out.

This episode is brought to you by SmartBear. Listen, load testing is tough. Investing in the right tools to automate tests, identify bottlenecks and resolve issues quickly could save your organization time and money. SmartBear offers a suite of performance tools like Load Ninja, which is a SAS UI load testing tool, and loaduipro, an API load testing tool to to help teams get full visibility into UI and API performance so you can release and recover faster than ever. Give it a shot. It's free and easy to try. Head on over to smartbear.com solutions performancetesting to learn more.

Hey Amvesh, welcome to the Guild.

Thank you Joe. Thank you so much.

Before we dive into your solution that you created for your performance framework, I just want to give people a little flavor for a little bit about more about your company so they get an idea for just how big of a performance effort is that you're dealing with there. So can you tell us a little.

Bit more about Gannett Gannett USA Today Network is a leading local to national media and marketing solution company. We are the largest local media company in America with 100 plus newsrooms spanning over 46 states. Gannett has over 2000 journalists with 100 plus newsrooms and a combined readership of over 150 million unique monthly users and 1 billion monthly page views.

So just listening to that I can already tell that why? The reason is you need to actually emulate a ton of users to try to really create real world conditions before you go live and to actually compete in the real world. So another question I always like to ask folks is how do you personally define what is performance testing?

Performance testing is a process used for testing the speed, response times, stability of the infrastructure and the application responsiveness over multiple platform of an application under load. Performance really is in the eye of the beholder. A well performing application is one that lets the end user carry out the given task without undue perceived delay. You can either be a firefighter and fix the performance related defects later in your application code or you can take measures so there's no fire in the first place. Consider performance during design Application Scalability Execute performance test to cover issues as early as possible Performance testing is critical to customer satisfaction. If your application's performance doesn't meet the expectations of your customer, they will move on to your competitor.

So another thing I see people struggle with is trying to decide what tools to use for their particular unique need or performance needs. So maybe what are some of your top tools? Or maybe compare some of the top tools you've used there at Gannett and what you end up selecting.

There are a wide variety of products and services which can help you build and run performance tests. I'm showing you only a small subset of these tools available. Firstly, Artillery I.O. is a modern, powerful and easy to use load and functional testing toolkit. Apache JMeter is written in Java by the Apache Software Foundation. It can measure performance for static and dynamic web applications. You can also use paid services such as Blazemeter to execute your performance tests. WebpageTest is another tool that was originally developed by AOL. It uses WebDriver to execute performance tests on multiple locations around the globe using real browsers. It provides free and paid services and the ability to host private instance of the tool. LoadRunner, which is another software testing tool from Micro Focus, can help you execute your performance tests.

Can you actually dive in a little bit more on to what is Artillery IO? You know I've heard of it but I actually don't know much about it. So what is Artillery I O?

It is an open source application with an opt in of premium services. It has detailed performance metrics including latency, requests per second, concurrency and throughput peak performance testing to handle maximum traffic for your backend application. For stability and reliability. It has the ability to write custom logic, post and pre test scenarios using JavaScript, which has a wide variety of NPM modules that you can use. It supports multiple protocols including HTTP, websocket, Socket, IO, Kinesis and hls.

So can you also actually talk a little bit more about infrastructure? What infrastructure is required in order to do or to work with a solution like this?

Running performance tests not only requires good test scenarios and a stable environment to execute the test against, it also requires a stable runner infrastructure where the performance test runs. When performance tests are executed, the runner requires heavy compute and memory resources. These resources are then utilized to mimic the behavior of a real user. Geographical locations from where the input is received to the backend. This is very common when the application under test is behind a GUIP based load balancer, memory and CPU is needed to create virtual users to generate the desired load for the application. In today's modern application there are many tools available to visualize the data in hand. Choose the right tool to visualize your test results. Controlling the bandwidth is needed when the behavior of the application is proportional to client's bandwidth.

Great. And also I think you created a custom solution around Artillery like almost like a framework. So how does that look like? How does that work?

To execute our performance test we have built a tool around Artillery. We call it the Fast Artillery, short for Function as a service. Yes, I agree we didn't find a good name for the tool and unfortunately this isn't an open source application as yet. There is a similar open source application available from Nordstrom called the Serverless Artillery which you can easily find on GitHub and use if you're on AWS. We use Google Cloud products and services to host our services on its platform. First, Artillery has two components, Cloud functions and an engine to run the test. First, let's talk about the serverless side. A wrapper around Artillery is built and deployed in Cloud Functions which is a serverless service available on gcp. This helps in quickly creating ephemeral Artillery that auto scales the infrastructure when the tests are executed, keeping the cost low. I will show you in later slides how we are able to execute performance test with a million requests per minute. The wrapper is deployed in two regions, US Central 1 and US East 4. This is used to generate load from multiple locations to our backend which are usually behind load balancers. At Gannett, we heavily use Docker and Kubernetes for hosting our application servers. When running performance tests, our backend applications are behind the firewall protected by the fastly API management. In order to have the network connectivity between the cloud function and the application servers, we use VPC connector. This allows the cloud functions to access the backend service through the secured gateway. Now let's talk about our engine behind the Fast Artillery. It is built in Golang and shipped as a binary inside a Docker container. This engine distributes the load on the cloud function by breaking down the test and then collecting all the results back from the cloud function, generating a single report. Since the cloud function is protected, it requires a developer or a tester to provide a service account with the invoker role on the cloud function. Finally, the engine then forwards the test results to New Relic for visualization. This serverless architecture helps to execute performance tests at massive scale while keeping the cost very low.

So as I mentioned earlier in the show, this is a recording that invested for a video session and what I did is I took the audio and I chopped it up. So the next section may be hard to visualize, but I think you can get it. So Invesh, can you walk us through a sample test?

Let's take a look at a sample test and then see how we use Fast Artillery to execute our test. In this test, we are creating a performance test for a target endpoint to be foo.com inside the phases block. We have provided the duration and arrival rate as 120 and 20 respectively. We also created a scenario called bar which will append bar at the end of the endpoint. This means that the script will run for 2 minutes with 20 requests per second or 1200 requests per minute. Since the cloud function has limited CPU and memory resources, each cloud function can only handle 20 requests per second. Wait, hold on. Didn't I tell you that we are running the performance tests at massive scale? Aha. We're going to use a Docker engine to convert the 20 requests per second to 1,000 requests per second shortly. But before we dive into the load distribution, let's recap and provide the base parameters to the container. First is the New Relic account ID and New Relic API key which serves as the authentication for sending reports to New Relic. Next we specify Google Application Credentials which is a service account to invoke the cloud function. We also attached a volume to the current directory in order for the container to read the test configurations. Next, we define the runtime parameter to enable new relic reports, the name of the test which is the guild conference and the script to run which is GuildConference YML and finally the number of workers is 500. This tells the engine to distribute the load across serverless architecture and thus result in 1000 requests per second. Although the engine can dynamically allocate the number of workers based on the request, we have provided it as a runtime parameter so the developer has more control on the entire process of their test.

So if you really did want to see this in action, As I mentioned, we do have instant access now available to you to get all the recordings including Avesh's. All you need to do is go to automationguild.com and register to get an all access ticket. Okay Amvest, can you actually give us an example of maybe some of the metrics or charts that you use?

The first chart shows us that 1.2 million users were generated for the entire duration of the test and the second chart shows us that the average latency to be 8.3 milliseconds. We can also see there were 500 cloud functions running at the same time during our performance test. Once the tests are completed the instant quickly drops to zero. Hence the cost for executing our test becomes minimal and it's only pay per use now.

How did this actually help you get ready for the 2020 elections and what did you find or how did it optimize the user experience during the election?

Elections to us is like Black Friday is to Amazon traffic spikes and thus backend application performance comes to test. Usually we receive over 1 million requests per minute. Since the data is constantly changing. Only 25 to 30% of the requests are cached and rest are served by the backend server. To test our applications, we used test generation techniques to quickly generate scenarios for 54 markets spanning over 600 endpoints. The first day we executed our performance test, the pods inside the kubernetes started to crash as soon as it started to receive traffic. This issue was quickly identified and resolved by the developers. We also noticed the application was using legacy services for configuration management. It was later changed to the new configurations from Google Cloud Storage Bucket and helped improve the response times. With the help of performance test we were able to identify issues with fastly caching. This resulted in elevated 503 response code. It was later resolved by working together with SRE Platform Engineering Team and Fastly Support team. Lastly, the nodes that were supporting the application servers were moved to reserved nodes instead of preemptible for high availability and disaster recoveries. Now we were finally ready for the elections. The response times status codes were acceptable to product and development team. As a result, our system processed and published updates across 300 sites in just over 3 seconds. At one point we beat Google by about 10 seconds. We were consistently ahead of our competitors using Associated Press source data. Our system performance was the invisible power center underpinning, quickly updating news coverage and results feeds and seamless consumer facing experience.

So Xanthony you see yourself adding on for the future to your framework. To make it even more awesome, what.

Does a feature roadmap look like? Fuzz Testing Fuzz Testing is a technique that involves sending random data to monitor for application crashes, failing, built in code assertions or potential memory leaks. Artillery provides a plugin which is called Artillery Fuzzer to execute the performance test. It has a wide coverage of random data that it uses to dynamically change the associated variables in the test script. We are currently building a reporting API which is built using Google Cloud Run and Apache Beam Dataflow. It will be used on our production applications to send network error logs, crash reports, intervention deprecation and content security reports. We used Fuzz Testing to ensure no bad data was coming into our system. We used this with Fast Sight, Leary and also identify a hashing issue with the dataflow pipeline. It was later resolved by reshuffling the data across multiple workers. We are also looking towards storing our performance test results into BigQuery. It will help us analyze performance test results from historical data. We will also use Data Studio for visualizing our results which is backed by BigQuery.

Arunavesh Before we go, are there any parting words of wisdom you want to leave the guild? And what's the best way to find or contact you?

I would like to thank my wonderful team at Gannett who work together to build innovative tools and services that shape the next generation of testing. Thank you all for your time. If you have any further questions, you can connect with me at LinkedIn LinkedIn in nveShmalhotra.

Thank you thank you Nvesh for your performance testing awesomeness. For links to Evident value we covered in this episode, head on over to testguild.com P15.

And while you're there, make sure to click on the Try them both Today link under the Exclusive Sponsor section to learn all about SmartBear's awesome performance testing tool solutions like Load Ninja and Load UI Pro. So that's it for this episode of the Test Guild Performance Site Reliability Podcast. I'm Joe. My mission is to help you succeed with creating end to end full stack automation awesomeness including performance testing. As always, test Testerton and keep the good. Cheers.

Thanks for listening to the Test Guild Performance and Site Reliability Podcast. Head on over to testguild.com for full show notes, amazing blog articles and online testing conferences. Don't forget to subscribe to the Guild to continue your testing journey.