How to Avoid Newbie Performance Testing Mistakes with Rebecca Clinard

Five, four, three, two, one. Drop that. Welcome to the Test Guild Performance Podcast where we get together to learn more about performance testing and site reliability with your host, Joe Colantonio.

Here it is.

Here it is.

Here it is.

Hey, it's Joe. And welcome to another episode of the Test Guild Performance Podcast. Today we're going way back in time into our performance testing time machine, all the way to 2016 for an interview I did with Rebecca Kleinerd all about how to avoid rookie mistakes in performance testing. If you're thinking about getting into performance testing or wanna know how to help your teams with performance related tasks, you don't wanna miss this episode. Rebecca is one of my go to performance testing resources. She's been performance testing for a while. She's currently a solution consultant at New Relic. So listen up. Let's face it, performance testing is tough. Traditional load testing tools create scripts that are bogged down in data. It's hard to read and require a lot of work. Even for simple playback. Load Ninja cuts out the dirty work by using AI to inspect and debug your code almost instantly. No coding necessary. No changing of hands or relays between teams either. Give it a shot. It's free and easy to try. Head on over to testguild.com smartbear and click on the link to learn more.

Hey Rebecca, welcome to the guild.

Thank you. Thank you for having me.

Before we get into it though, could you just tell us a little bit more about yourself?

Sure. So I have been in the performance engineering industry for 16 years. I used to be a Unix systems administrator and I moved into this career when a startup company basically called me up and wanted to me to do performance engineering for their application. And since then it's been a great career. It's a very lucrative career. Challenging and there's a really a lot of demand for performance engineers and a pretty low supply. So doors are always open.

Awesome. Now that's a great point you just brought up. For some reason it's really hard to find engineers with the skills you need for performance testing. I'm just curious to know. I got into performance testing by just happening to be doing test automation and just happen to see a loadrunner tool and just learning it. So how. How did you get into performance testing or why did you choose performance testing as a profession?

Yeah, so again when a startup company called me up and wanted some performance testing done for their product.

It was brand new to me. I was a Unix system administrator so I definitely understood the resource consumptions on the infrastructure. I understood how Web servers works, application servers, database servers. And so getting up to speed on performance engineering, where you're automating a realistic workload, was definitely new to me. But once I made that jump and I started to create performance test harness, I realized the value. In other words, people can load test a website and understand the scalability limitations without having to find out those limitations in production. And you're right, there aren't enough people that actually do this in the field. And I think the reason being is there isn't really any formal education out there on performance engineering. It's all kind of trial by error, which is I'm helping the industry in that way as well.

What would you say to someone that's in QA and they want to expand their role a little bit more and they want to get into performance testing? I know this is, I've heard from some people that sometimes a QA person may not make a really good performance tester. It's just a different set of skills. So any tips you can give to someone that wants to make that transition from say a QA automation engineer into performance?

Yes, definitely. So as a QA person, the first thing you're going to need is curiosity. So you want to understand really how what's going on behind the scenes. Typical front end functional QA testers are looking at websites, mobile applications and they're clicking and they're testing for bugs or testing for user experiences. Well, performance engineering takes it deeper. So the first thing to do is get really curious about those transactions. In other words, what are they hitting in the back end? Is it a database call? Is it logic being executed on the application server? What parts of the transaction responses are static which are denied dynamic. And to do that, the very first thing is to hook up a sniffer, like a network sniffer, like a Fiddler, or using Firefox, Firebug or Chrome network Inspector, it really doesn't matter. But you don't look at the visual of the transaction anymore. You look at the posts, the gets the API calls, the traffic and that's where you're really going to understand the transactions. And a lot of the tools out there, they come with recorders where you can record the transaction and play it back. That is not realistic and it's not all the information that you need. So all the people that start, they seem to just record transactions, play it back and then sometimes they get errors. But it's deeper than that, it's understanding the end to end transaction flow behind the scenes of that transaction.

Once again, I Think it's another great point. A lot of times people get a tool where they can put a load on the system, so they just throw like, I don't know, a thousand users with no think time in it whatsoever. And they cost, they bring the system to its knees and then people stop panicking. So how does someone learn how to really create a scenario where it's realistic to a performance test? Would they look at transactions over time? Anything you recommend in that regards.

So once you have the transactions, a typical user workflow, and you play back, it's playing back correctly. The next step is to make that scenario realistic. And what I mean by realistic, you need to get those scripts to act in a behavior of real human beings. So like you just said, a lot of people will set up a test and bring an application down to its knees with a very low number of users because they're not taking the next step and saying, okay, am I, I'm looking at a transaction and I'm about to like read an article or add something to a cart. So you need the beats, you need the pauses and the think times. And like you just said, actually to create a realistic throughput, if that application is already deployed into production, you can use the web logs to say, okay, how many items are added to a cart per minute? And you could reproduce that transaction flow in your load test case scenario. So having the right throughput is what leads to accurate results and representing so you can understand the scalability. And.

Really when you need to make configuration changes or add more to the infrastructure to support the load, great.

So I guess along the same lines, you're doing a performance test. A lot of times what I see people struggle with is to know what to monitor, or they monitor so many things that they're like, I don't know what any of this means. So it seems almost like you need a lot of different people with different skills looking at this output. Are there any key performance indicators that you always look at when you're doing performance tests?

Right. So on the front end, every tool is going to have the key KPIs built in. The key KPIs are the throughput, the TPS and the response time and the user load. So with your tool, you're going to gather key KPIs to say, okay, at this throughput. Throughput started to plateau, response time started to creep up, and what user load was that? And you correlate all three KPIs to understand the scalability of the application. Now when you're monitoring the Back end, you want to hook up every single server in the deployment. So maybe you have a cluster of servers for application servers and you have web servers which are load balanced. And let's say you only have one database server. So the key KPIs on every single VM or machine are of course CPU usage, memory usage, IO disk usage, and then you want to hook up the key KPIs per technology, the servers, the software servers that are running on those machines. So for a web server a good KPI is active request. A really good KPI is anything that's queued, queued requests. I always like to look at queued a queued KPI because that tells you you've reached a throughput where it could not handle it anymore and requests are waiting in line. Application servers, if it's a Java app server, you always want to look at the memory of the Java heap. In other words, is it all being used? Is it constantly garbage collection collecting? If you're using any kind of messaging systems, you want to look at queued messages, active messages in, messages out, and for database servers, longest running queries, again, any queued queries. So you're basically, you're looking for free resources and understanding if anything's becoming a bottleneck in the infrastructure. And the key is when you're doing a load test and you ramp up the number of users, you want to understand what becomes saturated first. Because if you just, if you're, if you're, if you identify a saturation point where something gets queued in a server, but you're not looking at the entire infrastructure end to end, that could be a symptom and not a root cause. So having timetables and graphs to overlay onto your results of your performance test makes it very much easier to analyze and identify the root causes.

That's. How do you know what is a symptom and what's not a symptom? It's just making sure you have all the information that you have. A lot of times, once again, a lot of people jump to conclusions based on a limited amount of information. So do you recommend people run a test multiple times and then get the average of that to find out what's going on or what's your approach to that?

Right. So a methodical approach to if you ran a test and you see in your infrastructure there are multiple saturations, response time is, is off, you know, off the scale, it's unacceptable and throughput is plateaued. Honestly, what I do at that point is I take it back, I Say, okay, if things went bad at 1200 users or even 12,000 users, I'll run another test. But as you ramp up to that number of users, you go a lot more slowly and this helps you to identify the root cause. So if you say you ramp up 10 users every 20 seconds, you and you methodically ramp up, you're going to, you ramp up more slowly. You're going to see, okay, what became saturated first in the timeline, in the graphs. If you ramp up too quickly. No, ramping up quickly definitely gives you information because it tells you when things fell apart. But if you are going to diagnose a bottleneck, you need to ramp up a lot more slowly to understand what happened first. It's always what happened first is the root cause.

So I forgot where I saw this, but I saw a video of you talking about how to analyze performance test results. Can you share with us a little insight on how best practices for analyzing performance test results?

So first, what it does is defines performance engineering versus performance reporting. And when I made that definition, because when you're in professional services or you work for a company and you perform a test and you isolate, okay, this system or this Application can support 23,000 conclusions current users and that's called reporting. Performance engineering is, well, why, why does it only or not? What happens? What do you need to change in the environment to support a higher number of users? So what I do in the course is I go into performance engineering and how to set up a performance test to answer those questions. A lot of gotchas, you know, stuff that stuff you could get tripped up on methodologies where.

You reproduce the results, how to analyze tons of data to understand scalability and what to look for. I call it clear the clutter. Because you get overwhelmed when you run performance tests. There's so much data to look at afterwards. So I say, okay, this is really what you, the only things you really need to look at. Also in the course, I show you how to create targeted workloads to answer your questions. So if you have a specific question on, you know this response time and why this creeps up, you can, you can design tests that clearly answer your questions. And another thing about analyzing performance test results is there is certain ways you can analyze it where it makes it a lot easier. You kind of, you clear the clutter and I say you reduce the noise. You work smarter, not harder, but you identify the key KPIs that are going to lead you to answering the questions that you need answered.

Awesome. I always found that the actually Analyzing of all the data and reports for performance testing is one of the hardest parts of performance testing.

Yeah, I totally agree. Analyzing the results is where the skill set is totally needed in this industry. You know, setting up tests, executing them, but analyzing. I manage a team of performance engineers right now for another company and it's the analysis skills that give the most value.

Absolutely. And it's so critical because your business might be making decisions on this. I think you mentioned something almost like about capacity planning, where what happens if we have to. It becomes the holiday and we need to cover, you know, 20% more traffic. They're actually making money based decisions based on this information. So to be able to actually look at all this data and give them valuable analysis of what's going on so they can make a valid decision on what to do with it, I think is critical.

I totally agree. Actionable results is what you want to get from a performance testing engagement.

So do you have any tips on how to make. You've analyzed the results, now you have a report that you're going to present to an audience that is not very techy, that actually are going to start making business decisions based on this. Any tips on how to present data so that it's more consumable or more user friendly?

Yeah, definitely. So you want to back up your results and you're going to have the graphs in there from the tools and all of that, but you definitely want to have an executive summary that everybody can understand. So you want to say, for example, these under the current configuration and hardware clearly state what the scalability is of, you know, the current configuration of your application, where the ceiling is. Perhaps you're expecting a headroom of 25% more traffic over the next year. Are you going to be able to handle it, answer those questions and then have the data to back it up with? Also, if the, if you're also monitoring the environment, the actionable information that you're delivering is what is limiting the current scalability. Let's say an application. The goal was to reach 10,000 concurrent users with a headroom, let's say up to 15,000 concurrent users for the future. And it blew it away. Everything was absolutely fine. However, you still want to be able to say, okay, heads up on this tier of your deployment, because from our testing and from the trending, it looks like this would become saturated. First, as you increase the load beyond this initial target workload and you just, in other words, write it in English. Don't just have the graphs, give the information.

Is there anything you see over and over again in your experience that most people do in performance testing are doing wrong.

I think it's interesting in the last couple years what's been mostly wrong? Well, I shouldn't say mostly wrong, but a lot of times is the clients don't understand. They know they need their performance or their web application or their mobile application performance tested, but they don't really know what they need tested. A lot of our responsibilities kind of educating them on performance testing, saying okay, we can performance test transactions which can be automatable, you know, and say something that can be tested over and over again. For example, if they have application that's very heavy on data, relying on data and it's very sensitive to data and you can only execute that transaction once using a piece of data, you can see where a lot of the time will be consumed with the, with the generation of the data and the managing of the data. So educating prospects and clients to say okay, following a typical user transaction flow, what are the transactions that we're going to automate here? Another thing is they might come to us and say okay, we need our website to support 100,000 current users. Okay, but we also need to understand throughputs, in other words, in a minute or in an hour how many thesis or whatever the application is are going to be uploaded. So it's not the requirements aren't just based on concurrent users that they expect in production, but also the throughput rates of their business transactions. So lately a lot of the things that have gone that needed more more defining and stuff are educating the client now performance engineers, I would say. I wrote this blog called the top 30 rookie mistakes for Performance Engineers and it got a lot of traffic and it's basically listing, you know, everything that rookies do wrong or not even rookies sometimes we all still make mistakes obviously.

Cool. So I know we can't go through all 30 of them, but can you give us a few of the performance testing rookie mistakes that you see most folks making?

So when you're performance testing and you're automating a transactions, a rookie mistake is that you would a status code comes back, an HDB status code of 200 comes back and you say okay, well that means it was successful. But let's just take for example that was a login and the login had the wrong password. So basically it redirected you back to the login page, that login page was delivered and it was a HTTP HTTP status code of 200. However, that that transaction was not successful. So definitely place validations or assertions, which are basically string verifies in the response to make sure those transactions are indeed successful. And that happens. You know, there was a performance test I heard about that was against a huge E commerce company and everybody was like, you know, high fiving. It reached to 40,000 current users. Excellent test. And none of the users actually logged in because they forgot to make sure that users were successfully logged in before they proceeded with the rest of the scripts, the rest of the transactions. The next one is actually you had already mentioned too, is pause and think times. People forget to add them or they use like 2 milliseconds and they think it's 2 seconds. And because every tool is set up differently, you really got to look at the tool and say, okay, is it going to take five seconds to make a decision on this page? Is it going to take a minute to read this article? So unrealistic loads because you forgot to add think and pause times. That's a rookie mistake that a lot of people use. Some other rookie mistakes is crying wolf when you spot a bottleneck. And when I say that is, I mean that you isolate a symptom and not the root, root cause. And this goes back to more methodical performance testing where you create a test that ramps up more slowly so you can see the trends, you can see the throughputs, you can see all the servers becoming more and more and more busy. And what server becomes saturated first? And rather than saying, oh look at, this is at 100% used or this is out of memory, no, that could be just a symptom. Another one I see time and time again. And the customers actually, customers actually drive this one. So this, this is where again we have to educate the customers. But don't make false assumptions while the test is running. While the test is running, the tool is doing its job. It's collecting the KPIs and the data.

But when it's presenting it to you, it can look kind of skewed. So in order to identify trends and really understand what's going on, let the test run to complete completion or you know, when you get a very high error rate, stop it and then go back and look at your and do your analysis. And that's where my course talks about too, is doing analysis the right way and working smarter, not harder and understanding, okay, what really happened here. If you try to analyze a test and make conclusions while it's running, more often than not you're really wasting your valuable time. Another rookie mistake would be when you create a script, you Want a user to log in and then they'll do certain actions and then they'll log out. Now in a real case scenario, a user will log in, let's say an easy one, like they're adding to a shopping cart. They're going to add a bunch of stuff to the shopping cart and they're going to check out and they're going to log out. Now a lot of times what people do is they'll create a script that logs in, add something to the cart, logs out, so they don't really reiterate on the body of that script. And therefore you have an unrealistic login and logout rate and not as much throughput on the business transaction side. So when you're creating these performance test scripts that mimic real users, make sure you put a loop in more commercial tools like, I like neoload for this and for a lot of other reasons too. But basically they'll label it init, which is login and end is log out. And then they will repeat on the action, on the body of it, without you even having to drag in a loop. If you're running an open source tool, then you're going to drag in a loop and make sure you do that yourself to create a realistic throughput and workload on that application.

Awesome. One of them that really struck home to me is when you said not crying wolf. A lot of times I'd be so extreme where I would say I'm not sharing any results until I get three runs and I'm able to analyze them and before I send it because I have some reason. Overeager managers always looking over the numbers and I'm like, hold up. Let's make sure what we're seeing is really happening before jumping the gun.

So true. I mean, people that when they don't understand the performance engineering process, they want results and immediately and clients, managers and everything else. But if you say, okay, if I run a test and I do the same thing, I do it three times at least before I report on results. Because we don't want anyone wasting time on anomalies that aren't reproducible. And a lot of times that's where the sinkhole is and they find out. They spend hours and hours researching this issue when it was just, it was a fluke, it happened once and it wasn't reproduced. Infrastructures are like that.

Okay, Rebecca, before we go, is there one piece of actionable advice you can give someone to improve their performance testing efforts and let us know the best way to find or contact you.

Okay, so the one piece of advice I have is relates back to what we talked to is do not be in a rush, okay? Don't be in a rush to slap together performance tests and get results. Spend your time on those scripts and making sure that you understand what's going on. You understand the business transaction flows in the infrastructure and then start off with a low, low.

Load and then increase it methodically. So take your time. Don't. Don't be in a rush because you're going to get the most valuable information if you do it methodically and with patience.

Thank you Rebecca for your performance testing awesomeness. For links to everything of value we covered in this episode, head ON over to testguild.com P PLAKEperformance20 and while you're there, make sure to click on the Try It Today link under the Exclusive Sponsor section to learn all about Smart Bears Awesome Performance Testing solution Load Ninja so that's it for this episode of the Test Guild Performance Podcast. I'm Joe and my mission is to help you succeed with creating end to end full stack automation awesomeness. As always, test everything and keep the good. Cheers. Thanks for listening to the Test Guild Performance Podcast. Head on over to testguild.com for full show notes, amazing blog articles, and online testing conferences. Make sure to subscribe to join the Guild and continue your testing journey. This has been a Joe Calantonio production.