The Beauty of Variance with James Pulley

Welcome to the Test Guild Performance and Site Reliability Podcast where we all get together to learn more about performance testing with your host, Joe Calantonio.

Hey, it's Joe. And welcome to another episode of the Test Guild Performance and Site Reliability Podcast. Today you're in for a special treat. We're going to listen in to a masterclass video recording taken from our 2020 PERF Guild Conference called Beauty and Variance from the one and only James Pulley. To actually see the video session with the slides for this audio, head on over to test guild.com P91 and I'm always looking for guests for this podcast, so if you or someone you know on your team would like to be interviewed on a performance or site reliability topic, drop me an email@joeestguild.com or all right, before we get into it, let's hear it for our awesome sponsor.

This episode is brought to you by the awesome folks at SmartBear. Listen. We know load testing is tough but necessary. So investing in the right tools to automate tests, identify bottlenecks, and resolve issues fast saves your organization both time and money. And that's why SmartBear created Load Ninja, a SaaS load testing tool to help teams get full visibility into performance so you can release quality software that faster than ever, make testing effortless and give it a shot. It's free and easy to try. Head on over to Loneinja.com and learn more.

Good afternoon everyone. My name is James Pulley. Welcome to my Perf Guild presentation on the beauty of variants. Earlier this year, when I was looking for a topic for the presentation, Mr. Colantonio suggested that hey, performance engineering. How about that path rather than purely performance testing? So I took a step back and said okay, what can I look at that has broad applicability across all testing tools and all performance tools as well. And really it comes down to a number of core measurements on time and standard deviation is one of those. And standard deviation, by the way, is the square root of variance. So hence the presentation.

A little bit About Me I'm a co founder of perfbytes with Mark Tomlinson. I'm a moderator for many forums related to tools and processes related to software, performance testing and performance engineering. I am a barbecue aficionado along with Scott Moore and Mark Tomlinson and many other people in performance related disciplines. I don't know, it seems to be an odd combination. Performance, barbecue and Scotch. That's a story for another day. And I'm an artist as well, so maybe some of that will come through in the presentation. So normally I deliver this presentation in a whiteboard format. So I'm trying to capture a little bit of the flavor of that as we go through. Notice my signature here. And let's start with our friends, the distributions. As we go through statistics, we all become familiar with how data is distributed in a system. And we have these items like the uniform distribution, very important for calculating random numbers during a test. You want them approximately uniform. We have our normal distribution, the classical bell curve, in this case, that everyone talks about the normal distribution. The problem is, when you get into natural populations, very rarely do you get a normal distribution. You get something more along the lines of the Poisson distribution. And it's important to understand a little bit about these distributions because we're going to be talking about standard deviation, which is a measurement of the spread of the data in the system. And it's easiest to use the actual normal distribution of the bell curve to speak about this because it's easiest to represent that environment. So here we have the normal distribution and we have the average right the middle, as we would expect. And then we have a measurement of the spread of the data, which is our population standard deviation. Normally you'll see a figure that looks like sigma at the bottom, representing the distance from the mean value. You may have heard the term six sigma. That's basically six standard deviations away from the norm for your population, things of that nature. And so we want to be able to make use of these measurements of standard deviation, this measurements of variance in the system of spread of data, in order to understand what's going on inside of the system. But you know what, I get this question all the time, but, James, my data looks nothing like that. I go and I do a distribution of my response times and. And it looks like somebody has been surfing off the coast in a storm. It's like all over the place. I got a little puddle data here, got a little puddle of data there. Here, puddle, there puddle, everywhere a puddle, puddle. And it just doesn't come together to look like a uniform distribution, a Poisson distribution, or a normal distribution, or any other types of distribution. How can I use standard deviation in this case? Here we see in red the average. We see some measurements of standard deviation inside of the data set and it's, you know, it's just kind of all over the place. It's like, well, okay, I get you. We're going to take a look at how we can make use of that standard deviation value in order to reveal what's going on inside of the system. And it's worthwhile to consider what an optimal distribution looks like in this case. So we can see once we have an optimal distribution, how we get to something that looks like waves cresting on the beach. Believe it or not, this is the optimal distribution of data for a response time tightly clustered around one value. Here we have unlimited resources, we have no resource contention and.

It just runs. We don't even need auto scaling turned on. We just have unlimited resources. We have unlimited cpu, unlimited disk, unlimited memory, unlimited network.

But you know what? Reality is nowhere near this.

What does an optimized distribution look like in an environment where we will have some sort of limitations, some sort of contention?

Well, it takes on a figure that looks like one half of the normal bell curve, kind of like the back half of the bell curve. The origin is around one point. It just doesn't go any faster than that. And what pulls this data out from the purely optimal is how resources are being used. And you can think of network has a gravity associated with it, disk has a gravity, CPU has a gravity, memory has a gravity. The higher our dependencies.

On these resources, the greater the pull away from the optimal solution in this case. And it also just so happens that anyone that has a high resource utilization is also something that usually is slow over time. As you get the system under load, it tends to get slower and slower and slower and slower. And let's take a look at what that looks like. So we start with our optimized distribution and then we get some enterprising developers who say, you know what, I can do some cool things, I can do this and I can do that and I can add this function and it will even get the status of the coffee maker in the next room.

All of that has a cost associated with it. The more CPU I grab, the more network, particularly that I grab, more memory that I grab, the more disks I grab. This has an impact overall on the scalability of a function and how the data is represented in a population distribution. And also it impacts standard deviation and say, okay, so.

We know that.

Resource dependencies impact scalability, and we know that resource dependencies impact standard deviations. So what does that do for us? Well, it gives us something that we can use as an analytical method. And why is this important? Well, not every testing tool has the ability to monitor a system.

Particularly if you go to the open source community. You know, you get response times, that's about it. You may get some standard deviation reporting.

But you don't get the ability to look deep into the system to Say, okay, how can we, where should we spend our time?

But this standard deviation value is pretty generic. As long as we have a measurement of time, we can apply it to a lot of different areas. If we have enough samples from JavaScript and the client front end, we can see what our resource dependencies are on the front end and tune JavaScript appropriately. If we can look at this on the way back end on storage, on query performance, we can add this analytical method along to the rest of our profiling to say, okay, where should we spend our time for the largest benefits? So we have this axiom, okay, high resource use drives high response time. We know that high resource use also manifests as high standard deviation. Because this resource dependency we get this pull on gravity that stretches out the curve and you know, from its optimal function only operating say at 1 second or less or 2 seconds or less, it's now being gravitationally pulled out with the resource utilization out to 20, 30 seconds under load, things of that nature. So if we can identify those elements and how we can make use of them, then potentially we can make our value to development that much more and our value and analysis that much more as well. So let's look at a couple of case study examples of where this data comes into play and how we can make use of it. So we're going to start with this data as a control element. Now, those of you who are, who are scientific method geared.

You know what a control element is. But in most cases for tests in it, control elements are kind of set aside. A control element is something in our test that tells us about the quality of the test or quality of something inside of the test, a reagent, something that we do to measure something beforehand and measure quality of something. But in many websites we have some natural control elements that tell us a lot about what's going on inside of the system. And I'm going to pick on the icon that shows up in the browser bar. When you go to a website, just about every modern browser will ask for an icon associated with the site. Not every site has one. You get a 404. A lot of sites do have one, it delivers it. But for some reason this ICO file tends to be not managed well in your cache plans. So it becomes quite a good thing to have as a natural control element on a website because it is very small, because it is requested by every user, it is requested associated with every single page.

The response for this should be inside of the response cache of the website shouldn't have to go to disk to load it if it's there. If it's a 404 error, that error should be cached. So it should be uniform and highly efficient in delivery. About the only dependency we have is network in this case. Hence you see that slight falling off for the network dependency. So this gives us something that we can look at across the entirety of our test and say, okay, this tells us that potentially the server is healthy even if other requests are slow. Because what this will also tell us is if this particular element, which should be highly optimized in delivery, being so small, being cached on delivery, and either a 404, it's an error cache, or something of that nature, it should be fast. It should be fast all the time. It should not deviate all that much. And it's a static resource. So if this starts becoming more like the wild cresting wave curve, we know that we have an overall resource dependency problem on the server. Server's too small.

Things are being aged out of the web response or the web, the web request cache very, very quickly, because this should be in the cache just about all the time, whether it's an error or the actual delivery. If it's not being cached as part of your cache plan, which of course is the ideal, but you will likely have, even on highly robust sites where this is in your cash plan, there'll be some other small element that you can look at and say, this should be delivered exactly the same way every single time, assuming our servers are healthy. If they're not healthy, this will degrade. And that'll be very telling about our infrastructure versus our application in this case. Next up, we're going to look at another common condition, and that common condition is we have something that is really slow. And I'm going to pick on something in Java in this case, but it could be Microsoft aspx, it could be any number of technologies, it could be Node js, but all that matters, it's an app server. In this case, we have what looks like a highly optimized function, you know, visualizing our curve and how the data is represented in the system. But, gosh, it's 45 seconds and our SLA is five seconds. Well, that's a problem because this not only tells us that we're slow.

But the shape of the curve being highly optimized around a central point with a low standard deviation tells us that if, in order to improve the performance, we actually have to have an architectural break in how this request is responded to.

In order for US to shift it left earlier than 45 seconds. We have to change the entire architecture of this request and that may require architectural changes downstream as well.

So if I'm making recommendations on where to spend time in the system, I should take that into consideration of do I, I can report this, but do I spend engineering resources on it? And that's, that's a question we'll come back to in just a couple of minutes in this case.

And let's contrast this with another element also on an app server, my remote call to which by the way could just be a query that's running on a store procedure on a database server as well.

But in this case, you see that we have a very wild curve represented. We have a very wide standard deviation. Our average is pretty much the same as before.

But this wild variation shows that we have a large opportunity for tuning and improving the performance of the code inside of this release. This does not necessarily require an architectural break. Notice we have response times very close to zero, you know, 1, 2 seconds in certain cases. But as we get resource contention, as the server gets loaded up, we have more and more of these long, long response times, which indicates we have something going on. On the resource side of the house. We have an over commit or a lock on one of the four main resource groups, I.e. cPU, disk, memory or network. Scott Moore likes to call these individuals the four horsemen of the IT apocalypse. Which is not a bad set of visualizations. You know, everybody gets a T shirt with their label on it in that case as they ride in on their, their horses. But because they are finite in nature and if you use too much, you will suffer and your entire system will suffer as well. Now there is a better visualization for this. As I was putting this presentation together, I was like, okay, how do I overlay this stuff? And I went out to Wikipedia and the wiki comments. I said, okay, let's look and see what's available on standard deviation. Well, there is a fantastic curve set that they have represented that I'm going to bring up next. And it really shows this condition of a highly optimized function versus one which is not highly optimized. And so we have this condition of we need an architectural break to improve performance versus we need to tune how the code is actually making use of resources in order to improve its performance and its function. Ideally, we want every request to look like what's here in red. Do we want it to have 100 second response time? Maybe not. Maybe that's 100 milliseconds. Could be change in scale here. But we want every response, ideally to take this really narrow curve, this one half, you know, this back half of the normal curve with a very low period associated with it and a very short standard deviation versus the blue curve, which is just wildly all over the map. And there are almost no, there are, there's almost no amount of resource that we can throw at that that would keep it from not degrading. In this case, bad code will generally fill the available resource pool that's available to it. And it's just a question of how long before it degrades, not if it will degrade the first function, the one in red. You can feel pretty certain it's going to operate in a very narrow fashion and it's not going to cause performance issues overall in this case. But it is something you do need to take into consideration to improve the performance. How do we get there? And we're going to take a look at that next. And this brings us to the practical side of the use of standard deviation in this case. How can we make use of this across a wide variety of solution sets to provide us a measurement of time for insight into what's going on inside of a computer? Well, certainly we're tool agnostic. Anything that can give us a measurement of time, we can get a measurement of the spread of those, those durations. We can, we can look and see what the standard deviation is. So that is very important.

Many performance testing tools on the market today, particularly the open source variety, only report response times. They don't have deep insight into what's going on in the back end. They rely on other tools for that. So if we can get a measure of measurement of standard deviation, we can get some measurement of insight as to what's happening on the back end. We can also look at it with tool, with items that do logging, items which do profiling, items which do deep diagnostic, all of them can provide a measurement of time and as such we can get a measurement of standard deviation. This is a tier agnostic item. As we move into rich JavaScript frameworks on the front end for client presentation, it becomes ever so more important to be able to measure the cost of those JavaScript framework components and what is going on inside of the client. What happens on machines that don't have as large a resource pool as others? What happens when you're running 32 browser tabs open? What happens when WebEx is also running? As a part of this, you can get a measurement of how your JavaScript is varying in performance. If you have something that is highly optimized, you'll get that really nice curve that looks like one half of the back half of the bell curve, normal curve with a very short duration. Otherwise you're going to get samples all over the place. And that tells you that you have an opportunity to optimize your code to actually improve the framework component performance on the front end. But likewise, we can take this also to the middleware. We can take it to the web server to see what's being delivered as well, and we can take it into deep diagnostic and into database profiling or storage profiling to see what is going on there outside of the web tier. Anything that you can get outside of the web architectures, rather, anything that you can get a measurement of time off of. Let's say it is delivery of stock trades from Europe to the United States, and there is a measurement of time between when the trade takes place and when you receive it. And you can track that in logs. Well, if it's coming from different systems where you're getting the different quotes from, you might be able to look and say, okay, which systems are less efficient than others in the delivery of this data? And how might we use that insight to say, hey, this looks like it needs an architectural shift to get any faster. This one looks like you might be able to tune the code that's actually going in the system right now because it's got this very high variance associated with it. Providing that insight back to the development so they're not chasing engineering dragons or ghosts can be really helpful.

The finer degree of detail you can get with this timing data, the more effective this analytical method. So if we look at a web page, a web page might have 200, 300, 400 elements on it, and it could be anywhere from 80k to 16 megabytes or greater. In the case of J. Crew Last Thanksgiving, a 43 megabyte GIF on the front page. So you're going to get highly variant performance in a case like that because you have all of these elements on the page and each one of them has their variances in performance. So all of that is going to be accumulated in the web page. I actually recommend at least dropping down to the web logs, implementing time taken across all of your web tiers, that includes your app servers and things of that nature. And this gives you at least an individual request and an individual response to look at initially. And you can pair this with any performance test that you have ongoing, because you can pull the web logs at the end of the Test from the web servers, you can pull the error logs, also have time taken in there as well, and you can actually measure the variances on individual requests. So those things that are hidden within a page that are actually driving the highly variant nature of a page, you would be able to see at the individual request level and say, hey, this requires an architectural shift. This, it looks like we can tune within the existing release. And at that point it gives you a roadmap of what you can look at in greater detail to see what might be going on. And that really brings us to the kind of the standardized analytical method that you can take a look at. And whatever tool you're using, whether it's production, whether it's test heck, whether it's even development, and you're looking at response times and measurements against budget. There's is we have two ways that we can look at cost in a system. One is we can take a sum of these time taken measurements and that tells us, you know, the total load on the system, how many seconds we're using of resource time. And that directly translates to cpu, disk, memory, network, any, anytime we can trim a couple hundred milliseconds off of something, that number is going to go down. But prioritization of that can be hidden if you're only measuring the cost. And here's how this works. Let's say I take that highly efficient 45 second response time and I sum all of those requests. Well, the average of that was the same as the other request which had the highly, highly inefficient.

Response time. Both of those are going to look pretty close if they have the same number of requests coming in a test. It's like, okay, well where do I prioritize this? This is where we can take that measurement of variance or standard deviation in this case and multiply that by the sum. Now, in the case of a highly efficient function, those numbers are going to be quite small, sometimes below 1 fractions of a number. That's going to have the effect of reducing the overall weight or cost of that element in relationship to the one which has the highly variant response time. So you could have items which were slower in average than the 45 seconds, maybe 20 or 30, but had very large standard deviations. And this in fact would push them up quite high with respect to this one element, which is architecturally highly efficient, but very slow. So let me repeat that. If I take a sum of the time taken for an individual request and I multiply it by its standard deviation, that gives me a rank and where this fits in from a standpoint of both time and its ability to be tuned within the current release for where I spend my engineering dollars, my sparse engineering dollars, to improve efficiency within a given release.

So.

That'S it. Very, very quickly. Just a quick recap of what we've taken a look at. We looked at standard deviation as a measurement of the spread of data within a system. We looked at how standard deviation is reflective of how resources are being used in the system, and we looked at a number of examples of where we could use these frequency distributions and the measurements of standard deviation to understand our system better, and whether we're having system issues, whether we're having individual request issues, whether within the individual request, whether we need to take a look at architectural changes, or whether it looks like it's suitability for tuning. We also took a look at how we might implement standardized analytical methods, whether that's in a dashboard form, in a tool, or just in Excel afterwards in order to get deeper insight into the system. Thank you everyone for your time today. My name once again is James Pulley. And that's it. Have a great day.

Thank you James for your performance awesomeness. For links of everything of value we covered in this episode, hand it over to testguild.com P91. And while you're there, make sure to learn more about our exclusive sponsor Smart Bear, and learn more about their performance testing solution that makes performance testing effortless called Load Ninja. And make sure to click on the Try it for Free Today link to learn more. That's it for this episode of the Test Guild Performance and Site Reliability Podcast. Once again, I'm Joe. My mission is to help you succeed with creating end to end performance testing and site reliability awesomeness. As always, test everything and keep the good. Cheers.

Thanks for listening to the Test Guild Performance and Site Reliability Podcast. Head on over to testguild.com for full show notes, amazing blog articles and online testing conferences. Don't forget to subscribe to the Guild to continue your testing journey.