Absolute Performance with James Pulley

Welcome to the Test Guild Performance and Site Reliability Podcast where we all get together to learn more about performance testing with your host, Joe Calantonio.

Hey, it's Joe. And welcome to another episode of the Test Guild Performance and Site Reliability Podcast. If you haven't heard, I actually turned 50 this month, so I try to take the whole month of April off. Because of that I didn't have a lot of interviews lined up. So what I'm doing is I'm going through my back catalog of my Perf Guild online conferences, taking the best of best of sessions, chopping them up and making them more of an interview style session. So this is a quasi interview with James Pulley, all about absolute performance and let me have James explain what the session's really about.

First, welcome to the Perf Guild presentation on Absolute Performance Triple distilled for your engineering enjoyment. What we're going to do is try to kind of boil down what is performance engineering today and some of the core attributes that we look at in terms of performance as a performance engineer.

Thanks James. Really excited to jump into it before we do though, but if you don't know, I'm also always looking for speakers and topics to cover in the performance reliability area. So if you have any ideas, any suggestions for me, I highly recommend you go to testguild.com, click the about and Contact menu option and send me a quick email on your thoughts on what you'd like to hear in the future. Or if you want to be a guest on a upcoming Perf Guild podcast, would love to hear from you. And also let me know if you're interested in having a Perf guild session for 2021. I did it every other year so far, So I did one in 2018 one in 2020. I'm debating if I should do one in 2021 or wait or wait till next year. So let me know your thoughts if you'd be interested in that as well. And so the two ways to do that once again is just go to test guild.com, click the about contact menu option or go to askguild.com and I should redirect you directly to that contact form.

This episode is brought to you by SmartBear. Listen load testing is tough. Investing in the right tools to automate tests, identify bottlenecks, and resolve issues quickly could save your organization time and money. SmartBear offers a suite of performance tools like Load Ninja, which is a SaaS UI load testing tool, and Load UI Pro, an API load testing tool to help teams get full visibility into UI and API performance so you can release and recover faster than ever. Give it a shot. It's free and easy to try. Head on over to smartbear.com solutions performancetesting to learn more.

Hey James, welcome to the Guild. Before we get into it, can you just tell us a little bit more about yourself?

My name is James Pulley. I'm a co founder of perfbytes with Mark Tomlinson. My current role I'm a practice manager for Tech Systems Global Services Performance Engineering Group. In this group we cover performance engineering practices all the way from the birth to the death of an application. And that goes all the way from requirements through development.

Traditional quality assurance practices, platform engineering, capacity planning and simulation of modeling. So it's got quite a wide breadth associated with it. And I kind of talk to the things that we look at across that entire life cycle, believe it or not, a little bit more about myself as I moderate a number of forums related to software performance engineering practices and testing practices. You may have run across me and one or multiple of those on Facebook, Yahoo, Google groups, LinkedIn, SQA forums, and maybe a few others in there as well. And of course I like barbecue. I think that kind of goes along with being a performance tester and performance engineer. There's something in our DNA that we gravitate towards barbecue. And for those of you who do not know my background, I had six years of formal art training, so I bring kind of an odd eye to performance test and performance engineering because of that background. So let's go ahead and get started. And I want to begin with something we're all too familiar with time. And if we look, this is a pretty long measurement of time. An hour, 23 minutes, 45 seconds and 678 milliseconds. This is where a lot of performance testers stop.

In our industry today. And that's a problem. They simply measure time. They get a record of time, they get a report of time, they print out a nice graph of time and then they hand it to somebody who looks at the graph and says why?

Why is it this long? And this is where you get the break between a performance tester and a performance engineer is on this question of why. So let's look at what really drives response time and then how that has suitability to how we ask questions and things of that nature. The things that drive response time should be well known to a lot of people who are attending this conference, but it's worthwhile covering them. We begin with cpu. It is a resource in the box or multiple boxes. And we are going to pair that with memory because this is also a component of our system architecture. Disk comes into play as well. And then our slowest resource, which is a network. Now, Scott Moore likes to call these the four Horsemen. Why does he do that? Because it is how your application uses these four resources together which govern what that response time will be. Under high loads, your application should have as light a touch on these resources as possible, because other users coming in will also be contending for the use of these resources. So if we hold on, if we grab a really small amount and hold onto them for a short period of time, then we're going to have a higher scalability to our system.

So as Scott calls them the four Horsemen, I actually refer to these four collectively as the finite resource pool. And no matter where you go in it, there is a limit to one or more of these resources. Even if you go to the cloud and you turn on auto scaling, eventually you reach the limit of a host where you can go no larger on a particular host. And if there's something which is a truism or an axiom, is that poor performing code will expand to fill the available resource pool. So if you have a well tuned app, you might be able to run a website on an Intel Atom processor or Raspberry PI support a high user population. But if we don't, it's possible that the largest resource pool inside of Amazon will not even support a small number of users. So in addition to the finite resource pool, there are very specific questions we should ask of these resources. And we'll begin with by taking a look at what happens when we allocate the resources. There is an optimum point for allocation of a resource. I like to call it the point where a user gets interesting. And when I say by interesting, if you're in an e commerce context, this is the person that is beginning to convert. If you are interesting in another business context, it's when you are beginning, beginning to go down the path of a particular business process. As you get closer and closer to the end, we should preference you more and more in terms of resources. We shouldn't allocate everything right up front and by God, we should not give everyone who comes in maximum resources. Because if we do and we don't have at least 50% plus one of our users who are going to take advantage of the resource, we're going to have issues. Many years ago, Mark Tomlinson and I were sitting in a room together at a company we affectionately Refer to as big shoe. And he turned over towards me and tapped me over on the shoulder and said, you know what? They give everybody a shopping cart. And I just looked at him, I said, that's too early. And as it turned out, that was exactly the case. It was something that impacted their performance, it impacted their conversion rates, it impacted them on their most revenue, interesting events, spot sales. But it was a historical artifact that was there in the system that just kept being in place because a marketing manager said whenever somebody comes onto the site, we have to know how many items are in their shopping cart. That means by default everybody gets a shopping cart. Everybody grabs what's in their cart from cold storage and we put a number up there. And that was too early allocation because on no e commerce site does 50% plus one ever begin to convert.

The next question we need to ask is related to how long we hold on to a resource before we collect it. Now a whole industry has started up surrounding this issue, which is issues related to monitoring your garbage collection. So I prefer to think of myself as a systems programmer, which puts me in the realm of C. So when I free something up, it just goes away immediately. I don't have to worry about the garbage collector coming around and you know, has it been mark null and you know, is it ready to go yet? Let's, let's get it out of here.

But this is an actual serious problem. Why? Because once again, you might have a marketing manager coming into play and saying, hey, we need to hold on to that session because sessions are expensive. Hold on to that session for 30 minutes. Well, that's probably longer than your average page to page transition time or even your maximum page to page transition time on a website. And, and as a result, you're locking up resources that can be used for other people coming through the system. Getting back to that issue of high scalability, when you have a highly scalable system and you have a high number of users, you need to hold onto your resources as short a window as possible because the resource you free is going to go to the next person who needs it. We all probably remember healthcare.gov this was a real problem on healthcare.gov how long they locked up resources. People came onto the site, they were asked to fill out an extensive amount of information in order to get a quote when most people were really just looking for information. And that held that session open for that user for 45 minutes, an hour, an hour and a half, until they were done completing their form and submitting everything back. This had the effect of locking up resources so other users could not get on the site and eventually the site simply ran out of resources and fell over.

And this was a cloud based site, you would think they would have auto scaling turned on things of that nature. But this is a real problem and you can find this on your own web server today. When you take a look at how your web server cuts sessions or how your database connection pool cuts sessions, if you hold on to them too long, you're simply going to run out of connection pool connections to go to the database. You're simply going to run out of connection handles to get to the web server. So be a little aggressive, analyze how long you're taking on the system. In my case, my favorite statistic is that web page to page transition time, how long it takes to go from page one to page one plus one or page two or page three. And that's usually somewhere in the two to three minute range for most people. If you happen to look in your server configuration and see that your session is being held for 30 minutes, it's being held for 1800 seconds, well, perhaps it's time to trim that down a little bit because you're going to have issues.

Next up.

We want to ask questions about how often you hit a resource. We refer to this as your frequency. We've all been in situations, or nearly all of us, where we have a subquery which is out of control, a subquery which is populating a view or it's a part of a business process to a database, and the subquery has to be run 10,000 times. So each individual query may not be that expensive. But when you add up the cost of 10,000 times, it absolutely does. I've actually been in a situation where I've observed this issue inside of a Java virtual machine as well, where we have a whole bunch of classes and subclasses or permutations of classes going on. And so we get thousands or tens of thousands or hundred thousands of calls till we get to the absolute primitive object and come all the way back up through the stack. So it's very important that you look at your frequency of calls as well. And then to bring us around to the very end, we have to ask the question of quantity. Are we over subscribing now? This is a very common issue involving network. Network is the most expensive of your finite resource pool because we have to leave the bus and we have to go across what to the computer is an exceedingly slow connection to either send or receive data and websites Being what they are today, and digital cameras being what they are today, it is very Easy to have 8 megabyte or greater images of complexity on websites. Now make no mistake about it, these pictures are beautiful.

But.

There is a problem with having an 8 megabyte image on your front page. It does slow it down. This issue of quantity also drives the content delivery network industry. How often we use your network, that's frequency, how much data we're passing, that's quantity. We want to reduce the amount of data being sent and we want to put it in a location closer to the user network. And ideally we'd like to reduce the frequency. So we got quantity and frequency by adjusting the cache model. So that's how performance gets improved. As we take a look at each, each one of these elements in the finite resource pool, cpu, disk, memory and network, and we look at a way to shave a little bit at a time from a lot of requests for each of those resources in order to improve performance. So if I can pull microseconds away from the cpu, if I can pull milliseconds away from the network, if I can make the disk more efficient, if I can reduce my memory footprint, all of that has.

Benefits in terms of my performance. Okay, so now let's look at this with kind of a jaded perspective, knowing where we are in the industry today for performance engineering tools. If I want a fully featured tool, I need to be able to all of these questions. And there are a lot of tools in the market which only support a subset of these questions. And for a performance engineer, that's really a problem because as I move and look at tools, I do need to have a balance on my tools. I need to be able to collect timing information because that's my end user experience in this case. But I also need to understand why or what is driving that. And that's my monitoring collection. But interestingly enough, it's really on the analysis where we're adding the majority of our value as performance engineers. Now think of a traditional performance test cycle. We have this balance of capabilities and we have this ever increasing value. I may spend a month building a test in order to generate my timing records, in order to collect my monitoring resources and things of that nature. And at the very end, I have a manager demanding to know the output in under an hour after the test has finished. That's a real problem because it is in that one hour where we have our limited window to deliver maximal value. And this is where most organizations are struggling today, is you spend all of this time, building the test, executing the test. In some cases you're executing a test without the benefit of monitors, without the benefit of analysis. You're just collecting timing records. So you have a limited, you have a limited opportunity to deliver value.

I want to suggest to you today, go back to your organizations and find the earliest possible opportunity that you have.

To collect timing records. Once you have that, find the earliest possible way to collect monitoring data to match up to that. That might actually be in the unit testing phase where someone is looking at an individual query to a database, where someone is looking at an individual web services call.

In development. And this could be long before you actually get to any sort of multi user performance test. There's a little dark secret that I should expose in this case, which is about 80% of performance issues that we find in multi user performance tests can be found with one user or one unit test. There is evidence at that small of a level, that atomic level, that there is a performance issue in the form of time too long where budgets have been exceeded. And then you could take a look at how resources are being used to find the root cause and send it back and get it fixed much earlier.

For those organizations which are struggling to find value in their performance testing practice, consider a performance engineering perspective of how do we collect this information early. And that information could be collected in terms of rum real user monitor integration. It could be collected from logs, it could be collected with other instrumentation inside of say a Java virtual machine. But ultimately you want to find the earliest possible location.

And time to collect that data.

On a typical basis. Here's how most performance engineering organizations develop. We begin with time.

Hopefully we progress from this point. We collect time, then we need to understand why certain things are going on. Monitoring is the key to this. And then finally we get to the point where we can deliver value or analysis. This is not a turn on the lights one day and you are delivering value the next. Typically for an organization mechanically, it actually takes quite a while to get proficient just at collecting time. If you're doing it in a performance testing context, it could be three months to a year for different individuals to mature in their testing use just their testing tool use. If you elect not to train or not to have a mentor for your group, then that time is actually probably going to be extended.

You may think that you're saving money, but you're actually impacting the delivery of value. Once you get to monitoring, we now have two components of that balanced that balanced resource or that balanced tool set that we need to have. And finally, when you get to analysis, this is really where you begin to deliver value. Once again, reinforcing the prior slide of the earlier, you can provide some analysis, even if it's on a single user basis. I would even argue if you can go back to the requirements and identify that there are no performance requirements. That is, you have no way to advise the developers what they should expect, how they should set budgets, no way to advise the software as a services vendor what the expectations are on your part for them. If those are absent, you're almost always going to have an issue and struggle in the performance testing and performance engineering phases of your project. So find that early on, the earlier the better. And if you can find it and fix it in the requirements phase, that will have miracles downstream for how it will shift your organization.

Now, as part of my last slide, I want to talk about the tools that are in use in the industry today.

And how those tools kind of fit together in a performance engineering context. So I want to start with what you might call a fully featured tool. A fully featured tool has the ability to either pull or collect or generate a timing record, a transaction as we normally term it in performance testing. It has the ability to collect monitoring data directly from the tool, doesn't require a third party tool, it's within the tool itself. Then finally, you have analysis that you're able to conduct within the tool itself. So think about that three legs on a stool. You have timing record collection, you have monitoring and analysis all within one tool. You have no other dependencies and you can move forward.

Next up, we have in the market a large number of partially capable tools for performance testing and performance engineering. And when I say partially capable, that means at least one of these three capabilities is absent. I have the ability to load and report, but maybe not monitor or analyze. I have the ability to load and report, but not monitor or analyze, you know, things of that nature. So if I'm looking at a tool.

If I'm only interested in response times, and I've never done anything else, and I'm a performance tester, I will look at another tool that only generates response time and say, hey, that's what I do. I generate response times, I collect timing records. That's perfect for me. The problem is, when you come to use it in a performance engineering context, it doesn't have all the data points that you need in order to deliver value. You're going to have to use another tool or collection of tools in order to add that.

Then we have a class of Tools in the market called load throwers. These are tools that can actually walk through a business process for a given interface, say, web, but they have no monitoring capability to speak of. They really have very limited reporting capabilities and analysis capabilities and visualization capabilities are almost non existent. So we have tools that, hey, I can point at your website and I can turn them on and generate 500 or 5,000 or 500,000 users. And I collect a bunch of records for how long that takes for all those sessions. Then I can maybe print off a report and hand it to somebody. And then the common question comes back why? You will find that these tools are typically paired with other tools that do monitoring. A third tool that will do visualization, you may have a fifth tool that does analysis, or fourth or fifth tool that does analysis and reporting. So this becomes like a Lego project where I have a tool to do X or a brick to do X, another brick to do Y, another brick to do Z. So you're not really getting the full capability within one tool. You have many tools to support and many tools to maintain.

And then finally at the bottom we have this class of tools called a denial of services tool. And my favorite tool in this category, because it has such a cool name, is Bees with Machine Guns. I mean, you can just, visually you can just see that whole swarm of bees with machine guns and they're all coming against your server. I mean, the person who named that is really just genius. But what separates the denial of services tools from the load throwers is the denial of services tools are really on a somewhat uncontrolled basis. They don't really collect timing records. They don't really walk through a business process completely. They have very limited reporting. They assume that you're going to run all sorts of monitors on the back end and basically they're just there to turn a fire hose or a flamethrower on your website. And that's about it. Now you can get some data out of a denial of services tool on the monitoring on the back end. Maybe how your cash model is working, how, how it loads up your CDM provider, things of that nature. But word of warning, CDM providers don't like these denial of services tools either. They see them as an attack and sometimes they will turn off your load generator sites as a result. So just keep that in mind. Key definitions as we move forward for you to evaluate what you're using in your environment, we have fully featured tools which have that balance of capabilities. We can either generate a timing record or collect a timing Record, we have monitoring capabilities within the tool, and then finally we have analysis and reporting capabilities. As we move further down, we have partially capable tools. They don't have the balance of those three capabilities and you get fewer and fewer capabilities as you go further and further down this list. As a general rule of thumb.

The fully featured tools are mostly on the COTS front, the commercial open source tool front, and the denial of services tools. And the other ones that are partially capable, you tend more towards the GNU or open source front. So if you've ever wondered why there is a break between those tools and some people say, oh, tool X is perfect, but I only generate timing records. So if I had a fully featured tool and I only generate timing records, I see no value to me in what I do with those other capabilities which are there, but I don't use. Likewise, if you're on the side where you have a tool which only generates X and you need to answer questions, now you're back to that Lego construction project of bringing in other tools to fill those gaps.

So lastly, kind of a look at the marketplace of tools out there. There are a lot of tools represented here. Some are on the open source front, some are on the commercial off the shelf software front, but they all have some capability for either generating a timing record, collecting some monitoring data or analyzing. Now, not every tool has them.

For instance, let's say I was using Gatling in a development context.

Might be perfect for exercising.

My web services call, but it might have limited capability to report the type of timing that I need, the type of analysis, the type of monitoring. I can use other tools to fill in that gap. That gets to that point I made earlier as a performance engineer versus a performance tester. We want to look as early in the life cycle as possible to begin collecting those items, collecting that timing record, collecting resource measurements for cpu, disk, memory and network, asking those questions of when we allocate a resource, when we collect it, how often we hit a particular resource and how large of that resource we're using. Because if we can find that early, and if we can find it substantially early, we can save the company quite a bit of money. We can also improve our value on a performance engineering context to the organization because they no longer will see you pigeonholed in a particular location of oh, James really only answers multi user performance questions. So don't even ask a question until you get to multi user performance. Well, that's not true. I can help you answer questions related to single user performance. In fact, I might have a rum agent installed inside of functional testing development so I can get data as early as possible and identify and point out, hey, we have some non performing code early, let's go take a look at it. On a single user basis. I might be collecting logs and looking at them using Splunk or the Elk stack, in which case I could potentially see collection of timing record data as early as the first lines of a web services call are built. I can see how the times are shaping, I can see if they're within budget and I can begin advising quite early on an active basis versus a passive basis where data comes to me. I can say, hey Mark, I see that service you're writing. It's out of budget. Should we take a look at it in greater depth? Mark may come back to me and say, oh no, no, no, I have a whole bunch of debug stuff in there right now to make sure that I'm getting all of all of the information back that I need. Once I pull that out, then we'll take a look at it. So have that dialogue early. Have that dialogue often. Use as many tools as you want to get them. But also keep in mind when you come to performance testing, it's probably best to have a full featured tool.

Just because it reduces the complexity of your test environment. To do this with now, I'm going to get a lot of arguments from that. I'm fully aware and I will tell you as a services provider, if you come to me with an open source tool and say you must deliver with an open source tool, I'm probably going to have to charge you more because I want to deliver value and I have to go add other tools to the one that you've told me in order to deliver on that value proposition of analysis, collection of monitoring data, timing records and bringing them all together in one location in order to deliver that value. Thank you very much for your time today. If you have any follow up questions from this presentation, please feel free to reach out to me jpullychsystems.com or jameserfbytes.com.

Thank you James for your performance testing awesomeness and all you do and everyone at perfbytes for all you do for the performance engineering community. And as I mentioned, this was taken from a 2019 PERF Guild Online conference. I'll have a link to this in the show notes. So so if you actually wanted to watch the whole video, you could still actually get the recordings for this event. And to do that all you need to do is head on over to testguild.com 64 and while you're there, make sure to click on the Try them both Today link under the Exclusive Sponsor section to learn all about SmartBear's two awesome performance test tools, Load Ninja and Load UI Pro. So that's it for this episode of the Test Guild Performance and Site Reliability Podcast. I'm Joe and my mission is to help you succeed with creating full Stack Automation awesomeness, which includes performance testing and site reliability. As always, test everything and keep the good. Cheers.

Thanks for listening to the Test Guild Performance and Site Reliability Podcast. Head on over to testguild.com for full show notes, amazing blog articles, and online testing conferences. Don't forget to subscribe to the Guild to continue your testing journey.