The Economics of Performance Engineering with Lloyd Watts

Welcome to the Test Guild Performance and Site Reliability Podcast where we all get together to learn more about performance testing with your host, Joe Calantonio.

Hey, it's Joe. And welcome to another episode of the Test Guild Performance and Site Reliability Podcast. Today we're talking with Lloyd Watts. All about the economics of performance engineering and probably touch on a bunch of performance issues or performance topics he's been posting recently recently on LinkedIn, if you don't know. Lloyd is the Founder and Chief Executive Officer at Neocortex Inc. And serves as its board on its Board of Directors. Prior to founding NeoCortex, Dr. Watts was the founder of Audience, Inc. Where he served in various capacities over the years from the CEO, CTO and Chief Scientist. I believe that company went public in 2012 and actually was acquired in 2015. So as you can tell, Lloyd has a lot of experience with multiple companies. Early in his career, he worked as an engineer at internal research corporation, Microcartel Pacific Research and a bunch of other companies. Really excited to have him on the show. Have you seen his post on LinkedIn? You can see he has a lot of interesting takes on performance engineering that we're going to dive into today. You don't want to miss this episode. Check it out.

This episode is brought to you by SmartBear. Listen, load testing is tough. Investing in the right tools to automate tests, adapt, identify bottlenecks and resolve issues quickly could save your organization time and money. SmartBear offers a suite of performance tools like Load Ninja, which is a SaaS UI load testing tool, and Loaduipro, an API load testing tool to help teams get full visibility into UI and API performance so you can release and recover faster than ever. Give it a shot. It's free and easy to try. Head on over to smartbeer.com solutions performancetesting to learn more.

Hey Lloyd, welcome to the Guild.

Thank you very much, Joe. Glad to be here.

Great to have you on. Before we get into Lloyd, is there anything I missed in your bio that you want the Guild to know more about?

No, no, you covered it all.

Very cool. So as I mentioned in the intro, you've been posting a lot on LinkedIn, a bunch of different topics. One of them that caught my eye was on the economics of performance engineering. So before we dive into that though, what's prompted you to stop posting more on LinkedIn on the on topics of performance engineering?

Well, my company, Neocortics, has a service that we think is going to be beneficial to load test vendors. And so that's my personal interest in the subject is that it's important to my business, but it was important early on, you know, that I get a good understanding of the dynamics in the industry, who the key players are and what they care about, what's important to them. So I originally started this conversation on LinkedIn, just talking to some of the local, you know, some of the load testing experts, people like Stein Sheppers, laurie custi@corvisor.com, paul McLean, several others, and was just trying to get an early indication of how they allocate their budgets towards instances that they spin up for load testing compared to license costs of tools that they buy. So my initial question was just about the relative expense budgets for those two components, right? So I asked this kind of open ended question, hey, do people spend more on instances than they spend on licenses for their software? And I was surprised and pleased by this sort of flurry of answers that I got back from a wide number of, you know, large number of people that all came back with, what about this, what about that? Like for example, Scott Moore came back and, you know, didn't answer the question at all about the cost of instances and the cost of licenses. And he came back and said, well, what about the cost of having your website go down on Black Friday, Right, that's a cost, right? And so we had this set of contributions to this conversation about what all the different cost components could be, right? So that was kind of the beginning of broadening my thinking about this. So there's not just the cost that you pay for the, you know, AWS that you spin up instances on the cost to your software license providers, you know, the neoload or Load Runner or the, you know, the places where you would spend your money on licenses, there's all these other costs, but some of them, like the ones that Scott Moore was talking about, well, they're a cost of not doing performance testing, right? That's a different kind of cost, right. And so that got me thinking about, well, there's these things that are direct costs and then there's other things that are really more benefits, more of a benefit of doing performance engineering. And it becomes a cost if you don't do it. So anything that is a cost, if you don't do performance engineering, I'm calling that a benefit. And then just to kind of close it out, the, the way that I looked at at the end is, okay, you've got a bunch of benefits and you've got a bunch of costs. And generally people analyze that as return on investment. If my benefits outweigh my costs, then I've got a positive return on investment. And so now I had the beginning of economic model of the benefits and costs of performance engineering.

Very cool. And for the folks that are listening, I'll have a image of this in the show notes. It's a really cool graphic that Lloyd put together that shows this equation that he has set up. So Lloyd, I thought we'd dive into maybe each aspect that make up that equation. See where it takes us. The first, first section you have I think is certifications. So how do certifications play into the economics of performance engineering?

Yes. Well, so this is one of the three benefits of performance engineering. So there's the benefit of certification, benefit of performance and benefit of stability. So it's one of the components of the benefits. And this one came from some early discussions I had about three or four years ago with a couple of different companies. They were all in the E commerce space. So companies that provide search engine widgets, for example, to other people's websites to make recommendations about products you might want to buy, for example. Right. Well in that case they're a third party provider to a bigger website which has the relationship with the end customer who's going to buy the shoes or the clothes or whatever.

Right.

And so it appeared they didn't say it to me directly, but what they did say, one of them said, for example, you have no idea how much I'm spending on load testing every month. That's a direct quote from the CEO of a company who is in that third party widget category. Right. And the way that he said it implied to me that he was being forced to do it. Right. It's not that he particularly wanted to spend a lot of money on load testing, but he had a certification need to do so. His customers that he wants to win business with to get his widget onto their website were demanding that he self certify his service so that if there was a big spike in demand, kind of a Black Friday event, that his search engine wouldn't fail under load.

Right.

So that's an example of self certification. If it's necessary for you to win business, in order for you to win business, you have to prove to your customer that you can handle load. Then one of the benefits of performance engineering and load testing is that you get a certification benefit, you get business you wouldn't get if you didn't certify your performance.

You know, that's probably a benefit maybe a lot of companies don't think about until they're in a Certain situation, I assume. Do you think this is a benefit most companies know about?

Exactly. I mean, until they have a customer, it's not a customer with a gun to your head. It's a customer with business that you'd like to win. And they're saying, well, you want this business, you're going to have to prove that you can handle the load that we expect to deliver to you. So it's a benefit to the company to do the certification. But the nice thing is that now that's a quantifiable thing. That company could basically say, okay, I have this much in revenue that I wouldn't have if I didn't do performance engineering. So we can put a dollar value that. That's $2 million a year of business I get from that company that I would lose if I didn't do load testing.

Absolutely. So besides the benefit of certification, I think another one was performance. So what do you mean by performance in this particular context?

Yes. So the benefit of performance, that's the second component of the benefit, total benefits. So that's a well known idea that a lot of load testing companies talk about. It's the benefit that you get from having a fast, snappy, responsive website. So the general idea is that for every second of delay that a user has when they come to your website, they're more likely to abandon the page. Right. If you have a really slow, sluggish website, they're going to want to go to another page and buy their shoes over there. And where there's a snappier, more responsive experience. So there's a translation, there's a scale factor. Basically every second of delay in response time corresponds to so many millions dollars worth of lost sales to a large shoe company, for example. Right. So now we have a relationship between delay time in seconds and dollars lost in sales. Again, we can turn that into a benefit of doing performance testing and performance engineering. If you invest in performance testing, if you test your website, you find out how slow it is and then you do the engineering work to improve it. If you improve your response time by 2 seconds and it's $2 million per second is your estimated cost of delay, then okay, well that's 4 million 2 seconds times 2 million per second. That's a $4 million benefit you get by improving your response time. So that's a benefit. And again, we can put a dollar value on that.

Very cool. So once you have the dollar value on that as well, I think the next one for benefits is stability.

Right, Exactly. So stability, this is the one that Scott Moore was driving home in the conversation. So his way of saying it was quite sort of sarcastically, he was saying, oh, you'd rather have a website failure and have to do war room triage after the fact that. Right. Oh, those are cheap. That's what he said. Right. So he's sort of sarcastically making the point that the cost of a website outage on a serious sales day like Black Friday is huge. The huge dollar value in reputation in lost sales on that day, lost longer term sales to people who switch over to another supplier, for example. Right. So he's making the point there again of a huge cost, but a probabilistic one. It's a, you have a cost that you might end up paying if you have the misfortune of having a big failure event that happens under load. So that's a cost that you might pay if you don't do performance engineering. And once again, my philosophy there is if it's a cost that you would pay if you don't do performance engineering, then that's a benefit of doing performance engineering. So I count that again as a, as one of the benefits. That's the benefit of having a stable website because there's a huge cost if you have an unstable website that goes down under load.

Awesome. So those are the three benefits. I think you also have costs then. So I'm curious to know how the costs bake into the benefits. I guess the first one is, was instances, I believe.

Yes. The cost of instances. Okay, so there's the three components. That was the first one that I, that I was actually directly interested in. That's what made me ask the question in the first place was how much money are people spending on instances? Most performance engineers, that's spinning up AWS instances or Google Cloud platform instances or Azure instances, but spinning up these cloud instances that can act as artificial load generators and then can measure the response times that come back. That's the essential operation that's being done in a load test. Well, if you're going to spin up tens of thousands of instances to simulate million virtual users to simulate a real Black Friday event for a very large company, well, you're going to have to spend some real money on commissioning those instances and keeping them up for an hour or as long as your test is. So my question is was at the time, how much money are people spending on commissioning those instances? And you know, what's the annual budget for that? And so I will tell you that no one really gave me a direct answer on that. But What I could infer from the answers that I did get was that depended very strongly on the size of the company. So if you have a small startup that has created a nice website and they have a few hundred customers, then they don't need to do a gigantic load test and they don't need to run it for that long. And the benefit that they get of preserving their good reputation is small because they're not well known in the world. If you compare that at the other end of the scale. So let's imagine you've got a $2 billion company, something huge on the scale of a United Airlines or something like that, where the cost of going down is enormous. They might have 50 performance engineers to make sure that their website is fast and responsive. And it's possible they might spend $3 million a year on spinning up instances and making sure that they're capable of handling enormous load in advance of major travel events, for example. So basically, the amount of money that people spend on instances, loosely speaking, I think is proportional to the size, the amount of revenue that they get, and the size of the company. So that's the first component is the cost of instances for load tests.

Absolutely. Cool. All right, so we went over. The first cost was instances. So how does licenses then play into this equation?

Right, so that's the most straightforward one of all, which is if you are using commercial tools like for example, Neoload or Load Runner or Gatling Frontline, there's a number of tools like that. But then, then you're paying some kind of licensing fee to be able to use those tools. And generally, of course, the people that are choosing to use them are doing so because they find that it's well worth it. You pay good money for good tools because they more than pay for themselves. So that's a straightforward one. You have to pay for good software if you want to use those good tools. You have to pay for the instances that you. That you spin up in order to actually execute the tests.

Did you have some pushback on this one? I thought I saw a response about licensing that just because something has licenses commercial to doesn't necessarily mean it's worth it.

Well, right, so this is one where there's a number of different opinions and in some cases emotions run kind of hot in some occurrences. Right. So basically, there are some people who really love and swear by their favorite commercial tool. And this is really worth it, and I'm really glad that I pay for it. And of course, that's just fantastic. And those tools are very valuable. They did another whole posting on why would you buy these commercial tools? And the answer is, the short answer is because they're totally worth it, right? If you get these benefits of great support, great integration with DevOps, CICD pipelines and so on, there's a number of great reasons. Great protocol support, for example. There's a great set of reasons why people would love to use the professional paid tools and be more than willing to pay the license fee. So I have a whole blog post just on that, and that's one of the most popular ones, is singing the praises of the commercial tools. At the same time, when I posted the general comments about comparing the, the various benefits and the various costs, I also got responses from people saying, yeah, but wow, these, some of these free tools like JMeter, there's the free versions of Gatling and K6 and Locust for example. So there's a bunch of free tools and well, wow, they're surprisingly powerful. And one person said one of the benefits of the free open source ones is that you can extend them. So if there's something that you need to do and the big commercial paid tools don't support it, it might be faster for your engineering team to actually modify one of the open source tools than to submit a request, you know, feature request to one of the professional vendors and wait for them to do it. So that's one of the trade offs that people make. There was another observation that I had which was that generally, I mean, it wasn't universal, but there was a general trend that the people who were favoring the free tools were often coming from smaller companies. Now that makes sense. If you've got a very limited budget, then the free tools will be accessible to you and you might not be able to afford the really great expensive commercial tools. Whereas the larger companies that have huge reputations on the line and they don't want to have their website crash, they have huge revenue, they may have $2 billion a year in revenue, they can absolutely afford to pay for the tools and get the additional confidence that comes from having these specially well supported tools. So this question of free tools versus commercial is a. Some can be controversial and people can take strong opinions about it. I tried to take the position of there's arguments to be made in both directions and I posted in favor of both points of view.

Absolutely. I guess I was spoiled. I used to work for a large enterprise like you said, and we had all licenses for everything and I used Loadrunner pretty much for 20 years. And then you go to JMeter and you're like, what the heck is this? So I can see the pros and cons for both, for sure. So I guess the next one for cost was, I think the last one is engineers. How do engineers then play into this whole equation?

Right. Well, so that's one that I must admit that I wasn't thinking of at the beginning. So as far as I was concerned, the limited view that I had at the beginning of the conversation was the costs that are being spent on a particular load test would be the cost of the instances that you're spinning up and the allocated cost of the license. So if you're doing one big load test a month and you've got an annual license fee, then 1/12 of your annual license fee would be the allocated cost of the, you know, of the software license for that load test. So I was just thinking in terms of money that is spent directly attributable to each load test. But then it opened my eyes when the first comment that came back was from my friend Laurie Costi, who is at the founder of supervisor.com and he said, no, actually the cost of the instances is negligible, and the only costs that matter is the cost of the licenses, which I had asked about. But then the cost of the engineers. So like the salary of your engineering staff. Right. Well, that wasn't even in my mind at the time. I wasn't thinking of that as an expense associated with a particular load test. But from his point of view, that was also one of the costs of doing performance engineering and performance testing is the salaries of the engineers who are on your team who are managing those tests. So, in fact, I owe it to Laurie for extending my view of that. And that's where that third term came from. So then I have the three components of cost, the instances, the licenses, and the cost of the engineering staff. And then we have the three components of benefit, the certifications, the. The performance, and the benefit of stability.

So when would someone use this equation and does it help them determine when they should use a commercial tool versus open source? Is that what this helps them with or just in general? Why? Maybe a commercial tool might be more beneficial for them?

Well, yeah, so I think that it does help. However, you could also, I think it would be fair to say that most managers that are doing this, making decisions about what tools we should buy, how many, you know, this year, should we hire more performance engineers or should we more for tools or should we do tests more often? Right. How do they make decisions about what to spend on as Far as I can see, I'm unusual in trying to turn this into a quantifiable thing where I can put equations on it and say we can calculate what the dollar value of the benefits are, the dollar value of the cost, we can divide benefits by costs and we can get a return. Right. Well, so that's an analytical approach that I take. I think it's fair to say that many project managers wouldn't necessarily do a mathematical calculation like that. They would simply eyeball how much, you know. Yeah, we need, we need more engineers here because we're the ones we have can't keep up with the effort that we're asking them to do. Or we should spend more on instances because we had a failure last time and you know, last year and we need to make sure we don't have that. Right. So there are actually two more elements to that first slide about the economics of performance engineering, which were the constraints on it. Essentially we've talked about the three components of the benefits, but there's another line here that says all of the benefits added together can't possibly be greater than the total revenue that you got.

Right?

Because essentially those benefits are represent revenue that you would lose if you didn't do performance testing. So the benefit can't be greater than the scale that the company is on. A $2 million a year startup can't claim that they're getting a $2 billion a year benefit of doing performance testing because they're not at that scale.

Right.

So that the revenue of the company sets the scale for how much benefit could you reasonably claim to be getting by doing performance engineering? Right. So that's sort of sets a limit on how much you can claim you get a benefit for. And then similarly there's a constraint on the total costs, the total money that you have to spend on instances, plus licensing fees, plus your staff of engineers. Well, that's your budget for, for performance engineering. And now there's a relationship between these two things. You know, no company is going to spend 95% of its total revenue on performance engineering. That's too much allocation towards that. And it looks to me like perhaps there's a rule of thumb there that people might spend half a percent or 1% of their annual corporate revenue on performance engineering. If they are really making a lot, you know, it's an online retail business, then they would be willing to spend, you know, if they're spending 2, if they make $200 million a year on an online retail business, well, would they be willing to spend $2 million a year. That's 1% of their revenue on their engineering team and their software licenses and their instances. Right. So there's a scale relationship between the budget that you'd spend relative to the revenue that you have for the scale of company that you're at.

And I guess this all goes back to the first story you told me about how someone had to do certifications. This actually could give him some concrete thing to say, hey, you're not doing too much load testing. You're doing just the right amount. Maybe you're right.

Right, Exactly. Yes, that's exactly it. So I think generally a project manager or an executive CFO at a company that's at that scale, they're not necessarily calculating a return, as I'm proposing with my math here, but they're doing kind of an eyeball. They're setting a bunch of parameters in their spending relative to what they perceive the risk of not spending it and how much can they afford to spend. And of course, that depends on the current conditions. As you go into a recession, then, well, you might spend less, you might have to lay people off and so on. So those things can vary over time. But in a sense, what I'm trying to do is quantify the eyeballing process that people are loosely doing all the time. Right. You could say the same thing in your own. Your own family, personal budget. You know, how much money do we spend on various insurance and food and rent and various other things? And, well, we all make sort of eyeball decisions about that's too much. This is not enough that we're spending on this thing. We need more of the benefit of that thing, for example. Right. So we all make those decisions in an informal way. I'm just trying to put it in an equation so we can look across companies and at different scales and see, is there some sort of rules to it. It's about 1% of the budget. 1% of the revenue goes to the performance engineering budget. That's kind of a rule of thumb that seems to come out. In fact, actually, that's. That particular thing came from a little case study that I informally did. That was another informal conversation with one of my advisors who was involved in setting up the load testing for one of these E commerce companies several years ago. And the inside information there was that the company had a thousand employees, $200 million in revenue, and they were spending $80,000 a month on AWS instances. So their cost of instances was $80,000 a month, which is $960,000 a year. So approximately $1 million a year that they were spending on instances, not licenses, not their engineering staff. The cost of instances was about a million dollars a year, and their revenue was about $200 million a year. Okay. So that means for them, their budget for instances was about half a percentage of their total annual revenue. So there's a rule of thumb that came from that particular data point that I had, and that approximately scales. If you had a $2 billion a year company, would they be willing to spend $10 million a year on instances, not 1 million.

Right.

And. Well, maybe they would.

Right?

Yeah.

I mean, I love this because it gives people in performance engineering or teams to be able to justify maybe why they're doing what they're doing in a way that upper management can understand. Right. If you say, hey, look, we understand this is a cost, but it's actually, when you break it down, it's actually a benefit because in the end, the return is greater than what you're actually investing in in this team to help you with performance.

Exactly right. In a sense. Several of the comments that I got in the LinkedIn conversation were like that. It's like, oh, this is formalizing a way for us to justify our budget to our executives. Right. Well, so most performance engineers are focused on the technology and the software and the techniques and the results they're trying to get. They're focused on the engineering. But there comes a time in the engineering manager's life when he has to go to his boss and, you know, talking about the, what's the budget for next year and are we going to increase your budget or not? And this is where having some additional tools to be able to say what? Quantify the benefit of spending a little more here. And in some cases, look, if we spend it right, we can get benefits that we weren't getting, and we might even be able to cut costs at the same time. That's where you get a massive return, an improvement on your return. If you can get your benefits to go up and at the same time, get your cost to go down, now you've got a really winning proposition. When you go to the CFO or to the VP of engineering who has to make a budget decision, you've given him a really strong argument to take the engineering actions that you want.

Absolutely. And I think someone else made a suggestion that people should do a proof of concept. So even this helps. Once you have the equation in place, you can also do a proof of concept to make sure that the tool you selected based on that equation Is the correct tool? I don't know if they tie in or is that just an extra type of activity?

Well, so. No, I agree with that. So once you have the general equation framework that describes what the benefits are, what the costs are, what's the return and what are the constraints on revenue and budget, then now you can, you can use that framework to do a number of different scenarios. So for example, the manager of this performance engineering team could say, well look, looks like we're being given an extra $200,000. So we could spend that on, you know, burden cost on another engineer and that might give us the additional benefits that we need. Or maybe what we should do is keep the team the same size as it is, but maybe we should use more high performing and potentially more expensive software package for, you know, so upgrade from one tool to another, for example, to get a greater benefit. So those kinds of trade offs. Or should we do more tests, should we spin up instances more often? Do more instances on a, you know, do larger tests and do them more often. Right. So you could do different scenarios, as you just said, and then you could try to quantify for each one of those scenarios how are the benefits affected by that choice. Right. And so that comes now to the couple of the other follow on posts that I did. I said, well, so let's look at it. Suppose you make a decision to go with a high performing paid software tool. Then how are the other costs and benefits affected? So you, you make a decision, I'm going to spend more on C licenses so that on that slide there's a little red arrow. You're spending more on C instances and that increases your costs. However, if because you've got great software, the number of engineers you need to have on your team goes down, it could be that it's net cost reduction, you spend more on software, but you don't have to hire quite as many engineers and so overall your costs might go down even though the cost of licenses goes up. Another reason this came from interesting conversation with the folks at Octopurf. They said, you know, the performance, the efficiency of the instance utilization is better in the paid tools than the free tools. Like they said, basically we've put engineering effort into making our load generator software very, very efficient. More efficient than the free version of JMeter. Well that means that, okay, if you buy their tools then, or you know, you pay for license for their tool, then your cost of instances will go down because you can get the same load with a smaller number of AWS instances.

Right?

So That's a great example where spending more on software licenses reduce your costs on instances or on engineers. Right. So these are the kinds of trade offs where various cost terms compete against each other, or spending more on one reduces the costs of the others. And similarly also spending more on, for example, cost of licenses can increase your benefit of stability. So for example, if you have even better software tools for doing performance testing, you could argue that our likely and you do the tests well, then our likelihood of having a website crash on a Black Friday event has gone down. So that means your benefit of stability, you have a more stable website. So paying more on licenses means you're less likely to have to spend a lot of money repairing the damage of a website failure. So yeah, so by I put beside these equations, little arrows, costs going up are colored red, costs going down are colored green. And similarly, benefits going up are green and benefits going down are red. Right. And so you can see, oh, for this decision I might make these benefits go up and these costs go down or up, what's the net improvement on the return?

Nice. Now Lloyd, you talked a bunch about how you've spoken with all these other companies and I know you touched on it a little bit, but how does Neocortex actually work with these other companies? Or just can you expand a little bit more on what your company does?

Sure, of course, yeah. So, well, we fit into the ecosystem in roughly the same place as where the other cloud providers fit. So like aws, Google Cloud platform and Microsoft Azure, what they do for the load testing companies is that they provide instances that allow the load testing companies to deliver load. So they put load generating software onto the instances and create artificial demand for their website and they measure the response times. So the load generator instances are what's being provided by companies like AWS and Google and Microsoft. Neocortex is in the same position in the ecosystem as where they are. We're another way of providing instances, but with some differences. The thing that's common about the big guys like that, the aws, Microsoft and Azure, is that their instances are servers that are located inside data centers. And what micro, what Neocortex does is we have a worldwide network of mobile phones that are located in people's homes. And we're incidentally, we're paying them for the right to use their processor to do work for our customers. So we have a shared economy business model. We pay them, we're not hijacking their devices, we're doing it with their consent. They rent their processor to us. So we can put our instances on mobile devices in people's homes and use them legitimately as load generators for a load test. Well, that's a very different testing scenario than putting all of your instances or all of your load virtual users of your load test into servers in a data center. We argue that the Neocortic scenario is much more like a real world actual Black Friday event. If you think about so what happens on Black Friday, it's not just that you get a lot of demand on your webpage, it's you get demand coming from millions of people all in their current living environments. Right. It's living rooms in Oklahoma and Boise, Idaho and New Jersey, people's homes all over the place is where the demand is coming from. It's not all coming from US West, Oregon, aws. It's not all coming from one place, it's coming from all over the place. And if you want to deliver the load in a realistic way, and especially important if you want to measure the response times, the most realistic way to do that is to do it using many mobile devices that are distributed all throughout the world. Right. So that's the Neocortex value proposition, is you really should be doing load testing using mobile devices globally distributed and located at last mile endpoints. So you're measuring the true response time. If you measure the response time to a really high performance data center, you're going to get an underestimate of how long it takes for the response time to the living room in Oklahoma. I'm using that phrase. I heard that first from Tom Lunibos, who was the CEO of sosta, which was another. A very famous load testing company, performance engineering company was acquired by Akamai. When I told Tom what we were doing, he said, oh, this is a great idea. I had this idea to make a company to do this in 2011. That's another story I'll go into in a second. But he said basically that's what you're doing here. The reason this is such a great idea and why he came up with it before I did in 2011, he said this is such a great idea because you're measuring the last mile response. And he said the phrase I use for that is you're measuring the response time to a living room in Oklahoma and that is not a data center in Oregon.

Right.

It's a completely different response time. It has all, you know, the last mile, whether it's on WI Fi or not. You have some reduced bandwidth, for example. Right. So if you don't measure it in this realistic way and you use high performance data centers and high performance instances for your load generators, you will necessarily get an underestimate of, of the response time. In the next day I'm going to be posting another thing that's going to show that picture very clearly.

So you were talking about sosta and why it was such a great idea. What I think it's a great idea as well is making sure using real mobile devices from different locations to really emulate a proper load, not just from a discrete location all the time.

Yes, exactly. Right. So it's an important principle and it has implications that are more subtle than might first appear. You know, so basically if just arguing for doing a more careful measurement, if you just express it like that. Well, okay, well you know, you could see someone arguing what's. Yeah. So the act, it's not quite as accurate as if you did it. Right. Doing it with the devices instances inside the data centers. Yeah, what's, what's the difference? Right. How big a difference could it be? Well, actually, here's why this is so important. So the importance of having an accurate measurement of a quantity that you are trying to optimize, basically you've got a very significant situation here where you have a measurement which is inaccurate and in particular is biased in favor of being an underestimate. It's a significant underestimate of the real world user response times. So if you have a biased, not just a noisy measurement, but you have a biased measurement that is inside a control feedback loop. That's the, you know, all kinds of terrible things happen if you put a faulty sensor with a bias in one direction inside an actionable loop. Well, the actionable loop is the performance engineering cycle. Right. If you're measuring, let's think about that. If you're, you're measuring the response time with your load test and we've got a target that this sla, the service level agreement target, says we want to have response time of three seconds. And the load test from the data center is telling us that it's at 2.9 seconds. Okay, great, we've met our goal and there's no point in doing any further engineering. Okay, well if that 2.9 seconds that's calculated and measured from the data center load test is a bad underestimate and the real performance is 4 1/2 seconds mean performance time, then the action of the performance engineer is now the wrong one. Instead of saying great, we've met our target and we can stop doing reductions in response time. We don't have to make our server any faster because it's fast enough, we meet our goal. Okay, well you only meet your goal in the presence of this optimistic underestimate of the response time from a faulty test. If you had a true measure of how what the real user response time is, you'd see no, we need to do more performance engineering. We have to remove another second and a half of response time to get from four and a half seconds down to three seconds. Right. So anytime, the general principle, anytime you have a faulty sensor, especially a biased one inside a control feedback loop, you can get disastrous results. If anyone needs an example of that, the Boeing Max crashes are perfect examples of that. A faulty sensor, that said the nose needs to go up when it was already up high enough. If you put that in a control loop, controlling what you're doing with the wings, you can crash the plane.

Right.

So no, it's a big deal having a faulty sensor inside a control loop.

Absolutely makes sense. Okay Lloyd, before we go, is there one piece of actual advice you would give to someone to help them with their performance engineering testing efforts? And what's the best way to find contact you or learn more about your company?

Well, okay, that's a, that, that's a nice invitation. Well, so let me begin with just take a Look at these LinkedIn posts about the business model, the economics of performance engineering on my LinkedIn page. So my name is Lloyd Watts, you can search for me and find me on, on LinkedIn. I think that's it's a worthwhile effort to really understand what are the costs of performance engineering, what are the benefits and how do they trade against each other. Those relationships are subtle and interesting and worth understanding. I've been surprised at some of the controversy that I've seen about it. And that's why I went to the trouble of making these equations, was to make it clear so everybody could see with clear eyes what the trade offs are. I think that's helpful. So I'd say yeah, check that out. Also there's this follow on set of posts. I've done the first one now on measuring response time and I'm about to post another one which is going to make this point about the possible errors from doing load tests in a particular way. So please check that out too. I think that this is important and it might open people's eyes to something that is common in the industry, data center based performance tests, which I think could be improved and we could actually help the field advance by paying attention to the fact that this is not quite optimal yet and there's things we can do to make that better. And the actionable thing here is companies that get that right, if they do the performance testing in the way that is most accurate, then they will optimize their performance to the benefit of their customer and they will be in a better competitive position. So if you do it right, your company will make more money. And the flip side of that is if you do it wrong and your competitors do it right, then they will make more money and you will be in a zero sum game and lose money. Right? So it's there's millions of dollars on the line here on doing this right, but we have to first understand the principles, what is the inaccuracy in the measurement? And for God's sakes, if it's going to be inside a control loop, it better not be biased thank you Lloyd.

For your Performance Testing awesomeness. Links to up there in value we covered in this episode. Head on over to testguild.com P55 and while you're there, make sure to click on the Try them both Today link under the Exclusive Sponsor section to learn all about SmartBear's two awesome performance testing solutions, LoadUI Ninja and Load UI Pro. And if you like this episode, why not rate us in iTunes ratings, really help in the rankings and help spread performance testing to the masses. So that's it for this episode of the Test Guild Performance and Site Reliability Podcast. I'm Joe. My mission is to help you succeed with creating end to end full stack Performance Testing awesomeness. As always, test everything and keep the good. Cheers.

Thanks for listening to the Test Guild Performance and Site Reliability Podcast. Head on over to testguild.com for full show notes, amazing blog articles, and online testing conferences. Don't forget to subscribe to the Guild to continue your testing journey.