Tuning Up with David Sweet

Welcome to the Test Guild Performance and Site Reliability Podcast where we all get together to learn more about performance testing with your host, Joe Calantonio.

Hey, it's Joe. And welcome to another episode of the Test Guild Performance and Site Reliability Podcast. Today we're going to be talking with David Sweet, all about his upcoming book from Manning Press, Tuning Up From A B Testing to Bayesian Optimization. David Sweet has a PhD in flow physics and has worked as a quantitative trader and as a machine learning engineer at Instagram. He has experimentally tested and tuned trading systems and recommended systems, and he's also an adjunct professor at Yeshiva University. Steve has a lot of experience in this area and I think you can have your eyes open to maybe a new technique or a new process that you can actually add to your software development lifecycle to make your optimized software even better. You don't want to miss this episode. Check it out.

This episode is brought to you by SmartBear. Listen, load testing is tough. Investing in the right tools to automate tests, identify bottlenecks, and resolve issues quickly could save your organization time and money. SmartBear offers a suite of performance tools like Load Ninja, which is a SaaS UI load testing tool, and Loaduipro, an API load testing tool to help teams get full visibility into UI and API performance so you can release and recover faster than ever. Give it a shot. It's free and easy to try. Head on over to smartbear.com solutions forward/performancetesting to learn more.

Hey David, welcome to the Guild.

Hi Joe. Thanks for having me.

Awesome to have you. So this book, like I mentioned in the pre show, may be a little bit outside my comfort zone, but I think the techniques you present in it be really beneficial to folks that are software developers. So just curious to know, before we get into it, David, is there anything I missed in your bio that you want the Guild to know more about?

Oh, I think the bio is complete and thank you for that introduction.

Awesome. Awesome. All right, so why did you write this book? I always ask authors because it takes a long time to write a book, especially when you have mathematics involved, and I'm sure it takes even longer. So why this book?

Well, I saw there's kind of two reasons behind it. One is I saw this connection between what ML engineers were doing at Instagram and what the quantitative high frequency trading engineers were doing in other firms. I'd worked at that. There's this process of ideation, testing something offline and then finally going online and either a B testing it or doing Some kind of parameter tuning. So one I thought, well, there's some audience for this kind of information. The other is I just had my experience over the years of learning these techniques kind of one by one, almost the way I learned them kind of followed the chronology of their development. You know, the development was over, you know, a hundred years. My learning was shorter, but it sort of followed that path. And I just thought, well, there's no. All this stuff isn't in, isn't collected in one place. But it's all addressing this same problem of once you get online, what do you do to sync up your model, your hypothesis about how things supposed to work with reality? How do you get those parameters to the right values? And if there's so many people doing this kind of thing, rather than have them struggle with a zillion resources like I did over a long period of time, maybe I can compact it into one book and make it a little bit easier for a bunch of people.

Awesome. So I guess the next question then is, who's the audience? The book, as I mentioned, this is more towards performance engineers and say reliability engineers. Who are you thinking of when you actually wrote this book?

Well, I had in mind three parties. One is quantitative traders. The other is machine learning engineers at social media companies. Like, I'm thinking Instagram, Facebook, Spotify, Twitter, that kind of thing, LinkedIn. And then the other is software engineers building infrastructure for these kinds of systems who are sensitive to things like latency. For example, I know in high frequency trading we want low latency, we want tight latency distributions. And when you're serving web pages or app updates to people, you're concerned about latency as well. People, they get bored very quickly. You know, sometimes in the hundreds of milliseconds range, they can get frustrated. So.

Absolutely. And that's the exact target audience of this podcast. And performance engineers and reliability engineers, I think they get a lot out of this because they have to create performance software. And like you said, especially when you get towards mission critical applications that need to perform really quickly and also near Black Friday or holidays, the systems need to keep up with the load and stress that are put on the systems. So I guess, you know, the book is broken up into eight main chapters. The first one is system tuning. So I'm curious to know how you explain what is system tuning is at a high level before we dive in a little bit more.

The way I think of it is a by analogy, so that there's this idea that's crept into engineering and software of tuning of Calling parameters knobs. I've seen them referenced in code bases before. I presume it's an analogy to an old FM radio where if you wanted to find a station, you'd sweep the parameter, which was like the frequency you're tuning into by hand by turning the knob left and right until you got the optimal signal to noise ratio, at least to your ear for whatever you want it to listen to. And so that's the analogy. And the problem we're solving in engineering is whatever system you have will have some parameters to it. Maybe it'll have a threshold over, after, you know, over which some signal goes for it to take an action or some weights on multiple signals when you combine them together. Or maybe it's parameters like the size of a cache or how much, you know, how much memory you're going to, you're going to buffer or, you know, various things like this. Systems always have lots of parameters like these and we know qualitatively the purpose these parameters serve, but we don't know the exact values they should take until we actually try them in production. Right. You can get some sense offline, you can simulate things, you can emulate certain processes, you can take production log data and kind of get a rough idea offline. But there's really no substitute for real production with the real thing.

Yeah, it seems to be a bigger trend nowadays, people actually testing in production. When I started off as a performance engineer, we'd build up the system and staging, we'd test it, we go into production. Nothing was ever the same what happened in production. So I see how this definitely applies as well. So I guess along these lines as well, you start off the book talking about different workflows and I'm curious to know where tuneup actually fits into those workflows. Can you talk a little bit about why you started off talking about the workflows? I think you started talking about the software development workflow and then also machine learning workflow and I think the quaint workflow.

Yeah. So in all three cases they follow this pattern like you've described here. The first step is you've got some ideation, you got an idea for a new feature or an improvement or something you think will improve it will improve your system. And the next step is you have to figure out whether you're right or not right. So I think most of the ideas you come up with will probably not work, at least in a mature system, even if they're good ideas. Not because, you know, we're all dumb and coming up with bad ideas, but because, because the systems are complicated, the world is complicated and you kind of have to grow this thick skin and say, well, maybe 10, maybe, you know, 10% of my ideas are going to come to fruition. They're going to mean something. So what do you do? So the next step is do some testing offline where it's cheap, where it's fast, and weed out the worst stuff. But you're not probably not going to get the right answer. You're not going to find the best stuff, right? You can weed out things that are obviously broken, if they're buggy, if they rely on some assumption that doesn't hold true, that kind of thing. And then the final step though is the production testing and so forth. You're going to go run live and maybe you'll do an AB test or some other kind of check on what? On this new idea that you've implemented. Now for the quant, the offline part, that middle step is a backtest. For an ML engineer, the middle step is some kind of simulation or data science type analysis of historical logged data. And in software engineering, offline, what you might do is you might run, you might run live data through the components and measure them. You can measure, for example, if you wanted to measure timings, you can measure timings offline, see how long the software that each piece of the software took and ask, am I improving this? Am I making it faster? Am I making the distribution tighter so it's more reliable? But in all three cases, once you go online, like you're saying, the world changes. The world is bigger and more complicated than what you're doing offline. What you're doing offline is a model, right? At all three cases, it's some model simulation, emulation of the real world. It's going to leave out a lot of the dimensions, it's going to focus on what you think is important. But there could always be more to it that you didn't, you didn't realize was important. And almost always there is.

So yeah, absolutely. Once again, it's another big trend for performance testing. They take a model that they got from production, they put it into the testing area and still somewhat, there's still issues when they go live. Like you said, there's a lot of different things. Maybe we can dive into why that is. But you also mention about good ideas and you've mentioned this a few times. I read it in the book and I also saw a live stream where you said, like Amazon, these bigger companies that have good ideas eventually turn out to be like a very low percentage of things they actually try actually turn it around to make it better software. Can you talk a little bit about why that is, maybe?

Sure. I can say that this kind of started with me like an inkling, like noticing this trend. I sort of describe engineering to my friends. It's like you come up with 10 good ideas and you test them all and maybe one of them works, right? And so I kind of thought, well, you know, what is this? How realistic is that? So I started taking a poll of colleagues here and there. I'd ask them, how often does a new idea that you've come up with make it through testing and be productive in the end? And the answer, Every single person said 1 in 10, except for one guy. He was pretty cynical. Generally speaking, he said 1 in 100, but everybody else said 1 in 10. And so I did some research on this for the book and I found Netflix does AB testing, they do production testing, and their report is about 1 in 10 of these tests lead to acceptance of a new feature component, that kind of thing, into production. Amazon reports 50% and Microsoft reported 33%. I think of their A B tests result in acceptance. And I think this is just what I attribute this to, is that the world is complex and you cannot keep it all in your mind, right? You can't. I feel like there's two layers of models really. Well, the first one is in your head, is where you're going through ideation and you test your ideas. You do like a little simulation in your head. You test your ideas against. What if I ran it this way? What if I ran it that way? Could this cause a problem? And at some point you get to the stage where you feel the idea has passed, your internal tests, your mental model, and then you're ready to go and spend time writing code, right? You want to avoid the cost of writing code and the offline. You have tests that you've written out that are bigger than you can hold in your head, kind of. And you want to pass those tests before you go online. But then online is just even bigger. It's even more complex. You've got beyond the complexity. You've also got non stationarity. Things change over time. That all of that logged data, all of that knowledge that you've stored in your mind about how things work, they could be relevant in the next day if things change. If you're a trader, the fees might change on the exchange, your counterparties might change. If you're running a system that's used by social media system, for example, that's used by a billion users. There might be trends, people respond to the news, they respond to changes in politics and so on. And that might change the way they interact with your product. And just it's not in the data.

So I guess if someone is skeptical they might hear this and go, then why even try? I mean, it seems like not high likelihood that this is even going to work anyway, so why even try? What's the payoff?

You know, that is such a great question. It gets to something so fundamental about decision making. You can ask yourself, you can ask, well, what's the probability of something working? Or you can ask yourself what is the expected value, right, of all of these things? Right. If you try 10 things and they take you some, some finite amount of time, nine of them don't work. So you have a 1 in 10 chance of something working. But that one thing, once you install it, once you incorporate it into your system, it just keeps on giving. It's the gift that keeps on, right? So that the next day it provides more value and the day after and so on and so on and it'll last a long time. And while it's providing that extra value, whether it's more money or more user engagement or lower latency, whatever you get out of it, you can then go and work on the next idea. And so these improvements, they compound over time. And even though there's lots of sort of detritus by the side of the road, of all of all the failed ideas, after working on something for six months or for a year, for 10 years, you can craft something fantastic with incredible performance. Interestingly, it'll be complex enough that you won't know how it works, so it'll be part of the complexity of now. It's not just the environment that's complex, it's your system that's complex. And then you're into any testing all the more.

Right, right. It's almost like compound interest. Like you start off investing and it's like oh, little by little, but in 20 years you look back like oh my gosh. And I guess that's something like Amazon I heard is really big on this testing and like you said, maybe they had a small percentage, but over time they've been around for 20 years now the algorithm has advanced so much now that it's like exponentially growing, I guess based on those tune ups or the changes they've made.

Yes, if you're a new idea, 1 out of 10 of your ideas works. It makes it a 1% improvement to your system. Well, you know, after several years, all those 1% improvements, you know, get double the performance, triple performance, critical performance. It's really fantastic. It's hard, I think, the human brain to kind of to capture and you have to sort of trust that. Trust that it's going to work. Yeah, I mean, the best sort of approach is, you know, put your head down and keep grinding out new ideas and testing them and look back every once in a while and remind yourself that it works.

Absolutely. So is there a framework someone can use? You talk about experiments a lot in the book. So in context, how do experiments work when someone's trying to create a model or try to tune their systems?

Okay, so the most basic experiment and the experimental type is an AB test. It's also the most, I saw, the most reliable, and I think the most easy to communicate to other people. The results are the most easy to communicate because what you do is you take two systems, your original system as it's running A, and then some modified version, call that B, and you run them side by side, or you run them, or you alternate them or randomly choose between them, but in the end you have some measure of which one did better, which one gave better latency, better revenue, higher user engagement. And typically, from what I've seen, companies will build their own systems to. The larger companies will build their own systems to do this. And in the old days you had to. Although now there are companies that offer AB testing commercial products. So they'll do the randomization, they'll tell you about how much replication to do, and then they'll compute all the metrics and analysis at the end for you.

So for AB testing, is that the same as feature flags, where you release a small amount of your code to customers and then based on how it performs, you roll back? Or is it something different than that?

That's exactly the kind of thing you'd want to do. So, and the feature flags are a great way to implement this. That way you can affect the production system without lots of interaction. Right. Just change the flag. And so. So, yeah, so what you do is roll out the customers. So you can test for multiple things. You can test just simply to see whether it works, but you can also test measure very specifically the ways in which the customers appreciate or don't appreciate that new feature. Like, do they spend more time using the app, do they buy more, do they click through on an ad, whatever your metrics are that you're looking to improve.

Awesome. And I know you cover this a little bit. We're talking about trying to measure one thing is hard. But then you mentioned a few times measuring multiple things. How do you know how to measure all these other things in context to know what is actually moving the needle?

There's two ways in which you wind up measuring multiple things. One is, do you have multiple ideas that you want to compare? Or say you have a parameter that can run between from 1 to 10, so it's 10 possibilities. So maybe you need to run 10 different tests. Or maybe you slim it down and you run you test 1, 3, 5, 7 and 9, for example, and see which one does best. The other dimension along which you could measure multiple things is you have multiple objectives to meet. So maybe you want to increase revenue, but you don't want to decrease amount of time spent on the site. Or maybe you want to increase engagement. But maybe there are indicators of like negative kinds of engagements, you know, people bullying, where people you know, or otherwise insulting people, that kind of thing, or passing around misinformation or other kinds of inappropriate information on a website. So that could lead to more engagement, but it's not the kind of engagement that you want for your product and for your users, right? So typically when you run an A B test, you're going to look at, you're going to have, maybe have some most important metric, but you're going to want to look at all of your metrics all the time. So in the idealized a B test, you have one metric, you run a T test, you've picked a threshold beforehand for this T value and you say, well, if it's statistically significant to such and such a level, then I'll be willing to accept it. And then you do and you move on and it's kind of automatic. In practice, I think each A B test is more of a discussion, right? It's a discussion about multiple metrics. Have we pulled back on some, Are we trading off improvements in one direction for regressions in another direction? And is that a good trade off? Right. So perhaps if you're looking at, I know in quantitative trading this looking at latency is an important thing. And you might say, well, we've lowered our median latency, but our tails have gotten larger. So, well, we can't have that, so maybe we want to. So that's not a good trade off, but maybe we brought the tails in and the median's gone up a little bit. For some situations that would be terrible, some situations that would be a good trade off. And it really depends and you really need to look at the metrics. And typically also you're going to discuss them with multiple people. Like, the more value that's on the line, the more stakeholders, so to speak, you want to have in on a decision like that.

Absolutely. So are there any best practices for a B testing? I think you spoke about bias, how to create an unbiased design. I guess if you start off with a biased design to begin with, they're wasting a lot of time. So any best practices on how to avoid an unbiased design or a biased design?

I mean, there's an ideal situation where you know everything about your system and you can identify the biases and make sure that they're not represented. It's almost never possible, and it's very simple as possible. So that the solution is randomization. Right. So the way to get rid of bias in your system is, let's say you have, you know, you have 100 users and you've got some new idea. What you could do is you could alphabetize the users and give the first half, expose the first half to your current system and expose the second half to your new idea. But how do you know what's being represented by these, the first or second half of the Alphabet? Another way to do it, another, you know, easy way to roll something out, is to expose your current system to one geography. Say the west coast gets A and the east coast gets B. But they might have different preferences, east and west. And this will make it complicated to understand whether any changes were due would apply globally, whether they're due to your new change or whether they're due to changes in the behaviors of the people on the east coast and west coast and so on. So the randomization just says flip a coin. And every time you are going to do an exposure, every time you're going to, a user's going to come in and interact with your application, expose them randomly to either A or B. And if that makes too much noise, maybe you want to assign users permanently to A and permanently to B, but you'll do it randomly so that you're not going to be fooled by their demographics or a specific user's persistent behaviors. You'll just break all those correlations is what the randomization is.

Nice. Now, this is all in chapter two. You go into detail. You also talk about false negatives and false positives. Could you talk a little bit about that before we go on to chapter three? A little bit, sure.

So false positive is when you say something looks good, but it actually isn't. A false negative is when you say something looks bad, but it actually isn't. So these are the two mistakes you make. The one mistake is you run a test and you say, oh, B is better. I'm going to accept it, but actually you were wrong just because of noise. And the other is you say B wasn't better, but you were wrong. So in the first case where you accept something new into the system that shouldn't be there, you're damaging the system. And over the long term, just because you didn't measure the damage in your tests, which is over a short period of time, over the long period of time, that damage will show up. It's like if you ran the test for longer, for long enough, eventually you'd see that B was worse. But if you put it in production, it's like you're running that test for an infinite amount of time for the rest of the lifetime of your system. So the damage from B will show up. So that's a false positive. False negative. On the flip side is if you had a good idea B, but then you threw it out there, you aren't damaging the system at all, but you're missing an opportunity to improve it. And so generally speaking, the false positives where you're actively damaging the system, they're considered worse. And stricter limits are put on your willingness to accept false positives than false negatives. So if you didn't make an extra dollar tomorrow, that wouldn't be so bad as if you made a dollar less tomorrow. So higher we just put on the false positive on protecting you from false positives. Now, the question you might want to ask is, why not just say I'm not willing to accept any false positives or any false negatives. And the problem is, of course, that it's probabilistic. And the best you can do is just set a limit, say 5%, like the most false positives in one half, for example, or 20% false negatives. That those are typical limits.

Very cool. So I guess once you have a B testing in place, what's the next step? I think chapter three and four kind of dive into that. Chapter two is more about multiples, how to handle multiple changes to the system while maximizing business metrics. So is that running multiple AB testing at the same time? Know how to handle that?

Yeah. Well, so the way I developed the book is AB testing is like the foundation for everything. And that chapter teaches about randomization, replication, meaning running the experiment over and over and over again until you get a Nice tight error bars. So you get these probabilistic bounds on false positives and false negatives and then everything else expands upon that and tries to make the whole process more efficient. Because as an experimenter you want to run the experiment for as short a time as possible so you can get the information you need to make your decision. Testing is a cost because in any a B test you could be exposing your system to something that's worse. So I see your overall performance of the system will on average decrease and especially take into account that what we were saying before, that most of your ideas are probably not going to be good for the system. So your tests really like 9 times out of 10 are going to cause harm to the system. So there are costs, so there better be a long term payoff like so that one that works better payoff for the long term. But at the same time you want to keep the testing time as short as possible. And it does two things. One is it's less harm to your system as you're doing the testing of things that turned out not to work. But the other is you get a sequence of tests and you talked about compounding before that the shorter the cycle of that sequence, the more cycles you can do per unit time, per month or per quarter, that kind of thing. And the faster you compound, the faster you make that compounding work for you. So yeah, so experimentation is a cost. We pay it because we want this compounding effect, we want these improvements. So all of the other methods in the book look at ways to make experimentation faster and more efficient. The multi Armed Bandits look at ways to say, well, while we're testing, maybe we can pay attention to the metrics we're trying to improve. And if we see that B is starting to look better than A, maybe we'll shift a few of the users over to B. Right. We'll be a little more dynamic and we'll adapt to the data as it comes in. Whereas AB testing is just going to wait till the very end to make any kind of decision. The multi Armed Bandits will kind of make partial decisions along the way. They think of it. So a B testing is kind of black and white. Either A or B Multi Arm Bandits sees all the gray areas in between as well, willing to slide users over slowly to be if they're doing well. And the Contextual Bandit builds upon that even further. The Contextual Bandits are really interesting. They're willing to look at multiple options, but they look at the decisions being made of whether to use two different versions of the system, potentially the contextual bandit says this. It says, well, what if we left both versions in the system? What if we had multiple versions of the system and we could change dynamically based on characteristics of the user. So whereas before we were doing randomization because we want to ignore the demographics of the user, contextual bandits take the opposite, not the opposite approach, but somewhat opposite first in that they say, let me know about the user, let me know all the features that represent the user and I'll choose whether it's going to be A or B. And you might have other versions, other features, C and D, other options, and I'll choose those based on a model that I build of the user. So you have like a regression that says, based on these user features, what's the best option to choose A, B, C or D? And I'll do that dynamically on the fly every time I see a new user. And so with these systems, you can tune hundreds, thousands, even millions of parameters in these models based on your logged data. It's a very special situation though, where the metric has to be a short term metric, some called the reward, and you get the reward immediately after you make the decision. So for example, you might say, I'm going to show an image to a user and see whether they click on it. You know, right away when they click on it. That's the whole reward. Something you couldn't do quite as easily with a contextual bandit is let's say you show a user an ad and you want to know if sometime in the next month they purchase the product or not. It's a little bit harder. What gets even harder is aggregate metrics. Like if I change the way I show ads, do users in aggregate spend more or less time on the site? And that's not the kind of thing that's great for contextual bandit. So what you see is these methods can become more powerful. Like contextual bandit can tune many, many parameters, right? And do a really good job of it. And precise. They can make lots of decisions precisely based on the user, but it becomes more specialized, Right. And so if you look at, I talk about something called response surface modeling, which is a little bit older, you can't tune as many parameters, but it's really good for continuous parameters. In that I mentioned before, let's say you had some threshold and it had to take a value between 1 and 10. Let's say it was continuous between 1 and 10 and could take could be 1 and a half, could be 3.7 or whatever. Now a response Surface model works great with that kind of parameter. It'll build a model, it'll allow you to find the optimal parameter even if you haven't measured it. Maybe you measure 1, 3, 5 and 7 and it can tell you 4 and a half is the best answer without you having to go measure four and a half beforehand, without you having to hypothesize that four and a half is a good number. So again, it specializes, but it becomes more powerful than a B testing. You can get these answers in a shorter period of time, but you specialize. So this is sort of the theme of the book is that each of these new methods builds upon the other, specializes a little bit and gives you some efficiency so you spend less time running experiments.

Awesome. So David, I just want to talk a little bit about your time on Instagram. How much of this did you actually implement at Instagram and was there a process in place or was it new and something that you had to get the team more involved in? Like if someone's listening, like how do people at companies like Instagram succeed with it and how can I, my company implement it and succeed with it?

The way Instagram works and actually as part of Facebook, there's all of Facebook has a team and been around for a long time dedicated to building experimental tools for use. A B testing Bayesian optimization tools. They've released their internal software as open source, something called Axe. And so Axe combines ideas of a B testing, multi arm bandits and whatnot into Bayesian optimization. Really combines those ideas. This is a Bayesian optimizer axes but you can use it to run an A B test. You could use it to run a multi armed bandit. It's a powerful piece of software.

Nice. So I guess also that I would think that most companies, if they aren't doing this, have to do this in order to be competitive. Machine learning is only going to help make it easier, I guess, to do this type of experiments and optimization. I would assume.

Yeah, I think it's due rigor for all the large company tech companies that you could think of, Google, Apple, Twitter, LinkedIn, most of them have published papers on the types of systems they use. And Google produces something called Vizier, which is their competitor, their version of Axe, their version of a Bayesian optimizer. And their paper where they published was cool because they said they had used the algorithm to tune a recipe for chocolate chip cookies by serving the cookies in the cafeteria at Google hq or maybe it was in New York, I don't know. But they served the cookies and Got feedback from the eaters of the users of the cookies. The eaters. And then we're able to run the Bayesian optimization make better and better cookies over time.

So that's an experiment I definitely could participate in there for sure.

Yes.

So talking about that, participating, if someone's listening to this episode, maybe they're not really strong in mathematics per se or statistics. What skills do they need in order to get involved in a B testing in general?

Yeah, I think this is something that any engineer can participate in. And your level of mathematics, I guess maybe will determine what level you participate. Certainly if you're at a big company or even a smaller company that has these tools now, typically these will be built by specialists, so eventually will be built by specialists. You can be a user of these tools without deeply understanding the mathematics behind them. That being said, what I try to do with the book was to write at a level for. For non specialists. Right. For. For users of these tools to understand what's going on enough to be better users of the tools. Right. So sometimes it's fine to just know superficially what's going on, but as an engineer, one, you're usually curious and two, you're usually going to do a better job whatever tool you have, if you understand how it's working underneath the hood, at least to some extent. And so I wrote the book as for someone who knows Python and basically has like high school mathematics, which is probably mostly all of us. I guess maybe the mathematics is rusty, but it's not gone, right?

Absolutely. Okay, David, before we go, is there one piece of actionable advice you can give to someone to help them with their ab testing, tune up experiments? And what's the best way to find contact you and get our hands on your book?

Oh, I say my advice always measure early measure often only the real world knows what really works. And if you want to see the book, you can go to Manning's site. They've got it in their early access program. So there's five chapters online and you can read those now and the other three chapters will be out soon.

Thanks again for your performance testing awesomeness links to everything of value we covered in this episode. Head on over to testguild.com P68. And while you're there, make sure to click on the try them both today link under the exclusive sponsor section to learn all about SmartBear's two awesome performance test tool solutions, Load Ninja and Load UI Pro. And if the show has helped you in any way, why not rate and review it in iTunes. Reviews really do matter in the rankings of the show and I read each and every one of them. So that's it for this episode of the Test Guild Performance and Site Reliability Podcast. I'm Joe. My mission is to help you succeed with creating end to end full stack performance testing awesomeness. As always, test everything and keep the good. Cheers.

Thanks for listening to the Test Guild Performance and Site Reliability Podcast. Head on over to testguild.com for full show notes, amazing blog articles, and online testing conferences. Don't forget to subscribe to the Guild to continue your testing journey.