Performance Testing Pipes and Balloon Animals with Scott Barber and Mark Tomlinson

Five, four, three, two, one.

Drop that.

Welcome to the Test Guild Performance Podcast where we get together to learn more about performance testing and site reliability with your host, Joe Colantonio.

Hey, it's Joe. And welcome to another episode of the Performance Testing and Site Reliability Podcast. Today I'm going to share with you a small piece of the roundtable we had at the 2019 PERF Guild Conference. Perf Guild is a annual conference I put on every year, dedicated 100% to performance testing. Just want to give you a flavor for the types of questions folks were asking, especially as we get into the new year, how you can prepare for your performance testing efforts and even more importantly as we get into the holiday season. So I'll be sharing with you some awesome answers from Mark Tomlinson and Scott Barber. Scott Barber is the CTO of Perf Test plus and is a recognized expert in performance testing and analysis. He combines experience and a passion for solving performance problems with a context driven approach that he sometimes calls a scientific art to produce accurate results. Scott is also an international keynote speaker, trainer, consultant and writer of articles for a variety of publications. He's also one of the very first folks I ever saw talking about performance testing. I actually started my career being a performance testing engineer and and this is one of the first training sessions I ever had was from Scott Bobber. And we also have Mark Tomlinson. Mark is awesome as well. Mark is a performance engineering and software testing consultant who's worked for a bunch of different companies like Microsoft and Hewlett Packard and is the host of one of my favorite podcasts, PerfBytes. So he has amassed a broad experience with real world testing scenarios of large and complex systems. And it was an honor to have both Mark and Scott part of the PERF Guild 2019 conference. And here's a playback of how they answered some of the top questions we got asked at the 2019 conference during the Ask Us Anything Roundtable. Like using functional automation scripts for performance testing the future of performance testing Ways to get started with performance testing how to get your team more involved with performance testing how to get started with performance testing if you don't have requirements and a whole bunch more. So listen up and discover how performance testing is like pipes and balloon animals. Check it out. Let's face it, performance testing is tough. Traditional load testing tools create scripts that are bogged down in data that's hard to read and require a lot of work. Even for simple playback. Load Ninja cuts out the dirty work by using AI to inspect and debug your code. Almost instantly. No coding necessary. No changing of hands or relays between teams either. Give it a shot. It's free and easy to try. Head on over to testguild.com smartbear and click on the link to learn more.

First question is from Rashmi. They want to know any guidance around setting up test data and thresholds for performance.

Test, test data. That's probably the most complex part. Given the more complex the system, the more complex the data interaction can be. So I would say just to start with test data, you can learn a lot from the functional test team and you can learn a lot from the business analyst as to how interdependent the data is for the transactions that you want to load. So if you're analyzing, let's say 15 steps within a flow of a transaction and you may have data in step three and four that have to be correlated with earlier steps. Right. So aside from very low level transport level data correlation, you might need invoices, accounts, customer IDs, specific kind of data to just drive proper load across the database itself. So across the tables. So I think I, I think to get started with that, you kind of, if you think about your workload model, go back and start talking with either functional testers or your colleagues, maybe even business analysts and start analyzing. Hey guys, what kind of data do you use to drive these transactions? And then you probably have to grow that data to become much bigger for a load test. At least that's a good place to start, I think.

Yeah, I'll say that, you know, one of the key elements is the interdependencies of the data, right? Because it's, I'm going to say, easy enough to come up with a data set or a scenario.

Now we're talking about a data set that's going to handle a whole bunch of users in multiple scenarios. And so you're often consuming data or changing data during the course of your test. So that's one of the, you know, it's not like functional testers don't have to think about this. They do, but they think about it in between tests. We have to think about it in the middle of our test. So what's the guidance? The guidance is to really know your data. But the key is not just to think about what does it take to make this transaction work, but it's equally as important to think about as this data changes, am I impacting my other users or my other scenarios?

The next question is, is from Tom, and this is actually, I was talking to a few vendors and they say they speak to their customers and they tell them they don't even do performance testing anymore. They just auto scale everything. So this is kind of along the same lines, I think, and that question is, do you see performance and test engineering as a solid, viable specialty for the future, or should we as professionals be expanding into automation and other parts of test and development?

Over 20 years now, I've seen what I consider the performance field change in focus area dramatically. I've seen when we were talking mostly about capacity planning and then when we were talking about code optimization and then talking about architecting for performance. And now we're in a new space where there's. And it's gone full circle. I remember in, I don't know, probably 2002, right, when all of a sudden people didn't want to do performance testing anymore because adding another web server was cheap. Hey, I just auto scale on the cloud, no problem. And to a degree, if, and this is the key, if you truly understand.

The architecture and the way all the pieces fit together.

You can say, hey, this piece over here I don't have to worry too much about. Or maybe I only have to test it initially to make sure I've implemented it correctly. Right.

But there's always going to be the interconnectivity of these pieces. Right. And one of the things that.

Served me well, right? I wasn't a computer scientist. My first degree is in civil engineering. Right. To me it's all water through pipes. And the first thing that we learned in the first of four semesters of hydraulics and hydrodynamics that I took, right. There is always.

One critical bottleneck in a system unless it is perfectly.

Optimized. And the likelihood of that degree of perfect optimization, especially in our field, is magnificently close to zero.

Because.

The systems that we're talking about don't do the same, exactly the same thing every millisecond of every day. So things change. So there's always going to be a slowest spot. And for as long as that is true, then there's going to be some place for us in software, computer information, however you want to think about it. Systems. Is it going to look the same? I doubt it. It's the thought process is the understanding that you gain from thinking about systems and the way information flows and where the bottlenecks come in and what slows things down that is going to persist.

Right. I'd love to say we're going to have the same titles and sit in the same places. Can't say that. But I can't say that there's a place.

Awesome. And Mark, I know you're dying to say something. Lay it on me.

Well, I think as long as there's crappy code, there's going to be testing and the majority of performance problems come from how early can I find crappy code that doesn't match, as Scott said, the infrastructure. There's going to be one constraint within a system of interconnected components. And if you can code exactly around that constraint and you can get the transaction work done without exploiting or running into that particular bottleneck, then you've achieved some perfect match. Not just the perfect elimination or balance of that, but the further our engineers, meaning people who write code, get away from the infrastructure in these auto scaling cloud, somewhere down the street some guy in a garage running my hardware, then the more our minds are not thinking about I need to make this code work in this system from a logical to a physical translation across the architecture. So if you're crappy at writing code, you're going to be really crappy. If you have no idea what that infrastructure looks like, you throw auto scaling at it. I get you. The CFO does not have an auto scaling budget, so eventually you can get auto scaling. And then someone's going to come around and say hey, you need to learn how to write better code. Because these serverless functions are going right through the roof in terms of cost. And I see that in two of the key customers I'm working with right now, including my own job. It's like hey, we start publishing the daily numbers of cost within different environments and all it takes is one dev pretending that they know how to do load testing somewhere with a Cosmodb on Azure and boom, there's $10,000 bill at the end of the month. It happens more and more frequently the more developer engineers get away from understanding architecture. So I think there's a brilliant and bright future of helping people not shoot themselves in the foot, which is no different than the last 20 some years.

So there's a questionnaire, I deleted it by mistake, but hopefully I remember it is that they're seeing a lot more trends with being asked to use Selenium functional tests for performance tests. Do you have any thoughts around this or why this would happen or is that a successful strategy? I think was the gist of the question.

Proceed with caution.

It can be done. And if you can do it within the definition of your scenario, meaning maybe you don't need to do exhaustive super scalable hammering tests, you can save a Lot of energy and a lot of time in just reusing some of those scripts. And there's some good practices around which scripts are viable for load and which ones aren't. But for the most part that fatter driver that you would see with a full Selenium or Micro Focus hp, we did the true client thing. There was some other stuff in BrowserMob, there was some stuff within Sauce Labs that drives the same, I think a webdriver based piece and some others. And you can see tools like JMeter that have the WebDriver plugin as well. It's great, it's fine. And it's an age old technique from can I get that last mile fat client performance latency. Right? Can I. Here's the network and the server latency and then whoops, I'm actually rendering some crazy stupid node app Java JavaScript pieces. Just hammering the client so that those are good techniques. If you have to run a really high amount of load, you'll probably spend a lot on load generators. Not that you shouldn't do that. Go ahead and proceed with caution. But it's not a one to one. You can't say I'm just gonna. I think a lot of people misunderstood the avoidance of correlation and the complex scripting at the transport level and say oh, I can just do this in Selenium and I don't have to do any of that fancy correlation stuff. And that was a problem for about 10 years from 2005 to 2015. And then applications started becoming more discreet. So when you did web service or data layer based tests, they were fairly independent from the client. But during that, at least in my experience, that little gap of about 10 years, applications were still intertwined in that middle client to data presentation layer to data layer app. And so correlation was a nightmare. And a lot of customers are doing one or the other now, they're oh, we're just going to hit the web services, that's great. Or we're going to actually do some full client end to end. But we don't run thousands of full client type load pieces. So I say proceed with caution and if it works for you, go for it. No, not, not an issue.

I'm going to add on to that and I agree with all the things you've said. One of the slippery slopes that I, that I've seen, aside from the, the aspects that Mark said is once you start using and I don't tool independent, once you start using your it's called functional automation Scripts for performance testing.

There'S this Dangerous slope where all of a sudden the answer is, oh, here, here's all my regression tests and I'm just gonna plop them in and throw a whole bunch of users at it and it's just gonna do what it does. And here's what you end up getting. You end up getting a gazillion logins, three searches, three orders, three admin functions, and it looks absolutely nothing like what you're going to see in production. And again, it doesn't mean it's not. There's no value there. But what it does mean is you need to be really careful about understanding.

What your test is doing, what you're trying to learn, and whether or not that test gives you the data that you need to learn or tune or.

Fix.

The pieces that you're interested in. And again, you know, it goes along with what Mark said. Proceed with caution. Not saying it can't work, not saying I haven't seen it work, but I've seen a whole lot of gross oversimplifications that gave people false confidence. And it's not just in this scenario, right? Design your test.

For a purpose.

Use it for that purpose, and when you have a new purpose, design a new test. Maybe you can reuse your scripts, but don't assume that you can use the same test for multiple purposes. Put timers in your functional tests, stick them in there. Every time you run the test, log the time so that every time you run, you can look at, look at a trend line and say, did it change? Now that might not be the right time, it might not hurt the production time, it might not be under load, but every time that you get a chance, collect your timing or your resource allocation or your memory use whatever it is, collect it, stick it on a trend with the build number in the machine, look at it on a trend line and say, wow, what happened in build 426? Something was off, right? Why not? Right? It's information. Don't lose information. Because maybe scripting with a functional tool isn't the best for performance.

But that doesn't mean that you can't get valuable information out of it.

Next question is from Tom, and Tom wants to know. My experience is in HP Micro focus load tests. Given only so much time in a day, what are key essential tools that Test monitoring analysis, JMeter, EPM, Splunk languages, JavaScript, Python R, and all the skills to grow and to have impact in the performance field.

For the future of everything you mentioned there, I recently had a great conversation with a couple other performance folks about the core tools still being at the heart of the heart of the tools. I can generate load and I can watch stuff. And there's been explosions in both parts of being able to do that technologically, particularly on the watch stuff side. Right. I mean, we figure early profiling, diagnostics, type solutions were maybe 20 years ago. Let's see, it's 20, 1999. We got, you know, performance, the Wiley stuff that all was kind of brewing and popping and finally becoming real products. But the ability to watch stuff now from Splunk is extending to that old John Allspaugh thing from Etsy back in his early web ops days. If it moves, graph it so you can see how it moves. Scott just mentioned trending, which is being able to see stuff move. And every one of the people I talk to is like, take the next step into R. Take the next step into predicting the future. Which is very hard to do when your code churn is and code rate of change is really fast. If the code stays the same and the only thing that changes is the user load, then you can start making some more solid production. So this is a really tricky piece. I recommend looking at Prophet, I think Profit, I think is the Facebook algorithm. They have it for R and Python. So you can look at. If you have a time series database that you can dump everything into. Splunk is okay for time series. Look at Cassandra or Influx is also a great time series database for these tons and tons of metrics. And then start looking at R. And some of the predictive side of things would be my advice for the future of what your work is going to do is being able to.

Provide meaning from a given set of measures or a set of test results. If it's not truly hypothesis based, you're looking at a trend of things going in a certain direction. I think on the load side of things, the stuff that's becoming really interesting is applying AI and machine learning to that element. Right. So actually being able to record and generate load, what kind of traffic should we create? So just take a bunch of traffic, hand it to a supervised engine or even a supervised engine and come up with interesting scripts and test cases to generate chaos load. It was great hearing the stuff about Gremlin on the chaos stuff. Chaos engineering is really cool. So I also see other types of drivers. There's always going to be new protocols and new kinds of ways to generate load and that's still extending and I see that getting easier as well. But I still think we're branching out from those two core parts of a tool and innovating on those for the future.

SCOTT it's all about really understanding what you're trying to accomplish and how to communicate it. I lost count a long time ago of the number of tools that I've used. And in the beginning, learning a new tool was hard, right? I'm not gonna sugarcoat that. But after a while, here's how it kind of panned out is hey, this is a load generation tool. I'm generating rest traffic, I'm generating, you know, HTTP traffic or whatever it is, I know what that looks like, I know how it's formulated.

I'll figure out the tool in a, you know, an afternoon. And not because I'm some super smart guy, but because after the third one you figure out that you know what, I'm just sending network traffic. It's just traffic. I know what the traffic looks like. I can figure out how to make a piece of software generate the traffic or not if it's, you know, unsupported or whatever. And it's kind of the same thing with, with an ella, you're always going to get new tools and if you over focus on the tool, you're going to lose that focus on what am I really doing with this tool, right? A craftsman, right? He's got his favorite hammer, Guaranteed he's got his favorite hammer. But I'm pretty sure you can hand him pretty much any hammer and he can get the job done because he knows what he's doing, he knows how to swing the hammer, he knows how to drive the nail, right? And I think that we need to be careful not to get too focused on what are my tools and rather focus on what am I really doing with these tools and how do I make that work and how do I get value out of that and then the tool kind of becomes an overlay. I realize it's over simplistic but you know, learn two or three after that. Just keep up with wherever your company.

Or your application is going to add on to that. Scott I think it could be also 80, 20 in Tom's question, like you say when you start out, pick one tool because the learning curve of all of the non tool specific stuff, just concern yourself with learning one tool and get really good at that and keep your mind open to I'm learning all sorts of stuff that's kind of cool agnostic. So at the beginning of your career it might be 80% JMeter and 20% general learning, whatever. But as you proceed five, 10 years down the road, it'll flip, it'll be 20%. Whatever tool doesn't take that much out of my brain. But the 80% of the problems now that you go after are communicating, like you say, meaning how did I present the visualization to get the most efficient reaction from the support people who should be freaking out that the site's going to crash? You know, how do I excite people about the information so that they can do good stuff or resign and run screaming from the building? Whatever they decide is up to them. But that challenge is the 80% when you get to a senior level. So your tool investigations or picking up and learning new tools will change as you sort of become a master in the general of all those kinds of things. Scott put it perfectly.

All right, next question. This one's from Michael. Wants to know, do you see value in large scale full integrated performance test in a pipeline driven microservices architecture? Many people think that if all the pieces are performant that there's nothing to worry about. But I believe this is a recipe for disaster. What's your take on this?

I'm going to go back to Scott Barber and Civil Engineering and I believe it's hydrodynamics of some kind. If you take balloon animals and you squeeze different parts of a balloon animal, you've got one microservice over here and one microservice over here and one over there and one over there, and somewhere in between there's a limited number of database connections, limited number of threads, limited number of infrastructure. And if you never test them all together, just like they're going to run in production, at least in some capacity now, you may be able to narrow off certain slices and organize them by risk area, organize them by impact, organize them by sensitivity to new code. Take Scott's FIPLOTS acronym, which is fantastic. I mean, you might be able to group them into smaller areas to make it easier or smaller. But at some point, if they're running on shared infrastructure in any way you want, all of the components that are sharing that infrastructure, most common one is a data source. You also may have physical infrastructure, machinery and network, of course is the most ubiquitous shared thing. Right? It's like the freeways. And you know, as society evolves, we have more and more traffic on freeways. There's still traffic jams. Even though we've tested each individual car, each individual car passes all of the tests. But we put them all on the road at the same time and we're stuck in a traffic jam and nobody's making money and There's a lot of pollution. So the idea that you can test individually and not test integrated and be successful is a polite fallacy that you, that you may be in a fantasy land in a honeymoon period and everything does work great. But I still recommend where you see a critical shared component that's being used more intensely than maybe more independent services. That's where I look for pulling the test together and doing some integrated load running at the same time.

Integration points are scary, period. And now look, I love microservices.

I'm actually working in a place right now where we talk about mini services because they're not quite micro, right? But here's the interesting thing, right? Even where I'm working right now, we have over the last year implemented this procedure, if you want to call it that, where we test every service kind of independently.

Or it ever gets integrated, and that's fantastic. But you know what? You start building these services that do these little itty bitty things and you know what happens after a while? Somebody calls this service that calls this service that calls this one that calls that one, that goes to the database, that goes into a queue, that goes back to the database, that goes to this service, that goes to that service. Next thing you know you got 47 services in a string and it's taking you three minutes to, you know, say, hi Scott, thanks for logging in. And it happens. And I don't care how fast you make an individual service, if you've got 47 of them strung next to each other, it's going to take a while. So there has to be some notion of the system.

Rafael wants to know, as a beginner performance engineer, how can I introduce performance testing engineering practices at my current job project? Where would I start? Where would you think they would start?

Make friends with a well respected developer architect and really get them interested. And I know that in some ways it's more psychology than we techies want it to be. But here's the reality, right? Find somebody who has respect in the company people listen to. Because if you're, if you're new or you're trying to inject something and you don't already have a reputation, people are going to go, yeah, yeah, yeah, that sounds cool. But whatever you go, you find that person, you get them interested and you don't get them interested by saying, hey, we ought to do performance engineering.

You run some little tests and okay, look, I'm a bit of a rebel. Maybe you don't have permission, you beg forgiveness later, whatever, but you show them this Cool data, right? You find something with some cool data and you go, hey, is this interesting to you? We could do more of this. And you get that champion. Next thing you know, one dev is integrating some low level tests at their level and they tell their friends, hey, look at this, I caught this bug. I didn't have to come in this time when we went live and it crashed and it just kind of catches on fire. I'd love for, I would love, after years of consulting, I would love to have a better approach. But I tell as close to the CTO as you can, get them excited and feed them things that keep them excited and it'll happen.

Yeah, I think if you're just getting started, you may be the only one that actually has some interest in performance. It's a challenging kind of thinking and a challenging tool landscape. And it's actually challenging from a process standpoint. And so a lot of people just kind of step away or they drop out of doing it for those reasons. I think Scott's point is if you can find a developer engineer or even like a technical product manager, even someone in infrastructure, another engineer who also has sort of that strange, masochistic kind of, I want to go dig deep in these terrible things and have some kinship and support, that's good. I think secondarily, if you're getting started, try to develop small assets like here's the top 10 list. And you can apply it to any app or database and say, here's the top 10 worst queries, here's the top 10 worst network segments, here's the top 10 worst method calls, here's the top 10 worst pages. And you could do a top 10 list to help bring product owners and other engineers like, hey, who maintains that? Like, what's the slowest? And then you can get this sort of social experience being you don't want to be in the top 10 list. And you might drive other people to be like, wow, I'm, I'm just studying and getting some measures in place. And then someone will say, wait a minute, you put me at number three, I'm not number three. And they go back in their sprint and they do a bunch of extra work and you're, you're, but you're doing that kind of communication is a gap for a lot of engineering. So if you can put an information radiator together, just a confluence page with, hey, here's the top 10 worst performing methods, here's the top 10 worst performing tables, you'll make some new friends because you're showing Concern for something, and that may be one way to go. And then you're going to learn a whole bunch of tools to help you deliver on those kinds of communication devices, which is cool.

Okay, another question, this one's from Perfdrummer. Perfect Drummer wants to know how to deal in a situation where there's no clear requirements, no usage patterns to learn from. Is a spot where you enter performance engineering engagement. So where you have no data, if it's a new app or looking at logs, where would you start for a new engineering engagement when you don't have that type of information?

You know, I've been going, I think all three of us have seen this throughout our career, right? It's real easy to tell somebody it's not performant when they say we want sub second pages or whatever it is.

But here's the truth. Even when you've got requirements, and this is the kicker, even when you've got them.

Unless they come from something legal, contract, they're just a swag anyway.

So the truth of the matter is it's all about the data. It's all about the trends and the feel, right? Okay, I've got no information. Well, you know what? I can collect a little information on my app, even if it's with a stopwatch. And really, you don't need stopwatches anymore because you got, you know, your, your developer tools and your browsers and whatnot. You can collect a little data, anecdotal data, but some data, right? You can say, wow, look at this. We're, we're showing four seconds. Our competitors are showing two seconds. That's interesting, right? Maybe I want to get some measurements and do some comparisons or, or maybe I'm really building a green field path, right? There's nothing to measure yet. There's no requirements. There's, there's, there's no idea what it ought to be. But you know what you've got? You've got two things. You've got, you've got the ability to start collecting data.

And you've got you and your friends, co workers, whatever, who are using, testing, whatever, and you've got your own intrinsic sense. If it feels slow to you on your network, on your laptop, in your garage, or your.

Skyscraper, it doesn't matter.

Guess what?

It's not going to feel better to your end user. You don't have to be quantitative to point out, this is not going to make my user happy or this is not going to be fun to use. And that I think is step one. We'll Quantify it later. Let's qualify it first.

Yeah, I like that. I like the human aspect of it as well, Scott. The idea that you can think about, there's real people depending on this thing we created and we may be five steps removed from a real end user because we're a service behind a service behind a thing. But at that point you're really looking at when somebody has to wait, is it enough to make them frustrated? I mean, I was never a big fan of just APDEX out of its origins, but if you can fine tune the APDEX parameters to meet that context for the user, then it gives you a good idea of what is frustrating to a real human and what is completely intolerable. And those are good place to start. I think the other aspect outside of human response time and frustration. So if you go back to your product management team and think about is this a little test market or are we really turning on and we're going to get a fire hose? So you can like, you can start getting some measurements where there aren't requirements. But if you're in a greater context like holy crap, we're going to get 5,000 transactions per second on day one, then you at least have some impetus to figure out what the challenges are. And then you can start specifying, well, given what we know about the performance of the system, it's never going to make it to 5,000 transactions per second without the following list of changes. And then you're sort of making the investment or the justification for that investment against the risk of a huge volume. So I think between response time and volume, just thinking about how does this impact real users in latency and how will we keep up with what is projected to be maybe an enormous market size? And that would be the other aspect that you don't have to have a clear requirement, but you can ask questions around those two contexts to figure out something to get you started.

Thank you Scott and Mark for your performance testing awesomeness. For links to everything of value we covered in this episode, head on over to testguild.com p like automation.

And while you're there, make sure to.

Click on the Try it today link under the exclusive sponsor section to learn all about SmartBear and their awesome performance testing solution Load Ninja. So that's it for this episode of the performance testing and site reliability podcast. I'm Joe and my mission is help you succeed with creating end to end full stack automation awesomeness. As always, test everything and keep the good. Cheers.

Thanks for listening to the test Guild Performance Podcast. Head on over to testguild.com for full show notes, amazing blog articles and online testing conferences. Make sure to subscribe to join the Guild and continue your testing journey. This has been a Joe Calantonio production.