Performance is a Team Sport with Mark Tomlinson

Five, four, three, two, one.

Drop that.

Welcome to the Test Guild Performance Podcast where we get together to learn more about performance testing and site reliability with your host, Joe Colantonio.

Here it is. Here it is.

Here it is.

Hey, it's Joe. And welcome to another episode of the Test Guild Performance and Site Reliability Podcast. Today we're opening up the Test Guild Vault for an interview I did with Mark Tomlinson five years ago about performance testing. Testing is a team sport. Mark is a performance engineer and software testing enthusiast and co founder of the popular PerfBytes podcast. Mark now helps companies to adopt modern performance testing and engineering strategies, practices and behaviors for better performing technology. I think this message is needed even more than ever. That's why I'm resharing it with you today. You don't want to miss this episode.

Check it out.

And also my friends at SmartBear just wanted to let you know that their SmartBear Connect conference that they planned on running live April 27th to the 28th in person has now become a full blown virtual event. And even better, you can get your free virtual tickets for free. Now all you need to do is head on over to smartbear.com connect to register today. And speaking about online virtual conferences, it seems to be the rage now, especially.

Because of the coronavirus.

I've been doing them since 2017 and I just want to let you know that we still have some open slots for speakers at the June 1st June 2nd online event PERF Guild, an online conference dedicated 100% to helping you succeed with all things performance testing related. I'm really encouraging first time speakers and minorities to apply to be a speaker. All you need to do is send me an email on a performance testing topic you'd like to speak about@joestguild.com and I'll get right back to.

Let'S face it, performance testing is tough. Traditional load testing tools create scripts that are bogged down in data. It's hard to read and require a lot of work, even for simple playback. Load Ninja cuts out the dirty work by using AI to inspect and debug your code almost instantly. No coding necessary. No changing of hands or relays between teams either. Give it a shot. It's free and easy to try. Head on over to testguild.com smartbear and click on the link to learn more.

Hey Mark, welcome to the show.

Joe. Hey Joe. It's Joe. How wonderful to chat with you man. This is great.

Same here. Today I'd like to talk about getting started with performance testing or how to avoid being featured on Perf Bytes. News of the Damned.

Yes, you don't want. And I would say there's two parts to that One, most people who are doing zero performance testing, they're almost absolutely guaranteed to end up talking with Satan at some point. But it's interesting because there's so many shared things you can do to avoid ending up in a disaster like that. And part of it is just getting started with doing some load testing proactively. But there's a lot of it that goes to operational practices and good practices and things you can do with infrastructure in operations. That's why I think APM is becoming application performance management or monitoring is becoming so much more popular, because you can avoid a lot of these problems just with simple configurations and infrastructure, even if you're not, excuse me, you don't have all the load testing practices together.

What do you mean by APM though? So would this be in CI or something that. Where you just would have monitors running, just checking, you know, threads that may be running for a long time or things like that?

Well, actually, I had a question recently. APM grows up from over 15 years ago. And years ago it was the people just running monitors in production. Surprisingly, Joe, you might not think this people never used to monitor production. It just would. They would just say, we tested it for the last 18 months and boom, it's live. So it was actually quite a revelation a long time ago to we should monitor the cpu, disk and memory. They would run Iostat or they would run it every few minutes or something like that. I remember it was a revelation for some people to actually monitor response time in production. Like, oh, we can do that. Those practices have come a long, long ways now. We've gone from just monitoring from outside the app. We monitor inside the app. So diagnostics and profilers like new relic Dynatrace CA has some of the old Wiley stuff that you can plug into your stack and you can monitor the actual method level calls, database traces. So you can do a lot of that stuff nowadays because computers are so faster, so much faster, you can actually monitor without a lot of overhead. And a lot of the cloud providers now give you a lot of the basic stats to do, quote, unquote apm. So take that practice from production, Joe, and start thinking, you know, who else is profiling in this whole company building software? Who else does profiling? And then you finally run into a developer at the water cooler and say, hey, do profiling. They're like, we do profiling all the time. We do unit testing, we do white box testing, we do, you know, debugging and profiling and all this stuff in development. We're like, oh well why don't, why don't you guys get together with the ops guys and start talking about profiling the code? And they're like, well, there's way too much overhead. So we figured out all those practices to make it more performance. But yeah, doing some APM now, of course, performance testers and even functional testers. Those environments, your QA environment. The same types of profiling tools can be used to. If you think of Microsoft recently did the whole, they wanted to eliminate the idea of not repro ing a bug or that being the problem that you couldn't repro a bug. And they wanted to eliminate that because they thought it was a major waste of cycle. When, when a tester would say, oh, I found a bug and then the dev says I can't repro that bug, so I'm just going to dismiss it. Not a bug. So they tried through the Windows, you know, event tracing and stack tracing and putting sort of real time profiling into QA and really making QA more like a white box development discipline behind the scenes. And so when I think of APM or performance management or performance practices moving upstream, it's actually gets to look a lot more like white box profiling and debugging. And some testers are having to learn that now they're using tools like Dynatrace or you know, Visual Studio, the actual Steel, our Profiler or. NET or Java Profile. And don't even consider that you would be running load tests. Just, you know, you might take your functional regression suite and turn on some profiling behind the scenes. And if you hit a bug, boom, you instantly have all the tracing that you need to look at and go, yeah, awesome.

Back in the day we use something like Site Scope to monitor the CPU utilization and then HP diagnostics for the, like you said the profiling piece. So are you saying that would be a good place to start for someone that say they don't have performance yet in their organization and they want to get their feet wet at a minimum, would you say, let's just start at least profiling the application and seeing where we go from there?

Sure. Think of it in two ways. If you monitor and make visual, if you make the data, the measurements on cpu, disk memory, network response time, throughput app stats like garbage collection, heap size, things like that, request queued, number of connections used in a pool, number of threads used in a Pool. If you start making those metrics available or make people more aware of them, what you're doing is influencing the thinking of those people who first of all would be interested at all in those numbers and secondarily would know what to do if those numbers were quote, unquote, good or bad. So if I, if you, let's say Joe, you're working in operations and I came up to you and said, hey Joe, something's running at 99%. Well, how do you feel?

I feel it's probably the CPU or.

Something that's, but it's, but, but it's not good. Right?

Right, Yep.

What if I said it's 99% uptime?

That's good.

Oh, that's good.

Right.

So just, see, so if you just make, if you just make people aware of a number, they may, or they may know what the number means. A second thing is if we're not really sure about the reaction, the fun follow through, we may be doing, you know, the sky is falling Chicken Little, which could be, wow, let's publish all these great numbers. We don't, we haven't really indoctrinated or come up with an elegant way to know how people are going to react. So before you even start load testing, if you start making numbers, make measurements aware, Be very careful about the communication pattern, your influence on developers or your influence on a product owner or somebody who doesn't and know the difference between someone who doesn't really know what that number means and what to do. You may give them a lot of power, put them in a management position. They're like, oh my gosh, we're going to put together a million dollar project to do all this stuff to go after that cash hit ratio. And let's say they just didn't understand cash hit ratio. There's all these esoteric black arts in the performance world, which everyone is a team of firefighting specialists. And I think we need to break that down by just simplifying some of these numbers and saying, hey Bill, when you see this number, if it's over 85%, what I'd like you to do is get three guys together, Mary, Pete and Shirley, and bring them together and we're going to do a little brainstorming on triage. Because if it's over 85%, we probably have something to look at. Otherwise, Bill, you can relax and go about your business. Then we're being compassionate to Bill because he doesn't need to be a performance guru, but he at least knows, hey, I'm Part of a functional team. And we're making good use of those numbers. And I think that's the reactive model. Whereas load testing or performance engineering using pre production or what if scenarios or production simulation early, that proactive model uses the same numbers, but it communicates something very differently when you start switching to doing load testing.

I definitely agree and I think this is actually a big issue. It seems small, but the communication piece, you give someone a tool that's new to performance testing and all of a sudden they're throwing 1500 users at application with all any wait time and you have all these numbers being generated and they're going to management saying our application's broken and everyone's running around in a frenzy. So this is a good way to ease your team into it and slowly teach them what's a good number, what a number means. And before you actually dive into throwing a bunch of users at our application and then trying to deal with all the output that comes out of it.

I think also people evolve. Like you say, if someone jumps into load testing, and this happens often with teams where somebody saw listen to one of our podcasts, listened to this podcast and they went back to their team and said, we're not doing enough about performance testing. We have to start doing something. And it lands on somebody within that sprint. Oh, I got a story. And the story probably isn't specific. It says do something about performance.

As a product owner, I'm really scared that we haven't done anything about performance yet. And that would be a valid story. But if you, like you say, you'll end up with false positives if somebody just jumps in and they're detached from the real world of production, what the real systems look like. So there's a couple things. If you're getting started in a vacuum.

And you haven't made friends with your ops guys or even just say the database administrator, somebody that touches production and just have an, you know, take them out to lunch to say, tell me how do I even know how much load to throw at a system? Or maybe the web people that know the web logs or load balancer logs can say, well, we can tell you about the traffic, we can tell you about kind of how many hits per second when the peak two hours is. That should might be the first part of research before you start running, you know, quote unquote real load. Now that to say, Joe, let's say you were a developer and you said, I'm just interested in running 10 threads. I'm not, I'm never going to communicate to anyone that this is Chicken Little production is going to fail. But I do want to do some concurrency tests. That's like N unit perf or juniperf will do some concurrency. And you can look at thread contention. You could look at, you know, connection limitations or something. Some mutex, you could look at very, very small level memory leaks, other critical sections within the code that you want to sort of blast out. There's nothing wrong to let a developer do that and even a tester to help out in doing that. But you're right, it's a problem. If we turn that into, oh my gosh, my little ten threads exploded. We're gonna, we need to escalate this to management that production is gonna fail. You know, that's probably not the best practice. So somewhere in the middle, those first couple of steps are how do I, how do I switch into, how do I generate some real load?

Awesome. So how do we make performance testing a team effort where everyone is engaged like you said, and everyone's aware that it's not one person's job for performance, it's the whole team we're baking in performance in our application from the start of our requirements all the way to the build. So Jen, any advice with that?

First of all, don't ever leave that person alone for too long. I've been that guy. And our hair grows really long. And you don't eat right, you don't get exercise, sometimes you wear a parka and you live in the lab all day long. If you're still like that, that. So it's. Performance testing is a, is a team sport. It's collaborative sport actually. Just like any other modern.

Innovative team or development effort whatsoever around products. Everyone is agile. We're talking that we're in the teenage years of real agile or extreme development. We understand there is more benefit, more. I hate to use a synergy word, but it is true. There's a lot of, there's a lot of interactive benefit that stems from human beings thinking together on problems and solving problems. Paired testing, paired programming. You should think about paired performance. If you've got maybe one particular app or set of components or services, one scrum that is like these guys are the mission critical scalability guys. Pull your specialist performance tester and say, will you come and just join a couple sprints with us? Even if you're not taking on a story, just help us figure out what are the critical things going on within this code. So pull them out. I would say also years ago, Joe, I had this, this story Or a metaphor of a guy that walked around with a bucket full of bottlenecks. But it's true. Think about the metaphor of a guy, a guy like me who's like, well, I've learned these 10 different environments, this storage, this database, this network, this app, this what? And I have a bucket full of bottlenecks that are already known. Like take SQL server intention in tempdb. So you have a single tempdb file instead of round robining across 10 tempdb files. It's just a logical constraint within old SQL Server. So I knew that bottleneck and I know that 90% of the customers with SQL Server I could walk in, tap, tap, tap, select star from sys processes and tap, tap, tap. You, it's like a doctor. Yep, you have the same disease and here's your prescription and thank you for your money. At some point, the SQL Server team fixes the default setting and no longer has that contention. And now my value is shot. So building absolute or isolated, the ivory tower of a bucket of bottlenecks only has a lifespan, it has a half life. And suddenly you're not really valuable anymore. So every performance tester out there needs to know that their current knowledge is going to be cut in half within the next year with new technologies. And so if you have a career in performance, I mean, probably good idea if you keep learning, the great way to keep learning is to be collaborative. And so the biggest tip is to respect somebody's past knowledge. But say, is there anything similar in IO contention that we might find in Oracle or in logging or in JMS or in some of the. Oh, wow. If we look at RabbitMQ. Yeah. There's also some file contention. Maybe we can set our queues up differently. You know, I think of that conceptually that's very similar to that tempdb convention intention from 15 years ago.

Yeah.

So the transference, it's difficult. So inviting your performance folks to transfer their knowledge in a little bit more abstract way, that's one way. Or just start including them in. What do you think? What should we do with the story?

Say, for example, this is all imaginary. This is not a real story that.

We need the little, the Wayne's world, you know.

Exactly. We have eight Sprint teams and there's a person that says, now we're going to focus on performance and each Sprint team is going to work on performance. That's their high level thing. And they get some number from high up saying, thou shalt perform in two seconds. And of course it doesn't work in Two seconds even using one user manually.

So how do you engage that team.

To realize that maybe that the performance criteria they got was unrealistic and how can they show proactively and in a positive way why that is and how to move forward to a more realistic number.

A good example of this is to show how this works out is that like E tailors online E commerce, we've done a pretty good job. And I'll say to the, to the testing tool or APM vendors out there, we do a good job of helping people convert the metric of their conversion rate into bottom line impact for revenue and actually connecting performance to revenue. And sometimes if the guy in the corner office, he or she sends down that thou shalt have two seconds. There's a couple different ways that that could come about and we should be aware of that in the scrum and say, all right, well why two seconds? And some don't ask questions, just get it done. Well, so maybe we don't need to ask the question where we're like, all right, for some reason there's a valid reason. It's either financial meaning a 2 second response time. Currently our average in production is 3 seconds. And vendor X or white paper Y told me that if I shave 1 second of response time off the shopping cart page, I can boost revenue conversion rate by 15% and that's my revenue goal. So you've got this very hard revenue or, or throughput number attached to performance. That dysfunction is to just say, let's bring more clarity. And you may not be able to go back up to the, to the executive to talk about that, but you can go parallel, go to your peers and say what? Hey Jim, tell me you got the monitoring tool. Tell me what, what's the actual response time in production like? Is two seconds. Crazy. And you might find out response time in production is 2.1 seconds. Okay, hey, wait a minute. We're only shaving 100 milliseconds off of this transaction, guys. This is not going to be hard at all. And let's start digging around where we can do a little caching or maybe limit some round trips and it might be very easy to do it. But let's say your response time in production truly is five seconds and you got to shave three seconds somewhere. That's a more difficult ever. So first thing would be figure out whether this is a serious hard to do problem or if it's actually easy by talking to your friends in the real world. And look at your beaconing monitoring response times in production and if you don't have any of that, just go out, go home from your Internet connection and take some samples on the page. Use ySlow. You don't need to do load tests, just take some samples of the page load times in five different browsers. So that's a good thing is to figure out the scope of whether this is a serious problem or not. I would say also then if you're across all of these sprints, you're probably immediately should say, look, if all of these teams are looking at performance, we might end up finding one team finds a pattern or an anti pattern that they're solving. There should be a collaboration across the teams. Maybe every day at 4 o' clock if you know there's the AHA team that we all, you know, go stop, get a cup of coffee, grab a beer, whatever you're going to do. We all go into, you know, hop on a Skype group or hop on a collaboration thing and say, hey, Susan found this awesome bug and we think it may be a pattern across all the data access layers for all the teams. And so share what you found, share how you fixed it. Minimizing round trips is a good example. Minimizing or caching some data that you don't have to retrieve every time from the database. That's the second thing I would do is start collaborating because those patterns will be, you'll find them across all the teams and share them as soon as possible. And what you find is shaving 50 milliseconds off. Even though Susan might say, Look, 50 seconds, 50 milliseconds is not that much. Then you're like, wait a minute, this is across all 15 teams on different components together. All those 50 milliseconds add up to 650 milliseconds. Now we're getting somewhere. So make sure you collaborate with the bottlenecks that you find. The third thing is get your validation piece together. So if you're not doing load testing to confirm that Susan's fix is working, then you need to start looking at. All right, I need a way to start. Build a little lab, pick one of your environments, start somewhere and get either JMETER or there's free. Everyone has free tools now. Cloud test is free tools, loadrunners, free tools. You can get started with those small level of load and start hammering some load at the fixes and create that environment for all of these bottlenecks within that two weeks, every night, hey, whatever fixes you got, let's get them in a build and put them in some production simulation and Start running some load. So those are kind of the three. Make it relevant to the real world, collaborate on any fixes you find and then set up, choose one of your environments for a highly repeated, you know, put all the fixes in and rerun it. Rerun it, rerun it. Using, even starting with some of the free tools that are out there, it's a great place to start.

So another thing in my experience is we always look at performance from a technical type of lens as testers and performance engineers. But what are your thoughts on how can we look at performance as a end user or as a user to get a better handle on things that our users will be noticing and how they'll be perceiving performance rather than how.

We think performance will be perceived?

We work with technology as testers, developers, whatever. Sometimes we discount our credibility as users and we know we're biased. And it's good to know you're biased, but it's also good to know, hey, this is a simple page. This thing is taking 40 seconds to load. And I don't care if it's just my environment or not. Something is not right here, especially developers. You think developers that sort of play with the code, they compile it and run, compile it and they're, they're debugging as they're building. Some people do that interactive, they don't just write all the code perfectly and push the compile button once. I don't see that anymore. That's the old COBOL and C programming, but in the interactive framework based stuff, they're molding it more like clay. They're sort of developing it as they go and get this running and now we'll add this and what changed and add this and what changed. Like I added this component or this extra method and now it does something. What is, oh, I'm using the wrong futures and the pool's not right and I don't have enough threads, things like that. So the idea that we get trapped in the physical thing we're building, the use of threads or physical resources, we lose track of time. And actually I find our influence as engineers, especially in technical, we will influence not only our peers in testing, but especially the business users, business analysts, product owners, even a technical product owner. We think about technology first and end users second. And I actually advise engineers when we're doing Sprint planning or doing investigation or paired work around what the story should be, to never talk about any physical properties at all. Just talk about time and volume. And that can be if your product owner or Your business analyst doesn't understand response time, then there must be something abnormal with them in their brain because everyone on the planet can relate to time. Time is something that you know, how long did it take you to drive to work? How long did it take you to do these things? We know the difference between the context for what would be perceived as a long time and what we usually perceived as a short time. And your business analysts, your product managers, or your, or even your sales guys should understand who they're competing with. And if the competition takes 40 seconds to get credit processing done, boy, if you can do, if you can do that in six seconds, that's tremendous. So if you go back to the CEO you were talking about or the executive that said thou shalt have two seconds, you might be able to go back and ask the business analysts, hey guys, do we really need two seconds? And they might come back and say, guys, the competition can't even do it in 30 seconds. So if you can do it in, I don't know where that two seconds came from, but get your partner in the business to give you that stamp of approval or some feedback to say, this is under 10 seconds, is it? Now you also know end user perspective and context matters for what they're doing. So if you're in an E commerce site, you want how many seconds for the buy me button shows up, right? I mean sometimes it's like we rendered everything beautifully and then the buy button takes forever. Conversely, I've talked with guys at stpcon last week who were talking about they actually trimmed such little content from the page. They actually saw people not spending time so they weren't selling. So E commerce has a whole different dynamic. Go to Back Office Joe and look at reports and say report runs that we cache in business objects or out of, out of the OLAP cube or the data warehouse. You may wait 20 minutes to get your report, but that report to the company could be worth a couple million dollars, you know, so it all is relative. And if your business people aren't thinking about, well, who is, who am I really serving and how much, what is long time to them? What is a short time to them them? And are we competing on this dimension of time in the marketplace or even competing against the clock with our, you know, more people, less time, back office kind of stuff. So I think time, you know, ignore cpu, disk, memory, network, app servers, all the technology stuff that you ever studied and just spend time thinking about time and the different context for what's the real value of being fast which again goes back to the executive you mentioned saying, thou shalt have two seconds. And if you're able to have as savvy a conversation as I just did, you'll walk into the CEO's office and blow their minds. They're like, oh my God, you're the new VP of performance. I didn't even know you could think about time this way. But there is no one absolute. I will say also there's a challenge with, with mobile and latency. That was my time at Shunra and I've known Shunra for so long as they got now acquired by hp, they really did a lot of the state of the art thinking around network impacts on functionality, on user perception, on customer satisfaction, on conversion rates. They, they kind of lived that more than anyone I'd ever witnessed in terms of really impacting the business and impacting end users. And then mobile came along and whoa, oh my gosh, latency is, is everywhere. So that's, that's the other part. You have to think not first about the human experience context for the end user and what they perceive as performance, and then then only secondarily start digging into what are the, what are the physical implications of, of my environment for serving that experience to a customer.

Are there any books or resources you.

Would recommend to someone trying to learn performance testing or to get them started?

Yes, I will give a shout out. We, on the Perfbite show, we interviewed Bob Westcott. You remember Bob.

Awesome book.

Yeah, yeah. And it is a great book. And it's the Every Computer Performance book and which was modeled in theory as he said on the podcast. It's modeled after like every bicycle repair book or something like that, which means it's for anyone to be able to relate to the concepts and then go a couple steps further into how this practice is done. So that is a, that is a great book to get started in performance. The second thing is like you mentioned somebody that might in a sprint just get handed. You're going to do load testing. Yes, and if you're in that world. I do. This is not a shout out to myself, even though I helped with the book. It's the visual of the Microsoft Patterns and Practices book on performance testing guidance. It was Scott Barber, J.D. meyer. That's. Yeah. And that one has, as you know, it has a lot more like methods and process for what steps to take and how to think about the tools you're using. I think that's a pretty good book also for beginners to take a step into the performance world. Those are the two together that I would recommend to anyone who came to me and said I want to learn performance.

Okay, Mark, before we go, is there one piece of actual advice you would.

Give someone trying to improve their performance testing efforts and let us know the best way to find or contact you.

Run a load test. If you're not running a load test today, run a load test, pick up jmeter, pick up n unit perf J unit perf and get some unit test. Run something with multiple threads and I don't care where you run it, I don't care how you run it, make it small. Don't try to blow everything up at first but just that first step of wow, what if I run multiples of this at the same time. It's a beautiful crescent. What if happens? It's a wonderful contemplative question. Starts from a point of exploration and that feeling, that moment where you get the freedom, I'm going to go, what if I do this? And it's multiple concurrent activities that if that opens your mind and you're excited about it. That's the spark I look for when I'm mentoring people and they're like wow, I never thought about that. And then they're going and try to fuel that energy. So the really the first actionable thing is just pick up a tool and run a load test. The first actionable thing if someone's already doing load test and they want to go further is to start I would say right now it's really popular. Learn the real world. Connect to the apm. See if you can make your load test relate to the real world. Figure out what real world performance looks like and those are the two actionable things. I think for somebody coming from zero to somebody who's been doing load test wants to go further would like what they're doing. For contacting me you can find me at mtomlins on Twitter mtomlins everywhere. That was from the old days when they made you first letter and seven did like that was your account in college. So I've been M tomlins forever all over the place. You can perfbytes is P E R F B y T E s and you can follow us on the perfbytes twitter and that links to everything out there.

Thank you Mark for your performance testing awesomeness.

For links to everything of value we.

Covered in this episode. Head on over to testguild.com P25 and while you're there make sure to click on the try it today link under the exclusive sponsor section to learn all about SmartBear's awesome load ninja Performance Testing Solution. So that's it for this episode of the Test Guild Performance and Site Reliability Podcast. I'm Joe and my mission is to help you succeed with creating end to end full stack automation awesomeness. As always, test everything and keep the good.

Cheers.

Thanks for listening to the Test Guild Performance Podcast. Head on over to test guild.com for full show notes, amazing blog articles, and online testing conferences. Make sure to subscribe to join the Guild and continue your testing journey. This has been a Joe Calantonio production.