Anyone can Performance Test with Eric Proegler

Foreign.

Get ready to discover some of the most actionable performance engineering Testing Insight Reliability device with some of the world's smartest engineers. Hey, I'm Joe Calantonio, host of the Test Guild and SRE Podcast, and my goal is to help you succeed in creating application performance awesomeness.

Hey, it's Joe and welcome to another episode of the Test Guild Performance and Site Reliability Podcast. Today we'll be talking with Eric all about how anyone can performance test. Eric actually was on my podcast, my Automation podcast way back in 2015 in episode 80. So it's been a while. Really excited to get back up to base on what he's been up to since then. If you don't know, Eric is a Staff Test Infrastructure Engineer at Credit Commerce in Oakland, California, and previously he's been a Senior Director of Quality Engineering, a performance testing consultant, practice lead, a Testing Tools Product Manager, and President of the association for Software Testing. He presented and facilitates at a bunch of different events like CAS, Test, Bash, Agile, PNSQC, StarWest, Star East, STPCON, Code Fest, all the good places and he's here today. Really excited to have him. So you don't want to miss this episode. If you want to know how to get into the performance testing game, you don't want to miss this episode. Check it out.

This episode is brought to you by the awesome folks at SmartBear. Listen, we know load testing is tough but necessary. So investing in the right tools to automate tests, identify bottlenecks and resolve issues fast saves your organization both time and money. And that's why SmartBear created Load Ninja, a SaaS load testing tool to help teams get full visibility into performance so you can release qual software faster than ever. Make testing effortless and give it a shot. It's free and easy to try. Head on over to loadninja.com and learn more. Hey Eric, welcome back to the Guild.

Hey, thanks for having me.

So, it's been a long time and I was really excited to see at CAST you're going to be doing a session on performance testing, so I thought we'd touch on that and probably some other things. I believe you're also doing a workshop on performance testing as well, so we're going to touch on that also. But Eric, before we get into it, is there anything in your bio that I missed that you want the Guild to know more about?

No, I think that covers it pretty well. I have a background in performance for many, many years. I've moved towards more general testing responsibilities these days, but my performance Testing and load testing are still near and dear to my heart.

Absolutely. Same here. But I noticed something interesting and I'm curious to know if you're seeing this as well. For some reason the past year or so performance testing has, it seems to me being increasing in popularity. I have some sessions open for Animation Guild and a bunch of them were on performance testing. So are you seeing the same thing? Is that something in the industry or maybe it's just me Talk to a lot of performance engineers?

I mean, I think anecdotally that's true. I mean I'm hearing from hiring managers that they're talking about load and performance experience to hire quality leads. Companies that made the full journey to the containers in the cloud are still talking about. Well, I don't, you know, I want to verify how the stuff scales myself. I want to improve out failure modes and things like that. So I think for. There was a point in the mid 2010s where it started to feel like, oh, now our observability has gotten so good that we maybe we're not going to be doing this as much. And at least the performance practitioners that I was around with, like the Perfbytes podcast I was on for a while, they were, that was becoming part of their tool belt as well. You know, they had to look at the, you know, the really good telemetry you can get in the front end these days, look at waterfall charts, look at how web browsers load. And they were seen to be spending more and more time there and less on pushing a button and throwing a bunch of load at a backend. But it's, yeah, like I said, it's coming back around, I guess that these things get to be cyclical. We'll see what it turns into this time.

So that made me think of another question. How do you explain what performance testing is? Because nowadays also when I talk about performance testing with some engineers, they just think of front end, how the one user is loading up in a browser. But it seems to me back in my day performance testing was basically load testing. So how would you explain what performance testing is?

I would say that load testing is a subset of performance testing and that performance testing is any sort of exploration or learning about how a system performs either with a single user or with load, you know, to satisfy requirements around response time, maybe even uptime and reliability. When you start to get to what load testing can do.

Absolutely. And, and you know, a lot of times, I know sometimes people don't like this word. Performance testing is thought of as A non functional requirement. But I always, if a tester is a tester, they should be testing all the things. I don't know if that's not realistic, but what are your thoughts on performance testing being a non functional requirement? Do teams ignore it or is it something that usually a dedicated team will would focus in on?

I think that people who really want to be focused on the practice of creating features, they want a lack of ambiguity, they want to be really clear what they're supposed to do. And if you're looking at how does an application respond, it's actually a very difficult thing to measure. One of the shortcomings of load testing with modern architectures is how much of the stuff's in the front end and how much it's subject to things like last mile network conditions. I think testers should be trying to help however they can, but you know, going down to a coffee shop and loading up the app on the WI fi when it's busy is probably. You can learn an awful lot about the performance of your app or your property. That's really hard to get if you're sitting at a desk on fiber. You know, even if the host the stuff's hosted in the cloud somewhere, are you using it the way a user would?

So that's interesting about requirements then do you see that being an issue? Sometimes I know performance teams, not performance teams. Someone will get a story in their sprint for performance and I'll just say this shall load in under five seconds and that's all they have.

I do see those in my current place. They'll come up, it'll show up as some kind of spike. Some fairly senior person, usually with deep client knowledge, starts using those, you know, those front end tools. We have to break things down and figure out how to either optimize things or change the loading pattern so that, you know, time to above the fold, time is reduced in some way, you know, fulfill that requirement. And the requirement even just to say this loads in five seconds. Well, do you mean the user can work, can do something within five seconds? Does it mean the user can see something in five seconds? Does it mean it's completely done in five seconds? Like there it's, you know, all requirements are incomplete.

Absolutely. So as they're flushing out that requirement, you know, usually teams want to know the why. Why, why does it matter? I assume people know why, but maybe they don't. Why would slow performance matter in an application?

Well, my experience, the last job I had that was specifically in performance testing. I was working for Sosta Right before they got bought by Akamai. And I worked with Tammy Everts, who's over at working with SpeedCurve these days. And she's put out a lot of data showing that at least in an E commerce site, you can tie revenue to response time. And we had a lot of data that was showing this big drop off as things started waiting for things longer and longer. Now that's a proxy also for the user's attention because when somebody abandons a site or abandons an experience because it's going too slow, there's some good evidence that you can tie those two things together. Now, if it's a business application and it's a captive audience, you're still just going to irritate them. They're still going to be telling their boss about how it really stinks that they can't get their work done because the stupid software is so slow and they're going to complain about it to their neighbors who are going to start complaining about it too. And it's never a great thing if you're responsible for a software experience to have it disappoint its users.

And I guess it's context driven as well. If you're a hospital or you know, something like mission critical, obviously you know you want it, you need to get done a certain amount of time because it's something that's very important. So I guess people need to take that into account as well.

Well, yeah, it's definitely across the context. It varies a great deal. I know somebody that writes C code for high frequency trading. He has a different point of view about performance.

Than people who are developing mobile games.

Absolutely. So you did mention something earlier about, you know, how does something really perform in a cafe on a mobile network to really understand what a user experience is. So I guess nowadays, you know, 2015, there was mobile. It seems mobile has really increased, especially with what we've been through a lot of applications now online. Do you see a gap now in performance knowledge in the mobile space? And does that require a different set of skills that would be compared to someone that's used to maybe just, you know, performance testing a web application.

So if by performance testing you mean load testing, I think that load testing mobile web service endpoints is pretty straightforward actually compared to trying to test like a collection of a bunch of different services that are presented on a single web page. I think if you're going to talk about the amount of the time, the amount of the response time that's spent in the user's hand, instead of in a server somewhere. Yeah, there's definitely a different set of skills. And even people who are writing that code, they don't always have everything that they need to be able to do that. Well, I've seen some like new relic browser type work where you can get loading times from your user base, but it doesn't seem, from the last time I saw it, it doesn't seem to be as rich as what you can get from a browser where it basically rebuilds the waterfall chart for you. Or if you go to webpagetest and you log in from whatever data center and you get every resource, how long it took as a sample.

So brings up another thought. You know, back in the day when I started, it's almost like you can almost do performance testing in production or at least gather the requirements for performance testing that Mimikyu uses better now in production than you could previously. Is that something you see, maybe teams miss out on? Or is that common knowledge nowadays?

I know that the people I talk to who are working on performance these days are more likely to be front end engineers than service engineers. The service engineers have gotten into this infinite scaling loop that, I mean, it's not true, but they believe that's out of their hands or just bump it up to the next size, that next biggest instance or whatever. The front end engineers who are working on that stuff are usually very connected to what Prod is telling us. Like this is our P95 in prod this week and those are the kinds of numbers that they're using. So it's less, you know, coming up with, coming up with a number out of thin air. Like Scott Barber told me once, just say three seconds, everyone will believe you. It'll end a lot of conversation. And he's right because it would just work. What's fast enough? Three seconds. Oh, okay. I mean, and the answer, it's under a second if you're Amazon and if you're in E Commerce, you're trying to drive it below 2 seconds. But there is some curve where it goes from I'm waiting to now I'm annoyed and it flips over to that state. It may as well be 15 or 20. And I'm not talking about like selenium coding here, but I started M. Absolutely.

And I assume with mobile it's even, even worse because someone, there's so many apps, someone would just say, you know what, I'm just going to uninstall this, never use it again. They have one bad experience with it. It's almost the tolerance level has has dropped even more, especially in mobile.

Yeah. I've spent most of my career working with business software where we'll tell you which software you're going to use and these days working for a public facing property with millions of daily users. I don't know any of them and so there's definitely some strong signal being sent every day and we are looking at number of uninstalls and how often people come back and what their experience was the last time they were here.

Absolutely. And Eric, a lot of the points we're touching on going to be covered in depth at cast. For the folks that don't know, maybe what is cast.

So CAST is the conference for the association for Software Testing. I'm a little biased because I used to be the president of the association for Software Testing. I think it's a great conference though. It's really the only explicitly context driven testing conference. But we're in the second or third wave of context driven testing now and there's still some interesting stuff happening. So like I love to go to a test bash or an agile testing days and you know, good stuff happens there too. And a lot of the context driven stuff has just kind of been accepted. Like people do exploratory testing now. You know, ISTQB teaches you about it. So in some ways the need for a separate context driven conference is less than it used to be. It still attracts a very interesting set of people.

What date is that again that you actually will do this session?

I'll do this session next Tuesday.

Oh wow.

That would be the eight.

Nice. So you mentioned exploratory testing. Can you do exploratory performance testing?

Is that a thing I have once I've had my scripts in place for load testing. So you said performance testing. So I'm going to cut that in half that for load testing. Even with you have a rig and a scenario, try different stuff all the time. Some of my most interesting performance tests were things like I'm going to let the stabilize at peak and then turn the knob up and turn it back and see what happens and kind of probe the system. And that's the idea with exploratory testing is you want to keep learning and as you respond to the things that you're seeing. So the first time I sit down with an app to do exploratory testing I'm looking for functions, I'm looking for how it behaves but I'm experiencing that performance as I go and I think that's kind of the point that I want to get to in this keynote is everyone who puts their hands on the software has an opportunity to inspect performance and give feedback about it. And if it's slow, when I'm doing exploratory testing, I'm absolutely going to report that. Because even under best conditions, very modern, you know, very modern handset, you know, not some Boost mobile, know Google clone phone, but you know something, some real test device. It's, it's, if it's slow there, it's definitely going to be slow out in.

The world for sure. So in the write up of this forecast you had a sentence that says performance is the net result. Many difficult or impossible components to control. Maybe you can unpack that a little bit what that means.

Okay, so I'm going to go to the smaller load testing world for a minute, for a long time. Would talk about if I could just replay the production logs, I would have a perfect test that exactly reproduced production. Except you wouldn't because you wouldn't have the weather, the variability in network bandwidth, the noisy neighbors, the different client locations that people are connecting from. You can try to take a particular measurement one specific way and then try to reproduce it. But the system's changing all the time and the things that are happening next to the app you're trying to work with are changing all the time. Like the load testing some people got very frustrated with, hey, this cloud based system does not show consistent response time. You're right, it does not. It's reached a level of complexity where you no longer can control it as well as you could control something in your lab.

So does that bring the rise in chaos engineering or. I like to turn better resilience testing, the need for that and especially as we go cloud based like you said, because it's not so, you know, you don't own the metal, the server yourself and everything. It's almost like you're consuming things. You probably don't even know what's being consumed by your application in the cloud.

Well, I think the willingness to test something and learn from it, as opposed to confirm a hypothesis is probably in line with what you're talking about. But once something's out of your control, the best you can do is to try to understand it or just take good records and say, okay, well this time it was 5 seconds, last time it was 4.2. I'm not sure why that is, maybe my stopwatch finger is bad, I don't know. But there'll be some plot of variability, what's normal and what's not very cool.

So, you know, I've been saying performance testing, you're right. Performance testing is just an umbrella of all these other little pieces that make up performance testing. One of them being load testing, obviously, which people probably tend to think that it's like a dedicated group. So how hard is it for someone to. Maybe some misconceptions people might have about load testing since this is a session on how anyone can do performance testing. How much is focused on load testing and how much knowledge does someone need to actually do a pretty good load test?

Not too much is focused on load testing. When people say they want to do load testing, I want to usually talk to them about, okay, there's some minimum level of investment you need to make to achieve a meaningful or useful test where you're going to want to simulate multiple activities. And if you're willing to just throw a little incremental load at a production site, it's a little easier than home. I want to recreate the production world from scratch, which even when we were load testing in closed environments could take months if you tried to run down every single strange act, strange thing that could be occurring. But I think it's easier than it's ever been that, you know, the load testing has kind of coalesced around JMETER as a standard. It's got an active contributor base and there's a lot of public clouds you could take your test to when you're ready to scale up. And compared to trying to set up load injectors in the past, things are way easier.

Absolutely. There's still things that people need to watch out for. Like, would you recommend me maybe for. You still recommend someone has like a testing strategy in place or some sort of risk assessment to know that they're focusing on the right things.

Oh, yeah, I think that's. You definitely want to do that so that you can budget how much you're going to invest. And if, if I have two days to do performance testing, don't pick up a load tool, learn how to use webpage test and start experimenting with different client conditions. Get a couple iterations into that. So I think what you're talking about. Yeah. Do I understand this whole thing? Do I understand how much I'm willing to invest in this and do I understand it's going to be. It's going to be some slog to get to useful results, even for an experienced tester. Just to get everything running and working reliably takes time.

So that gives me an idea for a post if you haven't done one yet. Or something like a proglar sliding scale. You got two weeks. Here's what you should do in two weeks. You have a month. Here's what you should do in a month. You have two days. Here's what you should do in two days. Is it that could you get like kind of that ballpark with someone? Right, you have two days. Here's what you look at. Yeah, but we. Here's something you should look at. You have two months. Then here are the type of things you should be looking at.

Well, if you squint and generalize very broadly, you might be able to come up with some stuff there. But I would be nervous that I would have somebody who either has access to a bunch of existing tests and go faster or who doesn't have the things that they might want to optimize what they're trying to accomplish. As long as it was very general like some sort of scale of this is twice as hard as this or this is three times as hard as this. Hopefully that would be useful.

Nice. So you know, once again with Martin software development people are in sprints. Do you see? Mostly, you know, back in the day we used to have one big performance test team. Mostly all this is thrown on the shoulders of developers now where they have to do the automation they need to do the testing they need to do everything. Is performance testing the same deal Now.

I don't see a lot of situations with that kind of staffing where that investment gets made where somebody is really built up perf and CI and they're doing any high level. The places where I have seen that happening, there's usually somebody who's like a quality coach or performance coach who's going to help people bootstrap that stuff and get a couple tests running so that you know, jump start, jump them ahead to where they have tests running. Because if it's I started from scratch and I want to build this, that's okay. Well how much time can you spend this sprint on this and how much time can you spend next sprint on it? It could take a while to iterate your way to something useful. I mean agile is great for reducing work in progress. It's not always great for maintaining focus over long periods of time.

So how often should people do be doing performance testing? I guess it's always right because you can always have a baseline to kind of compare it to. But someone wants to do a load testing, I guess it all depends. But any rules of thumb that you have.

Wow. Okay. So some of the bigger shops I've seen they will have local performance budgets and they'll verify their components performance locally pretty constantly. But the end to end tests where there's going to be some debugging and you're trying to get to some sort of scaled load that's a little harder to do frequently. So if you have good day to day performance information, you might be able to get away with doing load testing infrequently quarterly or something like that. But I've also seen places where people are running not super sophisticated load tests but some sort of load at some paired down test environment in their CI pipeline and even kicking stuff back out if there's some delta in bytes or response time that they don't like. That takes a lot of discipline to get there. I think that when I say that part of the talk about anyone could performance test is all you really have to do is record some response times or have access to a system that has those response times recorded and look at them and suddenly you have a bunch of performance testing data to crawl through and oh if you can, if you have enough context about your project and understand what's happening, we introduce this new framework. We, we upgraded this at this point sometimes you can make a connection at least going forward once you understand that data you can be checking it as things go out and see and look for deltas in it, look for impacts.

I love it. It's like let you start without any excuses, like you have a number, here's a number you at least can start with when you do it that way rather than waiting around for all the best of everything to have in place.

Well, I don't want to spoil my keynote but pull out your phone and go to webpagetest is you know going to be one of the things I'm going to ask people to do because that's a focus for cast is hands on. We want everyone to leave with having done something. And so yeah you could even if you don't do that and you just load your own website, you can if you, if you have some way to record the response time of that and you think about it, you're now testing because you're learning.

So you know you did mention this has happened next week. I don't know if this I'm be able to edit this and get it ready for next week. I didn't realize it was that close but if people are listening to this is like oh my gosh, I missed it. Like do you do this often? Is this something that they can get a recording for? Do you do ongoing sessions on this that people can get their hands on.

Oh, well, I have to say that when my time and on the AST board was over, I ran away. So I'm not involved with this year's conference and I don't know if they're planning to release recordings or anything like that. I've done talks like this three or four times a year for the last couple years. I kind of stopped doing talking about performance because I wasn't really doing it. And there didn't seem to be a lot of appetite for load testing. But as you said, there's suddenly a bunch of people curious, like, well, what can I do? How can I get started? So I don't know, maybe I'll be getting to do it some more.

Right. That's awesome. So I know you are doing something again, I think in December called Whopper Workshop on Performance and Reliability. Could you talk a little bit more about that? So if people missed out on cast, maybe they should definitely check out this session.

Oh yeah, they could check that out. So the Workshop on Performance and reliability, it's our 29th iteration. It's going to be in Montevideo, Uruguay, because the very, very skilled and friendly people at Abstracta said, hey, we'd really love to host a Whopper there. And that's kind of how we got here with Whopper. It's been running for 15 years, maybe more. 17 years. I wasn't there at the start, but I came in not too long after. And so we've been operating it for a while. It's back. This is our first one since the, you know, the thing that happened. The one before that was 2019. So we try to have one or once, one or two a year. And then each one of these workshops is based on specific subjects. Like this one is about iterative performance testing. So, you know, performance in CI or doing tuning cycles. Like Mark Tomlinson is going to come talk about his time working at Microsoft Labs and getting people's apps into that lab and then changing code and testing and helping people quickly improve their code performance in that kind of environment. There's a conference called Quality Sense which is going to take place at the end of the week. And a lot of the people who participated in Walker will be presenting that. And I think that one is going to be streamed. I'd have to check on that.

That's awesome. Okay, before we go, is there one piece of actual advice you can give to someone to help them with their performance testing efforts? And what's the best way to find a contactor.

So I think the first piece of advice that we've touched on is you don't need to do load testing to do performance testing. And if you set the bar so high to learn how to do load testing before you can collect performance feedback, you're leaving a lot on the table. And there's a lot of people working in performance who don't load test. You can do meaningful performance work that way. It's not that load testing isn't valuable for what it does, it's just that there's a lot more to performance than throwing load at a back end and ignoring the front end altogether. And then see the best. Usually the easiest way to contact me is on probably on Twitter. I guess we'll see what Elon does with it. But I I mean it's still my favorite social network, so I've got a super elite handle. It's like 3R 1C/P because my last name is hard to spell, so I guess something more complicated. I don't know.

Thanks again for your performance testing awesomeness links to everything of value we covered in this episode. Head on over to testguild.com P101 and while you're there, make sure to click on the Try them both Today link under the exclusive Sponsor section to learn all about SmartBear's two awesome performance test tool solutions, Load Ninja and Load UI Pro. And if the show has helped you in any way, why not rate and review it in itunes? Reviews really do matter in the rankings of the show and I read each and every one of them. So that's it for this episode of the Test Guild Performance and Site Reliability Podcast. I'm Joe. My mission is to help you succeed with creating end to end full stack performance testing awesomeness. As always, test everything and keep the good. Cheers. Hey, thanks again for listening. If you're not already part of our awesome community of 27,000 of the smartest testers, DevOps and automation professionals in the world, we'd love to have you join the fam@testguild.com and if you're in the DevOps automation software testing space or you're a test tool provider and want to offer real world world value that can improve the skills or solve a problem for the Guild community. I'd love to hear from you. Head on over to test guild.info and let's make it happen.