Context-Driven Performance Testing with Alexander Podelko

Five, four, three, two, one.

Drop that.

Welcome to the Test Guild Performance Podcast where we get together to learn more about performance testing and site reliability with your host, Joe Calantonio.

Hey, it's Joe. And welcome to another episode of the Test Guild Performance Podcast. Today we'll be talking with Alex Podelko all about context driven performance testing. Alex has specialized in performance testing since 1997 and has worked as a performance engineer and architect for several companies. Currently he is the consulting member of technical staff at Oracle and is responsible for performance testing and optimization of enterprise performance management and business intelligent products. Alex is also one of the leading thought leaders in the performance engineering space. He writes and speaks about it all the time, so you've probably seen him at multiple conferences. And he also has an awesome website that you should check out if you're into performance testing@alexanderpedelico.com so listen up.

Let's face it, performance testing is tough. Traditional load testing tools create scripts that are bogged down in data that's hard to read and require a lot of work. Even for simple playback. Load Ninja cuts out the dirty work by using AI to inspect and debug your code almost instantly. No coding necessary. No changing of hands or relays between teams either. Give it a shot. It's free and easy to try. Head on over to testguild.com smartbear and click on the link to learn more.

Hey Alex, welcome to the Guild.

Hello, how are you?

Awesome. It's great to have you on the show. Alex, before we get into it, could you just tell us a little bit more about yourself?

You did a great introduction for me. Yeah, so I'm doing performance testing and maybe we can say performance engineering in general for quite a long time. It is quite interesting area from one side, a lot of things happening around it from another side, not many are talking or writing about it.

Why do you think that is?

I. I would say my opinion it may be one reason why performance testing becoming less taxi right now somehow it going into shadow. By my opinion it is much more important nowadays, but we hear even less about it. Maybe it related to vendors because vendors usually had problem budget and reason to push performance testing and advocate it and promote it. And right now not so many vendors looks active in the place.

I I would think though with everyone talking about the hype about AI and machine learning in automation that in performance I think it actually makes a little more sense for performance testing because all the logs and data that are gathered during a performance test being able to leverage maybe a machine learning Algorithm to analyze that would be a hot topic. Are you seeing any of those trends or that type of focus or is it just still out there in the future?

I hope for this trend but I actually see somewhat reverse trend because before we had some advanced load testing tools that included a lot of functionality. Maybe it was not AI per se but.

Was pretty sophisticated analysis for example in load runner. And now we get.

Rather back to open source tools which why maybe great tools for some specific purposes but definitely lack almost any.

Analysis there so it should be combined with some other tools. So in my opinion it looks like it more happening on production side. Tools to monitor.

And performance management on production side and.

They are not.

So eager to integrate it with performance load testing. Unfortunately not. I believe that it should go that way but it doesn't.

Also interesting as you mentioned performance engineering back in the day I would just say performance testing or performance tester but more people call themselves performance engineers. Is that another shift you see in the industry? Maybe that the focus is less on testing and more on performance in general.

And especially in production.

I definitely see that trend. So it look I believe up to some degree it was all the time. If you really want to make difference you need to go beyond mere performance testing. Not only do performance testing but say do full performance analysis and work with teams to do to optimize system to tuning and all that stuff at least because test results, just raw test results are not very useful. You need to get down to specific root causes of any performance issue. So actually I believe that good performance testers always beyond mere testing and do a lot of let's say performance engineering work. And it is more so now because I think we in many organization we and with move to agile development we don't have siloed team anymore or at least not so clear cut siloed team. So it doesn't usually it doesn't work that somebody just run tests or create scripts. There are some cases when it works but usually team expect from that person to do more than just scripting and running tests, do some kind of performance troubleshooting and all that stuff around it. And actually there are opinions that it should be the same person that should fix the problem. I don't think actually that it will work but there are such opinions and another clear trend is automation. So there is another group of opinion that everything should be automated. So performance test should be done in an automated way which is a good thing. If we remember that it basically would be regression performance testing just one kind of testing. Not all we need but sometimes looks like people get consider that as only and final destination. But anyway there is some stress on kind of plumbing skills and getting performance testing to build in into this continuous integration, continuous development. So actually I think yes that it should move more in performance engineering and the person who working with performance should go beyond performance testing and trace that from the beginning performance requirements to the end when to see how things going in production. But actually I don't see that either. So it's somewhere in the middle. It moved beyond performance testing, but not far. In my opinion it's actually a good.

Segue because you have a concept called context driven performance testing. You did mention agile and try to. It's not just about.

As I say shamelessly borrowed this term from functional testing because by my opinion it describes the situation perfectly. Before with waterfall development we had pretty standard process with performance testing, let's say kind of traditional load testing when somewhere before release of the product performance team get project of performance testing of this product which was done with some kind of corporate level load testing tool like load runner or similar tool. Practically in any corporation you will could find then performance testing groups. They are in different places but almost everywhere you can find them they run tests if either product was good enough and was released or it get back to development. And I don't want to say that it was bad or wrong process. That was exactly right process for waterfall development because you couldn't do much before products would be ready. Actually I had long time ago some presentation on the topic of agile performance testing where I was talking what you could do before it to start earlier. But it was still some side things like checking performance requirements, maybe testing some technology but. But overall you couldn't do much. And now it changed. And I would say changed drastically because now almost. Well, at least in areas where I'm working everybody moved to kind of agile iterative development. So you have a working product practically from the very beginning. So you could work with a product and do performance testing from the beginning. And suddenly we get much more options than it was before. Theory behind traditional load testing was that you need to make test system as close to production as possible. Make workload as realistic as possible. And when I talk about that in some cases we need other kind of tests, for example just to investigate specific performance issue. I even got criticized for that. But when you have realistic workload it means probably a mix of workload. It is not so easy to use for tracing performance problems or finding best way to optimize the system. And now it somewhat changed from one side system became much more sophisticated. So in some cases it is even not feasible to test the whole system to do full scale test. We still would like probably to do that if we could. But at least it is not feasible to run this test all the time against full system. And and we definitely getting performance testing embedded in continuous integration process in one way or another. It is far from finalizing, but we definitely move that direction. If it was exotic some time ago, now it is more and more used. And if you work with Agile team from the beginning, you just just don't have time to run test manually with every label or build or whatever would be your process. So you need to find out some set of tests that you want to run. But still we have as I mentioned it just performance regression tests. So we just making sure that performance is the same and we don't have any problem with it. It's still not eliminate other kind of tests. For example full scale test against the system to make sure this system will handle full load. We still want to do that if we could or test system for memory leak run test for a long time. If you run it for some several days, you definitely don't want to make it a part of continuous integration. Probably should be in parallel to main process. So we have a lot of different options of what we can do now, which has different purpose, mitigate different risks and have different costs associated with it.

And in this sense term context driven fits by my opinion the situation perfectly. So depending on context on your system, you need to find the best way to mitigate performance risk. Doesn't mean that you need to do every possible performance test you need. Maybe you don't. And now it is even more part of performance engineering because some part other things mitigate performance risks too. So for example, if you have full control on your production system and you have good monitoring in it and you make do some kind of canary testing in production, your risk is slower. So maybe you don't need to spend.

Too much time testing system before that. But from another side I still believe that you need to do performance testing to eliminate whatever you could. Because moving testing is on real users. I still believe it is not. The best idea was somewhat popularized by companies that work in social networks or some other kind of customer entertainment which have of course very interesting and cutting edge technologies, but they can allow themselves that some not optimal customer experience. But in other words, when we have some financial system or some other kind of systems that mission critical or IoT system, when real things depends on it you don't want to test everything on customers. You need to do good chunk of work perform. But how much it depends on your system and what, what is your risk?

Are you able to do exploratory testing with performance testing? Is that one way that someone can use to get over this just full blown performance testing mindset that some folks still might have?

I always believe that we should do and I always believe that it fit performance testing even more than functional testing. Because actually until you start testing, you don't know how your system behaves. For example, to create dozens of scripts and run them all together is never was a good idea. You better at least create these scripts one by one. Run some tests to see how different components components of the system behave. It is not just pass fail. Performance testing never was just pass fail because it was major input to all configuration, capacity, planning, sizing, performance, troubleshooting. And when you start to run test, you start to understand how system behaves on that load and what are how sensitive it is to the load. And in most cases, if in functional testing you know how system supposed to behave in performance, actually you don't. You have some hopes or requirements. But when you start to test system, you have no idea if it will be performed up to this requirement. So it fails with two users. So only when you start testing, you get some information about system and exploratory testing. The definition is like getting information from about system along with testing it. So I would say you always have kind of exploratory performance testing before you build that full test scale test or maybe continuous performance test. Because you need to figure out what to include there. You could do it otherwise create everything and then start to figure out what is working and what is not. So that what you get if you start just formally create scripts and add them. Because performances more sensitive and depends on all moving parts. So when you try to do move all parts together, it is difficult to understand what is going on and how we can narrow down the problem we are working with. Because I guess that formal approach assumes that we don't have problems with the system. But it never happened. In my practice with every single new system we had problems maybe with mature system which already in use for long time and some small changes. You you have already that information. But it means that you just did that kind of exploratory performance testing before it. Or maybe you did it in production. You just say experiment on your customer and collected information on system behavior from real workload.

Absolutely. So it seems like a lot of things have changed. In the past 20 years I've been in performance testing. One of them also from traditional performance testing was environments like being able to stand up environment exactly like you had in production. And the other way to really test was to test in production. But nowadays we have cloud solutions.

Any thoughts around how the environment, how you can use context testing against the different environments now that teams have to deal with?

Yes, sure, every kind of test in a way require different environment.

From one side. Actually it is even simpler. Nobody by some reason, nobody talking about it. But in cloud environment you could create a test set up as close as production as you want. You just pay more for this. So before it was just whatever you have in the lab. So it was practically impossible to scale it up. And it took a lot of time to get new equipment and set up new equipment. Right now you could run some kind of full scale test or at least large scale test for short period of time without much cost in cloud. But it is interesting that somehow that point is not highlighted. But from another side you get new kind of performance testing and needs context needs that don't need that such big environment. So if you want to do it as part of continuous integration. So in that interactive development, just to run small kind of smoke tests to make sure that nothing wrong is with performance before deploying it. You don't want to run that full scale test because you need a lot of time and resources to deploy it. You need to do it quickly. And here I have some kind of discussions that if you don't do this with realistic environments, there is no use of it. You can miss a lot of problems related with full scale system. I don't believe that it is a good reasoning because. Because it is not all or nothing game here. Even if you run 10 user tests, you could catch quite a few performance issues or at least see if performance degrade. Even single user test is helpful. Because if single user performance degrade, multi user performance would probably degrade too. It may be a few exceptions, very exotic. But as a rule you even just single user performance test, right? Just running perform functional tests and checking timing would be helpful. Yes, you definitely could miss some performance issue. And then more performance tests and larger scale and more realistic tests you will run, the more problem you will catch. But it has costs associated with that tool. And it still doesn't guarantee you that everything will work perfectly in production. Nothing guarantee you. You could mitigate risk, decrease risk that something going wrong. But it's still not a full guarantee because you never could do everything. You can See in production in your test environment it always could be some weird combination that will impact performance in some unpredicted way. But I don't see it as a reason to not do performance testing. I would say one more reason to do different levels of performance testing to mitigate performance risk as much as possible in some reasonable way.

And I guess context driven also is I love I'll have a PDF in the show notes that you have on this whole topic but you have some good questions to ask about defining risk and your performance testing strategy so that can help guide the teams to know what to focus their efforts on.

Definitely. And again it change a lot because we have different kind of a system like IoT systems that just kind different.

Animal so you may need quite a different approach there or some other kind of system. So before they become distributed system in real sense of the word so they became dynamically configured systems so you even don't know what configuration is. So many tools, monitoring tools have kind of configuration discovery again and behavior system become different not necessarily behaving straightforward way it may adjust functionality depending on load. So all kind of resilience and reliability engineering some solutions of how you move load or when something fails in your system and that related to performance and related to performance testing.

Okay Alex, before we go, what's the best way people could find or contact you?

You could find me on twitter apodelcom, they could send me email again@apoddelcon, first initial and last name@yahoo.com and LinkedIn. I trying to accept all requests at least for people who somehow involved in high tech. So if somebody want to discuss something with me, we can use any of this medium. Probably email and linked will be better for discussions.

Thank you Alex for your performance testing awesomeness. For links to everything of value we talked about in this episode, head on over to testguild.com p likeperformance p09 and while you're there make sure to click on the Try it today link under the exclusive sponsor section to learn all about SmartBear's awesome performance testing solution. Load ninja and if this episode has helped you in any way, if you haven't already, why not rate and review it in itunes? Ratings really help in the rankings and help expose the show to more new people. And you can help spread performance testing awesomeness to the masses. So that's it for this episode of the Performance Testing and site Reliability Podcast. I'm Joe. My mission is to help you succeed with creating end to end automation awesomeness. As always, test everything and keep the good. Cheers.

Thanks for listening to the Test Guild Performance Podcast. Head on over to testguild.com for full show notes, amazing blog articles, and online testing conferences. Make sure to subscribe to to join the Guild and continue your testing journey. This has been a Joe Calantonio production.