Performance Testing in an Agile Environment with Michael Olechna

Welcome to the Test Guild Performance and Site Reliability Podcast where we all get together to learn more about performance testing with your host, Joe Calantonio.

Hey, it's Joe. And welcome to another episode of the Test Guild Performance and Site Reliability Podcast. And today we'll be talking with Michael all about performance testing in an agile environment. If you don't know, Michael is a product manager for Load Ninja, an industry leading cloud based load testing tool at SmartBear. And Michael is focused on helping people of all testing backgrounds to get the results and reliability they need from their sites to perform when they're needed the most. So really important topic for sure. Michael has a background in biotech. He also I think worked in 3D modeling at one time and he finally found his true passion at SmartBear. So he's worked in multiple different industries, has different backgrounds, a lot of diverse backgrounds from startups to large enterprises and really what drives him is his approach to performance testing through this unique lens that he sees working with all these different companies in his past. So really exciting to have him on the show. You don't wanna miss it. Check it out.

This episode is brought to you by the awesome folks at SmartBear. Listen, we know load testing is tough but necessary. So investing in the right tools to automate tests, identify bottlenecks and resolve issues fast saves your organization and money. And that's why SmartBear created Load Ninja, a SaaS load testing tool to help teams get full visibility into performance so you can release quality software faster than ever, make testing effortless and give it a shot. It's free and easy to try. Head on over to loadninja.com and learn more.

Hey Michael, welcome to the guild.

Hi Joe, thanks for having me.

Awesome to have you. So just curious to know you went to 3D modeling and then you said you found your passion in Smart Bear. What led you to SmartBear and performance testing?

So what led me to SmartBear and performance testing is as you said, I used to work in a large 3D modeling organization and from there I decided I wanted to go more into from information technology more towards software engineering and software as well and performance testing just I find very interesting and unique and how you can really get real hands on experience of the end user experience on your applications and really model the user experience and how really get a unique insight into how users interact with your applications.

So you know the topic I thought we'd dive into a little bit more is performance testing in the agile environment. What I love about you know, you work for a company that has a solution. So you probably speak with a lot of different companies at the enterprise level. So I'd love to find out the first thing. What are some maybe challenges you see with performance testing in an agile environment?

So I could think of three main roadblocks when it comes to this. One is a lot of people state they don't have the time or the resources to really dedicate it properly to performance testing in an agile environment. Second is they often will say they don't have the infrastructure or the tools. And you can also couple this with the skills required to really perform performance tests properly. And then third is they don't have the budget to hire additional people because of the first two factors. They often don't have the skills so they have to go outside with a third party, which can be quite costly. Or they just don't have the budget to hire the necessary testers to perform and write these script tests, these test scripts that can take a lot of time and resources because there also aren't many of these testers available for hire in general.

So what are those skills then? Is it just someone that's able to create the performance test or are there other skills that you seem that companies lack?

So mainly in terms of correlating scripts to properly mimic realistic scenarios that their users are experiencing on their site, which can be very complex and these complex scripts often require years of experience and a lot of protocol based tools and that's very difficult to find.

Absolutely. Now about the time, I guess if you're working in agile, do you see more companies trying to get performance testing early in the software development life cycle where it's actually part of a sprint?

I think that's been the shift is that it really, what needs to become a part of a company's model is test early and test often and really incorporate it in throughout the sprint. So there's a cycle of continuous improvement throughout every stage of the software development life cycle. So performance testing isn't taking a back seat but becoming more of a front seat and becoming a part of their normal workflow.

Absolutely. So you did mention, I think, three constraints that you see with most companies. So how do you address those constraints then in order to then get them to get buy in to actually starting to do some performance testing? Like how do you get them to, okay, you need to up your infrastructure or you need to up your skills or you need to budget more time into your sprints to address this.

So in terms of the first constraint with skills and time, you need to Be able to find a solution that isn't as time intensive or may not require these extensive correlating skills and writing test script scenarios. Maybe that's a low code or even no code option that can really save yourself time and be able to get up and running and start writing these test scenarios very quickly. And then in terms of the infrastructure, one way you can do this is by looking at by a lot of people don't have the proper test environment. And so if you that can be costly in order to create a proper test environment, but you can use your existing live environment and say testing off hours when you know there aren't as many people using your application. And so that's a cost effective way of being able to get real results while also being cost effective.

So I guess the question next is what are some benefits then that people get if you think they start incorporating performance tests into their agile type of environments?

So some of the benefits include that you are able to continuously improve your performance application. And really the main benefit with that is that continuous improvement is directly intertwined with quality because you're continuously monitoring your application, continuously finding areas of improvement, and you're consistently monitoring your application. Then you can make these fixes to your application throughout each sprint. Well, instead of putting it on the back burner, you're going to end up maybe having a last minute issue that really ends up becoming a much larger issue. And you can avoid that by testing throughout instead of leaving it all to the end and then realizing that you have maybe a main issue on your hardware or something like that that is very costly to fix and as well time intensive.

So I know when a lot of people hear performance testing they automatically think of like stress testing or load testing. When you're doing performance testing in Agile, what are the main type of testing buckets people usually perform? Is it stress testing or is it more like how is it performed for one user or things like that.

So primarily I'm thinking from a load testing point of view. So being able to monitor the amount of users on your application, the way they interact with your application, and factoring in the geographies they are accessing your application from as well as the devices they may be on. Because those are the three main areas within each load test you need to factor in. Because depending on which device they're using, where they're located and when they're, the amount of traffic on your, say your website that can seriously alter their experience with your application.

Absolutely. So I think this lends itself to start talking a Little bit more about Load Ninja because I don't know if open source solutions have that full rich functionality like being able to run on different devices, different locations. So I guess you know you are the sponsor of the podcast, so thank you for that. For the folks that don't know what is Load Ninja.

So Load Ninja is a cloud based solution focused on performance testings and being able to write scripts without even necessarily having a background in load testing. So a lot of load testing can be done through open source solutions and these open source solutions are focused on the protocol layer as well as many flag name load testing solutions. And that can be very problematic for people that have very little experience with load testing or they don't have the time to really learn these tools. And Load Ninja is able to get them up and running testing very fast because it's a no code solution, you don't need to write a single bit of code. And you can also test through the cloud, which means instead of through emulators you are testing with real browsers. So you gain a very realistic representation of how your end users are interacting with your website or your web application. So that in and of itself is very useful and being able to really understand your users experiences as well as being able for you to begin testing without having really extensive background in writing test scripts.

Great. You did mention user experience. Why should people care about that? Usually people think it's a non functional thing, don't necessarily need to focus on it. What makes user experience so important?

Ultimately user experience is important because the users are the ones that are interacting with your application and they're the ones that are going through checkout processes on your site, they're purchasing, say you have an E commerce site, they're the ones making the purchase, they're activity is directly tied into your revenue based activities and that is ultimately going to have a larger impact on your site. And because users can be also very vocal in their experience and how they have either had a positive or a negative experience on your site and they can also show by not coming back and you want those users to keep coming back and having a positive interaction with your site. You don't want them going to say a competitor because of a negative experience.

So you know, you mentioned a few key benefits of using a tool like Load Ninja. I want to make sure we cover all of them. What makes Load Ninja an ideal tool, say for performance testing in an agile environment? Sounds like it's no code, low code, easy to pick up and it works in the cloud. Did I miss Anything or something else that makes it really good for this type of performance testing within an agile team.

So one also great thing about loadninja is you have the ability to record and playback your test scripts. So instead of having to for each test, be able to, instead of having to write each test individually, you can record in Low Ninja's UI recorder and you can record your test script while then playing it back later. And you can save your test script for future test case scenarios. So it saves you further time not just in the present, but in the future when you have to run similar load tests. And you can also edit that load test in the future and tweak it based on different scenarios, which is also another key benefit. And like I said, I just want to go back to the no code. One key factor in why load testing is so difficult unless you have a lot of experience writing test scripts is the fact of correlations and having the right extensive correlations to properly mimic real user scenario. One other great thing about Load Ninjas, there's no correlations required, so you're able to get an accurate representation of the end user experience because it's done at the browser level and without having to write those extensive correlations.

I've been performance testing on one of my first gigs as a performance tester like 20 plus years ago. I won't go how long, but it was over 20 years ago and correlation was a big deal. It's funny that it's still a problem and it was very difficult to do. So can you explain a little bit more, what do you mean by through a browser? So back in the day it was just a protocol. You'd send whatever the HTTP and come back and then you'd have to correlate all the dynamic values and then parameterize them and all that. So when you say you work into a browser, is it like almost a selenium script? You're working with a browser just like the user and you record it and play it back. That's a performance test.

Yeah. So it's done through Headless Chrome. So it's done through the Chrome browser, which is the most common browser used by users when visiting web based applications. And that in and of itself is very useful in determining the real end user experience instead of say testing out the protocol layer where you have to make those emulators that they mimic the experience, but it's not a true indicator and true representation of the end user experience.

Gotcha. How does it handle the correlation then? Say it's sending a dynamic data background does it know like it needs a dynamic piece of information? How does it handle it?

Yeah, so it's done through dynamic correlations for the most part.

Cool. So another thing I've seen people struggle with performance testing is after they performance test is done, looking at the results and correlating all the information. When I run this mono test, CPU went up this high, but it was actually due to something on the back end. Is there anything built into Load Ninja that helps people maybe report the results quicker or actually understand what happened during the test run?

Yes, absolutely. So a lot of times with a lot of these older performance testing tools is that you get a lot of reporting fed back to you, but it's so much data that it's almost. You don't know what to do with it because there's too much data to really get down to the specifics of what really needs, what's really impacting your site or your application. Load Ninja does report as well, but it's focused on quickly being able to identify those bottlenecks that are impacting your user experience and on your site and being able to identify them in real time and then apply those fixes as in real time instead of being distracted by all this other data that really doesn't have as much of an impact on your application.

So what kind of metrics does it give you? Is it just a standard cpu, memory disk or does it give you other metrics maybe that team would be interested in when they're working in Agile?

Yeah. So it can give metrics based off response time, time it takes to load your webpage and your application images, things like that, as well as to the metrics you stated.

Cool. So I know another trend is running performance tests in CI CD pipelines is something Load Ninja can help you with. Are there any tips that people can use when trying to run in CI cd?

Yes. So that's another great thing about Load Ninja is that it's able to fit into CI CD pipelines as well and it's a seamless fit and it can also be able to help facilitate that continuous improvement cycle and being able to test, load, test and become a part of your software development lifecycle and your established workflow instead of just being an afterthought.

What does Load Ninja work against? I know with the protocol things I could run a test against a thick client application using really some shock quad kind of protocols are really hard to kind of correlate. So does it only run against browser based applications?

Yes. Yeah, you have to be browser based in order to use Load Ninja.

So do you have any case studies and maybe where you don't have to name companies where maybe they used an open source solution, then when they moved over to Load Ninja they were able to save X amount of time or they found X amount of issues or anything like that?

Yes. So and we do and without naming names in general, because of the no code required, Load Ninja can save up to 60% of your time creating test scripts and you can get up and running and create a load test within an hour instead of sometimes it can take hours, even days to properly test script load scenario. And that's again the time savings that Load Ninja offers is extremely attractive to a lot of users that maybe have tried typical protocol based solution but did not have a great experience with and are looking for something that's a little bit faster, easier and not as time intensive.

All right, so just make sure to clarify once again, the reason why it's so easy to use then is because it actually runs against a real browser and it does correlation automatically for you and it does better reporting, quicker reporting than maybe another solution. Anything else that's built into Load Ninja that helps with the agile performance?

So again there's the being able to record and play back your scripts so you can use it in future scenarios and throughout different.

My question was so you create a performance test, say it's a large script, how do you make it because it's codeless, how do you make it reusable? Does it create like a login function for you automatically? And then when you have to create another performance test, you know there's already a login for it. Is it just like a, a business process like Enter a patient is a performance test. So people could reuse that. How does that work? Reusability part of it, I guess the.

Question so because loninja is a SaaS product, you're able to save your tests and log into the solution online and you're able to use and reuse those tests that you have previously recorded.

So Michael, do you have any like tips or things if someone's looking at a solution performance tool should have that maybe would help them with agile performance testing that they could go through?

Absolutely. So one thing I would say is you don't want to sacrifice sophistication for the sake of simplicity. So I know that Lone Ninja is very easy to use, but it's also very sophisticated in the fact that you can really with its different servers located throughout the world, you can, you're able to see how different users and different amounts of users interact in your site on different devices and in different geographies are able to experience your application. And you need to make sure that the tool that you select is able to factor things like that in because that's really indicative of what the real end user experience is like for real visitors. A second you want to keep in mind is you want a tool that allows you to monitor and test where the users are now and in the future. So like I said about geography, you also say you only are, your solution is only available in the U.S. eventually you're going to want to scale that up and you want to be able to have it a tool in place where if you were to do that it's possible. So you may want to expand to Europe, you want, or even Canada much closer, but you want to be able to have a tool in place where that's a very simple scalable operation for your organization. And again you want to be able to simulate realistic use case scenarios. So one example is this. A lot of E commerce websites, maybe only 5% of visitors will go to checkout and actually make a purchase on your site. 95% of those visitors are not going to end up making a purchase. You want to factor that in and have the ability to factor that in when you're writing these scripts. Because if you have say 1,000 virtual users, you're not going to have all thousand users in that script go to checkout because that's not realistic of a actual scenario. And then lastly, you want to be able to have day and report that you can share across your whole performance team or if you don't have a performance team, your key stakeholders that has actual insights. So then you can really identify those bottlenecks and and fix those issues that are causing a negative experience for your users.

So Load Ninja, like you said, SmartBear covers every aspect of software development across your whole pipeline. If someone uses a tool like Load Ninja, does it tie into any of the other SmartBear products to get like a more end to end type of solution?

So Load Ninja doesn't currently tie into the products through a direct integration, but it you can use with a tool like TestComplete and a tool like BitBart. So you can be able to have a full testing suite in Terms of using TestComplete for functional testing scenarios and then Load Ninja for your performance testing and then bitbar for your mobile devices. So you're able to get a full picture of your application's performance. However, they don't all currently interconnect with one another.

Okay Michael, before we go, what's the best way to contact you and what's the best way that people can get their hands on Load Ninja?

So the best way to contact me is through LinkedIn. Michael Elekna at LinkedIn and the best way is to go to Loneinja's website, www. Loneinja.com and be able to request a trial through there. It's a free seven day trial and then you can get all the features of Lonenja and be able to start scripting and really get it and get into the tool and really determine its customizability and fit to your organization's needs. And yeah, also you can visit smartbear.com to access all our other great tools that are available throughout the software development lifecycle.

Thanks again for your Performance Testing Awesomeness. Links to everything of value we covered in this episode. Head on over to testguild.com P89 and while you're there, make sure to click on the Try them both Today link under the Exclusive Sponsor section to learn all about SmartBear's two awesome performance test tool solutions, Load Ninja and Load Ninja UI Pro. And if the show has helped you in any way, why not rate and review it in itunes? Reviews really do matter in the rankings of the show and I read each and every one of them. So that's it for this episode of the Test Guild Performance and Site Reliability Podcast. I'm Joe. My mission is to help you succeed with creating end to end full stack Performance Testing Awesomeness. As always, test everything and keep the good. Cheers.

Thanks for listening to the Test Guild Performance and Site Reliability Podcast. Head on over to TestGuild for full show notes, amazing blog articles and online testing conferences. Don't forget to subscribe to the Guild to continue your testing journey.