Automated Quality Gates in Performance Testing with Roman Ferstl

Welcome to the Test Guild Performance and Site Reliability Podcast where we all get together to learn more about performance testing with your host, Joe Calantonio.

Hey, it's Joe. And welcome to another episode of the Test Guild Performance and Site Reliability Podcast. Today we'll be talking with Roman all about automated quality gates and performance testing and all other kinds of performance engineering topics. Roman is a former software developer with an advanced degree in Astrophysics from the University of Vienna. So as you can tell, he's probably always been attracted to very complex problems. After gaining several years of software engineering experience, he founded Triscon in 2018, which is a company dedicated to performance engineering. Roman enjoys helping to evolve the newest approaches and performance monitoring and performance engineering, such as the automation of like test designs or DevOps approaches and integrating performance tests into the CICD pipeline. And as a former software engineer, he enjoys going beyond performance testing to find and understanding the nature and the root cause of a problem. You don't want to miss this episode. Check it out.

This episode is brought to you by SmartBear. Listen load testing is tough. Investing in the right tools to automate tests, identify bottlenec and resolve issues quickly could save your organization time and money. SmartBear offers a suite of performance tools like Load Ninja, which is a SaaS UI load testing tool, and Load UI Pro, an API load testing tool to help teams get full visibility into UI and API performance so you can release and recover faster than ever. Give it a shot. It's free and easy to try and head on over to smartbear.com solutions performancetesting to learn more. Hey Roman, welcome to the guild.

Hi Joe. First of all, let me thank you for inviting me to this great talk. I would also like to start with an invitation actually. If you or anyone else who's listening ever comes to Austria and wants to talk about performance, feel free to contact me. You're very much welcome. Come by to our office in Vienna for coffee. Have a little chat about performance testing. We always love to talk about this topic. Just google Triscon spells T R I S C O N and you'll find us.

Awesome. I actually saw a video. You're in a beautiful building in Vienna due to COVID 19. Are you still around? Like if I flew there today, would I still be able to have coffee with you?

Not right now since we are in the second lockdown phase. But apart from that, usually yes, then we're there. Right now it's home office and yeah it's coff called the Vienna's Millennium Tower. As you Mentioned It's a nice building. We are in the 23rd floor. So you got a nice view over Vienna there as well.

Nice. So there are two pivots in your career that I'd like to dive into before we get into the meat of the topic.

Sure.

The first one is, I think you're the fourth person I've interviewed that has an advanced degree in astrophysics.

Really?

Yeah. What's the deal with that? Are there any things you take away from astrophysics that make people want to be software engineers or vice versa?

That's a. Yeah. That's really astonishing that I'm already the fourth person with this kind of background.

Because there's not so many people out there. But anyways, actually there is. I've had to do with the topic of performance engineering as during my studies as I was working with a science team on a space project. So during my masters I was part of a team and we developed the onboard software for a space telescope and I was doing the algorithms, so called centroiding algorithms, which are responsible for the telescope not losing the star once it's observed a star in space. So there is a link, but it's actually coincidentally it didn't lead me directly to my work that I'm doing right now. But there is a lot of software engineering going on in astrophysics as well. Numerical simulations and stuff.

Yeah, I don't know if this has anything to do with that. I know they're starting to do mining of asteroids and so I'm sure there's different types of software that need to be created in order to do that. So what? Like that sounds like more fun to me than performance engineering. So what made you then make the switch over from software engineering to performance engineering?

Another good question. Actually. I was always present in both worlds, the scientific world as well as the business world, since I was working as a software engineer part time during my studies to finance it. And I just enjoyed the business world more in terms of the entire collaboration and communication aspects. It's something that I decided based on all the exciting projects that I had going on besides my studies there at that time.

Nice. So were you on a particular project that got you involved as a software engineer with performance and you said, wait a minute, here's an area I never thought about that I want to learn more about.

Actually I was always attracted to new kinds of technologies and. And probably, if I really be honest, not only about technologies and challenges that come with it, it's a fast living world in the it. I do enjoy that really Much. It's a lot of change going on there. Usually compared to scientific world, life is a little bit more slow paced, I'd say since you work on one subject or one topic for let's say at least a year or two, if you do a PhD, you focus on one thing for like three or four years at one time. And I found it more exciting to learn a lot of new stuff fast. And what I personally enjoyed really a lot was working together with my colleagues at this time during my work as software engineer.

So you start a company and it seems to be mainly about performance engineering. So do you feel you must speak with a lot of customers or see a lot of people developing software? Different situations. Do you see that they still don't know that they need a need for performance engineering help or do you think they see a need but they just don't have the resources that know how to do it?

I personally think that there is a lot of need that is unseen currently in terms of performance engineering. We obviously, of course we do have customers who see the need and we do a lot of work together with them. But I think there is still a huge amount who is not aware how important this topic is. And with the whole sre, the site reliability engineering coming up, which is in my opinion a really hot topic right now, I think also the entire performance engineering will thrive and evolve in the near future.

Absolutely. Like you said, I noticed a lot of companies may not be aware exactly what is needed, but they know there's some, some sort of need in the performance arena. So you did mention SRE seems to be a hot topic the past five years or so, four or three years, whatever. So do you see that people again confuse them what SRE is compared to traditional performance testing or they basically now the merging of two different disciplines into one?

I wouldn't say I think one. I think they both fit in together quite well. The traditional performance testing has to change a little bit to fit in the new SRE approaches and to the fast living DevOps world I would say. But in principle it's a synergy. For example, if, I mean it's all about site reliability. If you want to make sure that you can rely on your IT services, obviously you have to do performance tests as there are some bugs and problems and stability issues that you only can uncover in performance testing. So yes, I think it's, it's really important to combine these things and that performance testing should actually be part of, be mandatory as part of sre.

Definitely. You know, I notice also with Performance testing. A lot of times people still think it's like an activity that's done after the software is developed and you're just throwing a huge load on the system to make sure it's able to handle it. But I believe your session recently with NeoTest on actually integrating automation or performance scripts into your automation quality gates is kind of to me still an advanced topic. So before we dive into that, what do you see are some of the benefits of automating your performance test in a way that you can integrate it more into your performance engineering type of CI CD pipelines?

Yes, so I think that's what actually I meant by saying that the performance engineering and performance testing field has to adapt to the current times. And by saying that I actually meant exactly that there is the need for a lot of more automation in the entire process to avoid having performance tests just after everything is finished and then before to go live. You do a big load test and it's probably too late to react to the results appropriately. So yes, it's a clear need to integrate this process more in the early development.

So Raman, I think a lot of times, once again people think they have to wait to even get to cicd and sometimes people even pushing it. More left than that, are there any tools, techniques developers can do to be more aware of performance? Especially when they're making these like single page type applications where it really is like front end performance that is causing an issue, not necessarily a large load on the back end causing an issue.

So if you are a front end developer, there's a couple of tools out there, for instance sonar or something like that, just Google some. There's really quite a big tool set out there to evaluate your load times, page load times for front end tests. But what's really important is to cover the entire thing if you want to also do backend and service API testing, etc. There's like really a bunch of tools and it really doesn't matter what you take, just get started with one tool, no matter if it's front end or if it's API use, soap, UI use, JMeter use neoload, use anything that looks good for you. But I think the important thing is that you are aware that you should do it, that you should care about performance. And I actually feel the developers there when they're not doing it since probably they're in a hurry sprinting from one release to the other and to cover something that might be totally new even to performance testing with all the concepts that come with it because There is no single performance test. You have different test types, different test goals. You need to consider how to set up scenario and stuff like that. So a lot of time is actually needed to understand what to do to do it right. And I think that is probably one obstacle. That's why performance testing isn't as widespread as it should be.

Well, I agree with you 100%, especially when you start talking about integrating things into the pipeline where developers check in code and then you tell them, oh, by the way, we're going to add some performance scripts that are going to run. They probably start getting freaked out because they're going to slow down, usually how long it takes for that build to be integrated. So before we get into that, then when you talk about performance testing in a quality pipeline, what exactly does that mean? Or it's our quality gate.

What does it mean? Automated quality gate? It actually means, to sum it up in one sentence, to have a direct feedback in a pipeline, or not necessarily in the pipeline of your test, to automatically evaluate a performance test and decide whether it's good enough for an application to proceed to the next stage. So the gate approach simply means that if one application is transitioning from a development stage, so to say, to the next level, the testing stage, there might be a quality gate in between in this transition, preventing the application from moving from one stage or progressing from one stage to the other already, as there may be performance issues that should be fixed beforehand. And what's necessary there is the automation of a lot of things. If you want to automate performance test, the automation of the quality gate, this entire approach is consisting of a lot of automation processes. To get there, first you have to automate the test design and automate the test execution as well, and then automate the test analysis. So if you think about it, you have a performance test, you have to do your scripts, you have to maintain your scripts. Last year during the NEOTIS ramp up, I talked about how to automate the test design. The entire correlation stuff that you have to do for performance tests, that makes it really intense and time consuming in terms of maintenance. And you can automate this, for instance, by automating your functional tests, such as tests you have in Selenium or, or anywhere. There are nice integrations. We often use the tool NeoLoad since it offers such nice integrations. Obviously this is something that is not so easy. Automating the execution is easy. You just have to trigger your test somewhere. It can be from a pipeline, web interface, it can be from anywhere. And then automating the test analysis to really evaluate is it good enough or not, That's a really complex step. And this is, so to say, the final step. And altogether, if you end up having a score, because this is the goal, you want to make a decision based on all the metrics that are relevant for a load test. Those end up in a final score, and you can determine if it's good or not to proceed to the next stage automatically. But why is it so complex? It's actually simply so much different compared to traditional functional tests or unit tests. Because if you do test automation on browser level, for functional tests, you have simply a pass or fail scenario even. It can work or it can't work. But if you do a performance test, there is a lot going on. You have multiple tests executed at the same time from probably hundreds of virtual users. So it's actually not one single test run, it's hundreds of thousands. So you get a lot of data. And to evaluate this, you have to apply statistics. This is one thing and one challenge. And the other is that you have a whole bunch of metrics that are interesting. So why is that? Because in performance testing, you care about stability, you care about concurrency, error rates, you simply don't want to break your site. And next thing is, of course, you care about the performance, about the speed, and you also care, and this is the third thing, the resource about the resource utilization. You want to know how much resources am I using from, so to say, from my red hat OpenShift cluster, something that is also related to cost. And all this should be covered in performance tests. And for all these things, there are different kinds of metrics available, and you need a way to automatically evaluate those and combine them into a final decision. And this is something that performance engineers are doing manually for years. And there's a good reason for this, because if you start to automate this, of course there's also coming risk with it if you are not considering all the different things and statistics that might be necessary to make a good decision.

And once again, I totally agree. I think that another challenge is getting the environment. So when you do scale up your stress test or your load test, that you have the resources to do that. So I guess what are the tools and pieces that you use to create an automated quality gate and performance testing? I think one of them was captain. But before we get into that, are there certain tools you use across the different stages and what does it look like?

Yes. So basically you need three components. So one is you need a tool for executing the load. Most of the cases we use new load. We also showed this at the Neotus pack event. With NeoLoad, you need an APM tool to gather your metrics. Usually we are using Dynatrace for this and you need a third component that is responsible for evaluating all these metrics that I mentioned before and herefore we use Captain. So it's a combination of, for instance NeoLoad, Dynatrace and Kapton. But it can be a combination of any tool that stresses load or generates load, an APM tool and a tool for automatic evaluation of your metrics. So what we did is we had a demo application, it's called the Triscontesting Turf. We use it actually for recruiting challenges. There are bugs in there that can be triggered if you do a simple load test. And if you're able to do that, you actually pass to the next stage in our recruiting. So we did this at the pack event. We performed the load test with neoload to the Triscontesting turf. It's a simple application. You can do registrations. They are just a registration form that you fill out and submit and you will end up seeing a lot of users in a list. So we generate this load. The entire application is constantly being monitored by Dynatrace, meaning that we see on all service levels, front end as well as backend, as well as database. We see how many resources, how much resources are consumed, we see the individual performance in terms of response times as well as error rates on each individual component. And if you have a highly reproducible test, by simply making smart test design, you can do the same test over and over again to do some baselining. So you will be very aware of anything, any little metric that changes from one release to another. And the third component, Captain has to gather this metrics and it can gather it from any APM tool. Captain is open source project, you can check it out online and it can gather metrics from so called SLI providers. So slis, it's a terminology that comes from Google sre. So we're going to talk about slis, slos. Let me just wait for it for a minute. Finish this. So to sum it up, what we showed is that you can stress an application, for instance with NeoLoad, have your application being monitored by an APM tool by Dynatrace. This is happening anyway, if you have it. And then after the test you send an event to Captain to tell Captain when the load test was performed and it will automatically evaluate a score based on these metrics. So where do these metrics come from. I already mentioned the whole SRE topic a couple of times. So basically, if you're a performance engineer, this is not new to you. You have evaluated your load test previously, maybe using performance counters that you gather manually, or you already have an APM tool and build your own load test dashboards. Since there is so many metrics that you may want to consider, it's really hard to tell any software that evaluates it manually what metrics you want to have and what are the goals. So we thought there must be an easier approach to this, otherwise no one's going to do it. And we had a workshop together with Andreas Grabner, who's one of the main contributors to Captain. And the idea was that you simply gather these metrics automatically from your load test dashboard because there's high chances that you will build it anyway to analyze your test and maybe it's already even there, so you can simply reuse it. And in this dashboard where your metrics are plotted in your nice graphs, etc. You simply add a tiny additional piece of information of what's the goal on this metrics in a way that Captain understands. And after that, what's happening after the test is that Captain is pulling out these metrics and combining them into a final score. So there is actually a lot of things that are going on there. I've mentioned pipeline and I just want to sum it up. You don't even necessarily need a pipeline. It doesn't matter from where you trigger your test, because if you're a performance engineer, maybe you do not care about pipelines yet. So simply start your load test as you do always have your APM tool monitor your application as it's already been done anyways. And you can add the third component completely independent from that, automatically evaluate the score by simply triggering this single event, telling Captain when the test was performed from here to then and where. The SLI SLO dashboard is.

Very cool. So when you talk about scores like is it a main Overall score like 80? And then when you dive in it, it breaks it out into where you scored bad or well, and then does it give you suggestions on how to actually fix it or is it just like, can you just talk a little bit more about what you get with that scoring?

Yes, absolutely. So what you're going to end up with is I will stick with Captain now, since it's what we choose here. And what you see after the test, after it's been evaluated, is in the so called Captain's Bridge, you see all the individual metrics that you defined that are important and necessary and interesting for a load test. You see what value they have and this is where it already happens. So you have to previously define if you want to have for instance, the average or median or 90% percentile of the response time, error, etc. This is something that you have to predefine. This is the so called sli, the service level indicator. So what to measure and how to measure it. In addition, you define a goal. So for instance, let's stick with an example. The response time should be less than 300 milliseconds in 90% of all cases. So this would be a proper slo. So captain will evaluate then and figure out if this individual goal, which is just a part of the overall score, is actually met or not. And then you have also the option to put in some weighting that you want to have. Some SLIs being more important than others. And overall, if you have hundreds of apartments SLIs with no weighting. Yeah, so all are similar. And then you would each individual SLI would simply be a score of one. And you always, no matter how many SLIs you evaluate or define, you always, always end up with a score from 0 to 100. And to determine whether the test fails or not, you can also individually define if you want the test to be green, if the score is 80 or above, or if it's whatever you need it to be.

So when I spoke to Andy about Captain, like fairly new when that just first came out and so are you actually using it in production or is this a proof of concept? And like, can you attest to how well Captain has been working for you in the real world?

Yes, it's constantly evolving and we are actually using it already for one of our customers and we're using it to evaluate our performance tests, which we do in test stages. Captain has a lot of other use cases. This would. What I'm talking about is the so called quality gate use case. If you go to the Captain's homepage, you will see topics about auto remediation, autonomous cloud. So it's all about deciding whether a new build is in production being better or worse than the other blue, green deployments, etc. We're not doing this yet. We are focusing only on the automated quality gate approach to decide whether our test was good or not. And we do this in test stages. And as I mentioned before, what was really necessary to make it usable in the real world, as you said, was having a nice way of defining this slis and slos with this so called dashboard approach that we came up with in the workshop because in the previous version you had to define all this manually in YAML files. So it would be a lot of effort to maintain this and to initially set it up. For instance, for small load test with a couple of charts from 5 to 10 where I plot different server metrics and each I already end up with a YAML in behind that I don't care about anymore, which has more than 500 lines of code and you don't want to write that manually. And since this is available now, it's really feasible and with not a lot of time that's necessary to evaluate your load test with this approach. So yes, it works in the real world for us.

Very nice. Speaking of the real world as well, it does seem like it's kind of new to get teams on board to do this automated quality gate type approach to performance. So any lessons learned that you think people have been struggling with that you say there, you think, oh, if they just knew about this, they would make an easier transition to this type of approach.

I think actually the entire community around Kaptain is working hard on this to make the installation, etc. Really easy. I think with only one or two lines of code you can actually set up Captain on a new server. And Captain runs on kubernetes. So you would think that you have to set up Kubernetes from the scratch and then deploy, etc. But no, I can say and that it's a lot easier with the installation that the community of Captain provides here. And all this gets spun up automatically. And so for us it was actually the main challenge in the corporate environment where you may have not access to anything from anywhere to care about firewalls, since there's API communication that's necessary. So Captain needs to reach your APM tool and if you want to have it triggered from a pipeline, you also need to have a communication between Captain and the pipeline software there. And of course you need a server somewhere to set it up if you want to have it on premise, but this is actually just something that you have with any kind of software. So I would say it's pretty straightforward and easy to give it a try.

So we're only a few months out from 2021. Any thoughts on the future of performance engineering or any trends you definitely see happening that folks need to be aware of?

That's an interesting question. I've seen trends and I have a lot of wishes. I think one trend is actually getting more feedback from production integrating SLIs, SLOs, taking them, also using them in performance tests. That's a clear trend that I see, kind of fusion approach for site reliability engineering. And so a clear trend is to involve performance tests in development as early as possible. But it's a trend and most of the people are still doing it. I think as you mentioned earlier, that you develop your software and then later on, at a certain degree of evolvement, maybe even just before it hits production, you do a load test. I still see this quite a lot and I think this is going to change over the time. And from the other perspective, from the technological perspective, I think what's really, really going to be exciting is browser based load testing.

There are already solutions out there that can do browser based load testing compared to the protocol based load tests, which obviously come with more maintenance, etc. But there is no solution out there that works in all environments. And by all I mean cloud based as well as on premise, which is really important as well. So I'm really excited about that and I've seen a lot of people working on getting more feedback from production also in terms of usage. So the thing is that you want to reproduce the real world in your load tests, right? So from your APM tools and also from your logs, et cetera, you want to recreate the scenarios that are actually happening in production in your load tests. And there's actually people out there, for instance at Neotis, who are trying to make that happen. I'm really excited to see that.

Kirum, before we go, is there one piece of actionable advice you can give to someone to help them with their performance testing efforts? And what's the best way to find contact you or learn more about your company and the services you offer?

All right, and advice is if you are already a performance engineer, I think that your role will become more and more important. So you can be excited about that. I personally am excited a lot to work in such a great and interesting field. And maybe as an advice, it's it's fun thing because as a performance engineer, if you think about becoming one, sometimes you are seen as a hero if you are actually able to save an application a lot of trouble of going live with a bad version. But sometimes you may also be seen as a villain if you stop an important application from going live and it's because of significant performance issues that came up. But in the end I figured up.

People will always be grateful even if they are not in the first place. So even if you may look like a villain in the first place, you will eventually maybe be seen as a hero just like Batman. So for anyone who enjoys the job as much as I do, being a performance engineer, keep that in mind. What you're doing is really important. It's a really great field and it's going to become a lot of attention in the next years. I promise that. And the best way to contact me and my colleagues, which I would like to thank at this point as well. I would like to say thank you to Fabian, Arthur and Christian, such great colleagues and excellent performance engineer. If you want to reach out to us, you can reach out to me on LinkedIn. My name is Roman Ferstel, Spells R O M A N F E R S T L or you can simply come to our webpage@triscon triscon-it.com and we offer services all about performance so APM performance testing workshops, trainings, etc. If you simply want to have a chat, just make sure to contact us.

Thanks again for your performance Testing awesomeness. Links to everything of value we covered in this episode. Head on over to testguild.com P53 and while you're there, make sure to click on the Try them both Today link under the Exclusive Sponsor section to learn all about SmartBear's two awesome performance test tool solutions, Load Ninja and Load UI Pro. And if the show has helped you in any way, why not rate and review it in itunes? Reviews really do matter in the rankings of the show and I read each and every one of them. So that's it for this episode of the Test Guild Performance and Site Reliability Podcast. I'm Joe. My mission is to help you succeed with creating end to end full stack performance Testing awesomeness. As always, test everything and keep the good. Cheers.

Thanks for listening to the Test Guild Performance and Site Reliability Podcast. Head on over to test guild.com for full show notes, amazing blog articles and online testing conferences. Don't forget to subscribe to the Guild to continue your testing journey.