Performance Pretzel Testing with Andrei Alexandr Danilov

Welcome to the Test Guild Performance and Site Reliability Podcast where we all get together to learn more about performance testing with your host, Joe Calantonio.

Hey, it's Joe. And welcome to another episode of the Test Guild Performance and Site Reliability Podcast. Today we'll be talking with Andre all about Performance Pretzel testing, which is BDD performance testing using Java cucumber, rest assured, Locus 4J and more. You don't want to miss this episode. Andre was born and raised in the Agile era, which makes me feel old. I don't even know how old he is, but that makes me feel real old. He has multi industry experience in banking, real estate, telecom, insurance, Internet security, marketing. There are a handful of different platforms he's done back end, front end, mobile, desktop. If you ever go to his LinkedIn profile, you know he's involved with a lot of different testing. He's my kind of tester. He finds himself really enjoying the state of the art of software development. Classical QA and testing approaches he's into as well. And he does a lot of different activities with manual automation testing, handling product functionality to non functional requirements, executing architecture solution implementations and leading QA departments and activities. You don't want to miss this episode. Check it out.

This episode is brought to you by SmartBear. Listen, load testing is tough. Investing in the right tools to automate tests, identify bottlenecks and resolve issues quickly could save your organization time and money. SmartBear offers a suite of performance tools like Load Ninja, which is a SAS UI load testing tool, and loaduipro, an API load testing tool to help teams get full visibility into UI and API performance so you can release and recover faster than ever. Give it a shot. It's free and easy to try. Head on over to smartbear.com solutions performancetesting to learn more.

Hey Andre. Welcome to the Guild.

Hi Joe. Nice to be here.

Great to have you. So before we get into it, is there anything I missed in your bio that you want the Guild to know more about?

I think you got it pretty right. Apart from it, I'd like to explore wild ideas such as this one that we'll be talking about today.

Absolutely. So you do have a functional testing background, it seems. What got you into performance before we go into your solution for your performance product? I know I started off with Performance Test and I used loadrunner and I really enjoyed using it. So why did you decide what were the issues with loadrunner that made you want to look somewhere else?

The issue with Roadrunner and also other performance testing tools, it's like the general problem with the performance testing, let's say in general.

Right.

One thing that we have this tool that is like external to the team, the team needs to get skilled out of it. It usually required some external team to do it. And it's usually this how it was set up, let's say more in the classical approaches from the past. And it's not part of the, let's say things that teams own inside an organization, something that usually something else. And they feel like I'm not skilled enough. I don't know this much, this tool and I'm focusing on developing an UI or developing this service and I don't have time to learn this. And since this is not part of my code base, I might end up not doing it. So in the end we have specialized teams that do this amazingly. But I think it's a feeling of ownership. Right? And a feeling of ownership and a feeling of bringing this activity somewhere more closer to products, to requirements, making them more involved into this discussion that is somehow left for engineering teams to figure out without any product involvement.

So in order to do that, I guess that leads us to. Which I've never seen before. Well, maybe I have BDD with performance testing. Usually when someone thinks of bdd, they think of functional test automation. So is that what you thought you would use in order to get people more involved with performances to use BDD on the front end?

I've used BDD in multiple locations in the past a lot for front end, also a lot for backend, also to test some very specific tools that were inserting things like for example, in a database. The nice thing about BDD is that it shows the business benefit of the software that you are building and it's describing it in a way that a user can understand the business can understand product can understand, and they can understand the benefit behind it. Right. So indeed this is where I thought that I could be linking BDD with performance testing because it's not something that it's not valuable, it's something that we should be thinking from when we create the user story. It's something that should be there in the process of discussing, maybe in each grooming session. And we kind of need to talk about it. We cannot just leave this for a later stage when it cannot be maybe fixed or it cannot be solved as fast as possible. As fast as if we would have fought for it in the beginning. Right?

Yeah. So when you're planning up front, then your teams actually have on their backlog. Let's have a performance story and then you write that in bdd.

What I would say is the idea behind Pretzel and BDD with performance testing is that we don't have separate stories for performance. You have this embedded in the same story. As easy as you say, given a user logs in when it clicks the Google button, you get to log in with your Gmail account, for example. You can do that for multiple users. So inside the same ticket, inside the same story, we can define this acceptance criteria, let's say in the BDD format. That includes performance as well. So it's not thought of something extra? No, it's again going in the same philosophy of I think in Agile, in splitting, testing and development in different tickets or different stories. The idea with this is to help them actually bring them together and understand that this is a piece of value that we want to increment the product with. The piece of value also includes performance here, that multiple users are actually be able to do that at the same time in how many users for example, at what rate?

So can you give an example? Given I'm logged into my application as five users and I go to check my email, then response time shall be less than five seconds or something, I don't know. But what does your BDD look like for that type of story, for example?

I would say that it will look exactly the same as the BDD for one story, but it will tell us more about the multiple users we will be trying to spawn. Right. So for example, when 100 users try to log in at the rate of 10 users per second for 5 minutes, then the login is successful and it's within the predefined sla, for example.

Nice. So I'm trying to envision what you have set up. You have multiple kind of technologies going on. So the front end is your BDD and I believe using Java and Cucumber.

Yeah.

So to implement the actual feature, then what's behind the scenes that's running the performance test as multiple user?

Okay, so behind the scenes we have the automation technology that you use to write your functional tests. So example that we provide in Nightcode Labs in GitHub you will find service test. So an API test. We are using the rest assured to test the functional side and we are trying to reuse the same infrastructure that is already done with the service object. Already done for functional. We are reusing it with performance. So how do we reuse it? We plug in into Pretzel, which is an MVM dependency that me and David Marcus, which are the knightcollabs have created in order to help us to do all of these things. All of the performance Pretzel uses in the back Locust. And Locust is already a performance tool that is used out there, but it's very typical for the Python, let's say ecosystem.

Right?

So how do we use that? We like Locus because you can create a worker for it, that it can be in a different programming language, that it can be your programming language and you can connect to Locust Instance, which called by them as the Master, which actually runs the tooling. So with this small interface that you can build, you can program in any language and talk to the actual logic that is in the, let's say in the locust Master. For this we are using a library that was also out there, opened by the community, which is Locust 4G. So already an interface with Locust that helps us to communicate and to execute all of this. You might wonder, why do we need to create this dependency? Why can't we just use locus 4G? The idea that me and David started to think is, hey, let's create a test project that uses this and does the performance testing with Cucumber. We ended up with a lot of helpers, with a lot of things that we needed in order to make this perfectly run within Cucumber and with the other things that we have implemented, like for example, the reporting. So Pretzel helps you actually to make this easier, so you don't need to rewrite the same things that me and David did to make this idea a reality perfect.

So it sounds like if someone's using Java and Cucumber already, all they need to do then is call a maven dependency. And by doing that then they have access to be able to run performance tests utilizing Rest Assured. How does Rest Assured and Locus communicate?

They don't really communicate, just Pretzel by itself is running as a task. This Rest Assured code that you have written in functional tests. So imagine you want to make an API call to do a specific action over service. For example, let's make a post to update a name of a user, right? What's going to happen inside Pretzel is that we are going to say, okay, this code is going to be repeated multiple times by Locust. So Locust will know that it needs to go and do this multiple times. So you don't need to code again the same let's say functional test that you already had. You already have the, let's say the service object. With the logic of making the post, getting the Test data if you need it, so on and so on environment. What you just need is now I want to Execute this into 100 users for time. That can be five minutes. What's the hatch rate? Maybe 10 users per second. And you want to check finally the time. So this will be where are the difference? Right? You want to control the performance test and you'll be reusing in the end all the methods that you have already created with rest assured in a functional scenario, even the assertions.

So when it does run a performance test, does it spin up containers specific to running multiple users, or do you have to specify that ahead of time where to run them?

It doesn't. What it does, it takes from the scenario the amount of users that you want to hatch, and based off of all that information that you provide regarding the performance, it launches Locust Master that is able to understand that information. When this test is finally executed, we go on to the next test and we destroy the previous instance, let's say of the configuration. So the number of users and all of the for example, reports that we got while saving what is interesting for us. So the data to generate the reports finally, and so on, so on. So we are by design Pretzel isolates the scenario itself within Locust, executes it, and after executing it takes the reports. While taking the reports, the current instance of Locust is shut down and the next scenario comes in. When the next scenario comes in, we are launching another Locust instance and we are going to feed it with the information for the next scenario.

Cool. So what does the reporting look like at the end result? Is there a special annotation you add in order to get the performance data and your results?

Exactly. We use the annotation, the tags that come with the cucumber and we tag each one of the scenario. Like for example, at performance, you can tag it as long as you want. How the infrastructure looks like is that inside the runner you will be able to find specific method that after each class save certain information if the tag is for example, performance. Right. So what are we very interested here is we are not super interested, maybe in the functional scenario is to generate these graphs, graphs that will show the users and will show the people who execute this the different response times for each one of the scenario, in which minutes you can generate different graphs. This is what we show by default for the performance scenarios. By doing this inside the runner we are using for the project Extend reports, which is kind of standard, let's say around the Java area and scenario area to show information of execution in a nice way. But any other report can be benefiting in report format can be benefited by the same logic. Let's say you just need to slightly change this in your test project inside the runner.

Very nice. So how does it work then? How often do you run your tests? Is it when you check in all the performance tests run as well since they're just part of your normal story? I guess it just plugs in in a pipeline and does it run at every check in every night? What's the cadence for your test runs?

Actually we recommend to use this however it best suits for organization. We understand that some organizations have different deployment pipelines and this can fit wherever they kind of need it. It's ready to be run after each of let's say functional scenarios are run even before. Or it's ready to be run also maybe on schedule, overnight or when it arrives to a specific environment. This is configurable. So most probably inside the project itself. One would have, let's say a file describing the CI technology that they are using, describing what are the different steps and for which kind of branches, maybe for which kind of environment to be executing. So this would be one of these steps and maybe it would be no an if. No. For example, if we are in the performance environment, run this right. Maybe this is already configured there. There are pros and cons to use this as part of each one of the, let's say, deployments of production. And I think it's good that we leave teams the choice of when to put this inside their pipeline. Some teams might need it much more often than other teams. So Pretzel and also the example project that we kind of provide is ready to be put wherever it needed by the team.

Nice. I believe Pretzel, it's fairly new, right? I believe it's a few months old, am I correct?

Yeah, we launched it around early this May after we were working on it for a couple of years or months as, let's say, side projects. This idea, both me and David have worked together in the past and part of our work we were thinking this is a real problem that we might be solving and nobody is doing this out there, so why not try it out and let's see if we can make it. We were able to make something. We had something running around January, but we were not super happy with the shape of it. It was like what developers usually call monolith that we wanted to refactor. We wanted to put it more nicely so it could be reused by anybody out There finally, around May, we were happy enough to release, let's say, an alpha version of it. And we are looking forward to understand if there's any people out there that might be interested in this approach and why not work together to develop this and make this usable in an enterprise, let's say, environment, right?

Absolutely. I think it's a great idea. So because it is so fairly new, is there anything on your roadmap or functionality you'd like to see added? Maybe because it is open source that people can contribute that you'd like to see added to Pretzel?

I think we have around a couple of things that we saw and we would like to improve. One of it is concurrency. So as I was explaining previously, it allows just, let's say sequential execution. And we saw that, yeah, this is beneficial for, let's say, for a fair or small amount of test cases. But when this starts to grow, this cannot be longer part of your CI pipeline with every execution, if it takes hours, maybe to execute. So the concept of parallelization would be very nice. Now another part, my feeling is that it's a bit heavy. So as you could notice, you can also see from the documentation we explain that we execute, so we have the Locus Master. So the execution itself, it's happening in the same machine where the pipeline is executed or where you are trying to run the code from. For example, if you're doing an maven clean test, yeah, you are going to get over there inside the same machine. This test run, which is okay for starters, but it's definitely improvable because we understand that there might be people running this in a small Jenkins server and running performance test. It's a heavy thing and might put these machines down. And also it doesn't really allow us to do the parallel thing that will help us.

Right.

We have some other things, like some tweaks that we like to do with the code. We have some things with the locus 4G library that we would like to improve. They are having this concept of task that we kind of inherit and we are not super fan of, but this is a very nice opportunity for us to contribute ourselves to the other project to see if we can somehow shape it to make the use case for Pretzel affair more nicer. And on the other hand, it's like any automation tool. So if you have been to famous speeches like by Michael Bolton, it makes a check, you know, it doesn't really tell you where the problem is. So it will be nice to enhance on that and maybe provide at least Some guideline or at least for us to explore how we could relate this to maybe some monitoring tools that are out there and we could possibly link like logs and the checks at the same time so we can potentially easy up the work of why this happened that testers need to do.

Very cool. So when you first started this, how did your team embrace it? Was it really hard to convince them that, hey, we need some sort of performance built into our framework?

The idea of this started a bit as a need and it started more as a need of, hey, can we reduce the, can we reuse what we already have and not redo all of functional test or the functional test infrastructure in another tool? Like, for example, we were looking at code based solutions for this, like Gatling, for example. And the challenge there was, yeah, we will need to learn Scala, we will need to learn Gatling, and we will need to rewrite everything what we have inside Gatling. So isn't there maybe something that we could be reusing for Java and that will help us, let's say, put the test in the same way? And then the natural discussion came out of bdd, came out of, yeah, but we have these reports. Wouldn't it be better that also performance tests benefit from these reports? And then even going a step further, we started to discuss. These are things that we never have talked about in our grooming sessions or with products or with product owners, with business analysts. This is requirements that never come to the team. So how can we make that maybe BDD is kind of already solving us this problem? Could we reuse the same approach for performance? So this is when we started to explore this idea with the team.

Okay, Andre, before we go, is there one piece of actual advice you can give to someone to help them with their performance testing efforts? And what's the best way to find or contact you or learn more about Pretzel? Cool.

So one of the few things that I would definitely recommend when starting with this is look what solution out there and don't be afraid to innovate if you need to. Even a wild idea can help you to be better at performance testing. For me, and I think for a team I was leading, the most important thing is how did we get this, how did we get business and how did we get product? To talk about this concept and not leave engineering alone in this discussion, the way that we might be helping inside this process is building BDD and performance testing. There might be other solutions out there that might be solving the same problem. And feel free to explore, explore and don't stick with the known because the known might not solve your own problems. Feel free to contact me in LinkedIn. I will pass contact to Joe and as well as in GitHub this project. It's inside the Knight Code Labs organization. Feel free to open an issue. If you like the project and you are missing something, feel free to make a fork and contribute. Let's see how we can work together to make this idea happen and.

We are looking forward for your pull requests.

Thank you Andrej for your performance testing awesomeness. For links to everything of value we covered in this episode, head on over to testguild.com P46. And while you're there, make sure to click on the Try them both Today link under the Exclusive Sponsor section to learn all about SmartBear's two awesome performance testing solutions, Load Ninja and Load UiPro. So that's it for this episode of the Test Guild Performance Insight Reliability Podcast. I'm Joe and my mission is to help you succeed with creating end to end performance testing awesomeness. As always, test everything and keep the good. Cheers.

Thanks for listening to the Test Guild Performance and Site Reliability Podcast. Head on over to testguild. Com for full show notes, amazing blog articles and online testing conferences. Don't forget to subscribe to the Guild to continue your testing journey.