Unit Level Performance using QuickPotato with Joey Hendricks

Welcome to the Test Guild Performance and Site Reliability Podcast where we all get together to learn more about performance testing with your host, Joe Calantonio.

Hey, it's Joe. And welcome to another episode of the Test Guild Performance and Site Reliability Podcast. Today we'll be talking with Joey Hendricks all about performance testing on the unit level and and a really cool open source tool that he created to help you with this. If you don't know, Joey has been working as a junior performance engineer for two years at Accenture, working mainly for prominent Dutch financial institutions for the entire time as an external consultant. Because he's newer to the field, I'm really excited to get his fresh take on his views on performance testing and techniques that older folks like myself may not be aware of. So really excited to have him on the show. Also, this interview is based on a session he did at this year's NIOTIS Pack event, and I'll have a link in the show notes to that as well. So if you want to learn more about a developer centric approach to performance testing, you don't want to miss this episode. Check it out.

This episode is brought to you by SmartBear. Listen. Load testing is tough. Investing in the right tools to automate tests, identify bottlenecks, and resolve issues quickly could save your organization, time and money. SmartBear offers a suite of performance tools like Load Ninja, which is a SaaS UI load testing tool, and LoadUiPro and an API load testing tool to help teams get full visibility into UI and API performance so you can release and recover faster than ever. Give it a shot. It's free and easy to try. Head on over to smartbear.com solutions performancetesting to learn more.

Hey Joey. Welcome to the Guild.

Thank you for having me, Joe. So a great honor to be on your podcast.

Awesome. Yeah, thank you so much. I was really excited to see the session you did for NIOTIS and really wanted to really share with more people as well on my podcast. So thank you for agreeing to do this. Before we dive in though, Joey, is there anything I missed in your bio that you want the Guild to know more about?

Yeah, not really. I think you are spot on with the bio. I love it.

Cool. Awesome. So I think the first thing we need to go over is a lot of times when people think of performance tests and they think of a stress test or a load test, I don't know if they're necessarily familiar with performance testing on a unit level. So at a high level, what is performance testing on a unit level on A high level?

Well, it's basically defining three sets of tests. So you'll have a boundary test or a regression test that you could do on the unit level. And the third one is where you go and start profiling and checking out and playing with the results. So that's basically the three levels, what a unit performance test is. So it depends a bit on what you're trying to do and what you're trying to test, but that's about it.

All right, so let's attack these levels then. First off, who performs these type of tests?

Well, it's typically done then by a developer, so it could be included into your unit test framework, where you would define a couple of tests that you would have for key parts of your functionality, which, when they fail, they would produce results that you could use to debug your application. So it's typically run by developers.

Nice. So, Joe, as I mentioned in the intro, you have been working at a major consulting company for a while now. When you were on site with customers, did you notice that there's a gap in this type of performance testing in this area?

Actually, I see nobody doing it. I don't think it's something that I've mainly noticed it just throughout the community and online. So it's not something that I've just picked up in my experiences as a consultant. I see this a lot when I'm on Reddit or when I talk to people online or offline people at clients. I see that people barely touch profiling tools and they only touch it when it's absolutely necessary, and they're not always the easiest to use and they're quite complicated. So as I have experience with Python, I was like, let's try and find a way to make it a little easier, more accessible, and to get a very clear and cut way to pull out performance metrics out of one of those tools.

No, I agree. I've been in software engineering for over 20 years and this is an area I hardly see anyone do. I mean, maybe I worked at one company that kind of did it, but not even at the level that you described. So I guess how then do you get more people involved in this type of testing? Is it just more awareness that you think people aren't doing this? Or like you said, it's so hard maybe to start messing around with it at first.

It's difficult. So the type of testing I'm describing. Right, like, I'll recap quickly with the regression test on the code level and maybe a boundary test or profiling just Profiling itself, I see it hardly done and I think it's a bit about the difficulty or that people are very focused on functionality and delivering functionality that they forget to do this because they're just battling through the next to come. They're battling to go through the sprint to make sure that they can deliver the functionality that they promised. And sometimes I see the performance is then put at the back seat until it really becomes a problem. When I pick it up with a load testing tool or find it in appdynamics, that kind of stuff happens then. So I pick it up in the later stages where it's obviously very expensive to go back and fix it because we have to find it first. And if a developer would have picked it up earlier while he was doing unit testing, Serka what I would call a unit performance test. If you would do one of these unit performance tests and would find that bottleneck earlier, he could have solved it in like 10 minutes. But now it takes me and maybe three other people to find out where this bottleneck is. Obviously quite a waste of time and resources that you want to prevent from being wasted.

Absolutely. And it always falls back to once again, been doing performance for a long time. Back in the day they used to wait until the whole application was developed and then they'd throw on a and they're like, oh, it doesn't work where your approach is more agile, more shift left almost where before you even get to that stage, you're actually baking in performance into your application, your code so that maybe it can scale better than if they're able to handle it at the unit level. Am I understanding that correctly?

Yeah, you can consider it the same thing as a test driven development of what they use for functional testing. So before we start any piece of functionality, we are going to establish in the agile world, the definition of done. And inside the definition of done should not always only be oh, it should work this in this way. It should also contain the. It should run this and this like this. And it should run between five or four seconds or it should come back within a certain amount of time. You want that in there?

Absolutely. So, you know, once again we talked about three layers of three types of tests, boundary regression and profiling. I guess maybe before we touch that, we should talk a little bit about quick potato. What's quick potato?

So I still laugh at the name. I have something with potatoes. Like I picked it up on social media that people were posting at long posts that were very long, they would post a potato or an image of A potato from like, sorry for the long post, here's a potato.

So I started coding and I was like, if the code runs long, that image came to mind. So thinking about a framework that would performance test long running code, I was like, yeah, it has to have the word quick in it. And then the first thing that came to mind was potato. So I glued those two words together and that's it. It's nothing grand behind it, mainly a bit of funny. But quick potato is a performance testing framework which I built for the performance for the Python programming language, which basically allows you to use those three layers, it allows you to set boundaries on your code and if one of those boundaries is breached, then your code is your test has failed. It could be a minimum value or a maximum value or an average or a certain percentile. Can program that in, no problem. But the second would be then a regression test. So I would run a regression test on the code and if you make a change to your code, let's say you put timesleep somewhere in there and in the previous iteration of your code it didn't have that time sleep in it. So if you would run it again, you might have a second delay because you might sleep for a second because of the time sleep because of that delay. Your code would then be failed because you have a difference between your first test and your second test. And that difference is what I would look for in those regression tests inside the framework. I'm using a T test to do that, so a very simple statistical test to do that, which is quite cool because actually what allows you then is if you have those type 2 tests inside your unit test framework, you would create boundaries that you could check if your code doesn't overstep certain boundaries. And you could check if any changes you just made to the code have impacted the performance. And that way you can, if something goes wrong, those tests will be automatically failed. And you could generate with quick potato then reporting. And one of the reports that I generate with it is our flame graphs. So I generate a flame graph of what happened during the test and then you can just quickly see and compare another flame graph that was generated on the previous test and you can compare them in one view and see, ah, that function is new and that's when it's slowing down. Was expected that it was going to slow down. Then you just pass the test and you can continue. So that's a little bit of that idea behind that, that you can automatically check two tests and see if there are any changes. Finding bottlenecks a little bit more earlier and making sure you don't waste your time in the later stages of development.

Very cool. So is this tool then only for an application that was written in Python, or do you use this framework to create your test to run against other applications regardless of what technology they're written in?

For now, it only works against Python because I'm calling a Python Profiler. So it uses the built in Python profiler C profile. For the people that are interested in Python, I use C Profiler, where we spin it up on demand and then I profile the code that's coming after and once the test stops, the profiler stops and then you can see what's happening. That's a bit how it works.

Nice. So how would someone incorporate this into their lifecycle if they're Python developers? I assume this is only this wouldn't be shipped to production. How does it work? Like how do you integrate it with your application then? So to get the benefits and not slow down anything or cause any unintended consequences.

So a bit technical, right? I'll go a bit technical into the Python stuff. I'm using a decorator for now. I also have plans to not use a decorator, but I'm using a decorator, which basically, if you're unfamiliar, it's a function that you can put upon another function. So it's a little bit of a nested function system that the nested function, the decorator can then profile the code that's running inside it. Once it then exits, it throws the output that that function gave back of the decorator and your code will run functionally as normal. So it will have zero impact. But you need to put those decorators on the methods that you wish to test. So if you have certain methods that you only wish to profile, then you need to put those decorators on it. I'm working on a way to also spin up the profiler without using the decorators. But I really like the decorator approach because you'll have a intrusive line of code on top of your business logic, your actual code that tells you, okay, this line of code is performance critical because you see the name of the decorator in there and I named that decorator performance critical. So you'll have above every function in your Python code, you'll have a line of code setting performance critical. That also with those profilers there, they're only enabled when you're running your tests, so you would have to turn on a boolean and then your tests will kick off the Profilers and it makes sure that it will only run during your tests. Obviously, when you have your profiler attached or integrated so deeply into your code, it also gives you some benefits. It allows you, and Quick Potato also does that. It allows you to spin up the decorator on demand. So if you have certain service that you have, say you have a Django REST API that has a certain transaction in there and you want to only test, only profile that particular piece of code in live in production, so you're having issues with it and you're taking the hit of the performance loss that you're going to get because you're turning on the C profiler. Because obviously when you profile stuff it automatically becomes slower, but you're taking that risk of making it more slower so you can get more debug statistics out of your code. You could use that information to also then debug a live production system by particular on purpose, slowing down one or more transactions to look at it during for debugging. It would be obviously much cooler if that overhead would be much lower. So you could do what APM toolings are doing and just flick it on and see live information about your code and automatically profile. But that's what Quick Potato cannot do as of now, but something that I would actually find really awesome to put into the project.

Nice. So this is all documented in your GitHub as well, so it's very clear how to integrate it with the code. So once you have your decorators, then I guess it's just like any other tool. When you run in cicd, then if you have that flag on, it'll run your test automatically and throw reports if there's any issues. So it's not only local on the developer's machine, but also like you said, you can run in production, but also maybe in your staging environments. And when before you merge up to code, you can check to make sure there's no performance issues before it goes completely to your main branch.

Yeah, it would work the same as any other unit test. So I designed it in a way that it would click nicely into your Unit Test framework. Also in my GitHub I have an example where I just use the Unit Test framework that's standard into Python. I just use that library as an example. I just click into that. So wherever that can run, Quick Potato can run and you can integrate those unit performance tests, as I call them.

So I'll be honest, I've always had a hard time getting developers just because I've worked for dysfunctional companies to do any type of testing. Of course they do some minimal unit testing, but it's hard to get past that. Have you found or do you know a way to get developers more involved then with the performance testing? Say, look, not only can you do your normal unit testing, we have this great utility or this great tool you can use to help your performance as well. Do you see that they normally will embrace this or how much encouragement do they need in your experience?

That's a very hard question. So my personal opinion about this is that performance and the quality of the code you've written go hand in hand. If you have good performing code, it's usually also well thought through and developed better. It's not always the case because mistakes happen or problems might arise that you didn't expect to, but higher quality code is usually also higher performing code. So if you're keen as a developer to write better and more higher quality code, then you would take performance serious and you would also do unit testing. Serious functional unit testing, obviously. But you could also then integrate besides your functional unit testing, also the performance unit testing, making sure that both are well kept and that you're executing them well. So the concept of this is new for me too. So I've been pushing this for about a month, two months from the point of this recording, and I haven't really gotten much feedback from developers yet. So it's very hard to get them to embrace this. It's also very hard to get them interested. But as soon as people have a problem, if people burn their fingers on something, they tend to not want to burn them again. So it's also a matter of time that people encounter problems and want to prevent those problems for themselves, that these times of too also get a little bit more footing. But like I said, it's quite a hard question and it deserves also, I.

Believe a better answer because only time will tell then. But yeah, I agree it's a hard thing to get done. So I guess one way to find out is if, like you said, if it actually found something before it got into production or it helped someone to help debug an issue quickly while it was in production. It seems like it hasn't been live that long to get that type of feedback yet.

No, no, no. It's basically life since the native pack. So okay, it hasn't been on the open source market that long and there's still a lot of improvement that it needs. But I'm really happy to work on this because it's becoming a little bit of my pet project. Which is fun because I can get a lot of my technical know how out of this and create cool stuff with it. As you can believe, there is a lot of room for improvements and making it cooler and more fancy and letting it more do more cool stuff. But it's obviously an amazing basis to taste and test this feeling, test this concept out and see if developers would appreciate it and that it actually brings the benefits that I think that it could bring. Because that's still obviously up to debate and not fully tested yet.

So the information it's captioned is pretty cool. Like you said, there's some visualization with I think you said it was a flame graph. What are you using up for that? Is it tableau or is that built into the application as of yet into the framework?

I've built it in completely. So what it's doing. So I got introduced to flame graph through a framework called Dexter from AppDynamics and Dexter is a cool plugin for AppDynamics. Anybody using AppDynamics and wants to pull out flame graphs out of their AppDynamics instances. Daxter is great at doing that, really like using that. So that's the first time I was introduced to the concept and I was intrigued by it. So I searched up and I found out that the guy who made those was Brendan Gregg, a guy working for Netflix. And he's quite a big guy in the performance engineering namespace. And so I read up on his blog and I kind of knew how these things work and I was really intrigued by them. So I was like, these things are really powerful because it allows you to get one image of the entire profiling and you can see exactly where slowdowns happen or where CPU time is being spent. These flame graphs are usually a bit more complicated to interpret. So inspired by his work, I set to create my own flame graphs for Quick Potatoes. So they're built in, they're renderable by code, so you can automate that process. So if you want them automatically spinned out every single time a test fails, then that's possible. Or you want to spin it out every single time a test is run. So you'll always have flame drops of each iteration or each test. I can better say that's possible. So I don't use tableau to do that. I do use tableau for other analysis in the performance testing for my performance testing work, especially when looking at raw data.

Yeah, like I said, these graphs are really cool. Once again, I'll have it in the show notes. But you know, another Issue I've seen people have just to change topics a little bit is and this has happened all the time, not having realistic test environments. So usually they get in trouble because they're even if they do test, they're not testing in a reliable testing environment. So any tips on how to create more reliable test environments? When you're doing this type of testing.

It depends a bit on what you're trying to test. So if you're trying to test something end to end, right. And you're doing a full blown performance test, then you would obviously want the most production like environment you have, right? And.

With DevOps and more infrastructure as code, it's becoming more and more easy to spin up a production like environment for full blown load testing. And that makes it a little bit more easier. But also for unit performance testing, being able to have infrastructure as code ready that you can just click a button and a database is spinned up, it's quickly filled with an X amount of data and you could run a interpreted test performance test against it. And then you can know I'm running against an environment that has 50% of the data or 25% data. Those queries are going to be faster than I'm going to be expecting them. So if you have those kinds of environments that you can spin up really quickly, it's very easy to do a unit performance test against an environment which is reasonably production like and then you can validate, okay, I'm generating queries a lot of times. Are they in the boundaries that I'm expecting them? Are they performant? Obviously I'm not going to think that my queries are going to come back at the same speed as they would in production. They might more data. So it could be slower. But I could validate if they're expecting in the way that I'm expecting them, expecting to behave in the way that I'm expecting them. So if they run for five seconds and I'm expecting them to run for five seconds, then it's okay. But if I am expecting that they would run for five seconds and they're running 15 seconds, 20 seconds, then I know that something is wrong, maybe with my query, maybe with my environment and I can go investigate and I can validate if everything is working and.

With a sound of mind I can see, okay, there's a problem with my testing environment, my code's running fine, but I at least have checked it. So if you just check if my code is working functionally against any random database, then.

You'Re not doing a really well, good job in testing. So a testing environment is quite difficult when you're doing real end to end testing to get it right. And to do that it's basically checking the environment and checking if everything is expecting an is production like and asking questions and questions and questions when you're going to do full blown load testing. And if this can be spinned up with code, like with infrastructure's code, then you can do it much quicker, which is obviously easier, but a little bit like that.

Great advice. So I guess this just popped in my head. I guess it's the opposite. It should be probably a good issue. We talked about how it's hard to maybe get developers more into this type of performance testing. Are there any issues or where you see. Maybe someone can go overboard with this type of approach with bacon in performance into their code where it's not helpful or it's overboard.

Yeah, you can go for both with these. So when you reach a catch 22 scenario and you add these checks and they do not make sense and you add criterias to your tests which are always going to pass, then just keeping these tests and having them around is useless. Or having an environment which they cannot be used in, it's also renders these tests useless. So you would have. If you're going to test run these tests against an environment or against a database, then you need to trust that environment. If you do not trust that environment, those tests will be rendered useless and it's not going to be at the same value that you would want them to have. So that's about it with the catch 22.

Great. So I guess before you even get into catch 22 though, are there anything inherent with Python that if someone like coding things that are known to be bad for performance before even doing any type of testing that maybe a Python developer needs to know about?

I don't think I have enough experience to give a good tip on that because I use Python mainly. I use it daily to, to automate stuff and to write these cool kinds of tooling. I don't really have a clear thing. Don't do this because this could involve performance. The only two tips I would give a developer, the only one tip I could that comes to my mind right away is if you're using any type of framework that could generate SQL queries, is to make sure that whatever it's rendering that it's making sense. Because sometimes these packages could render something, render a query that's not as effective as you would want it to be. I've seen a case of that happening once in my career. So when I write queries in Python, I always check if they're a bit logical, just as a sanity check on that side that it's running in a way that I'm expecting it, you know.

Absolutely. Once again, that's usually comes out to a performance issue with the SQL statement somehow being dynamic or too huge or something pulling back too much data. So that's great advice. Okay, Joey, before we go, is there one piece of actionable advice you can give to someone to help them with their performance testing efforts and what's the best way to find or contact you?

Well, first, the latter question the best way to contact me is through LinkedIn. Just send me a message. I always respond quite promptly. The second on your first question. The advice that I would give to other performance testers or also developers is to test early, try to catch these problems as fast as you can, and fail as often as you can. That way you'll spend less time debugging issues that you shouldn't have been debugging.

Thanks again for your performance testing awesomeness. Links to updated value we covered in this episode. Head on over to testguild.com P56 and while you're there, make sure to click on the Try them both Today link under the Exclusive Sponsor section to learn all about SmartBear's two awesome performance testing solutions, LoadUI Ninja and Load UI Pro. And if you like this episode, why not rate us in iTunes ratings, really help in the rankings and help spread performance testing to the masses. So that's it for this episode of the Test Guild Performance and Site Reliability Podcast. I'm Joe. My mission is to help you succeed with creating end to end full stack performance testing awesomeness. As always, test everything and keep the good. Cheers.

Thanks for listening to the Test Guild Performance and Site Reliability Podcast. Head on over to test guild.com for full show notes, amazing blog articles and online testing conferences. Don't forget to subscribe to the Guild to continue your testing journey.