Testing our Performance Test Tools with Nicole van der Hoeven

Welcome to the Test Guild Performance and Site Reliability Podcast where we all get together to learn more about performance testing with your host, Joe Calantonio.

Hey, it's Joe. And welcome to another episode of the Test Guild Performance and Site Reliability Podcast. And today you're in for a special treat. We have Nicole Vanderhoeven joining us to talk all about testing our test tools. Nicole is a performance engineer Extraordinaire with over 10 years of experience in breaking software and learning to build it back up again. She's worked as a performance testing consultant in Australia and Europe, all around the world, and she spent the last few years helping teams all over the world scale up their load test on the cloud. She currently lives in the Netherlands and remotely works as a developer advocate at K6IO or K6, which is one of my new favorite performance test tools. So we'll probably dive into that as well. So she loves load testing because she can get involved in all kinds of different things. So she draws some all the different fields like economics, video production, linguistics and productivity to reframe load testing concepts. I think she also speaks like 5 to 10 languages. So really smart. Load testing engineer coming your way. You want to listen to this episode? Check it out. This episode is brought to you by SmartBear. Listen. Load testing is tough. Investing in the right tools to automate tests, identify bottlenecks and resolve issues quickly could save your organization, time and money. SmartBear offers a suite of performance tools like Load Ninj, which is a SAS UI load testing tool, and Load UI Pro, an API load testing tool to help teams get full visibility into UI and API performance so you can release and recover faster than ever. Give it a shot. It's free and easy to try. Head on over to smartbear.com solutions performancetesting to learn more.

Hey Nicole. Welcome to the Guild.

Thank you.

So it's awesome to have you. Nicole, is there anything I missed in your bio that I really botched that you want the Guild to know more about?

Not that you botched it, but I just wanted to say I don't think I'm very smart. I think what I am is very stubborn. So just on a personal note, I'm Filipino and I'm a woman and I did economics, not computer science. So I just wanted to do a shout out to everyone out there with non traditional backgrounds that were stubborn and somehow managed to find their way into tech.

Absolutely. I didn't have a degree. I went to like three semesters at Berklee College of Music for guitar. My friend Gave me questions for an interview back in the day when it was like before the tech bubble. So I actually got in knowing nothing about computers, so I had to agree with that. I don't know if I'm stubborn, I just was desperate is more what I was like. But Nicole, what got you into computers then?

Um, actually I found. I lucked out. I found a mentor. You probably have heard of him. Stains, capers.

Well, no. Yep, yep, yep. Nice.

Yeah, he was my first boss in IT and he taught me everything from scratch, actually.

Nice. So what drew you to performance testing? Of all the things you could have got into it. I know your mentor is a performance engineer expert as well, but is there anything specific that you said? Oh, not only did I try this, but I, I also enjoy it. Of all the things in it I could have got into.

So I was always interested in tech. My bro is a developer and my father was an engineer. So it was a very techy family, which, you know, is kind of uncommon in the Philippines. But I knew very early on I didn't want to do pure development because I love people and I feel like testing is a perfect bridge. Like you have to be able to talk to developers and speak their language, but also with the business, you have to keep end users in mind. Like it was just a hodgepodge of different disciplines and I, I like the mixture.

I don't mean to keep harping on the linguistics, but I'm just fascinated by anyone that can speak more than one language. I just remember sitting in front of my grandparents, like, no idea what they're saying. None of my grandparents are relatives. So what even motivated you to learn more than one language? And how does that relate maybe to performance testing or tech? How. Maybe it helps you with your perspective.

Yeah, I think it's actually really related. Well, I can't really take credit for learning English. That was kind of something that was decided for me in the Philippines. You really need to speak English. Everybody does at school. And so that was sort of something that I got for free, but I really took to it. And of course it helps that I have an Australian husband and we've been together for, you know, 17 years. We've known each other for 17 years at least. So. But I mean, languages are languages, right? And so in performance testing, maybe you don't get the in depth knowledge that developers would, but I think it's way more important that you are exposed to a breadth of technology, technologies and, and protocols and languages. And that's what I love about it. Like I may not be able to speak everything natively, but I know enough of a lot of languages and, and I love people. So I feel like every new language exposes like opens the door to a lot of other people that I can talk to and might actually listen to me.

Absolutely, yeah. I love that. And I love how you mentioned, you know, learning all these different languages gives you an of what's out there and different perspectives. So it kind of relates to test tooling in the sense of there's a lot of test tools out there that people can learn or spend time on. How do you determine which tools you should be focusing your time on of your limited amount of time that you have?

I try to look at all tools at least briefly, but you're right, we're limited by time. There just isn't enough time to go to know every single tool on the market incredibly well. So I honestly let the market decide a little bit. There's some tools that keep getting mentioned for open source tools. Stars on the GitHub page are sometimes a good indication of that. And just I try to look at the feature lists as well. I think that as a load tester it's really important to be to have your own sort of framework for remembering all of these tools. Because I don't know about you, but my memory has been not great. I'm finding as, as I get older, not that old yet, but I still forget things that I knew really, really well not too long ago. And so I think it's really important to take notes and have some sort of list of features that you look at and you can say, okay, this tool does that, this other tool doesn't do that, but it does do this other thing. And that way you have it in your head.

Absolutely. I think with performance testing tools, people get themselves maybe into trouble because you could put a massive amount of stress and load on a machine that isn't realistic, cause a lot of fire saying, oh, our application is unstable. So when you come to choosing a tool like how do you know what kind of performance you should start with? Maybe when learning a tool or what do you look for maybe in a performance test tool for newbies?

Well, the first thing is how usable is it? And we always look at the end user experience in testing. Right. But when we're looking at test tools, we are the end users, but we never think about testing the test tools themselves. We should have requirements. And that's what I was sort of getting at with the framework. We should have a set of requirements and assess tools based on those. And I think that we like to think that we're tool agnostic, we load testers. But are we really? Because I think load testers are just as likely as anyone to stick to things that they already know. And once you start saying, well, that's what I've always used, you know, that's a problem, right. That should be a huge red flag. I think that there are things that we don't look at for a test tool. Things like.

What is the CPU utilization of a test tool when you're running it, or memory utilization or how easy is it for a developer to use or a tester to use? We kind of just take it for granted and use them to test our applications, but we don't actually test the very beginning of that chain.

So what do you recommend then to test the beginning of that chain? Do you do like a proof of concept that you run it through and then do you have criteria that you go, okay, yes, pass past, past. You can proceed to the tool we officially use within our organization.

Yeah, so I like to have one site that I always use and one sort of business process that kind of. I always script in every tool that I want to spend time on. And the reason that I try to stick to one site is that you can then compare one script with one tool to another directly. So it's a direct comparison. Because if you test different sites all the time, well, you're testing pairing apples and oranges. So I always start with a proof of concept. I always look at things like what sort of parameterization options does it have? Does it do dynamic correlation? Does it accept test data? So I have a list of my requirements and I think every load tester should have their own requirements.

Awesome. I think you actually, you do have like a list of tools on the market that you do an analysis of and then you do a comparison of them. So I guess obviously it all depends on your company. Though. If you say, if Nicole said this is Nicole approved test tool, performance tool, would that be the right tool for me? Like a banking industry or something? Does it matter? All these tools applicable to all situations or all teams different?

Yeah, absolutely. All teams are different, all industries are different. Even the testers, the specific testers and developers and engineers that are involved are different. They all have an effect on which load testing tool should be used. So I definitely don't think that anyone can ever say there's one tool that's the best. And I think anyone who says that is looking to sell you that tool. I really think that you should look at your situation, kind of approach it more as a toolbox. Like testers should have a toolbox of different tools. You wouldn't use a hammer when you need a screwdriver. So I think that that's the best approach.

I said, I wish I had a red flag I can throw up because I'm guilty of always going, oh, you should use Load Run. And the reason why I say that is I started, I used Loadrunner 20 years ago. Over 20 years. Oh my gosh. But anyway, and I liked it because it had everything built into it. So it has monitoring, you can script in it, it has parts, it's like end to end. And from my experience with open source tools, a lot of times you need to cobble together all these open source solutions to create one end to end type of performance solution. Is that the case? Am I doing it wrong or do you have any recommendations for that?

I wouldn't say you're doing it wrong. I mean, I still use loadrunner quite a bit and I actually learned load testing using Neolo. So I'm not against proprietary tools at all. It's just that I think that it appeals to a certain maybe market segment or you need, let's face it, they cost money. And if you are in a big company where money is not an issue, or they already have paid for these tools, then I think there's a lot of value in having the entire ecosystem already there. I think the appeal of Lode Runner is not just Lode Runner itself, but the fact that you can tie into Performance center and even loadrunner analysis as opposed to virtual the Vuegen. I think that it's the whole system that you're paying for. Right? But I personally have tended towards open source tools because that's where I see the innovation happening these days. I think it's way easier to push a commit to GitHub than to package up a very nice polished standalone application and deploy it to all your customers and get them to download it. So there are some very technical reasons why open source projects just develop faster, and that's not even taking into account the fact that there's a community out there that can contribute to it too.

So speaking about open source solutions, I would think the first one people gravitate towards or think of is JMeter. And I think you probably originally started with JMeter as well, and then you went over to K6, which I'd like to dive into. But first, any thoughts on JMeter and then why someone may want to start with JMeter and then I like to probably compare it against K6 to give people an idea or, you know, what the differences might be.

Sure. I've actually done a video on this and an article on this because it is something that I've been thinking about too. I love JMeter. I want to correct you a little bit and say I haven't moved to K6 because I am still looking at JMeter. I have several versions of it installed and I'll install the next one as well. And I think that just because you try out a different tool, that doesn't mean all other tools are dead to you. Right. But there are a few things about JMeter that was starting to wear thin with me and one of them is actually the thing that drove me to it in the first place. And probably it's the same for a lot of people. And it's the ui. I think that we have this notion that if something has a ui, it's easier to get started with because you can just sort of mess around and look at all the menus. And maybe that's true at the start, but I was finding that it was causing a lot of problems later on. I don't know if you've ever also used it for load testing when you were collaborating with someone else, but it is a pain to version control if you're trying to collaborate with somebody else who's also scripting in the same test plan. Then you either both have to be looking at the xml, which is so unwieldy, or you have to make sure that you're on the same version of Java, the same version of JMeter, the same version of the plugins, the same plugins. If one of those things is not the same, then you can't even work together. It's not a tool that I think is the best for continuous testing or collaborative testing.

Great. So I like how you said you're not JMeter's in debt to you. I guess that's the Italian thing in me. Oh, you're dead to me. But GM is not dead to you. So you have both of them. So when do you know when to use one over the other? I get you give some reasons why why it may be hard to use JMeter for certain situations. It seems like as people, if you have a company that's shifting left and they're trying to get the developers more involved in performance testing, that maybe K6, a tool like K6 might be a better option. Thoughts around that?

Yeah. So one good reason to use JMeter is if you need distributed load testing right out of the box for free because I don't know if there's any other tool that does that. JMeter has that built in for no cost. And yeah, you do have to provide your own infrastructure. That's still pretty amazing. If you need that, then go for JMeter. And I think it's also important to think about the developers. And I know it's weird because this is a testing tool, right? But a testing tool shouldn't just be used by testers. Everybody should be involved in writing and contributing to tests, whether it's functional or non functional. And I think one really good reason for using K6 versus JMeter is the development developer experience. It's really attractive for developers because it's written. The scripting language is JavaScript. So many developers already know how to code in JavaScript, so they're much more likely to contribute tests or write their unit tests earlier on in the cycle. So I find that K6 is much better for more modern agile continuous development and testing type frameworks.

Chris, you did mention K6 as JavaScript. Do you see any limitations between languages? I know some use Java, some use Go, some adjust record and playback.

Well, I think that so JMeter, I would say is still primarily GUI based, so you don't actually have to code to use it. What I find more limiting is what it runs on because it still runs On Java and K6, for instance, it runs on Go under the hood. So the scripting language for K6 is JavaScript, but it uses Go and that actually makes a big difference. I know one of your previous guests, I love the episode with Stefan Lundell from Gatling because he was talking about how important it is to test your test tools and to think about how about breaking past that one thread is one virtual user paradigm. So to step back a little bit with a lot of load testing tools including JMeter, one virtual user is generally simulated using one OS kernel thread. I think that that's an outdated approach and Stefan Landell points that out as well because it's really inefficient. So when you have a sleep in a virtual user and you normally want some sort of think time or pacing, so that's very common, then that thread just does nothing and it's just a waste of resources. So do you actually play board games?

I don't know.

I do. I love board games. I have several hundred.

Doesn't surprise me.

One of my favorite sort of categories of board games is worker placement. And basically that just means, I promise this is related to load testing. So basically that just means games where you assign workers to different tasks. So you might assign one to be a farmer, another one to take care of cattle. And I was looking at strategies for doing well in these types of board games and I think it really relates to performance of applications because the first thing that you want to do is get more workers. So you spend a worker to train more workers. And I think it's the same with application performance. The first thing that people look to do is multi threading, parallelism, increasing the number of threads and then when you run out of you increase the number of cores. But that's physically bounded and that's the problem with the one thread is to one virtual user paradigm is that there's going to be a limit and what do you do after that? So I think that the Gatling way is a significant improvement because it's message based architecture, so it's not dependent on how many virtual users you're simulating. They basically use a small amount of kernel threads to generate messages, not users. Because it's really the load that's important, it's not the number of virtual users can be a bit arbitrary. So I do think that that's an improvement. But I still think that Java as a language has a lot of performance issues. For instance, I think that Java, so Java based tools have the JVM to contend with. So when you have the Java virtual machine there, that adds management overhead. So you don't just have to think about tuning your application, tuning your test tool, but you also have to tune the jvm. Because I think it's like a rite of passage for all JMETER users at this point you had to have already run a test and gotten the error like keep memory exceeded and you have to go back in the configuration files and change that. That's not something that you have to do in a language like Go that's compiled already and doesn't need to be interpret, so it runs directly on the native hardware.

How much experience would one need to know though, when choosing a tool to think about these types of situations? You give really good reasons why. But I don't know if it's like the typical person starting was thinking about threads or resources that way.

Yeah, I don't think that you really need to know the reason why. I'm just sharing what I know just because I wondered why and I was curious. But I think where it starts, at least where it started for me was after a few thousand users, sometimes even 1000 users, depending on the Exact scenario that you're running with JMeter, I was seeing my tests kind of reach a bottleneck. Not even a bottleneck that's related to the application. It was just with JMeter and I don't think you necessarily have to know why that is or the differences between the languages, but you have to do some tests to know that it's there.

Absolutely. So switch gears a little bit. I think you started a new gig at K6 maybe a few months ago now. Maybe, I don't know, time flies for me as I get older, so I don't know if it's could be half a year for all I know, or a year. But I know as you were making that transition though, you actually were learning and you were blogging about it like every week. What you learned with K6. Can you just talk a little bit about that process? Because I think it's actually helpful for people going through a similar type of transition.

Sure. It was actually about three months ago now and it seems crazy to think of it, but it was actually every day that I was blogging about it. So K6, when you join the team, everybody, not just testers, developers, even if you're a manager, everyone does what's called a week of testing. And it's meant to be completely internal. So it's meant to be a week where nobody actually helps you or shows you the ropes because we're trying, they're trying to recreate the experience of a brand new user. And so I loved that idea. I think more companies should do it. And the whole point of it is at the end you're supposed to tell everybody what you thought and what you had issues with. And I thought that one, it would be a good way to kind of impress my new employer. And two, I thought that it could be a good way to get some content out there about how to use K6 for someone who is completely new. I mean, I maybe wasn't starting completely from scratch because I think other knowledge of other load testing tools transferred to a certain degree to others. But I thought that it was really helpful to kind of expose all of the issues that I encountered. And I did encounter some. And to K6's credit, they let me publish it without censoring me. But there were things that I pointed out that I didn't like or that I couldn't figure out how to do. But in general I was actually pretty surprised at how much I was able to do.

Yeah, I love that. I think you actually said they want you to be honest. So do they take the information for all these new employees and then help to modify their application. I think that's a great idea if that's what they do.

Yeah, yeah, they do. And it kind of depends on the person. Like we had a developer who wanted to focus just on the Windows user experience, which was great because we don't have a lot of Windows users. And then we have someone like me who was already coming with a lot of biases from other load testing tools. And they loved that too. They wanted me to compare it to JMeter in particular, because that's probably the one that I'm most familiar with. And I think it's just a very open culture. They're very open to listening and taking notes and incorporating it. And actually a few of my suggestions have already been worked on.

Awesome. So we talked about all different tools, but I want to know from your experience that you've, if you've used multiple tools, are there any like, first principles that you think every performance engineer should know or have in their toolkit regardless of what tool they're using?

Yeah, I mean this is, we're talking a lot about tools right now, but actually I think that the most important part is not even about the tool. I think you should be, not just you, but your team and everybody that's involved. All the stakeholders are absolutely clear why you're testing. Because you might think that, that your purpose is to make it faster, but maybe the business thinks that it's to prevent people from going to social media and complaining about them. And yes, one could lead to the other. Slowness has typically led to people to very bad publicity on social media, but it's a subtle shift because it's not just about making it faster. Then it's not just about response times. It's also maybe about front end performance things. And then even after that, the workload modeling is so important because even if you can do exactly what you're thinking on the load testing tool that you're using, I think that there's still a chance that you get it wrong. And maybe there are some irrational behaviors that real users do that you didn't account for that will cause the application to fail.

Okay, Nicole, before we go, is there one piece of actionable advice you can give to someone to help them with their performance testing efforts and what's the best way to find or contact you?

Sure. I think this is a bit of an unusual piece of advice, but I'd say take notes. I think test your test tools and take notes on what you find because you will forget and you will have biases later that you'll need to keep in check. I think part of being a load tester, one of the best parts, is being able to continually try new tools and come up with your own requirements as a user for one. And that's the only way that you'll know that you're choosing the right tool for the job and that you're improving. Oh, and the best way to contact me? Probably Twitter. Oh, I guess you'll put a link so I don't need to spell it out, but I'm N. Vonderhoeven and I actually just started a weekly livestream of K6 office hours over on the K6 channel on YouTube with my co worker Sime, whom you've also had on the show. I like questions. I'm new to live streaming, but what we can learn together and I'll answer any questions if I can.

Thanks again for your Performance Testing Awesomeness links to everything of value we covered in this episode. Head on over to testguild.com P61 and while you're there, make sure to click on the Try them both Today link under the exclusive Sponsor section to learn all about SmartBear's two awesome performance test tool solutions, Load Ninja and Load UI Pro. And if the show has helped you in any way, why not rate and review it in itunes? Reviews really do matter in the rankings of the show when I read each and every one of them. So that's it for this episode of the Test Guild Performance and Site Reliability Podcast. I'm Joe. My mission is to help you succeed with creating end to end full stack performance Testing Awesomeness. As always, test everything and keep the good. Cheers.

Thanks for listening to the Test Guild Performance and Site Reliability Podcast. Head on over to testguild.com for full show notes, amazing blog articles and online testing conferences. Don't forget to subscribe to the Guild to continue your testing journey.