Performance Testing Foundations with Dave Westerveld

Welcome to the Test Guild Performance and Site Reliability Podcast where we all get together to learn more about performance testing with your host, Joe Calantonio.

Hey, it's Joe. And welcome to another episode of the Test Guild Performance and Site Reliability Podcast. Today we'll be talking with Dave all about the foundations of performance testing. He also has numerous experience in other forms of testing like API testing that we'll probably touch upon as well as Dave is an experienced software tester, has been working in the industry for over 13 years now. He's worked on scientific computing software that runs advanced mathematical calculations that can scale up to compute across thousands of CPUs in parallel. Dave also has been involved in various automation and manual testing initiatives and currently works at D2L which is one of the leading online learning platforms. And you can imagine with the last year with COVID and everything, online learning platforms have been seeing a massive increase in traffic and load. So Dave has some awesome real world experience on how to continue to grow and learn as his performance testing strategies in this kind of crazy, crazy world we find ourselves in. So really excited to have Dave join us today. He's created a number of different software testing related courses, including a course on getting started with performance testing that we'll be diving into today. You don't want to miss this episode. Check it out.

This episode is brought to you by SmartBear. Listen, load testing is tough. Investing in the right tools to automate tests, identify bottlenecks and resolve issues quickly could save your organization, time and money. SmartBear offers a suite of performance tools like Load Ninja, which is a SaaS UI load testing tool, and LoadUiPro, an API load testing tool to help teams get full visibility into UI and API performance so you can release and recover from faster than ever. Give it a shot. It's free and easy to try. Head on over to smartbear.com solutions forward/performancetesting to learn more.

Hey Dave, welcome to the Guild.

Hey Joe, thanks for having me on. Excited to be on here.

Awesome to have you. So Dave, as I was looking through LinkedIn, I usually do just random searches for keywords like performance testing and I was really excited to see your course. I think he published it in 2019, so I thought we'd dive into it a little bit today. Before we do though, Dave, is there anything I missed in your bio that you want the Guild to know more about?

No, I think that covers it. I've been testing for quite a while deep in the industry. I love learning about testing of all sorts like you Mentioned. So excited to talk a little bit about performance testing here today.

Now I've been looking at your blog as well offbeat testing and a lot of the posts are about API testing. So I'm curious to know a lot of people when they get into testing, all they do is focus in on the ui. And I love how you're focusing more on backend automation and performance testing. So why focus on this type of testing rather than UI testing?

Yeah, it's a great question. I think I've started to kind of think of this as, or myself as being a full stack tester. Talk about full stack devs or you know, front end and back end devs and that, but I think a full stack tester is pretty valuable thing and so I see all these different forms of testing as they're testing. There's, there's specialties within testing, but performance testing, API testing, security testing, test automation, all these different things. I think a well rounded tester will have some experience in each of them. And so in some ways I've just wanted to dive into these different things, learn them. I've needed to learn them for my job and I've certainly spent a lot of time in the last couple of years really diving into API testing. I actually finished a book on API testing with Postman a little while ago, so I've kind of, I guess niched down a little bit in terms of API testing. But I, I think all of these forms of testing, there's something to at least knowing the basics of these different kinds of testing so that when you come across it, when you see the need for it, you've got the ability to at least contribute to it and be a part of the conversation on those things.

Absolutely. And I love this concept of full stack tester. So another thing I've been seeing up all, there's been a lot of talk on LinkedIn. I don't know if you know, Paul Bruce used to work for SmartBear, but he talked about how he doesn't like the term non functional testing because it kind of minimizes these things as mental nice to have but not necessary. And I think performance testing falls into that bucket. So why do you think people should focus in on other forms of testing like performance testing?

I like that too. I mean we do in the industry it gets called non functional testing, but things like performance testing are crucial to the success of modern software. I mean I think Google, I think in my course I talk about this, but Google has, you know, stats on how quickly people leave a site if it's not performant. And I mean we even recently on my team we were working on some new stuff and we had it out and it ended up under certain circumstances not being that performant. And it meant that people couldn't use it, like people just couldn't use it. We actually had to turn some features off. So you're looking at functional versus non functional testing. And in that scenario we actually sure, functionally the feature worked, but we had to turn the feature off for, for a few of our clients just so that they would be able to use the other features effectively. And I think that kind of illustrates the importance of this performance testing. I mean, it doesn't matter how well your app does functionally if you just can't interact with it. So performance testing, I think API testing definitely falls in that realm. It's one of those things that's kind of hidden away and you don't realize what's going on. But if you don't have a well designed API, you can end up causing a lot of problems for your end users. That, yeah, that could be avoided if we maybe focused a little more on these non functional aspects of testing.

Cool. So I asked everyone this question. What is performance testing from your point of view? I think sometimes when people get tripped up, they think it's like a performance testing team that has to create this lab with all these servers. And back in the day it was like putting a load on the server. It wasn't really from a user front end per se, but now with these Java applications coming along, it really has to do with the front end performance as well. So for you and where you're working now, what do you classify as performance testing?

Yeah, that's a great question. Because every term needs to be defined, right? Everyone has their own interpretation of what different terms mean. I think in software testing we see that, but I think that's not just unique to software testing, but when it comes to performance testing, the way I think about it, I mean, at the end of the day I think a customer focus is the best way to approach it. So what does it feel like for a customer? Using the site is kind of the core thing that I would try to look at with performance testing. Our company, the company I'm at right now, we have a performance testing team whose job it is to kind of drill into the details and figure out some of this stuff. They've got, well, lab, I mean everything's in the cloud nowadays, but they've got a lot of expertise in that and we rely on that team. And I just this week, well, last week I guess was in a meeting with them, kind of going through understanding some of the performance issues we were having with ours. And so using those specialties is great, but I think at a individual team level or individual tester level, there's a lot of stuff that you can do, kind of from the end user perspective that can give you insight without needing the details of. Okay, how does this exactly influence my CPU usage, my memory usage, all these things that are in some ways maybe less important now that many applications are on the cloud and you can kind of scale up compute resources as needed and you've got other strategies that you can use to deal with that, but using it to try and help yourself understand where's those bottlenecks, where's the things that are slowing it down so that my end users of the software are feeling like it's slow.

So, David, you just mentioned something interesting, that a lot of people think these resources are infinite, but they do cost money when performance gets bad. I guess if you're consuming a lot of it. So it's something you need to watch out for. So what are things you could look for then that aren't related to maybe resource utilization if you feel like have that pretty much handled?

Yeah, that's a good question. And that's a good point too. You know the. Just because you can scale up things, your operations team might not be so happy if you need to start scaling up thousands of dollars worth of compute resources. I think one of the things that you can really look at is that end user impact. So there's measurements you can do, like you said on the front end, looking at, okay, what does it look like at even just at a UI level? So when I click on this, how long does it take before the page is fully rendered? Or how long does it take before the DOM is active? Right before I can start to interact with the page, you can measure some of those things and I think an maybe underutilized performance tool is monitoring. So we can build those metrics right into your site and see in production. And that's kind of maybe in some ways the holy grail of performance testing. Right. It's really hard to do performance testing in production because if you're stressing your system or giving it a big load, you're going to impact users on the system. So you can't test in production, which then means that a lot of performance testing is done on different test labs and stuff, which is obviously a great compromise. But those test Labs don't have the same setup, they don't have the same resources that your production system does. So you've got to do a lot of extrapolation to figure out if results from your test lab are relevant in production. So if you can get things. And we've done this actually on our team where we have some performance metrics built right into our telemetry so that when a user does certain actions, we can see how long it takes for that action to kind of resolve and them to be able to interact with the page again. And so we can see in production how things are performing. Obviously, there's a lot of challenges with that too. Right. Because if a user is working on a mobile network or if they're far away from things or out in the forest or something, there's going to be slowdowns that happen just due to network speeds and a lot of other factors that we don't have control over. But I think that's a very interesting thing to think about when it comes to performance testing and the ability to kind of test in production.

So I have a few questions that keep popping up in different areas. So hopefully I don't feel like I'm being scatterbrained here, but I want to dive into a little bit the outliers. But before I get to that is I'm just curious to know, did you start this initiative from. Because it sounds like you. Like you said, you have a performance team. You're part of a team that is probably a development team. You're a tester who started this push for performance. From a user perspective, was it you taking the initiative? Was it from the top down saying, hey testers, you need to get more involved in performance? How did this become part of. Almost seems like a cool culture that you have going on there.

Yeah, I think it's a bit of a mix actually. So there was a recognition that we were having some performance issues. This was even before COVID a couple years ago. So we have a broad range of clients. Some of our clients have millions of users, millions of students. Right. Some of them have a couple thousand. And so you got this broad range. So some of our higher end clients or heavier users, user count clients, were finding that certain things were not that performant. So I think at that time we kind of got, okay, let's get a couple of people to do on a performance testing team that can kind of dig into some of this stuff and understand what's going on, specialize in those details. But then we realized that you can't just throw one team at a problem and expect it to go away. It's got to be a cultural change across the company. Having a performance testing team that people may or may not know is there, may or may not be able to use. You got the silos. That can happen with some of that when you've got a separate team. And so we started thinking about that in terms of the telemetry stuff that I was talking about. That was an initiative that kind of came from our team. I'm not going to say myself, only I was one of the people saying, let's do this, but certainly just as a team, we were thinking about, how can we better understand what's going on? Because we had a client that was saying, hey, this isn't working for me. It's not like it's too slow. And we're looking at it. We're like, well, it's fine on our test sites. Like, what's going on? How are you using it differently? So we're trying to understand what they were doing. So we're like, let's put some telemetry in so we can understand a little bit their workflow a bit better. And then while we're doing this, let's also measure the performance. Then we can actually see, okay, you've got slowdowns at certain times of day, maybe when you've got high usage, where a lot of people are using the system at the same time. Or maybe we can understand across your monthly cycle, Right? So in educational institutions, which is our clients, there's certain cycles to when you grade different users and things like that. So trying to understand this stuff, it was a bit of a mix. There was, you know, a recognition that things were getting in. Sometimes some cases escalated up to leadership, and leadership's like, okay, we need to do something about this. But then there was also at a local team level of us saying, we want to solve this problem too, Right? We're here to make our clients happy. And to do that, we need to understand where their pain points are as best we can.

I guess what adds extra complication in your particular situation is if you have a normal SaaS application, it's usually, you know, what the flow is going to be, you know, how users are going to use it. It's on your platform. I used to work for a healthcare company and different hospitals would implement the software differently. And it sounds like that's the case with yours as well. So how do you know when something's like just an outlier or it's Actually an issue with your software, you know what I'm saying? Like, how do you know? Like you work, you have all these customers, all different user cases. Even if you test it in production, you never find this issue because it's so unique to that customer, I would assume.

Yep. Yeah, it is. There's actually a fair bit of similarity between education and healthcare. I mean, obviously in healthcare maybe the outcomes are a little more immediately important. Right. If you mess up someone's education, they're not going to die on the spot. But they're somewhat similar in the sense that there's a lot of regulation around it. Most educational institutions are governmental of some sort, so there's certain regulations around how things work there. So obviously not to the same degree as in healthcare, for obvious reasons, but there's similarity there and there's a lot of different educational institutions that implement things in different ways, like you said. So trying to understand the outliers is really difficult. It's one of the challenges. I've been in meetings with some of our clients too, where they are concerned about way something is working and they're like, why didn't you test for this? Well, we do have thousands of clients that all do this in different ways. It's just not possible to test for everything. So to some extent we do have to be a bit reactive. Right. When you see a problem, you react to it. So to some extent that's just unavoidable. Now that's not the ideal, maybe the way that we exactly want to go about it. And so we are trying to also have that hybrid approach of understanding ahead of time as well. And I think that's where this performance culture that we were talking about a little earlier comes in to play of when you're creating something as a team. We need to think about the fact that, that there are users that are going to use it in ways that are, I guess, heavy usage ways. Right. So we can't just assume that because it's working functionally with, you know, and we're trying it out on our Test site with 50 users or something in a course, that's going to be fine. And we've started to realize that a little bit, even just in terms of say our definition of done, when is a feature done and ready to be shipped and we need to have performance testing included in that. We need to have some kind of load testing, understanding that it will scale and work well. And if we don't have that, we have to say, no, it's not ready to ship yet because we just don't know if we're going to run into major problems when we send it out.

So as you were speaking about, these areas are really regulated. So it's not like you can just throw a monitor on someone's internal network to see what's going on with performance. So do you build anything into the system itself, like for logs, so you can make it easier to know when there is performance issues to track it down?

Yeah, I mean, we do our best, the best that we can there. Like you said, there's obviously privacy issues you need to watch out for. Make sure we're not logging private information, personal information, gdpr. We operate in Europe as well, so we have to make sure we're compliant with those regulations. So there's some challenges there. But we do have some monitoring tools that we can use to drill in to help us understand the workflows. You can't always, but it can help sometimes.

Yeah.

I think it's a really important thing to think about for testing in general is how these tools can help us better understand what our clients are doing so that we can find those edge cases, those things that our clients do that we are surprised by.

Absolutely. And I love focusing on the edge cases, but sometimes people aren't even getting the base case down. So if someone's listening to this, maybe they don't have a huge performance team with them. Are there any questions testers can ask before getting started on this project? I know it's part of your DoD. I think that's awesome that they can also have some questions they maybe can ask to help drive them before doing performance testing to guide them.

Yeah. So we don't have anything, maybe explicitly defined in our team, but I think it's good. That idea of when you're getting into performance testing, I feel like it's a bit of an overwhelming field for someone who's maybe newer to testing or the idea of performance testing these loads and how do we generate all this stuff and it can get really complicated and then all these results that we get out and how do we analyze them? But yeah, I think for getting started with performance testing, it's a bit of an intimidating thing for testers. For those that are newer to the field or newer to performance testing, there's just a lot of stuff there. It's a big field. And so I think when you're getting started, some of the main things you want to drill in on are just understanding what the impact is on clients thinking through that aspect of it. So I think when you're getting into the field, you have to realize as a tester, you're not going to know much. And that's fine. Like you're starting something new and you're not going to know much, but you can actually still contribute something with your curiosity, with just asking questions about how is this going to impact the end user? How is this going to look like? I mean, you have to know your particular context, right? So how is this going to look like for the kinds of clients that we have? So, I mean, in this podcast, I've been talking a lot about educational clients because that's my particular context right now. But if you're working in E commerce, for example, you're going to ask some different questions, you're going to drill in on some different things. You're going to think about what happens Black Friday, what happens at Christmas time, what happens when we get these different kinds of scenarios. And I think really bringing that something that I think a lot of testers are good at, that customer focus into it. What is this going to feel like for our clients? And how are we going to be able to do this? And then also those questions of curiosity, what happens if we're successful? Sometimes we release a feature and we're like, ah, this is a great feature, but we don't think about, okay, what happens if this feature is actually really successful and people really like it and all of a sudden we have a whole bunch of users and then we're like, oh, no, it doesn't work when we get, you know, a hundred thousand or a million users or something. So just asking some of those kind of creative, probing questions can be a really helpful way to get into the conversation around performance.

Definitely. So besides, once they have performance, they know what they're going to do. They start doing it. There's numerous types of performance tests that can be done. How do you know when to do one over the other? Like, what are the different types of performance testing? Should you do all of them and maybe dive a little bit into that?

Yeah, I think I'll preface this too with I'm not a performance testing expert. I'm a software testing expert and I've learned performance testing. And so, and I want to say that because I'm going to talk about some different kinds of performance tests. I may or may not have the same definition that you've heard in other conversations. Right. For people listening to this. So I'll try to kind of define what I'm talking about as I go along with these different types of tests. But I think there's some helpfulness too, maybe, hopefully in a conversation like this where someone who's not deeply embedded expert in the field can still talk a bit about this. And for those that are newer to the field, maybe it makes it a little more accessible. So that's partly why I put together a course on it. Even though I'm not a lifelong expert in performance testing. I just wanted to share where I'm at, what I've learned along the way to help those who are maybe a step or two further behind in the learning process. But yeah, I think there's a lot of different kinds of performance testing, and that's one of the things that can be challenging about it, is when you say performance testing, what do you mean? I mean, I've talked about it in terms of what's the end experience for users. But then when you actually drill in, how do you test for those different things? How do you actually figure out what's going on? You can do different types of loads on the system. I think that's a helpful way to think about it, is I'm loading up the system, I'm creating some kind of load on the system to help me answer questions about how the system's going to respond when, if those loads happen in real life. And so I think about it again. Performance testing is testing. You design. If you were doing functional testing, you would design your test, then you'd run your test, then you'd look at the results and probably feed that back into your test design, maybe make another test based on those results, or maybe find a bug and need to fix it in the application. Same thing with performance testing. You're going to look at what questions do I need to answer with this test? And so if you're in the context, say of like we talked earlier, E Commerce, where there's certain days that you're going to see, you know, you're probably going to see a massive spike in usage. Well, then you might want to make what I would call a spike test. You're going to want to come up with a performance test load, a load profile that spikes up your users really fast, that you know, starts from a certain baseline level of users on your system or calls on your system or whatever. And then it's going to spike that up quite quickly to levels that are maybe 100 times more than your baseline level over a very short period of time. So if you're trying to answer that question, you may create that kind of load profile. Another thing you could look at is stress Testing your system. So you want to take your system and just put it under some stress and find out where the failure point is. At what point does the system stop behaving normally and start behaving in problematic ways. And so with something like that, you could just ramp up the number of calls on your system. You could linearly ramp it up over time and see, okay, where does it fail? Okay, that's the point at which I may want to investigate. Is that an acceptable failure point? How does it fail at that point is another helpful thing you could look at, right? Does it fail gracefully? Does it cause major problems? Does it break other things? And then you can have endurance testing, I call it. So the idea of just fairly heavy load over a long period of time, I mean for our company, we saw that with COVID hit and here we are 15 months later and we're still at loads that are quite a bit higher than our pre Covid levels. So endurance testing, how does your hardware, your system perform over long periods of time under load? And yeah, there's other things. Scalability testing, we talked about that with the cloud. And if you can kind of throw more resources at it, how well does it scale under those circumstances? Does it respond quickly enough? Do you have stuff in place that will help you know that you're suddenly spending a lot more money and maybe see if you need to go in and do something about it? All those kind of different loads that you can generate depends on what question you're trying to answer. Really?

Absolutely. And div, there's so much more we can talk about. But that's why you created a course, performance testing foundations that can be found on LinkedIn. I noticed there's over 12,000 learners here. Have you got any questions or any common questions from people taking this course?

Often the questions I get are just kind of around some details of the course, I guess. But I think one of the things that I've had heard feedback on from that course that people have appreciated about it is it uses JMETER in the course, obviously a well known performance testing tool. But I really don't focus the course on the tool. Looking at these principles, the things we talked about here today, but also I talk about statistics because again, if you're going to get into this stuff, to understand it, you need to know just a little bit about statistics. So talk just a little bit about some of that stuff. I've certainly had people ask questions about how to put it into practice in your day to day job, but that's, you know, that is Something all these different contexts change a little bit. Exactly how you put it in. But that's what I want to do is kind of lay. Okay, here's the foundation for how we can think about performance testing. And honestly, I think learning the different tools is in some ways, I'm not going to say the easier part because it can be quite challenging to learn those tools. But if you've got that foundation there, you should be able to apply it. You know, if you want to use Blazemeter, great. If you want to use JMeter, great. If you want to use whatever other tool Smart bears tools like, great. You can figure out those tools and apply the same principles from one tool to the next.

Okay, Dave, before we go, is there one piece of actual advice you can give to someone to help them with the performance testing efforts? I think you just gave one, but maybe another one. And what's the best way to find or contact you?

Well, if you're trying to learn anything. So this isn't particularly about performance testing, but if you're trying to learn something, you got to do it. You have to try it. You can go check out my course or other courses on the Internet or read books, and those are great things, but you've got to actually do some performance testing, whether that's finding a test app somewhere that you can use, pulling up a different tool, trying it out, seeing what you can do, what you can learn about it. But I think if you really want to learn something about it, you have to just actually do it. You'll find out you don't know a lot of stuff, and then circle back to the resources, look at them with fresh eyes. Okay, I was trying to do this. How do I do that? If you circle back, look at the resources. And so I think having this learning loop of reading, studying, learning, listening to podcasts, reading books, taking courses, but actually practically applying it, because I can read a book, it sounds great. And then I actually tried to do it, and I'm like, oh, I don't understand this yet. So I think that's a really important thing is actually getting your hands dirty, doing the work of performance testing or trying it out in some way.

And, Dave, the best way to find or contact you?

Yeah, best way to find or contact me. You can find me on LinkedIn. Just search for Dave Westerveld. I'm sure he'll come up. I have my blog, offbeattesting.com and I have the same handle on Twitter as well. Offbeat testing. If you want to reach out to.

Me, there thanks again for your Performance Testing Awesomeness. Links to everything of value we covered in this episode. Head on over to testguild.com P66 and while you're there, make sure to click on the Try them both Today link under the exclusive Sponsor section to learn all about SmartBear's two awesome performance test tool solutions, Load Ninja and Load UI Pro. And if the show has helped you in any way, why not rate and review it in itunes? Reviews really do matter in the rankings of the show and I read each and every one of them. So that's it for this episode of the Test Guild Performance and Site Reliability Podcast. I'm Joe. My mission is to help you succeed with creating end to end full stack Performance Testing Awesomeness. As always, test everything and keep the good. Cheers.

Thanks for listening to the Test Guild Performance and Site Reliability Podcast. Head on over to testguild.com for full show notes, amazing blog articles and online testing conferences. Don't forget to subscribe to the Guild to continue your testing journey.