How to Inject Robustness Testing Into Your Projects Dawn Haynes

Five, four, three, two, one.

Drop that.

Welcome to the Test Guild Performance Podcast where we get together to learn more about performance testing and site reliability with your host, Joe Colantonio. Here it is. Here it is. Here it is.

Hey, it's Joe. And welcome to another episode of the Test Guild Performance Podcast. Today we'll be talking with Dawn Haines, how to inject robust testing into your performance and set reliability projects. Dawn. Dawn is the CEO of Perf Test plus and is one of the most highly regarded software trainers in the industry. With over 30 years of experience, her real world approach to testing provides testers with the tools and techniques to help them generate new approaches to common and complex software testing solutions. I've been a fan of hers for years. I'm really excited to have her on the show to share her performance testing awesomeness. But first, let's hear it for our sponsors.

Lets face it, performance testing is tough. Traditional load testing tools create scripts that are bogged down in data that's hard to read and require a lot of work, even for simple playback. Load Ninja cuts out the dirty work by using AI to inspect and debug your code almost instantly. No coding necessary. No changing of hands or relays between teams either. Give it a shot. It's free and easy to try. Head on over to testguild.com smartbear and click on the link to learn more.

Hey, Dawn. Welcome to the guild.

How you doing?

Awesome. It's great to have you on the show, Dawn. I've been wanting to interview you for a long time, probably since I started in 2015. So it's great to finally have you on the show. But for the folks that don't know you, before we jump into it, could you just tell us a little bit more about yourself?

I have been in and around software for a long time and to hear, you know, decades have gone by, that's. That's sort of weird. But in that I've seen a lot of things come and go and come back again. I call them peace signs and bell bottoms. It's like. And we call things, you know, new names. And to be honest, we have the same issues that come back for us because I think software development is challenging. As you said, performance is challenging. Let's not make the parts of it that can be easier challenging. And so having tools and things like that definitely help. I think that evolution across the decades has turned me into something of a coach. And so while I do a lot of training, I do consulting, I speak at conferences, I just think of myself now. As a and quality coach, anybody who is in the biz and you know, dealing with their job, I have things that I think can help them work through stuff for themselves.

Matt, I love what you say. You are like a coach. With experience comes, like you said, you're able to see trends. I've seen the same thing. How about robustness of our systems? You would think over time our systems would have more standards and more ways of doing things that would make our systems more robust. So have our systems become more robust over time that you've seen?

Well, I'm just one data point now. I have seen lots of different kinds of systems. I interact with lots of different kinds of software every day and I observe disappointment. I have disappointment every day in my encounters with software. And many times I could categorize that disappointment as a lack of robustness or a lack of usability or a lack of good performance. But for the most part it's in that non functional quality characteristics space. And it's disappointing to think that after all this time, why aren't we tighter in these spaces? We know a lot about it, we have a lot of experience with it. But. But there's one domain where all of those things come to the forefront and that's the mobile space, right? If a mobile app isn't responsive, if it's not robust enough to handle what a user is trying to do, and if it is not insanely intuitive, then guess what? If I don't have to use your app, my next interaction is going to be uninstall, right? I'm done with you. I'm done with you. And the mobile space knows this. Now, with that said, that doesn't mean I don't encounter junky apps all the time that I uninstall, right? They're just going to atrophy. But for the organizations that care about their apps being.

You know, I don't want to say number one, but you know, five star and you know, millions of downloads and lots of love all over the place. They get it. And I think we could learn a lot from that. Still, every single day I take a screenshot on my phone, I take a screenshot on my computer of some way that software disappoints me. And I got that inspiration to like catalog this because I think we forget real quickly. We're like, oh yeah, it does that. The heck with that. I won't do that. I'll do something else we forget. But I capture these things so I have examples to share with people at conferences and training classes. To help them be mindful of how robustness or a lack of it, how performance or a lack of it, how usability or a lack of it shows up in applications and helps me make decisions about what I do next. So the book that I got that inspiration from is called why Software Sucks. Why Software Sucks by David Platt. If I remember correctly, he teaches software engineering at either Harvard Extension or maybe an affiliate of mit, but in the Boston area. And three quarters of his book is a rant at budding software developers that says, your user is not you. Stop writing software for you. Stop putting error messages in front of users that are for you, that belong in a log file or should be sent to tech support or something like that. Focus on that user. And the rest of the book is a plea to customers to be very vocal and very specific with software vendors about how software hasn't met their needs today. So that's what I've tried to bring along with me. And I'll say I took two screenshots yesterday.

I guess as you mentioned, it is a non functional requirement. So a lot of times with non functional requirements, of course, people sometimes have a hard time even testing it. What is robustness testing then for these type of issues?

Yeah. So I run this workshop at STAR conferences, How to Break Software Robustness. And it's inspired from James Whitaker's series of books, how to Break Software, but amping it up a little and having his software attacks just be one mode where you could approach robustness testing. Out of five that I share, the first exercise that we do in this workshop is can you define robustness? It's a hoot because some people really got it and chances are they got it because they've had some past experience, right, that's gone sideways. And it has been their mission to address that. So they've had to figure it out. Most of us.

Will start to go in lots of different directions about robustness. Of what? Robustness of an interface, robustness of maybe lots of users. And I see people who literally put down as a definition of robustness attributes of usability, attributes of performance, attributes of reliability. And I think it's funny, those are all different things to test. But for me, robustness does not align with the, I'll say industry glossary definition. The industry glossary definition is how does software respond to in the face of invalid inputs? Or I think it says unexpected or stressful environmental conditions. And I'm like, okay, in my career I've done a bunch of that kind of testing, put in invalid inputs, but what bothers me is that I witness a lack of robustness in software daily when I'm providing correct inputs, when there are no stressful environmental conditions.

To me, it's a sense of stability, sometimes shades of consistency throughout an application or a user experience that actually gives me confidence that the thing is doing the right thing, the appropriate thing. And if the appropriate thing takes a certain amount of time, I might forgive it if I believe it's robust. Right. So, you know, I, that's kind of the place that I'm in. For me, it's about applications delivering value to customers consistently and not disappointing them or frustrating them in any particular way. That's what robustness is to me.

So. But how do you test it? How do you test non functional requirements like that?

Can you give an example as far as testing it? Well, can you imagine somebody writing a requirement that way? I don't think we get robustness requirements. Right. I mean, most people do the negative.

Testing. Let me put in an invalid input and see what I get. They are inspired to do that based on the positive aspect of the requirement. It's just the inverse check of the requirement. But there's not necessarily a requirement that says, see what happens when you put in invalid inputs and make sure the error message is appropriate. Have you seen that?

Actually, no, in my 20 years of experience, I've never actually seen that.

Right, right. That's not a thing. So I, I experienced this this once and I still to this day can't believe it occurred. I was on a website and it was asking me for a file name and I put in what I thought was a valid file name. And I used.

Underscores or yeah, yeah, not the dash, but the, the underscore mark.

And it said this is an, you know, this name is invalid. And it gave me an error dialogue that covered the entire screen, whole screen, I couldn't even see what I typed. And it listed all of the valid characters I could put into that box. Now not for nothing, I mean the software didn't fail. It gave me information just not in a useful way. And to me that's a chink in the armor of robustness. Right. I'm no longer having confidence in this application. I'm concerned about the way they want to interact with me and gate my usage. But I'm pretty sure nobody wrote a requirement for that either. So I think robustness requirements very much like usability requirements and performance requirements, if they show up at all, tend to be weak. My favorite performance requirement, I love to hate the system must Be fast. Stop.

That's not even a thing. It's not a requirement. Scott Barber would call it a desirement, but it's not a requirement. I can't test an architect a solution to be fast. And as a tester, I can't test fast. It's too ambiguous. So bad information for development, bad information for the architecture team, bad information for the technical team, bad information for testing. Don't know how it's going to work out when we get it into production. No clue, no correlation, right? Same thing with usability. The system must be user friendly. Don't stop. I don't know what that means. You fire up a piece of software and it goes, hi, Joe, how are you today? Right? Are you having a good day? You're looking pretty good, Joe. You know, this is, we're gonna, we're gonna go for this thing like I'm in it with you, right, friend? What, what does that mean? And people are weak on describing all kinds of aspects of quality, right? Requirements. No, requirements are kind of a yes, no thing. But aspects of quality, where there's a measurement, I think we're weak across the board. So what would be a requirement for robustness? Maybe we just look at how we handle invalid inputs. Maybe we look at how we handle stressful environmental conditions, right? If we're to use the industry IEEE 610 glossary definition. But if we expanded it and said this is about the user experience, how robust are we for user interaction? Do we fall down when we get surprised by something? Or do we handle surprises in a way that can do two things. Inform the user about what's going on right now and make some sort of suggestion to them about what to do next. That's one thing. And on the other side, if we're surprised by something, can we do anything other than fail? Or can we expect that throughout the course of a piece of software running that we will be surprised by something and build in? I won't even go as far to say self healing. Self healing could come later. But build in mechanisms to check. Have we just been surprised? And if we were just surprised, what should we do about it next? So what are the options in the face of surprises? So if we built software with defensive baby bumpers, right? Do you ever go bowling? You know, like with the big balls?

Yes.

Right. So like if you bring a kid bowling, what they typically do is they put these bumpers on the bowling alley and they sit in the gutters so the kid can't throw a gutter ball. Like wouldn't that be cool. We all want that. Like, not the way not to throw a gutterball. So what if we tried to design software interfaces and internal mechanisms with clippers on the bowling alley to just say, hey, let's not be able to throw a gutterball for the most part. I mean, anything catastrophic could happen at any time. You throw enough at us. But I don't feel like software engineers should be surprised by users doing things they didn't plan for.

To me, it's like you've ever heard of or taken a defensive driving course. To me, it's like a defensive coding course. That's how I think of it.

Is it more of an exploratory testing technique for robustness testing or are there any other techniques? I guess a popular one now is chaos engineering is as the hot kind of robustness type of testing comes to my mind.

It is. So I mentioned I go through five techniques in my little workshop. So the first one I call random strikes. In there, I talk about things like the cat on the keyboard test and something I call whack a mole and just doing bad things and yeah, that. Random chaos. So if I had a chaos monkey to do that, I would absolutely do that. But I put random strikes into two categories. One category is you're doing random things, but you actually don't know if it's going to be stressful to the application. You don't know. Right. So you're just, you're hunting. And I think that is a very exploratory thing. The other is you are doing something you know is stressful. You know it's stressful. You just don't know how the application, you know, could, should, or would respond to it. So I put those in those two two categories, if that makes sense. So if I had a tool or a bot or an AI or you know, any chaos thing that would help me do that more effectively and with a larger sample set instead of the handful of things I'm going to try. Yeah, I think that would be an amazing start because random strikes are fun, but they're probably not tied to a requirement. It's an investigation. But I call them out as specific things. Expected stress and unknown outcomes. Unexpected stress, unknown outcomes. Then I shift into talking about specifically exploratory testing. And I walk people through creating an exploratory testing charter focused on robustness. And the goal is to target a feature or a transaction or a scenario, whatever it is. And then how do you amp that up to do an investigation and an assessment of the robustness of the software you're dealing with right now. And I find that this is really tricky for people, that it is not easy to transition this idea about robustness testing into an explicit charter. And so I share an example with them. So have you ever used like a tap to pay function so that the technology that they're using behind that is usually NFC near field communication? So my charter is focused on processing an NFC transaction. And I outline we might have a robustness issue if NFC doesn't connect or if NFC fails to deny an invalid connection request. So two sides, right? And either would be flagged to me as we got a problem here. Things that I might look for, look for an incorrect error message or look for a sluggish response. Okay, now turning that into actionable test ideas. So remember, I'm doing this all under the guise of a charter, which means it's not free form exploratory testing. I'm planning to test for robustness. So my test ideas attempt connections with multiple devices concurrently. I've actually done that. I've been stranded at a gas station.

Literally trying to do that. My results were not happy. Attempt connections from multiple apps in quick succession. Oh, this one's not working. Let's try this one. I did that. Try to process a transaction using invalid credentials or data. That's the classic essence of robustness as cataloged by the ieee. So I think that we could do this inspired, on the fly, exploratory evaluation of robustness. But I find that when people do that, they typically are testing functionality, usability, performance, reliability. They end up evaluating other things. And that's the best thing. I mean, we're evaluating, you know, whatever's important, whatever's interesting to us. But if we really want to go after robustness, I think we actually have to define what it is, what is our robustness, acceptance, quality bar. And if something falls below this, we'd say we have a robustness issue. And if it exceeds that, we go, I think this thing is pretty stable, sturdy, ready to handle a variety of user interactions and scenarios, and even invalid and unexpected scenarios.

That's how I'd characterize robustness. And if we put it down on a piece of paper before we're actually testing it, I think we could probably give it to our architecture, design and dev folks and say, let's spark a conversation about this. Because if I'm helping you understand what I believe robustness is, then we can have a conversation about how much robustness are we expecting needing to have in this application? And be mindful of that throughout all of our work. So I think writing it down is pretty powerful.

So how do you get people to care, though? I mean, I find people having a hard time to care when it's actual, you know, it's a bug. You have a real, you have an actual hardcore requirement. So now you have, oh, someone will never do that, or what does that mean to me? Or it's part of the system or how do you know is it risk based? Like, hey, look, this happens, you're going to kill someone. Obviously that's a hardcore robustness issue.

Yeah. And so that's the last part of my workshop is all about how do we deal with getting this to be part of the conversation, Right? How do we not just pay lip service to things like usability and performance and potentially robustness, but how do we get everybody to understand what it is and have a level of commitment to it? So the first thing I talk about is what are the barriers? And I think part of the barrier matrix is as we were just talking about, we don't put a focus on these things at requirements time, at, you know, project inception time. We're not saying we're building a performance system, we're building a highly usable system in all of these contexts. We're building a highly robust system. I don't think we start out with that intention. So if we don't start out with that intention, when we detect these kinds of problems, I think we're going to get those kinds of responses to it. So my list is, my first one is called fading. It functions as design. Right? So we didn't intend like, okay, so now you're just inviting me to not like your design, right? Let's talk about that. Let's. Like, I don't want to have a debate about is this a bug or is this not a bug? And that's my second one. Nav Not a bug. Nar not a requirement on a user, would never do that. Twinip that will never happen in production or wom works on my machine. I don't want to have any of those debates. I just want to say we have some information about how to evaluate robustness in our hands right now. And what I want to ask the team, not the developer. I do not want to have this debate with the developer. I don't neither one of us truly get to decide what is acceptable for delivery, rollout to production, for users and usage. We don't get to make that decision. It's really the business and sometimes we defer that business decision. Directly to a customer, right? We say, you tell us if this is acceptable with real user acceptance testing, where they sign on the line and decide to accept your software and pay you. Right? But if that's not the scenario, we have to decide what that is for ourselves. If I'm dealing with a developer face to face, I'm like, I think we have a concern here. And they're like, no, no, no, no. I'm like, hey, I have a donut. Can you just come with me? Right? And I'm going to a little popcorn trail of donuts all the way to the product owner, the business analyst, the product manager, the project manager, whoever the key stakeholder is that has ownership for the outcome of this project. And I want to say, do you think this is an issue? Sometimes people are stuck in what I call the triangle of the project, right? We've got the deadlines facing us. We don't have more resources, budget, whatever to deal with this thing. They're overloaded, right? And they don't want one more little thing to add to it. So this is all I ask them to consider. If this thing were to be reported by customer X or user Y, if it were to be reported after the product is in production use, what would our collective response be? Be? And so if our response would be like, the doctor, you go to the doctor and say, hey, when I lift my arm up over my head, it hurts. And the doctor says, well, don't do that. We could just say to our users, well, don't do that. Right? That would be us shrugging and going, yeah, we don't care. So if you don't care, don't care, that's fine. You cannot care. And if you don't care as a business now, I care less as a tester, does that make sense? Like, I can tune not only what I'm looking at, what I'm looking for, what I'm evaluating as good or bad, and what I'm deciding to report and share with the team. I can gate that based on information from the stakeholders. We care, we don't care. Please tell me what you don't care about so I'm not wasting all of our collective time. But if you say, oh, well, you put it that way, if it's that customer, well, yeah, of course we do something about it. And all I say is, well, then I invite you to consider doing something about it right now instead of waiting. Right? Would you rather be in hot fix patch, hair on fire mode already disappointing a customer, or do you want to be proactive about it. And look, it's not up to me to make the decision for them, but I think if I pose the question in the right way, it's awfully convicting. Right. And it lets them know the answer. Right. The solution, the action is totally within their scope of control. Not mine, not a developer. Right. It's up to them. So if I can, if I can share that out with them, I think then we can start to have a dialogue. Oh, what's this robustness thing you just put in my face that I don't like that smells bad.

What now? What do I do with that? Then I can back up to say, well, if we had thought about these things at the initiation of this work, we might not be in this situation right now. I typically try to use real project pain or projected project pain to help people decide how they'd like to operate in a future context. You know, I can't say that'll work for everything, but that's what I try.

So, dawn, what would you say maybe your top five things you look for when you're doing robustness testing?

Okay, so in the industry, I've not come across anything that says, you know, here's the.

Definitive guide on how to do robustness testing. But I caution, if the team didn't build robustness in, why are you doing robustness testing? Right. So that's just kind of a precursor. But if you are trying to evaluate robustness.

I would look to maybe an evolutionary approach. And so I have five techniques that I share with folks that range from the easiest to implement and start with to the hardest and potentially most expensive and time consuming. So the first one I just label random strikes, things like whack a mole. I know I'm hitting something hard. Clearly it's stressful to the application. I'll see what happens. And what we used to affectionately call monkey testing. So I'm just going to hit a bunch of buttons. I don't really know how stressful that's going to be. Or maybe I'll just move through the application in an uninformed, random way and see what happens. So random strikes, like Chaos Monkey and that sort of thing, I think is a good place to start to just scan. Do we have any concerns, or does it all look good here? Does the software stand up tall and say, yes, thank you, I would like another.

And then if you wanted to be a little more proactive and spend a little more time, can you weave some of these ideas into some exploratory testing charters and explicitly look for evidence or a lack of robustness in certain areas. In areas that count, right? So use your exploratory testing superpowers for good and aim on the threat of aiming. You could go on a bug safari. So a bug safari, I bring in Whitaker's how to break software attacks. Things like 17 Ways to Attack software through a user interface. Looking for deep egregious bugs in software at the code level. Like you can't see them from where you're testing, but you can detect them. And that's what attacks are for. Attacks go for the jugular. So bug safari, you're loading up your weaponry. You know they're evil, bad, you're pointing and you're aiming and you're hunting for the bad. Bad, right? Error guessing is another scheme that I talk about. And error guessing marries the things we do in bug safari with the intention of the exploratory testing and says I do an analysis of my application and I literally design specific test cases to evaluate robustness. I put it all together. Clearly that's more effort, more analysis, more time. But hey, if you have robustness criteria in your requirements, it makes sense to have robustness tests. And I label that error guessing, right? I'm guessing that there's a robustness issue here. Here's the test that tells me if it's there or not. And hey, if it's not, guess what I just found? I just found robustness, right? Which is a pretty cool anti test. So I like that. And the last one is called fault injection. And fault injection is make something super bad happen in the system. Like, how about I restrict memory to like completely? How about I restrict disk space to just a teeny tiny bit? How about I restrict bandwidth on the network and see what happens to the application running when I do that? So fault injection tools harder to use. We need more technical skills. Some of them are very expensive, some of them you would build in house. But that gets at the stressful environmental conditions aspect of robustness testing. So that's what I call them, my five techniques. Your mileage may vary, but that's how I've bucketed them over time.

Okay, Don, before we go, is there one piece of actionable advice someone could put into place right away to help them with the robustness testing or performance testing? And let us know the best way to find or contact you.

My one suggestion that you can do anytime, anywhere with a quality concern is mip. It. I got this from Scott Barber. MIP is mentioned in passing. So he did it with performance and asked Somebody to mip, how's performance today? Literally somebody that didn't have any, wasn't on the project team, literally just in the building. How's performance today? And at the end of a week, I believe he said performance increased over 800% by saying the word right. Not, not even do you know, not even showing up and saying we need to care about performance. Nothing, Just, just MIP it. So how about robustness, right? Can you MIP it? Can you just get the word in conversation? Can you help people understand that? They don't really know how to define it either, you know, and, and, and maybe out of that emerges a commitment on the part of the team. And I think when we have a commitment right, then we have an opportunity. So that's my one suggestion that I think works with all things you can MIP usability, you can MIP performance robustness, anything that you care about that you're not getting any traction with. So that's what I try to do there. And best ways to find me, I'm on LinkedIn. That's where I have some links to articles and other interviews, some videos of presentations are all there. There's also a way to message me through LinkedIn and Twitter. Twitter is Dawn Mhanes. So altogether Dawn M. As in Marie Haynes. I'm not very active on Twitter unless I am at a conference. So when I'm at a conference I try to scoop up the cool wisdom that I think would be interesting to share out. But for the most part I'm pretty quiet on Twitter. But if you send me a message there, I will get it immediately and respond as quickly as I can. So those are the best two ways to reach me.

Thank you dawn for your performance testing awesomeness. The links to everything we covered in this episode, head on over to Test Gifts.

P. 10. And while you're there, make sure to click on the Try it Today link under the exclusive sponsor section to learn all about SmartBear and their awesome performance testing solution Load Ninja. You definitely want to check that out, especially its AI functionality. So that's it for this episode of the Test Guild Performance Podcast. I'm Joe and my mission is to help you succeed with creating performance testing and site reliability awesomeness. As always, test everything and keep the good. Cheers.

Thanks for listening to the Test Guild Performance Podcast. Head on over to testguild.com for full show notes, amazing blog articles and online testing conferences. Make sure to subscribe to join the Guild and continue your testing journey. This has been a Joe Calantonio production.