Rise of Continuous Non-Functional Testing with Rick Broker

Welcome to the Test Guild Automation Podcast where we all get together to learn more about automation and software testing with your host, Joe Calantonio.

Hey, it's Joe. And welcome to another episode of the Test Guild Automation Podcast. Today we'll be talking with Rick Broeker, all about the rise of continuous non functional testing. If you don't know, Rick is a solution architect who provides thought leadership to organizations wishing to take agile development to the next logical level by automating the deployment pipelines, which is critical, especially as we enter the new year. So, really excited to dive in. He has a background in development and operation and Rick is uniquely qualified to provide architectural guidance to many enterprise organizations, organizations of all sizes. So really excited to have Rick on the show. You don't want to miss it, Check it out. The Test Guild Automation Podcast is sponsored by the fantastic folks at Source Labs, the cloud based test platform that helps ensure you can develop with confidence at every step from code to deployment. For every framework, browser, OS, mobile device and API, get a free trial. Visit testguild.comsourcelabs and click on the exclusive sponsor section to try it for free today. Check it out.

Hey Rick, welcome to the Guild.

Thanks. Glad to be here.

Awesome to have you. So Rick, before we jump into it, anything I missed in your bio that you think you want the Guild to know more about?

No, I mean, I think you did a really good job on that. I mean I've been doing this for a while and you know, I started out process control stuff and moved into the sort of automation continuous. I've actually built a few of those in the past. I'm glad to be work, glad to be done with that part of my career.

Awesome.

Now I'm just running them.

Right. No doubt. So Rick, I, I think nowadays non functional testing gets a bad rap or people get annoyed by the term. So I, I thought before we dive into it, maybe you could just give me a definition of what you think non functional testing is. So what it means, you know, at.

Some level, you know, you gotta test everything, right? So I mean people get wrapped up in this idea of testing specific functionality of the app, but there's a lot of stuff around the usability of your apps that aren't really tied to specific use cases that you're trying to code for. And so I think like non functional testing is about the total health of your app.

Yeah, 100%. I think the issue probably is when people think of non functional testing, they think of a manual process. And you're known for helping companies really Automate things and build it into that pipeline. So I guess before we get into that question, I've been hearing a lot more about non functional testing this year and I'm not sure why. Do you think there's a reason why maybe we've been hearing more and more about non functional testing?

Yeah, I mean, you know, as we move forward, you know, when you get, I like to include in the, not in the functional testing, sort of the non functional bits about performance and so forth. When you're in a competitive environment with a lot of apps that are competing for space on your mobile devices, you need things that aren't going to suck up your battery life, aren't going to take all the disk space or the static storage space, all the RAM space. So these non functional things, being able to measure how much of your app is and get the lowest profile that you can and get across the content that your users are looking for is really important.

Absolutely. And so what I was leading to before I asked that other question was a lot of this stuff sounds like it's more of a manual process. And because people need to release quicker and faster. Can you automate things like battery, like you mentioned, battery life and things like that?

Yeah. So our testing environment allows you to keep track of all these performance metrics on your device while you're doing the testing. We're using physical devices in the cloud and you can keep track of the battery life and the memory usage and CPU usage. And so that allows you, while you're doing your regular functional testing, to keep track of all these metrics and see if there's any deltas from one version to the next.

Nice. So I guess that helps companies to shift things left then, because it sounds like almost like a developer. Does this solution help tie it into your, the developer's ecosystem so as they're developing, they can actually run it on a simulator or emulator and get the results fairly quickly before putting into the pipeline.

So first of all, we're not active actually running on emulators or simulators. We're actually running on actual devices. Right?

Oh wow, nice.

And even though those devices are actually in the cloud, you can connect them right to a developer desktop. So while you're developing your code, you can see how much memory you're using, see how much CPU that device is using, and just pull a random device out of the cloud or a specific one. You know, sometimes you run into problems where a particular device doesn't behave as all the other devices do, because they're all sort of Unique. You can actually, you know, play with that one device and see what it is about that device that's actually behaving differently.

Very cool. So being older, I'm older. Being older, I'm saying me, not you. But I think of performance, I think of old style. Like okay, we're gonna put some servers together in a staging environment. We're gonna run, you know, a thousand current users and do a soak test and all these kinds of really intensive kind of tests where you need to ramp up a lot of scripts. So when we talk about performance testing nowadays though, especially when you brought up like battery testing, is that what you mean when you say performance?

Yeah, I mean that's in the way that the app actual actually behaves on in the device. Today we actually have, you know, a lot of these devices are talking to back end cloud services so I can auto scale my devices. So that performance that we used to historically think of is not really as important as it used to be. I mean it's still not trivial but it's, but you know, you can scale things out and you can get a lot of, you want to make it efficient so that's not necessary. But really where the performance bottlenecks are going to happen now is actually on the device. So how fast does a, does a UI render? When I start to render a ui, I might actually be waiting on some stuff coming from the back data sources. I don't want that UI to wait for that to happen before the user sees it because they're going to hate that experience. They want to start seeing the UI start to show up while in the background it's pulling stuff up. And so we want to measure a lot of these sort of performance things about how long it takes a page to come up, you know, how many, how much CPU cycles does it take to do that, what's the memory associated with that and then, and kind of keep that performance on the device about how that works and that other performance is a separate sort of issue.

So I always hope that testers think about performance testing. But you've mentioned from a user's perspective many times already. A few times already. So I'm just curious to know, like another term I've been hearing a lot about is digital experience. So especially when it comes to mobile, why is that important? And I guess it would differ from device to device. So how do you even test all those different scenarios as well?

Well, I mean, you know, you got to have several devices and you know, all the, all the big companies that are doing this sort of stuff actually have a number of devices and we provide an opportunity where you can have a bunch of devices in the cloud that you can test across a wide range of devices. But, you know, the average user is going to pick up, pick up your app for some reason, they were download it and they're going to spend a couple minutes looking at that thing. Right. And if it doesn't meet expectations in those couple minutes, if it doesn't load quickly, if it doesn't provide the pages that they're looking for, the data they're looking for, they're done, it's deleted, it's gone. So you have an opportunity to get on their phone and an opportunity to present them with information. If you blow it, you're out the door.

Absolutely. And it seems like it's a. The patience of a user is much less than it was before. So like you said, now it's so easy just to delete an app if it doesn't work and try another one. So I guess along with those lines, it seems like there's even more focus on mobile testing lately, and I wonder if it has to do anything with COVID or this digital transformation that we're going through. Any thoughts on how maybe Covid has helped accelerate why this is becoming even more important and why maybe hearing more about it?

So Covid has been just a monster as far as is accelerating this because, you know, with everybody locked in and nobody's going to stores and so forth and, you know, depending on where you are, you have some level of flexibility to get out and about, but people are used relying more and more on their phones to order food at the grocery store and have it picked up in the parking lot or have it delivered to your house or you like, you know, we have a large burger chain that we work for and like, if you order their app, not only does it allow you to have their fries delivered to you in a secure way so that the delivery driver doesn't sample one along the way, but also that it picks the best delivery mechanism. You know, I want to get into people's names on which delivery mechanisms they might be, but the idea is, is that people are looking for different ways. And so just, I mean, like, I've seen like clothing companies, you know, just like, I, I can go to a store and get that. Why do I have to have the clothing company app? And so everybody seems to be getting into this and all of them, especially when you get into these things where it's like, I could go to a different app. And get the same thing. Why do I want your app? Right, and so it's gotta be a rich experience and it's gotta be quick and it's gotta get. Deliver some people immediately. And, you know, you have an idea about how to do that, but you know, you got to test it and make sure it performs adequately on all the devices.

So you mentioned battery life as a key metric. You know, if people are using mobile apps, a lot of times they're relying on WI fi or their cellular network that may not be as reliable. So are there other things you could do to automate those type of scenarios? Maybe like lost packets or dropped Internet connections or anything like that?

Yeah. So our cloud testing tool allows you to simulate a wide variety of things that could happen on your phone. So things like having a network connection, bad WI Fi or no WI fi. Even simulate where you might be in on the planet, put your geolocation and see what kind of things pop up. You know, there could be things where you want to like, well, if I'm walking down the street in front of a large coffee shop, it should pop up on my phone that I got a coupon to get a mocha latte in this coffee shop and so I could just like walk in there and get it right. So there's a lot of different things that you can. A lot of the different instrumentation on this. The phone you can play with too. So. And that's one of the kind of cool things about actually having physical devices. You can, you can insert these things and see how they kind of play out in your app.

So can you give an example of how this works in a CICD pipeline? Like, is there certain stages that these tests are done? Are they still done at the end of the. Before it gets released to the user, or is it before? I'm just curious to know if you can, like, because it's a. It's an audio. I wonder if you can talk us through maybe some of the steps of how it works.

Yeah, so there's a number of places where you could introduce this kind of testing. So obviously one of them we just talked about is developers on their desktop when they're working, might want to be able to pull up an app, just sort of like to validate that their code is actually doing what they expect it to do and so forth. And then when they commit it as part of the build process, you could run through a number of sort of tests. You could do use case tests and so forth. But then as you build through the pipeline, you could Actually in any number of places do tests. And I think it's important to be able to push those tests as far left as you can in the process. So I can fail earlier and that sounds a little bit pessimistic, but you know, the earlier I fail in the whole process, the quicker I have a chance to fix it. And probably it's more top of mind to what I actually am working on right now.

So how do you get developers to know they need to do these types of testing? You have the tooling, obviously, but how do they know? Like how. Have you seen your customers maybe get their development team on board on like, yeah, you need to do unit testing, but not only that, like we have this technology to help you with this other type of UX type of text testing.

Yeah, I mean, so that's, that's an interesting thing because first of all, you know, historically the testing and development are actually two different groups, right? And so now we're, you know, part of this whole DevOps thing. We're moving everybody to the left. Everybody's becoming a developer, right. And this is actually a kind of a new skill set, quasi new skill set maybe for testing guys. Because traditionally they just manually log into applications and do things, right. And so now we're saying, well, you know, instead of just doing that, why don't you write some appium scripts to do that or some junit scripts to do that. And so it then becomes more, at least for now, it seems to be much more of a collaborative effort between the development teams and the QA team. Because as we're trying to ramp up QA people to know how to do these, these sorts of tests also think like in, you know, a lot of best practices around software development is the first thing you, you get a use case, right? What's the first thing you should do? Write the test case. The second thing you do is, is build the, the actual thing use case that you're supposed to be building. So we need to start thinking about that. And not only that, when we're building that test case, we should be thinking about what are the performance implications of that. You know, this is the non functional part of the functional test case that you're doing. Try to roll those things into transactions so that I do. What's the performance impact of doing that?

So, so that's interesting because once again, back in the day I used to have a separate tool to do the scripting of performance and I had another tool to do the functional piece. So it sounds like you're saying you can take an Appium script or a Selenium test and retrofit it to make it to use it for non functional testing.

Yeah, I mean, you know, all you're really doing is you're telling our testing environment to collect the performance data while it's running the tests.

And you mentioned transactions. I'm not sure how many people know what that means. Can you explain what a transaction is and how that relates to a functional test?

Well, I mean like so transactions are basically whenever you're writing the test, you mark the beginning and end of a transaction. So it could be as big or as short as you want it to be. But it's good to make them consistently small things and try to leave out things like just waiting for user input because that'll make your transaction longer and be of no value. So it's mostly from a place where you get some sort of input to the place where you get some sort of output and measure what happens along there. So that could be a number of of calls back to back end things or it could be a single one, could be just rendering a page.

So I don't know why this popped in my head back in the day once again. I used to have to ship out a computer with that had an agent on it to. I worked for an insurance company, so to an insurance company right across the country and that agent would go in and would run a Windrunner script and would time how long it took to do a certain thing and it would report it back to us. Is that still a concern or is that, is that still possible with this type of testing that not only are you trying to test from your point of view, but also is there a way to. We talked about shift left. Can you shift right and also kind of monitor in production to see how things are going.

So we do have ways of doing that. But that gets a little bit out of our testing tool. I mean so the testing tool does that in the sense that we don't really have an agent, but we have rest calls into our cloud that allow facilitate all this kind of testing that we're talking about. But we also have another product that's actually security product that allows you to build in security features like a basically a landmine of security traps, apps and whenever one of those gets tripped, one of the events can be call home.

Nice. No, that's cool. So that's a good point. You brought up security. Like I said, it may be outside of your area, but we talk about non functional testing. UX user experience is Very important. But also I would think security is important. Should people be focusing on that as well as part of their non functional type of testing?

I think protecting the security of your app is a critical thing. This is a little bit different than testing, but I mean I can certainly speak to it. We actually have a product that actually goes into like mobile pages, like JavaScript things because there's a lot of applications now using like Electron and so forth to do JavaScript. Even mobile devices a lot of times they're written in JavaScript so you can protect those. And then there's the native code things like in Androids, most that's written in Java, in Swift or Objective C for the iPhones and be able to protect those with a blueprint of protection schemes. And I like to refer to as like a landmine of, of security guards, right. So like you know, you may trip one and think you got away with it, but then there's another guard, a pop on top of that. That's as well. If any one of these guards was tripped then I need to do something and each one of those security guards can actually trip some sort of thing. And like the most obvious thing to think of is like am I running on a jailbroke phone? Because when you start thinking about the threat vulnerability surface of what you're going to have to deal with, those devices are actually not in your control. You know traditionally IT devices were in the control of the IT organization in that business, right. But today I don't control your phone. You know the people who have apps on your phone, they don't control your phone. You control your phone, right. So they need to have some, they have a gate, there's a gateway on that in that app back to their data center, right. And so they need to protect that. And so we need to have ways to do that. Now the couple things that you can do on testing as like test to make sure it's on a, not a jailbroke phone and then the security does that. But also you know, you want to make sure that people can't decrypt the application, decompile it and find like keys that you might have embedded in there or what's your rest APIs that you're talking about or what's the logic about how you call those rest APIs to be able to totally obscure all that stuff. So, so even if someone gets the phone and they decompile the app, they got like nothing.

Cool. Nice. So a little bit back to, I get off topic there back to the non functional Kind of performance type testing. I'm thinking if we missed any key metrics. We talked about battery life, it sounds like we talked about response time because we talked about transactions. Is there anything else that you think people should measure or be able to measure when they're using a tool for this?

Yeah, I mean, certainly you want to be a little bit cognizant about how much space your app takes. I mean, clearly bigger apps, you know, eventually, you know, with all the photographs on the your phone and videos and so forth, you know, that becomes a problem, but also how much memory it takes, you know, when it's running and being able to do that, you know, sometimes apps will go through, you know, downloading a big chunk of data and rendering it and they don't really need all of that and they only need a little bit of it. So, you know, think about, you know, this kind of gives you an opportunity to say, well, am I really downloading what I need versus just a whole bunch of stuff to have it available in case I need to it.

So I would think a solution also would have over time, historical over time. So is there a way to know like this previous build had this, the metrics with this and now I just did a new build and now the metrics are even worse. So you know, does it alert people? Like how, how do you know, like when things get worse is if you're trying to automate as much as possible. So you only get alerted when something really is like a bad thing.

Yeah. So being able to look at it over time is important because you might actually have one build that is, is totally fine and then you do something in the next build and it just the CPU spikes or the battery usage gets terrible or just whatever, I don't, who knows. And so being able to look at all those performance metrics over time is a great sort of thing. You know, are we, are we having a spike? Is there a gradual increase? You know, the most logical thing you would think about is as you add features, there's probably going to be a gradual increase in the memory footprint of the device. Right. And so how do you manage that? How do you keep that under control and you know, decide what your limits are going to be? You know, you might have to do some sort of packing things in or changing the way you load things and so forth to control that memory usage.

So, you know, we talked a lot about mobile testing. I know when you go from device to device, sometimes the app looks different. So when we talk about non functional testing, do you ever do anything like with visual testing to just ensure that, you know, it visually looks correct from one device to another.

So I don't have anything automated that does that sort of thing per se. But every time we run a test within our cloud, it keeps a video record of the test and allows you to capture screenshots. And even being able to capture screenshots, I think is. Or the video is really handy because if you do still have this sort of segregation between the testing team and the development team, be able to go back with a video or a screenshot. Yeah, this. What this is. No, can't do that. Can't. Sorry. You got to do better and be able to show them exactly what you mean about what's not okay. That's really helpful. I mean, just think about it, communication. I mean, even though we deal a lot in written words and so forth, I mean, a lot of people also respond better to video or, you know, visual sort of stimuli. And especially when you're talking about UI experience, it's all visual.

Yeah, it kind of doesn't apply, but it does. I work with a lot of remote workers for my small business and just send them a video is so much easier than having to type it. So I can imagine when you're working on a development team to say, hey, not only is this what's happening, but you actually could see it here at timestamp 2.30, you know, so that's. I definitely see that as a big help. I also wonder, does this help at all with accessibility testing or is there other functionality that could help with accessibility testing?

Yeah, we were starting to see more and more sort of automation of accessibility testing. There's actually standards out there now about accessibility testing and you have to be cognizant about that as a large company. I mean, I think was it Target and another one that slips my mind right now had big lawsuits against them for failure on accessibility testing and they lost a lot of money on this. So I think having some level of accessibility testing is going to be critical going forward. And so we have some things that allow you to do things like tested for colorblindness or testing people's visual site and doing kind of automating the voiceover commands. You know, if you get like an iPhone or something, you can speak to it and it'll. It'll like, you know, next or check or what have you and go to different functionality so you can actually test for some of those sort of audio cues and audio return codes, you know, so when it's. When it highlights something the phone will say what it's doing. So having all that and of course that gets added into the, the video record that happens to it. There's also an audio record that comes with the text. And being able to automate some of those things, those spoken things we have, we don't do them with audio files, but we do them with special key commands that we send that represent those audio files. But that's how we do it.

Nice. Yeah. For some reason, once again, I think maybe because more people are using mobile because of COVID Accessibility testing has been big on when I speak with engineers on my podcast. So like, who does all this testing nowadays? In CICD accessibility testing, we talked about user experience, testing, performance. Is it all on the shoulders of developers or do you see testers also contributing or does it depend on the organization in your experience?

I think so. First of all, it does depend, always depends on organizations. But I do see that there's still the sort of development world and the QA world. My push on merging the performance data, the non functional stuff into the unit testing kind of is driving those two guys together. Right? Because historically the developers do like unit testing on functions that they create just to know that it works right. And the QA team is more about the UI and seeing that it, that it looks right in the end. I think merging those two things together at some level is important so that when we get as we go through the CI pipeline, if we can automate that, we can automate the whole thing. So even though maybe a developer's testing some stuff and a QA person is going through the thing, doing it themselves, if we can automate both pieces together, we can regression test the whole app so that we know that when I added features, features Z, I didn't break feature B. Yeah, absolutely.

So I guess another issue and this might be off the wall. I don't know if you have any thoughts on this is AWS has been down twice in the past two weeks and another trend I've been seeing is a lot of companies going cloud native. And when I think of non functional testing, reliability testing, do you see that as something people should be testing? Like is there a failover mechanism that should be in place in your opinion or do you have any thoughts on that? Just like I said, it's an awful wall question, but it's very topical right now.

Boy, you're going to take me into a whole different world. So, you know, in addition to the testing, I do a lot of, a lot of DevOps stuff with digital AI, right. And so one of our sort of things that we encourage large organizations to have a multi cloud sort of approach, right. And make sure that you, you have load balancing between the clouds. So if you did have a performance issue with AWS for some reason that it went down, got hacked or whatever, you know, I think it was a DDoS attack. You would just switch over to GCP, Azure, Oracle, Tenzu, whatever.

So how hard is that? Because I would think like it's like big companies went down. Hulu Twitch, like I mean maybe I, I'm probably underestimating how difficult it is, but I mean is it a difficult thing to do or is it just people probably didn't plan for it?

I think mostly people don't plan for it. I mean a lot of people, they decide, I mean I see this a lot, A lot of people decide that they're going to do a cloud or even if they say they're going to do multi cloud, but they mean by multi cloud cloud is this group of developers is going to be in Amazon, this other group of developers is going to be in Azure, right? It's not this one group that developers is going to be in Amazon and Azure. And if something goes wrong with one, then they'll be fine. I also think that a lot of people didn't really anticipate, didn't really believe in the possibility of a particular cloud provider having a problem. I mean if you look at Amazon, how big Amazon is, how many data centers they have have scattered all over the world, just in the United States, there's what about four or five big data centers that they have. So you would think that yeah, I'm pretty safe with Amazon, especially if I have a, you know, multi data center approach and multi zones in each data center. But it turns out that maybe not so much.

Yeah, no doubt. So yeah, it's curious, curious to see how people handle this going forward because that was kind of bizarre, by the way.

I've been expecting something like that for years.

Yeah, I'm sure. I mean I would have thought that as well, but it seemed like no one and people thought about it, but probably didn't think about it or architect it from the beginning. Maybe it takes some sort of architecture like you said, as you're developing your microservice to be able to have a switch, to be able to easily switch between the two.

But I don't know, maybe I'm just paranoid.

I know I am. Absolutely. So Rick, you know, I think we only mentioned your company once and not Even you mentioned in passing, can you tell us a little bit more about where you work and what solution you've alluded to?

So I'm a solutions architect with a company called Digital AI and so we're, we call ourselves a value stream management company. And that means that we have a number of sort of verticals that we work with. So there's an agility product that allows you to model the whole safe agile process, you know, for ideation to completion. And that then ties into our DevOps tools, which is a release orchestration product and a deployment automation product. And that the release automation ties into all the things that you would think about automating and then, you know, in your release process, you know, so it includes things like building things and talking to deployment tools like our deploy tool, and also to other things like ticketing, other ticketing systems, like maybe a service request ticketing system or maybe tying into some, you know, security management things or possibly even to an automated testing suite like our testing, our continuous testing product, which allows you to test things like desktop browsers and mobile devices and so forth in a number of configurations. And that the deployment tool that I mentioned mentioned is one of the things that allows you to do sort of this multi cloud thing because I can specify some stuff for one thing and then change the targets around underneath so that they have multiple things in the environment which could be different clouds and deploy to all those different sort of environments. And then we have a security product that we kind of talked a little bit about when we got off on a tangent, that allows you to protect all your applications, even though they might be running in somebody else's environment, protect you from jailbreak things, protect you from people decompiling your applications, trying to figure out how it works so they can reverse engineer the thing and allows you to make that threat surface a lot smaller for you. And then the last piece is an AI analytics thing that sits across all of this stuff, that ties all this data together. So then I can do things like look across all those test environments that I have and see what kind and tie that in with the number of threats that we have. Or what's the. How successful is one particular team been at releasing code so that I know that they don't have to have a whole cab process. We can just let them go because they have a 99% success rate when they go to production. So that's, that's what we do. And you can do the whole platform or you can do pieces of it.

Very cool. Okay, Rick, before we go. Is there one piece of actual advice you can give to someone to help them with their non functional type of automation testing efforts? And what's the best way to find our contact you?

So the first thing I would say is every time you do testing put the non functional testing in right. So always be collecting that performance data for everything. I don't even think about a differentiation between functional and non functional test is just testing. So the best way to contact me you can I can find me I think it's zaircodebender on Twitter or just email me at our brokerage Digital AI.

Thanks again for your automation awesomeness. The links to Evident value we covered in this episode. Head on over to testguild.com a380 and while you're there, make sure to click on the Try for Free Today link under the exclusive Sponsor section to learn all about Sauce Labs awesome products and services. And if the show has helped you in any way, why not rate it and review it in itunes? Reviews really help in the rankings of the show and I read each and every one of them. So that's it for this episode of the Test Guild Automation Podcast. I'm Joe. My mission is to help you succeed with creating end to end full stack Automation Awesomeness.

Us.

As always, test everything and keep the good. Cheers.

Thanks for listening to the Test Guild Automation Podcast. Head on over to testguild.com for full show notes, amazing blog articles and online testing conferences. Don't forget to subscribe to the Guild to continue your testing journey.