Degrees of Automation

In a land of testers far and widely journey seeking answers, seeking skills, seeking a better way o the Test Illumination Testing Podcast guiding testers with automation awesomeness. From ancient realms to modern days, they lead the way. Oh, the test is Automation Testing Podcast.

What does degrees of automation mean and what are the top three questions you need to ask to find out? That's what we'll be talking all about today with the one and only Paul Grisaffi. If you don't know, Paul is a Senior Principal Automation Architect at Nerdery, where he is following his passion for providing technological services to testing, QE and QA organizations, including automation assessments, implementations, and through activities benefiting the broader testing community. Like interviews, like this. He's also an accomplished keynote speaker and writer and Paul has spoken at multiple local and national conferences and meetings and also at Automation Guild. And he's also frequently shares his thoughts on his blog responsibleautomation.workpress.com, we'll have a link for that in the show notes down below. And this episode is based around a blog post he recently wrote called Automation State. Answer me these questions and we're going to go over a bunch of other things as well. Really excited to have Paul back on the show. You don't want to miss it. Check it out. Hey Paul, welcome back to the Guild.

Hey.

I'm always happy to be here man. Thank you.

Awesome. So I just checked and officially it's been a little over a year since last time you've been on the show. So what you've been up to since then?

Jobs. So I started a new job at Nerdery and I am basically helping the sales team learn how to help sell test automation and quality in general, but also helping everyone else on the team be able to just work at that higher level. And I'm blessed to have several really high functioning people on the team. So that's super cool. And I will also be working on delivery as soon as there are some additional clients for me to work on.

Nice. All right, I'm already off topic, but what does that mean you're helping sales? Like is it helping with the messaging where you know we add benefit? But maybe the way they're talking about it doesn't necessarily jive what people are struggling with. Like what is that? How does that work?

Oh you know what it's really more about where the value for testing and then the extended value for automation is right. Automation is expensive. Let's not lie. Automation is expensive. But there's a value proposition for most organizations to do some amount of automation. Different organizations will require different types of automation and helping those clients and our salespeople talk to those clients about where the value proposition is for them, not just what you read off of a general thing on the web. Even the general stuff that I write. Right. Something specific for them that has a specific value proposition to help them meet their business goals. That's what I'm helping to do.

Nice. I think that leads in nicely to the main topic of this episode, and that is the degrees of automation. I assume since you work with a lot of different clients, a lot of different organizations, you need to kind of get a pulse on what's going on. And I think you in your recent blog posts have like three main questions that you use. I thought we'd jump off with those. To start the first one, in the blog post you talked about, do I have the right automation? So what does that mean and how does that help determining like where an organization needs more investment or they have the right automation kind of mix.

So that's part of what I do is to go in and look at what they're doing and say, are you getting value from what you're doing here? And how is it directly or perhaps indirectly helping you meet one of your business goals? Because the way I look at it is if there's a business goal, pretty much every line of code that gets written in an app or a website or whatever should be helping meet one of those business goals. Testing needs to help that business goal. And automation is direct support for testing, helping them be more effective and more efficient. So do you have the right automation? Is are you doing the right thing for your value proposition? There's a second part to that which is what you're doing or what you're considering doing may be valuable, but if you did this other thing first, it would be more valuable and then give you more effort, time, money or whatever to do other things, either other automation, other testing, other development from a total cost of ownership and an opportunity cost standpoint.

Awesome. So when you're talking about this question, is it more around like risk or is it like money areas of the application?

It depends.

Like where do you know where to focus in on to know if you do have the right automation or not?

Well, and these are the conversations we have to have with organizations, whether it's client organizations. While I'm a consultant now and previously when I was actually a full time employee at organizations and companies, it's what are we trying to do, how are we trying to get there, what what are we trying to make more money? Are we trying to lose less money? What are our problems? Do you want a faster time to market? I might suggest something different than we have quality problems where we have bugs in production. So that's the kind of conversation and then recommendations I would base on that.

How hard is that conversation to have? Do people get offended or do they. Is it eye opening or is it. I guess it depends on the situation.

So yes, it's the old age, old, it's depends question. But really it does depend. However, one of the things as a consultant to come in and do is make sure that you're not demonizing the client or saying you've done the wrong thing is to come in and to partner with them, actually be their trusted business partner, to say, all right, let's talk honestly about a few things here. What are you trying to get done? What are the problems you're having? And let's talk about how we might use testing to help facilitate that. And by extension, where can mechanization automation help you further that? Again for the effectiveness or the efficiency? And yes, sometimes it's a year baby's ugly conversation, but usually by the time they get to a consultant like myself, they know they have an issue, but they may not know what it is.

How do you figure out what it is though? Do you actually look at the code base after you have this discussion and say, okay, where are these tests running? Like, how do you know how to even map what the right automation that they currently have? Like, how far do you have to dig in to find that?

It's funny you bring that up. So our leader has put in place this sort of scale of certainty to uncertainty, right? So for an uncertainty standpoint, there are a lot of questions. It might be what's called an assessment. But assessment kind of is a dirty word for some clients. It's really, let's go in and start, see what you're doing, and then build you a high level roadmap of how we can help you get to the next place, right? Then sometimes we have people who know what their problem is, they just don't know how to solve it. And then we come in and ask different questions and approach that in a different way. And then there are the clients that say, look, I have an implementation issue of a bandwidth issue. Here's our backlog of technical debt, our backlog of test automation we wanted to have, but don't have come in and help us build it appropriately. All of those conversations are different because they're different Situations even on top of the fact that it's different humans and different companies and different business goals. So how do we have the conversation? We start with what hurts, right? What are your pain points? Where should we spray some medicine? And then we start digging through there.

So I assume that scale is proprietary or like are there any, anything online that they offer if someone wants to do a self assessment before they call you in so they have a little more background or research before they jump into consulting?

No, I see what you're saying there. We don't have anything online. Is it proprietary? Well, I just told you what it was, so I guess it's not too proprietary.

Whoops. No, no.

But really it is how we would approach potential clients and say, well where are you on the certainty scale? And then where would we come out and help you evaluate yourself and get you to the next level of certainty? And when I say we, I mean me, I mean the company and it's all the companies I've worked at. Right. I'm trying not to turn this into a nerdery sales presentation. Right. Because nobody wants to hear that. But even if you're not looking for an external consultant, if you have an in house expert, even if you have an in house expert, these types of considerations are very valuable to you to step back and say, let me ask these three questions that Paul put in this blog. Cause it was going to be the six degrees of automation and then that was just too many and then it was 3, 3 degrees and that was wrong. And so I just turned it into the questions.

Nice. So in that first question then do you ever delve into tooling like do I have the right automation and am I using the right tools? Is that like a separate conversation you would have?

It's all part of the bigger conversation, but it's not where I jump in. A lot of times potential clients or actual clients will say we're unhappy with our tool. Okay. Either the tool doesn't fit your need or the way you've used the tool doesn't fit your need. Let's figure out what that is. So that would all be part of that investigation and recommendation. So we'd have all the conversations about what are you trying to do, what are you not trying to do, where your pain points, what your culture like. But we would also look at, I keep saying we, whoever's doing this right should look at the tooling as well to see if it is appropriate to meet those goals. And if it's not, perhaps a pivot is necessary, perhaps a Supplement is necessary.

Nice. All right, so you have the first question, do I have the right automation? Second question is, do I need more automation? I guess the answer is always yes. Right, Paul, how's that question always go?

Well, the answer is obviously always yes. Right. I'm an automation guy, but in reality, in reality, like I said in the article. Okay, great. We've taken our, for example, our smoke suite, and it took a human 30 minutes to do it. Now it takes a computer 8 minutes to do it.

Sweet.

You know, T shirts for everyone. What do you do with that gained information? Is it that your smoke test suite was constrained, the quantity of it was constrained by the 30 minutes. We can only wait 30 minutes. We can only afford that. Now it's eight. Would you add extra things in or would you say no? Eight is awesome. Now we can do more deployments or more releases or whatever. How you use what you have saved from automation is really, again, it's context dependent. There are all sorts of ideas and ways to do that. I mean, I worked with a team one time, I was an employee there, and it took them eight hours to do what we might call a regression for this one component. And I watched what they were doing and I said, let's figure out how to automate this. And it took a few weeks to do it here and there, little updates to the app, updates to the automation, and then it could do the checking in one minute once we got it done. All right, what do you do with the other seven hours and 59 minutes? Well, what happened was the delivery team or the developer said, well, hey, now we can give you more. We can give you more releases so we can get stuff out the door faster. So time to market went down because that initial regression went from a day of two people to a minute of one person. So what do you do with that savings? It's kind of up to you. What is your. How do we help you meet your business goals? That's how. And then what you do with the savings or the economy or the efficiency, you figure that back into your larger business plan.

What are some examples of that? How they would invest that extra time savings.

Right. So that was part of what I was, I guess I didn't say very well before is okay. Now, instead of taking two releases every three weeks to test, now we can take multiple a week. And potentially that gives us either faster feedback or it allows us to deploy more frequently, just depending on what you want to get out the door. In our case, it was, oh, cool, I can give you more Releases to test incrementally so that when it goes out the door, we've got more things covered. Other companies might say, oh, cool, now I can release three times a week as opposed to twice every other week.

So in this step or in this question, are you also trying to determine what kind of automation they have, or are we just talking about functional automation?

Anything. In my world, automation is the judicious use of technology to help people be more effective and more efficient. And again, in my world, those people are generally testers, but I'm also looking at delivery teams as a whole. So if you write a script that some data generation or data checking for you, that's automation, man. You did something by hand and now a computer does it for you. That's automation. Is it traditional test automation? Nah, it falls into that category that I call automation assist. Right. It's stuff that helps you be more effective and more efficient.

And I love that type of automation. Do you think people are missing out on that type of automation when they look at their. When you go in, you're like, oh, my gosh, there's so many things they could automate that not necessarily functional automation that's going to help them determine whether or not they need more automation.

Finger in the breeze. I would say way over 50%.

Wow. All right, very cool. So how do you get people to buy into that? Is that a hard sell to say? Hey, look, I know it's not functional animation, but, you know, you probably should automate this process with the Python script or, you know, whatever it is that you're doing, the automation, there's a trust aspect there.

And a lot of times you have to talk about what people want to hear. They want to hear about Selenium and Playwright and Appium and name your other vendor. I want to come in, I'm going to automate my clicking and my pressing and all the other things. Cool. Nothing wrong with that. However, where are you spending your time? And when you start asking, where are you spending your time and where is help? Where would help be along your pipeline or in your process to be faster, to be better, to be higher quality? The conversation becomes easier to have there because you're not coming in to say, dude, you should automate this other thing or you should use a script. And now you shouldn't be using whatever UI testing tool because that's a waste of your time or whatever it is. Look at what they're doing. Really be that trusted business partner. Right again, whether you're a consultant like me or whether you're a full time employee where your job is helping get that software out the door at the most effective way. Asking the questions, thinking about how it works and saying, you know what, that's valuable. However, this other thing might be more valuable if we did it first. And you start talking about opportunity cost, which it's a very deep concept, but it's very simple to talk about it just in the ether, right? It's if I'm doing thing A, I cannot do thing B. Is thing B a higher value than thing A now? It gets really complicated from there, right? So the opportunity cost there is. You don't get the cost savings or the revenue from thing B because you're working on A, which is not necessarily a bad thing. It might be that A is the better value, but you have to think about it that way, about, okay, what are we doing now versus what are we going to do later and where's our savings and where's this and where's that? It's dangerous as well because you can start getting into that sort of that sucking down to that pit of we always do the thing that gets us the money now or that saves us the money now with long term penalties of oh, we should have upgraded our version of the framework or whatever and now we're kind of screwed for a little bit. So you have to keep a bigger picture on what's going on and moderate it. But yeah, the core of opportunity cost is if you're doing A, you can't do B at the same time.

Gotcha. So in this step, is it ever like you have a team that's not efficient, where maybe they don't have the skills to meet, get them enough automation? Like, hey, you have this big team. Why do you only have a few automations in place when you could be leveraging it to get you more? Do you ever get into those type of conversations or hey, you're an enterprise company, you're using this open source tool and your team is better suited for say maybe a vendor tool or vice versa. Do you ever get into that type of discussion?

If you've seen it, I've seen it. So yes.

I can't say I've seen everything, but man, I've seen a lot of things. And yes, I would have those conversations about the appropriateness of what you're doing with automation. And sometimes it's no, some cost, got to do it. We have to keep with this because the cost to change is so high. Or oh, hey, I hadn't thought of that. What if we Supplemented with this, or we switched over to that for these certain types of, these certain types of activities. Because I've worked in that world as well where we say, all right, for this, this tool is doing a really good job, let's keep using that. But for this, the tool is not doing the job we need. Let's switch. Yes. Now we have two tools. All right. In a company of any quote size, when I say size, I don't necessarily mean people, I mean number of programs, applications, services you provide. At some point you're probably going to need more than one tool. Are you going to need 10? Probably not. I mean, unless you're Google or Amazon or whatever, where you have thousands and thousands of people and hundreds of products and stuff. Raise your hand if you work for Google or Microsoft. Most people don't. So we're really looking at somewhere sub 10 products in most people's world. When I worked at the medical company, we had closer to 30, but that's because we were mergered and acquisitioned into a sort of a smaller conglomerate. So we had different technologies and we needed different things to do that stuff. But on average, one's probably not enough and five probably starts getting into the too many.

Nice. And Paul, this reminds me of a conversation we had a while ago, I think, where you said companies shouldn't measure themselves against other companies. Like you're not a Google, if you're like a healthcare company per se, so you shouldn't be trying to do what Google's doing. And I guess that goes to your first, all three questions. Do I have the right automation? Is that, am I understand that correctly? Do you remember that conversation at all? Is that ringing a bell?

Oh, every conference I talk, I go to, when we get to the Q and A part, it's. I get to that. So yes, if you don't work for Google, don't do what Google does, it won't work for you or it won't necessarily work for you. Right. Go and look at what they're doing and adapt the parts of what they're doing that are appropriate for you. Then go look at what Microsoft is doing and adapt those parts to you as well. There's nothing wrong with looking at what other people are doing. There's not even anything wrong with copying what they're doing. As long as you sat down and had a reasonable thought, process and discussion about we're going to do this because. And here are the risks that we are prepared to undertake. And really, if you look at it, Joe, at the core of what we're doing with testing is how do we identify risk? We're not in the risk reduction world, we're in the risk reporting world. How do we explain what we've done and just as importantly, what we have not done? And here's what we have found and here are the things we think we should do next. Are you okay releasing software today, Mr. Senior Leadership?

Love it. Absolutely. All right, Paul. So we went over, do I have the right automation and do I need more automation? And the third one probably is what you see a lot is, do I have too much automation? And so what is this all about, this question?

Of course not. How could you have too much automation? Right. Well, there are a lot of different ways. Automation is an expense. Think about it as insurance. You have an insurance premium. Do you want your deductible to be $1,000 on your car or $250 on your car? What's your financial situation? Are you in a position to be able to pay a higher deductible if you have a wreck, or is it better for you to pay it out slowly on a higher deductible with a higher premium monthly or yearly or whatever it is? Different people have different situations. So do I have too much automation? Probably most people out here have too much automation. There's redundancy. There is. There's stuff that used to matter but doesn't anymore. But nobody really goes back and looks and says, we have 5,000 test scripts. Very proud of you. Are they all valuable? Right. How many test scripts should I have for this app? I don't know.

One.

5,000, 100,000? I don't know. Right. It's all about, where's the value in the risk reporting? And then how do we get the machines to help us be more effective or more efficient at the risk reporting? And going through your automation periodically, either inorganically, where you say, okay, once a quarter, we're going to go audit this stuff. Or we say, hey, we've got these test cases that periodically fail or they always pass. Well, if they always pass, maybe we should go and look at them and make sure they're actually checking the things we want them to check. So, yes, most of us have too much automation simply because we haven't turned around to say, hey, we have this giant trail of cool stuff. Is it all still helping us do our jobs?

So this is a step I see a lot of people struggling with, especially if you say, start deleting things because it's almost impressive. Oh, we have 5,000 tests. Okay. They're all just testing the same area that's not getting you anything. But how do you break through that type of culture? Is that something you see or heard teams struggle with?

I have seen it. One of the companies I worked at, we had a problem with that where there were so many tests and so much data dependency that nobody wanted to touch it. It was only adding, it was never deleting. And then we did sort of an analysis of what if we did delete some stuff. So then the financial guy came in and said every outage we have, we potentially actually lose. Not like losing effort or whatever. We actually pay back to the client these thousands of dollars, we're going to keep doing what we're doing. So that was a business decision, right? It was a terrible technology decision. It was a terrible logic decision. But from a business standpoint, it was hard to attack that because we're still doing well. Now at some point that table's probably going to turn over the other direction and we'll have to have a different conversation. But yeah, that's how you approach it. You approach it with business. It's taking you four hours to run your smoke test suite in automation. What could we cut out of there? Well, we can't cut anything out. Well, we probably could. What if we got it down to two hours? Oh, well, then we'd be able to do more of X or less of Y or whatever it is. Well, let's start looking at it. Right. And every line of code you don't write is a line of code you don't have to test and support. Similar with the code that exists, every line you delete is one that you don't have to test and support. And make no mistake, right? Test automation is programming. Whether you're using AI or drag and drop or whatever, there's a programming aspect to it. Everything that's less, that doesn't diminish your value is less. Right. It's more time to go do other things. So talking about it from a value standpoint in helping people deliver their business goals takes a sting out of the cuts about the deleting things. And not everybody's prepared to do it. So if you're not prepared to do it, I understand. And we'll, we'll go, we'll try to help you find efficiencies and effectiveness elsewhere.

So, Paul, you know these questions, I don't think you just ask once and you're done. Do you put anything in place to be able to monitor throughout quarters to see, hey, you know, ask these three Questions again? Or where do we stand in relations to these type of degrees of automation insights?

Right? So I wrote a blog a while back about eating stale automation and stuff like that. The idea is you gotta treat your automation like a software project because again, that's, that's really what it is. When you look at your app, your product, the thing that you either give to your clients or sell to your customers or whatever, you look at it and go, wait, we're having some efficiency problems here or revving some code bloat there. We need a port to this other framework over here. Treat your automation the same way, either organically, where, okay, every, every release, every push, every whatever, evaluate some portion of it, or inorganically, like I said, where once a quarter we're going to say pick 6,000 scripts randomly and go and just see if they're doing the right thing. Or 6000 is a high number. But just whatever, whatever works for you and your organization, your team. And I keep saying this, but it goes back to your business goals and how do you be more effective and more efficient? Because really that's what the automation's about, right? It's just, where's the time crunch, where's the effort crunch in any sort of activity? It's the humans, right? I mean, we have to sleep, we have to go to the bathroom, we have to eat, we have, we get sick. Computers don't do that, right? Anything you can mechanize, mechanize it responsibly and then let the humans do the things they're better suited to, or at least the stuff that computers are not, and then go on from there.

All right, Paul, I can't end this without a question about AI. I've been asking everyone this lately. AI in testing or AI in general, Overhyped, underhyped or properly hyped AI, aio.

So it is currently, in my opinion, overhyped. Because I mentioned this with. Oh, was it Scott Moore? Scott Moore and I did a thing late last year or early this year or something where what's the effect of AI on software and on testing specifically? It's going to be disproportionate, much like automation was, right? Oh, wait, now the computers can do all the testing and a bunch of people get disproportionately affected, they get laid off, they have to go do other things, whatever. AI is going to be the same thing. We are very early in AI. We should not be scared of it, we should be judicious with it. And I don't, I really, I kick myself Every day for not remembering this. Somebody posted that you should treat AI like a new hire or an intern. Get them to do things for you, but you have to check them before you would put them into production. Same thing with AI, you should not blindly take code and go, sweet, it works unless it makes sense. Right. If I get a new hired, go write me a prototype and it does what I expect it to do.

Cool.

I got chatgpt to do that for me a couple of months ago. Just, hey, I need a TCP responder. I could have written it myself in a couple of hours. It wrote it in five minutes. So why wouldn't I use it for stuff like that, but for anything that's deeper and longer ranging, especially when you start thinking about maintenance. Unless you feed your entire system continually into whatever AI you're using. And maybe Copilot does this. I haven't spent a whole time, a lot time with Copilot, but even if it does this, that's awesome that it's thinking about your bigger picture. But a lot of times it only can respond to what you put in. And if you only put in a subset, it's going to respond with that subset, possibly irresponsibly, but it's missing the larger context because you didn't provide it. So it's a little overhyped, but it's going to be really cool eventually. And I don't know when that eventually is.

Well, I think it's pretty cool. It actually wrote the theme song for me. I just said, write a song about Automation Guild and I. I think it's pretty.

Oh, no, totally. Like I said, it is cool. Especially when you're doing stuff that's not mission critical. Right?

Yes. Right, Right.

That song, cool as it was, is not going to help operate SpaceX. Right. It's not going to help the rocket go up. Right. I would not let AI do the rocket things without a human checking it.

Right, Right.

But building a song, building me an image and all, that's awesome because I don't have those talents.

Awesome. Okay, Paul, before we go, is there one piece of actual advice you can give to someone to help them with their degrees of automation testing and what's the best way to find. Or a contactor.

Absolutely. So it's the same thing I always say at the end of the show, or some variation of it. Be responsible, don't do automation just because something or someone said you should automate, look at what value it's going to provide to you, and then decide what and how to automate based on the value. If you want to find me, you can find me on Twitter Grisaffi. You can find me on LinkedIn. I'm actually on Mastodon and Blue sky as well and obviously right here on the Testing Guild. And I'm actually doing a show for Testing Guild later in October. I don't know what the topic is yet, but it'll be something so watch this space.

Thanks again for your automation awesomeness. Links to Ed Valley we covered in this episode and head over to test guild.com a501 and if the show has helped you in any way, why not rate it and review it in itunes? Reviews really help in the rankings of the show and I read each and every one of them. That's it for this episode of the Test Guild Automation Podcast. I'm Joe. My mission is to help you succeed with creating end to end full stack Automation awesomeness. As always, test everything and keep the good. Cheers. Hey, thanks again for listening. If you're not already part of our awesome community of 27,000 of the smartest testers, DevOps and automation professionals in the world, we'd love to have you join the fam@testguild.com and if you're in the DevOps automation software testing space or you're a test tool provider and want to offer real world value that can improve the skills or solve a problem for the Guild community, I'd love to hear from you. Head ON over to testguild.info and let's make it happen.