Welcome To The Test Automation Illusion with Paul Grizzaffi

Get ready to discover the most actionable end to end automation advice for some of the smartest testers on the planet. Hey, I'm Joe Calantonio, host of the Test Guild Automation Podcast and my goal is to help you succeed with creating automation awesomeness.

Hey it's Joe and welcome to another episode of the Test Guild Automation Podcast and today we have the awesome Paul Grisaffi joining the Guild again to talk all about. Welcome to the Test Automation illusion and also a new development in his career I think you're definitely going to want to stick around to hear about. If you don't know, Paul is a Senior Automation Architect at veco. He is following his passion for providing technology solutions to testing, QE and QA organizations, including a bunch of different things like automation assessments, implementation and their activities benefiting the broader testing community. He talks everywhere, he shares his knowledge everywhere. He's an accomplished keynote speaker and writer and Paul has spoken at both local and national conferences and meetings. He's an advisor to Software Test professionals in stpcon as well as a member of the Industry Advisory Board of the Advanced Research center for Software Testing and Quality Assurance at UT Dallas where he is a frequent guest lecturer. When not spouting 80s metal lyrics, which as Paul is known for, he enjoys sharing his experience and learnings from other testing professionals. Well, his place to go to hear all about his thoughts is his blog, which I'll have a link for in the show notes, but it's responsibleautomation.WordPress.com really excited to have Paul back on the show. You don't want to miss it. Check it out.

This episode of the Test Guild Automation Podcast is sponsored by the Test Guild. Test Guild offers amazing partnership plans that cater to your brand awareness, lead generation and thought leadership goals and get your products and services in front of your ideal target audience. Our satisfied clients rave about the results they've seen from partnering with us, from boosted event attendance to impressive roi. Visit our website and let's talk about how Test Guild can take your brand to the next level. Head on over test guild.info and let's talk.

Hey Paul, welcome back to the Guild.

Hey Joe, thanks for having me. I always enjoy coming on to talk with you.

Love having you. The Guild loved you as well. Now I know I did a pretty extensive bio, but I tend to botch things. Did I miss anything?

Oh, you hit all the good stuff there. And the VECO is the career thing, right? So about two and a half weeks ago I started a new gig at a company called Vaco, where I'll be doing a lot of automation architecture and then helping to grow that QA organization along with the director of QA that's already there. So I'm really excited about that.

That's awesome. I know a lot of people right now going through transitions in their careers, switching jobs, a lot of things going on in the industry. Any advice. I know this is we didn't plan this, but any advice for people switching to a new job, since you're pretty fresh, how people can hit the ground running? Maybe.

You know, it's interesting. There are a lot of different approaches for different jobs. What I did was I networked. I used my network, and then people that knew people got me in touch with other people. So I talked to a whole bunch of people, and then finally a role that was a good match for me and a good match for, as it turned out, Vaco, we were just a good match for each other, and it worked out that way. So networking for me was number one. But the other sort of piece of advice that I see a lot of other people give, and I agree with it, is when you look at that list of job qualifications, you know, this many years of that, and you can do this, you can do this other thing, you don't have to necessarily check all the boxes. Right. It's not an all or nothing thing. If you have most of those skills or you have most of the core skills and maybe not all the desired skills and stuff like that, go ahead and apply. Because you never know what you get picked up on, that someone says, hey, they don't have this one skill, but while they got these 12 other things that I really might be able to find some use for as well.

Absolutely. And I like how you said it was a good fit for you as well. I don't know if a lot of times a lot of people don't take their needs in consideration. They're just trying to get a job. I know it's hard for some people. They need a job, obviously. But I think making sure it's a good fit for yourself as well, not just the employers, is something people should look for as well. Do you agree with that?

Absolutely. You spend a lot of time at the job, it's got to be at least somewhat of a fit for you. Right. Because every job is going to have some part of it where you go, ah, this isn't my favorite. But overall, it has to be more positive than negative or more positive than challenges along the way.

Awesome. All right, so I'd like To dive into your latest article, the Grand Illusion. It's based off the Styx album, the Grand Illusion. I'll be honest with you, I was never a fan of Styx. They were a little before my time. Then I saw them in concert opening up for Def Leppard and Tesla, who I love Tesla. And they were better than both those bands. So I guess that's an illusion. Don't judge a book by its cover if you don't really know them or see them live. But why pick this particular album or this band? I guess to make a point for this particular blog post.

So a lot of things sort of went into me getting onto that. I was having an online conversation with somebody about some particular aspects of test automation, and their opinion was very. First of all, it was a very apt opinion for their situation. It was absolutely, you know, what. What you're doing and what you have at your disposal. I would do it that way as well. Most of the clients that I work with and have worked with don't have those means either right now, or they don't have the budget for it, or they don't have the expertise to work with that tool set. So we have to do other things to help them be more effective or more efficient at their jobs. So I kind of looked at that and went, if everybody's looking at what everybody else is doing and thinking we need to do what they're doing and they're doing, we need to do what Google's doing and Facebook and Amazon and whoever else we're missing one of the real key parts of what the automation is going to bring to us, and that is the value for our context, for whatever it is for that company or that team and what they're trying to do. Because everything you look for, you think your neighbors got it made. And that tripped my memory over to the lyric from the Stick song. And then I looked at, wow, a lot of this really would be an illusion, because it really looks like one day Google twinkled their nose, right, and boom, they had all this automation up and going. And it didn't happen like that. It doesn't happen like that because it can't.

Absolutely. I love that. So I thought we'd dive into each point of the article on welcome to the Tesla Automation Illusion. First one, which I'm sure both our careers, we've seen this since the beginning, is, I think you mentioned automation is often sold as a grand illusion. I don't know if you see a rise in this now with ChatGPT and AI making this even more pervasive within the industry. But can you do 100% automation testing or is that a grand illusion?

I believe it is a grand illusion and it will be probably for certainly for the rest of my career and for my kids career as well, you know, whatever they wind up going into, but sort of that career time there. Because if you think about it directly or indirectly, who is using the software, who's consuming the software? Humans. So at some point we have to identify the fact that it has to be human worthy, human usable. Some eyes have to be on this software at some point. And then if you're going to do it with a human, a tester, does it make sense to also automate that? Maybe it does, maybe it doesn't, but you have to get the right value out of it. You'll hear me all the time say value, value, value, value, because that's really what it is. Is it valuable to 100% automate something? Never mind the fact that 100% of what? 100% of what is automatable? 100% of everything you can think of. Are you going to point the chat GPT at it? Get them things that maybe you didn't think of and you say cool, I thought of all the things now. No, it's just not feasible at this point. And introducing things like these large learning models and ChatGPT and then all the other ones that are there, they're going to muddy the waters because what you're going to see is you're going to see a lot of people taking exactly what these models and these tools give you and say cool, that is all I need to do, the AI told me so. And a lot of things are going to be missed just because they're imperfect anyway. But then what was their training data like? What are their heuristics like? What is all the confidence? I think what it's called confidence factor. There's a mathematical thing about when it decides if something's right or wrong or in or outside of that data set. It's going to be a while before any of that is really going to be trustworthy, such that you can say, okay, it's helped me with this set of what section of what I do and I don't have to go and double check it or I only have to spot check it because I know statistically it's right more than it's wrong.

So how does someone as a tester avoid the illusion if everyone else is tripping or has that illusion and you're the only one that Sees it clearly, like your company says. You know what I mean? Everyone has a. They're all tripping. They're all seeing like, wow, man, AI is going to do everything. You're the only one that's like, wait, how do you resist buying into it and say, okay, yeah, let's just go with the flow here first.

I'm not against it, right? I'm all for, oh, we have this new capability, let's test it out. But the same way you wouldn't take a new piece of software and from scratch, just build it and throw it out and say, now it works, are you going to use it? Right, you're going to fool with it. You're going to see what it can do, what it can't do. A lot of people out there are working on what it can do. Myself, the little bit of time that I spent with these tools and some other people sort of with a similar mindset is, okay, let's see what they can't do. Right? Let's ask it. I did this. I said, write a paragraph about test automation in the style of Paul Graffi. And it said, nope, I don't have access to the Internet, I don't know who that is, and I can't write in the style of other people. I said, okay, write a paragraph of test automation in the style of William Shakespeare. And it did it. It's like, okay, are you wrong? Did you tell a lie? Did you? If you start, you know, personifying the AI, right? So right there we know that there are some conflicting things that it will tell us about what it thinks it can and can't do. I also did some tests where I plopped in some code that had a bug, but the bug was based on the name of the method, did not match what the loop in the method was doing. So code review would catch this, but Chat GPT did not the first time. Then I put it in again after asking some other questions and it caught it that time. So it was weird about what it can and can't catch. But remember, these are relatively early days with this sort of consumer available command prompt AI.

Awesome. So I guess even more controversial is when I started my career, it was mostly, you need testing skills. And then it became, no, you're a developer, you need developer skills. Testers are developers. Yay. Let's learn all the algorithms and everything. And then now it almost seems like it's swinging back because we're going to have all the heavy lifting done for us with some of these tools, but we're going to need to use our brain to say, as a tester, is this correct in my context, is this correct? Is this find in risk? So do you see a swing back to maybe people having to focus in more on testing skills or am I overthinking it?

So I think in general, whether it's testing skills or development skills or whatever, these types of tools are going to help, are going to force us by and large to work differently. If there's less typing and more thinking, does that make you any more or less of a developer or a tester? Probably not. It just means that over time, as the tools have gotten better, you built a different skill set. We don't. By and large, the average developer does not write an assembly language anymore. Right? There's no need to. We have all these higher level languages that do in most cases for most applications, do just exactly what we need them to do, give or take. But we don't have to program in these stone knives and bear skins anymore. Right? The AI tools are going to help us hopefully deal with sort of the rote stuff, the DRE of programming, and potentially allow us to be more expressive, more creative, focus more on problem solving and less on key typing. At least that's our kind of hope. We start finding a sweet spot here. Now, does that mean that again, does that make us more or less of developer or more or less of a tester? No, it's a different skill set and maybe over time we get different titles for what we do, role names, but that's going to matter to some people and not matter to other people as well.

Absolutely. Great advice. Like you said, even developers themselves don't really like go low level. It's mostly cutting and pasting, putting libraries together. I think testers need to understand developing concepts just like developers do. So yeah, definitely. Great advice. So I guess the next point then is I think you mentioned in the article how a lot of people may look to a larger company and think, oh wow, this company is automating everything and they're eliminating testers and sometimes creates unrealistic expectations for other companies. So I guess can you expand a little bit about this and how people can avoid that as well?

Oh, I think it's less about avoiding it and more about avoiding the illusion of that is what you must do because those famous companies do it a lot of times. Back in the old times when we had a lot of in person conferences, that question would come up almost every time in one of my sessions and I would always ask the attendees, raise your hand if you work for Google, Microsoft, Facebook and zero people have raised their hands in any of my sessions. Now does that mean none of those people who come to my sessions or whatever? But anyway, nobody did. I said, okay, so you don't work for those companies. So you have different needs, you're shipping a different product, you certainly have a different budget, you certainly have a different tolerance for risk and you certainly have a, probably a more conservative approach to software development simply because you're not them and you don't have pockets that are that deep. Now should you be more like them? Now that's a context question. Do you want to evolve and pivot and behave like them because you find you've done your homework, that you are sufficiently like them or want to be sufficiently like them, that adopting their practices as is is something that's going to be valuable to you? Or should you look at what they're doing, take the pieces that make sense for you? Right. Standing on the shoulders of giants, so to speak. But not, not necessarily going in blindly to say, I'm going to copy this template and we're going to do exactly that. Because that's probably a bad idea. You're probably not going to get the value you're looking for out of your automation. And either your automation is going to fail or it's going to look like it failed. Right. Another illusion, because, well, automation doesn't work here. Well, maybe it does, maybe it doesn't, but maybe it does and you just went about it in a way that wasn't context sensitive.

Absolutely. So I worked for a big healthcare company and they're like, we're gonna do continuous delivery. Cause that's what Google does. And we're like, okay, we need to work with regulations. We can't just do continuous delivery. So they still went far with it and it was a disaster. Not because continuous delivery is bad, but like you said, for our context, because we were trying to copy someone else, it didn't fit. So is that like one example you would see?

Yeah, that's a prime example of you tried to do it exactly the way they did it and it didn't work. Instead of saying, oh, I see what they're doing, let me do some homework on that and change the model to fit what we're doing here at large healthcare company. You might have been more successful that way.

So it's almost like using them for inspiration, but not blindly, just copying them out of just.

Exactly, exactly. Inspiration is great. Again, that's standing on the shoulders of giants thing. I don't see any reason to reinvent something that is working or at least could work for you if it's appropriate for you.

Absolutely. So I guess another grand illusion that people get sucked into is tools. I'm one of them. I love tools. But there's always a, like you said, a shiny automation tool available. So when someone sees a new tool, say like playwright, playwright seems to be the hottest one right now, and people are like, oh, how would I do to learn playwright? They automatically just think, I'm going to switch. When do you know is the right time to switch to a new tool if there is? And like, what are your thoughts on transitioning from one to the other or even seeing if it makes sense to transition from one to the other?

So it's a delicate balancing act between not going too early, right? Not being on the bleeding edge where you're taking all the bumps and bruises of learning it and the bugs and all the things in the early days right before other people that have the opportunity to do that, like say, hey, I'm just starting off in a brand new greenfield automation initiative. You got a lot more choices, right? Because you don't have the sunk cost already. You don't have the wait, what do I do with my existing stuff? How do I train my staff on the other thing too, right? I've already eaten these costs. Do I really need to eat some more costs? So it's a different financial model for that. And I'm no financial expert, but I know enough words to say, yes, there's, there's a cost here and it's the opportunity cost. Because if you're taking time to learn and build in a new tool into your tool chain, there are other things you're not doing. You're not extending your existing automation, you're not doing additional testing, you're not doing additional. There's something you're not doing there because you can't do two things at once, right? You can't. There's a cost, but there's a cost the other way around too. If you stick on, quote, your old tool too long, it's going to get more and more expensive to make a switch when it is time to switch. So that's a different opportunity cost. And that's where you really have to sit down with your leadership, with your dev leadership, with your pipeline and infrastructure leadership and your product leadership and your automation leaders. Everybody needs to sit down and go, what does the product roadmap look like? As far as we can tell? And I don't mean features. I Mean, okay, let's look at what oh, we're using React. What's the React roadmap? When is the new React going to come out and is it going to break our existing automation or is it going to have stuff in it that the existing tool is not going to do or won't be able to do ever or something like that. Those will start impacting and crafting your solution. And your answer to say, when do I make a switch? And the answer might be never. It might be that you've got so much invested in what you have. It is a better financial approach for you to make do with that, knowing that there are gaps that you can fill in differently or and not everybody wants to hear this, sometimes two tools is the right answer. Hey, for 90% of what we're doing, we're using our existing tool, even going forward. But for this subset of stuff that the tool can't do, we're going to supplement with this other tool, knowing that we're going to have to re implement some things and there'll be some parity and some additional maintenance there. But it may be worth it if you know that you're going to get additional gains in efficiency or coverage or whatever it is time to market that is going to realize some real financial gains for your company. And yeah, you eroded a little bit from that because you had two tools, but the delta is so large that you it's well worth it to do it. So there's really no stock answer there. The right thing to do is keep an eye out, look at what's coming out, look at what the vendors are given. Even if you're an open source copy. Right. We do everything open source because reasons look at what the vendors are doing because at some point that may tip over as well again, either as maybe not as a tool replace, but perhaps as a tool supplement and never be caught where you don't have an opinion on one of the current tools that's out there. Maybe not all of them, but at least for the ones that likely would be good candidates for you in the future. Have a couple of paragraphs, have a little white paper for each of them where yes, you took some time and you had one of your experts download it, prototype with it and give an opinion on it so that you're prepared to if something comes along that says, ooh, this might be a good time to change. What would we change to?

Well, I love that almost having like a file of research that you've already done, someone comes to you right away you can say, wait a minute, here's some pros and cons I've actually looked at. So that's a great approach, I think. I guess that's also one illusion I don't know if you cover this is a lot of times people just go open source is the only way, when in fact, sometimes if you do your research, a vendor tool may apply better to your particular situation. I don't know if you see that as illusion or not, but no, I.

Did not cover that in there. But that's actually. Maybe I'll write an addendum about that one. But no, I find it a lot as a consultant that you're going in, you're going to help them with this. But they've already made the decision, well, you have to go open source. Why? Well, because it's free. Okay, yes, but let me explain actual cost and effort costs and all this from a conceptual standpoint for you. And there may be things you can't do because these open source tools can't handle xyz, but this vendor supply tool can. And just various things like that. And it's funny, and I grudgingly say this is that the development departments, organizations will grit their teeth, but they will get that software developer Visual Studio enterprise license to do their job. But they're not going to spend a few hundred bucks a license for a test tool. They're just not.

No.

Can't Selenium do that? No, Selenium cannot do thick clients. All right, so now we have to go go Appium, right? Okay, but Appium, now we're going to introduce WinApp driver, right? Or we could go with this vendor supply tool that's going to cover some of that for you. No right or wrong answers. It's, you got to sit down and understand what your cost and your value propositions are 100%.

And I just thought of another example. As you mentioned, you need to do a cost analysis. What makes sense sometimes people exclaim is the right answer. Well, this is a while ago we used to do QTP and they said, oh, we're going to just go open source. But we worked once again for a hospital and everyone was on older versions of ie, which Selenium was really not good with. So they made the switch over and it was terrible because not that Selenium was terrible, but because QTP handled the older browsers much better. And as you said, if someone had done the research, they could have said, yes, Selenium, maybe for this other project would be the right choice. But this current One, it's not going to work.

So. Right. And that's another thing that I, I don't remember if I've written about it, but I've talked about it a few times, is in a, in a company of any sufficiently large size, with a sufficiently large number of disparate products, you're probably going to have more than one test tool, one automation test tool, simply because you're just going to have so many different things that one tool and one framework may not be enough. You may wind up losing more value than if you brought in a second tool.

Absolutely. So I guess another illusion is, oh, the automation tool can handle it. We can have anyone use it because it's so user friendly. Do you see that as an illusion, maybe giving someone what's considered a powerful automation tool and thinking that they'll just be able to get up and running with it without any issues?

Yes, because I find that as an illusion and it's a rather dangerous one because every time we make a good strive into saving effort with automation, that comes back up again. Oh, now I can have anybody write these tests. Well, you can have anybody click the keys, right? You can have anybody drag and drop or record or tell whatever GPT that you want a test case to do this. Well, it can't be just anybody, right? First of all, they have to have domain knowledge. Second of all, they have to at least sort of know how to test. Okay, maybe they don't need to know how to write in a programming language, but no matter what we're doing under the hood somewhere right behind the curtain with the wizard and all that, it's software and we're developing software, we're sequencing instructions. We have to save those artifacts, we have to look at the results of those artifacts, we have to maintain those artifacts. And yes, the AI machine learning changes the dynamic on what it means to maintain those things, but those operations, those activities still have to be there. So if you say, hey, person that doesn't know anything about testing, but yeah, you worked in the medical industry for a while, hired you in, and you're going to test this revenue cycle management tool for this healthcare company, go. That's not the way it works.

Sure. So, Paul, I thought we'd have you since you're a music expert and I thought I chose Styx how you would rate ChatGPT's understanding of sticks with software testing. So what I did, I said, I went to ChatGPT and I said, give me other analogies other than Paul's on how testing relates to stick Songs. So here are the top five. I want you to rate how ChatGPT did.

All right, let's go.

First one it came up with is Come Sail Away. It says it's like exploratory testing. The song Come Sails Away can be seen as an analogy for exploratory testing where testers embark on a journey to discover the application's behavior without a predefined test plan. In context, how would you rate that.

Rating based on title? I'll give it an 8 rating based on song content. It's like a 2. Right. Because the song really is about aliens.

So ChatGPT once again failed in that context. And you need to think about it. Awesome. The second one was Renegade said penetration testing. Renegade can be compared to penetration testing, where testers act as renegades or ethical hackers to identify vulnerabilities in the application or system.

So again, based on title, that's a good eight, nine. We're talking about lyrical content. That's another two because that's about dude killed a dude and now he's on the run from the law. Like. Yeah, no, that's more like a black hat. Right. Now you tell me this black hat, I'm going to go. All right. Yeah. All right. I'll give that one like a seven.

Awesome. All right, now this one might be a stretch. Too Much Time on My Hands, it says it relates to performance testing in that the song Too Much Time On My Hands can be seen as a metaphor for performance testing as it emphasized the importance of time in software testing.

I'll give it a five. That shows a little more introspection into what Too Much Time on My Hands means and sort of matches some of the lyrics.

Don't know if you know the song Show Me the Way it says it. Usability testing in that it can be associated with usability testing where testers evaluate how user friendly the intuitive and application is.

Yeah. So Show Me the Way was the first single off the first record after Tommy Shaw left. So the one after us. Kilroy was here.

Yeah.

Again, I'm gonna give that like another five because I don't really remember the lyrics of Show Me the Way.

Absolutely. And the last one is the Blue Collar Man. Long Nights is like regression testing in that it can be likened to regression testing, which involves the repetitive process of retesting previously tested features. Just as the blue collar worker in Dora's Long Nights of Hogwart.

Okay, I'm gonna give this one a good solid seven because. Yeah, yeah, it is a lot of work and it's, it's thankless. A lot of times that it's working into the night. Yeah, I'll give that one a good solid seven. But it missed the most obvious one, right? It missed Mr. Morobato.

Oh, yeah, for sure.

That would be the most obvious.

All right, so that's just a quick test of ChatGPT, how you still need a tester to know the context of your situation, not just spit out stuff because it sounds right. Absolutely. So what do you think of that, Paul? I just thought that'd be a kind of fun segment to sprinkle in to make it a little more engaging.

Absolutely. No, I think it's great because you've heard some of my stuff before. Music inspires me, automation inspires me. And a lot of times I find these joins there and I include them in my blogs or keynotes or whatever.

All right, Paul, as I mentioned, you speak at a lot of different events. You're well known for speaking. So any upcoming events you'll be speaking at that people should check out?

Yes. So this coming week I will be speaking at inflectricon in Washington D.C. i believe there are still tickets available. You can check it out@inflectricon.com and they have taken over part of the old STP software test professionals conference, the STPCON track. So you can go and you can get all your Inflectra knowledge, but you can also get a non vendor General testing automation DevOps track as well. So you got choices there. And that is, I think I'm speaking on the 21st. And then in May I'm going to be speaking at the QA Global Summit, which is online. The basic track is free. The junior or the senior track does have a cost associated with it and you can check that out at Geekle.

Awesome. All right, Paul, before we go, is there one piece of actual advice you can give to someone to help them with avoiding any type of automation testing illusions and what's the best way to find or contact you?

So it's the same actionable advice I give every time I'm on the show here and that is be responsible with your automation. Don't do stuff just because somebody else did it. Do it because it's going to provide value to your team, your company, your organization, whatever the value target is there. And the best ways to get me are on Twitter, Grisaffi, LinkedIn or if you go to my blog, I've got all that information there and that is responsible automation@WordPress.com thanks again for your automation.

Awesomeness for links to everything of value we covered in this episode, head on over to test guild.com a 4:44 and if the show has helped you in any way, why not rate it and review it in itunes? Reviews really help in the rankings of the show and I read each and every one of them. So that's it for this episode of the Test Guild Automation Podcast. I'm Joe. My mission is help you succeed with creating end to end full stack automation awesomeness. As always, test everything and keep the good. Cheers.

Hey, thanks again for listening. If you're not already part of our awesome community of 27,000 of the smartest testers, DevOps and automation professionals in the world, we'd love to have you join the fam@testguilds.com and if you're in the DevOps automation software testing space or you're a test tool provider and want to offer real world value that can improve the skills or solve a problem for the Guild community, I'd love to hear from you. Head on over to TestGuild Info and let's make it happen.