The Four Phases of Automation Testing Mastery with Jon Robinson

Foreign.

Get ready to discover the most actionable end to end automation advice for some of the smartest testers on the planet. Hey, I'm Joe Calantonio, host of the Test Guild Automation podcast and my goal is to help you succeed with creating automation awesomeness.

Hey, it's Joe and welcome to another episode of the Test Guild Automation Podcast. And today we'll be talking all about preparing for automation, testing success and a bunch more with John Robinson. If you don't know, John has helped build and lead a ton of different teams across all aspects of QA, CX and marketing over the past 18 years and is currently the chief storyteller for Next IT, which specializes in helping organizations improve their overall quality. He is the former chief storyteller at Provar and has worked with organizations like Nissan Scripps Networks and Victoria's Secrets to battle test his methods. He really knows his stuff, if you don't know. Also, John gave an awesome workshop at this year's Automation Guild. I think it was the highest rated sessions. It was. He was tied with someone else and I thought, hey, let's get him on the podcast and pick his brain. More about how to prepare for automation testing and a bunch more. You don't wanna miss this episode. Check it out. Hey John, welcome to the guild.

Hey Joe, thanks for having me, man.

Awesome to have you. Hey, what the heck's a chief storyteller?

You were not the first person to ask that question. Obviously I got the idea from Microsoft a number of years ago and it came about because the one thing when I was at Provar, we really struggled to figure out how, as you know, testing, qa, test automation, it's a very hard topic to describe to people, to take to market and kind of explain what it is you do. And so the thing that we kept coming back to was how do we tell our story? How do we tell the story about what we provide value wise and how we help people, et cetera, in looking and saying, you know, storyteller seems like a really, really good idea. I started looking into it and it really kind of fit with what we were trying to accomplish, which was how do we kind of have a singular go to market message that everybody can relate to. It applies to the entire organization and it works really well at home too. My kids like to call it the chief liar, so I kind of just stick with it now. But it does. I like to tell stories. Those that know me know I like to talk a lot, but I like to wrap that into something that's fun and Engaging and interesting.

All right, already off the script. So if I'm a tester, it seems like this would be a skill that testers should be good at because they sometimes are overlooked. So how does a tester implement this?

I would absolutely. This is the one thing that probably relates more than anything. I talked to a friend of mine right after we kind of came up with the title, and he said it's the one thing that really hit home for him, because in a lot of cases, QA people and testers often have to tell a story about what is the impact of this thing, or how do we effectively test this thing. And sometimes user stories aren't enough. Right. Sometimes you have to really put yourself into the Persona of the person you're acting as from a test perspective to really, truly understand what it is you're trying to accomplish. And so it is a skill that I think a lot of testers should do more of, but I think a lot of them are really good at it already. And it's really just recognizing that a lot of the things that they're trying to do is tell story. And the best way to communicate things sometimes is through story.

So how do we learn to tell stories? See, I'm in the retro, and I want to highlight all the things automation has done. How do I do that?

A lot of it is starting with. In a story, you have the opening, you have the setup, you have the. The core plot, you have the resolution. And it's really taking a lot of those same elements and applying them to what you're trying to accomplish. So in the case of an automation project, for example, you're going to go in and you're going to say, you know, we went into this trying to get to this place, and this is the approach that we were trying to do. And along the way, this is what we tried. These are the things that went right, these are the things that went wrong, and just kind of giving some context to the answers you're bringing to the table. And a lot of times it really boils down to just explaining context. So often there's drops in. We give the stats and the numbers, but the context behind it's often missing. And people that don't know how to do it, the easiest way for them to do it is to just tell a story around how they got there, like what was the reasoning behind what they did, and follow the hero arc, so to speak. And it works out better than you'd expect.

Yeah, I think it's a valuable skill, especially now with AI it's going to be the human aspect of being able to tell a story that's going to resonate with another human being, I think for sure.

Yep, absolutely. It's the one thing that we, you know, stories have been a part of human culture for as long as we've been around and have had written in spoken language. That's how we communicate things. And the people that do it effectively are the ones that tend to be the most successful when it comes to delivery on their projects. Because it's not just the wrap up. Part of it is telling the story to get people engaged and involved to begin with. Why do they want to get involved? Why are they interested in doing this thing? Selling your story up to management and leadership. Why should they invest in this thing? That context is so important and we need to do more of it.

All right, so how do you battle a story against another story? A lot of times a business has been sold a story by a vendor about automation. Like this is what automation is, this is its purpose. And then you have to come back with another story. Well, all right. This is actually what automation testing and its purpose. How do you tell the story of, like how do you define what is automation testing and its purpose?

The ones that are most effective are the ones that are grounded in reality and can be tied back to specific successes or failures or point to details that everybody on the planet has been sold a story or a bag of goods from a vendor that promised X, Y or Z and it was exactly that, it was a story, it was a made up fictional thing. But when you're trying to tell that story internally and you can wrap a personal commentary around that, around when we did this, this is what we did and this is the outcome that we saw. And give specific examples where you've seen success or you've seen failure, those tend to resonate more than the marketing speaks, so to say. Right. They connect better because people relate to things that people do more directly and can speak to firsthand knowledge of it. So when you ask me a question about something that I just told you a story about, I should be able to give you more detail about that. And if I can't, then well, maybe there's not much to that story.

Absolutely love it. So are there any misconceptions you've come across about what is automation and how to think about automation?

Oh, oh, yeah, lots of those. I would think the biggest one is the, the misconception that automation is this magic bullet and it's going to solve a lot of things. Overnight. One of the biggest probably challenges that when I was at Provar, especially because we were selling an automation tool, one of the biggest roadblocks and barriers we had to get over was this idea that we'd sell you the tool and you'd be effective immediately. And that's not really reality. And no tool can do that. Right. It doesn't matter what your solution is, there's always a ramp up time to get to where you want to be. Right. And part of that is establishing how do we establish where we want to go? Like, where are we now, what are we trying to accomplish, where do we want to get to? And then those tools can have a fighting chance of success. But the number of tools and solutions or automation initiatives that I have seen fail because there was absolutely no planning that went into them. There was a, we're going to automate this thing and we got a bunch of either X developers or really smart guys with Selenium, and we're going to build this new framework and we're going to do amazing things with it. And six months later you have four scripts. You know, sure, you can do a lot more than that, but if you don't know where you're going, you're going to flounder a lot.

Absolutely. And that's why I loved your session at this year's Automation Guild. You went over, I think like part of Your session was 90 minutes of a strategy for test automation and yet some key components that you went over. Like, I think it was like, you know, scope, risk analysis, test coverage, techniques.

Yep, there was a whole. So we basically break it down into four, four chunks and it's, you know, discovery, design, develop and deliver. We call it our 4D model, but you can call it whatever you want. But there's really those four phases where you do a lot of your information gathering up front. You understand the story of where you're at. When you get there, you decide, okay, where do we want to go? What do we want to try and get out of this? What's our goal? Believe it or not, this is one of the things that even well seasoned veterans in the QA and testing space don't really grasp the concept of maturing along the way because they're like, oh, we're doing automation now and we're just going to continue doing automation. Well, there's a continuum of that and there's different things that you can do to improve along the way. And so we put together a maturity model that just kind of helps people understand where they're at both from just a general QA standpoint, but also just in the test automation space. Like, how are you moving along that continuum and are you improving and are you moving toward what your goals and objectives are as an organization? Because if you're not furthering those, why are you doing it?

So I guess, what is that maturity model then? What do you mean? Like, people aren't growing?

So let's walk through. We broke it down into six phases. We've got, you know, you're not doing any QA or testing at all, right? You're basically, you've just started some development projects. Now you've done enough work that you need to do some testing. You don't know what that testing necessarily looks like, but you're doing some manual testing. It might be developers, it might be product owners, it might be Joe Blow down the road, who knows? You realize at a certain point that's not going to cut it. And so you have to do something more. And that's when you start to think about and look at automation. And what is this thing called automation? How do we get value out of it? And this is the point where you get the first misunderstanding with automation is that we're going to start automation and that's going to allow us to get rid of all of our manual testers. And that's. Anybody that's done automation, anybody that's done manual testing very clearly know that's not a reality because there are things that just don't make sense to automate. But you'll start there. And that's the first kind of proof point that you'll have to get past. You'll start to split now, you know, you're doing some automation, but it's probably still 80% manual for the most part. As you move forward, you might get that more to a 50, 50 split, what that looks like from there. Okay, what are we trying to accomplish now? Is it just so that we don't have to spend the time doing the testing? Is it so that other areas of the business can take advantage of the test automation that we have in place so that now the development processes are a lot more simple, they go a lot quicker? How do we enable other people to contribute to this automation suite that we're putting together and those initiatives? And now you're starting to get into some of your CICD tooling and how do you work with your DevOps teams and start to flesh that out in a wider set until you get to the point where you're completely integrated in CI? Cd you check a new test in it gets run when relevant. Right. You've got it now tagged according to when this test should run based on which features being developed, which areas of the applications are going to be impacted. All of those things are now taken into consideration. That continuum. Right. A lot of people get into it and don't understand there's things further down the road that need to be considered. And so they'll go into it and they'll. One of the things we talk about is your short term versus long term objectives. You go into it and you start building all of this work and you start doing all this automation, but you're not really thinking about the long term goals for that. And then all of a sudden you realize, oh, well, we need to do this. Well, that means we have to start over. Because we weren't considering that a lot of this effort that we went into it, you might have chosen a tool, right? And you did your automation based on that tool as opposed to choosing a tool for the automation you need. Well, that tool might not integrate with your tooling for your DevOps. Right? So now most tools are not open compatible with other platforms. And so all the work you just did, whether it was a year or five years, you're going to have to start over. Because that wasn't a consideration for where you were going to go long term. Because the thing was, our company bought a tool, you use that tool to do your automation.

That's a great, great point. I don't know. Do you recommend people look at roadmaps? Because I work for a healthcare company and they start off the greenfield.

I'm sorry about that.

Yeah, it was brutal. Thank God. Not anymore. But we started with the Greenfield application. It was a web app, right. And then a year down the road, like, oh, now we need to integrate with a thick client application so a doctor could dictate notes to this app and you're like, gosh, this framework is built around totally web, totally threw us off. So was that in the design phase? We messed up.

Yeah, no, I mean, I think it's so one of the things that I highlight in kind of my process is as you go through it, a yes, look at your roadmap, look at the things that are known. What are the things coming down the pipe? Understand, okay, this is where we ultimately want to get to, but we have to do some stuff right now. What are our options for the short term? Do we preclude any of that stuff? But then as you're going through it, you mentioned you Know, you got midway into this process and you realize, oh, they're getting ready to introduce this new thing that we're not capable of integrating with right now. Like, we can't support that. So then you have another question that you need to ask. Is that something that requires, do we need to do that? If we need to do that, what is the impact of that and can we pivot the stuff we've already done or. Or is this a new initiative? Right? In reevaluating as you're going, as new information becomes available, always do a reevaluation. Because one of the things that you should always consider is sometimes automation is not an effective use of your resources. It is not always the right answer. Sometimes it is more overhead because that thick client and the work that would be done to create automation for that, because you've already got this well established platform, the work to either integrate them or do a new initiative might be more than the total time it takes you to test it, period. If you're not going to use that for a long stretch of time and for multiple purposes, it may not be worth doing it. Right. So then the question is, why are you considering automation for this? Is it just because you've got a well established automation process that we automate and so therefore any new thing introduced, we have to automate? That isn't always the right answer and I think that's a reevaluation that people need to do as they go through and new things get introduced.

No, it's a great point. And once again, there was a team that created this thick client application, just like you said, and we're like, do we even need to test it? It's already been tested. So we got to a point we had, after multiple stories going back and forth, realized, look, we just need to test up to the point where we send a signal, make sure it's received. They create a little API that would test it. We didn't have to interact with the client application because it's already there.

What you just described is 100% what I wish more people would look at the actual requirements for what needs to be tested as opposed to, I have an application that I have to test through the ui because a lot of times an API call on one end of the other is all you need, right? I work a lot in the Salesforce space and the number of times people will try and test the Salesforce functions and capabilities, the things that Salesforce has built and tested as part of their workflows. When guys, you don't need to know how to just create a new lead or a new opportunity. Salesforce has done that six days from Sunday. Like, there's no. There's absolutely no reason to do that. Use the API, make a call, create a new object, and then move on. Like, that's all you do.

But how do you not get sucked into the. That's not how a real user would do. Use our application. It's like, oh, my gosh, it's maddening. How do you handle that? Objection.

I think it's understanding what. So you look at it through the lens of if you look at most test cases that are put together and I challenge you to do this exercise on any test case that you have in your suite, 60% on average of that test case is not the actual test. It's just getting ready for the test. It's all the data prep that you have to do. It's navigating to the right screen. It's all of this stuff that has to get there. And so a, you're doing a lot of things that really aren't actually the testing. So if you're going to treat it as a real user would treat it, you have to start splitting your test cases into two different things. One, you have the functional testing, which is this is how the thing was built. The developer said they were building this thing, and I'm testing to make sure that they built that thing. That is one set of test cases based on your user stories, functional requirements, et cetera. Totally makes sense. But the other side of that is how does a user actually use it? That regression suite that a lot of people just tend to migrate from their functional tests into a regression suite. Really, if you stop and think about what you're trying to accomplish is should be a. Here's a list of all of the functionality that user's supposed to be able to do within our system. It's a pretty static test suite, right? As new things get added, you add new functionality. As things get deprecated, you take it out. But those things should align with how the user uses the system. Sometimes those will align with your functional test cases. Not often. But that test suite is static. And looking at it through that lens, that's a different set of testing, though, than the functional test that a lot of people say. Oh, we do inspiring automation. Really? Because how's that working out for you? Because I bet about 70, 80% of that's maintenance. Like, your actual testing is probably pretty low.

So what do you mean by that? So you don't recommend In Sprint Automation or do you recommend being 1? N1?

I like n + 1 or even later for automation purposes, depending on what you're trying to accomplish. Okay, right. So if you've got an application or system that lends itself very nicely and you've got a strong development team that can hand off things to you that are not going to change, then great. In Sprint automation is a noble goal. I think it's the where we should desire to get to the realities of that though are think of how many projects you're on where you come in and in Sprint 1 you develop a feature and in Sprint 2 you realize they left out a bunch of the pieces of the user story or functional requirements. And so now they have to make adjustments and changes to that. Well, now your test's got to change and then, you know, you come back into it and in part three, they're like, you know what, that actually doesn't work the way we thought it was going to work. We need to make a change. So over the course of four or five Sprints, every single time that functionality changes, how much like actual net new automation are you doing versus just maintenance in that scenario? And that is happens on so many user stories and so many pieces of functionality that in isolation it doesn't. You don't realize it's happening. But when you step back and look at actually how much time are you doing new things and creating new tests as part of that in Sprint Automation? Because those things weren't done. It happens all the time.

Yeah, absolutely. So it's hard to balance between the optimal and where you're at. So how do you avoid tech debt in that case? That's what people will come back for. Well, no tech debt will build up and once it's automation, they're going to be like, forget about it, it's too much. They're not going to focus on testing test. It's going to be pushed down and down and down until you're kind of in a jam.

This is where I like to focus. If you're going to start on automation, focus on regression and focus on building test suites that test the functionality from the end user's perspective as if it is all said and done. That means often, unfortunately, waiting until the end to build some of that automation. But one here, let's talk about another fallacy or misunderstanding about automation's value to a development project. Its value is not to the version that you're pushing out right now. That is not your value. The value for automation and regression in the Long run is the next version, so that it cuts down on the time you have to spend on the next go round doing all the work that you did this time. Right. So the work and effort you're doing now, sure, it might save you a little bit during this release or this particular version, but the reality is its real value is the next one. And that's where a lot of people misunderstand where they're getting the actual value from. So I would rather push the automation to later in the process where things are more stable, less maintenance costs, less rework costs, and then spend it so that when I'm running it the next time, I know I'm actually running a valuable test, which allows me to skip the tech debt part, because I'm not worried about testing the individual changes that they're making. I'm worried about testing the functionality that is supposed to be there and in place. And so when you make a change the next time, does it still work? Right. Did you break anything?

So the benefit more is it's a safety net that developers have confidence in knowing. If I push code, I know I have strong tests that will give me some sort of confidence in knowing.

Okay, yeah, because you're right. Tech debt, in and of itself, nobody goes back and tests that.

Yeah.

Like, it's stuff that developers do on their own, off to the side when they have the bandwidth to do it. And it's not something that normally justifies it. The best way to test it is with your regression tests. But the regression tests are only as good as the regression tests you have previously built, if you don't already have them.

Absolutely. So how do we measure then how we're doing? Like, how do we have a pulse on it? Do you have, like, I know buzzwords, roi, but everyone wants to tell a story about ROI or what they saved or what they did for the company. How does that work?

I mean, those are all. I mean, one thing I would say don't do is try and automate to a metric or a number because no matter how noble your intentions are, that kind of approach is always subjective to or subject to being gamed. Right. You need to make things look a little bit better. And so it's easier to bump up the number of something so that the percentages look better or. And you don't even necessarily realize you're doing it. It's just the nature of how things are built and done because you're not moving towards an overall goal of how do we improve the overall quality. This is one of my Biggest challenges just with QA in general is how do you measure quality? How do you actually give something so that you can say we are improving? One thing that I've done is in the past I've taken the approach where forget I don't care about bugs during the development process. I don't care how many bugs you come up with. In fact, we went so far as to call the development type bugs things that come up during the development process, all the way up and through staging. If there was something that came up through that point in time, I just called them an issue that the developer and your development team and QA team fix together. Call it a day. What I cared about, the only number that really mattered is how many things escaped into production, right? Once a bug, that's a bug, if it's out in the wild, that's a bug. And that allowed us to keep track of was our ability to minimize the number of things getting into production improving. I don't care whether or not the volume is going up or down before production. What I cared about was once we got to production, because you're never going to catch every bug, right? Anybody that says we're going to have a perfect release and you should have caught that bug because you were responsible, it's a pipe dream. You're never going to do that. But are we catching more or less getting out? That's the important thing. How severe were those issues that we're getting out? What was the impact on the business? And those were the kind of metrics that I cared about because those told me the things that we're doing before we get to production, those are working right? If I'm finding a bunch of stuff before, but yet I'm still getting a bunch of P1 level bugs reaching production that really, I'm not really improving all that much before I get there. But If I find 100 different issues in, you know, the development process and I get one that escapes versus 10 last time, that's a 90% improvement. I'll take that.

Absolutely love that. So, you know, besides, you know, being able to measure and know how you're doing better at, I find that sometimes actually managing the whole testing process can be kind of daunting, especially with a lot of open source tools. It's almost like back in the day when I had a vendor tool, I had like an end to end test management that was able to keep track of all the requirements. It was all seamless. So any thoughts on test management nowadays in Agile environment?

I wish this was A better developed space, I guess is the best way. Testrail did a good job on it and made some good strides. The new guys kind of stagnated since they took over. I think the Gurak guys went and did test mo and it built a pretty good platform there for test management.

But to your point about multiple open source tools and all these different things, one of the things that I found to be a successful way of looking at it is to treat your different tools that you use as their own development projects in and of themselves and give responsibility to the teams to build and own that platform. Because at the end of the day it is effect. Whether you're doing manual or automation, it's still a creation type project, right? It's still a development project. A lot of automation is very code heavy, a lot of manual is not. But the basic principles are the same. Treat it as its own project and its own thing and so you manage and track those things and have owners for each of the things the same as you would a development project. Because we tend to just bring in a bunch of tools and say here, use this. And because of the high turnover, there's no ownership of them. Right? Nobody really is making sure that the test cases are up to date. Right? Where are all the test results stored? How do we run the right set of tests at the right time? What is automated, what's not automated? What tools are we using to do performance testing versus security testing versus functional progression, smoke, whatever? Those things should be defined and have owners of them. So when you do have changes, you can manage it.

Love it. As I mentioned it, John did a 90 minute session at this Jazz Automation Guild. A lot of people don't know you could still get access to all the recordings after the fact. Instant access. So if this sounds like something you want to take a deeper dive on, definitely head over to automationguild.com, get a ticket and you can actually get John's video. Very interactive and had a lot of questions from the folks there and you get access to our 247 community and John's in there, so you could probably ask some questions directly in there as well. Okay, John, before we go, any potting words of wisdom or one piece of actual advice you can give to someone to help them with their automation testing efforts and the best way to find and contact you?

I would say best piece of advice is don't just automate for the sake of automating. Go into it with a plan. Take the time up front to really think about what you're trying to accomplish. A little bit of time and energy spent trying to establish a plan is way more effective than just blindly jumping in. You'll save yourself a lot of heartache. But beyond that, you can find me on the Heartbeat community. You can find me on whatever they're calling what used to be called Twitter these days. Jumpmancol. Find me on LinkedIn under John Robinson next it. And yeah, love to chat about this stuff, so if you are interested, just holler and but we'll talk for days on end. I promise.

Thanks again for your automation awesomeness. The links to everything of value we covered in this episode. Head on over to testguild.com a494 and if the show has helped you in any way, why not rate it and review it in itunes? Reviews really help in the rankings of the show and I read each and every one of them. So that's it for this episode of the Test Guild Automation Podcast. I'm Joe. My mission is to help you succeed with creating end to end full stack automation awesomeness. As always, test everything and keep the good. Cheers. Hey, thanks again for listening. If you're not already part of our awesome community of 27,000 of the smartest testers, DevOps and automation professionals in the world, we'd love to have you join the fam@test guilds.com and if you're in the DevOps automation software testing space or you're a test tool provider and want to offer real world value that can improve the skills or solve a problem for the guild community, I'd love to hear from you. Head ON over to testguild.info and let's make it happen.