AI Testing Strategy: Stop Being a Cost Center, Start Protecting Revenue

Hey, in my experience, most QA teams waited for proof that AI actually worked before they touched it. But my guest today did the opposite. She started rolling it out to 150 engineers while everyone else was still arguing about it or whether it was real. And her reasoning was almost boring. The worst case is you keep doing exactly what you were already doing. Hey, I'm JoeBot, and welcome to another episode of the TestGuild Automation Podcast. Today I'm joined by Nadine, a testing and leadership expert with over 25 years of experience in the industry. She leads a global quality organization spread across 4 countries, but the thing that really got my attention was not the size of her team. It was that she got ahead of the AI curve while most companies I talked to fell behind it. And in this episode, she breaks down how she was able to tell the real AI tools from the vendors from others that were just relabeling ordinary automation, why she refused to hand risk analysis to an agent, and exactly what to say when your CTO asks why your team cannot ship like Netflix. If you've ever presented test coverage numbers to your executives and watched them stop listening, this one's for you. You don't want to miss it. Check it out. Hey Nadine, welcome to the Guild.

Hello. Hi Joe, thank you for having me.

Really excited about this. I'm always looking for real-world experts and you have a lot of it, so I, I want to dive in. The first thing that really caught my attention with you is you seem to really have gotten ahead of AI acceleration in your company where I think many companies I speak with kind of fell behind. So how did that happen? Like, tell me the story, like how you got into AI and how did you adopt it where a lot of people were just ignoring it at first, maybe?

So Joe, philosophically speaking, right? Like, of course, like I've been here for 25 years in this industry. I always believe that as a QA leader, as a professional, we have an extra extra onus on us to grab that seat at the table and make sure you have that, right? So, be it with every tech change that has come in with us, right? Like when the cloud architecture came in, mobile came in. So, AI to me was no different, right? And I've always believed that my teams posture-wise should be ahead of the curve. Not behind. And whenever I join a company and I speak with my executives and leaders, I ensure that my stakeholders are aligned, that shift left is just not some fancy term we use, but hey, I'm going to be actually implementing it. So for me, it was very, very important that I keep my technologies and my teams, you know, always abreast. You'll be surprised, but I am always looking at industry saying the latest tech stack. And when I, you know how we get all of these inbox requests from a particular vendor who wants to speak to you? I never ignore them. I actually listen, hear them out for two reasons. I wanna understand what is the trend that's going on right now? And then I wanna see how can my team adapt How can we leverage and how can we stay ahead of the curve? So really and truly, the motivation was to always stay ahead of the curve, right? And never be scared of a technology. In fact, embrace it because it's only gonna help you leverage that efficiency and bring you ahead. And I'd never view my team as just test folks who are writing tests, developing tests. I actually say this in every interview, everyone I speak to is we are product quality experts. And when I say we, I'm talking about the larger ethos of the quality industry. We're product quality experts. Nobody knows the product as much as we do. So, you know, that's important. And then in order to keep that posture, it's very, very important to keep sure, make sure that you embrace the latest technology and trends. So, when AI came along, it was just, it was never chasing a trend. It was actually staying ahead of that curve. And it was the same shift-left approach that I took with AIM adoption as well.

How did you know though, it was a real trend though, and that you weren't just chasing something?

Well, that's—

Like, how do you know?

That's exactly why all the POCs were very important. So, I actually took a very phased approach, right, in adopting AI. So, first off, I never made sure that there was only— like, I didn't silo it with only a few engineers and say— I kind of said, like, you know, I'm gonna spread it and I'm gonna have, like, a— we called it a QA AI acceleration charter. So, you know, making sure that we had a train-the-trainer model. But I gave them all a small task. If you have an area you're working on, go take a look and see what is the industry trend that's going in. Ask for a proof of concept. A lot of folks were talking AI, but it was actually just a powerful automation. And that automation we already were doing. So for us at that layer, when I knew that it wasn't like real AI, it was actually just a bunch of engineers automating it, I know how to call that distinction. But at the same time, when newer LLMs started coming and they started maturing, I did not hesitate from adopting them. What's the worst thing that's gonna happen? You're not gonna be able to do what you're already doing, but you'll just continue in the same trend. That's fine. It's not going to be a deal breaker. So we took a very shift-tiered approach. Our first phase was just to embrace whatever technology you are seeing, the latest LLMs and agents that you're seeing and saying, okay, let's just get some low-hanging fruit done. Let's understand, like, get well-versed with the tools and see it. And then maybe, hey, if we have a comfort level with it, let's move on to the next phase. But at every point, we were making sure that there were demos going in. So there could be— at every other Friday, I would have the team do a demo. The reason behind that was the rest of the team could drill holes or say, "Hey, this could work on the aspect of product that I'm working," or, "This can be used better this way." So that way, it was a very transparent work that was getting done.

How do you empower your teams to embrace this? Because I've come from cultures that are completely opposite where they don't really— they tell you what to do. It seems like you really allow your teams to learn and to embrace and to do changes and to try things, but sometimes testers are afraid, they just want to keep their heads down. How do you change maybe that mentality to actually not be afraid to embrace it and try things?

There are two reasons, right? You always have to encourage innovation within your team. That is another reason that, or keeping that helps the quality team keep that seat at the table. Because if you are not just being a checklist curator or completing that validation, but you're actually bringing code to the table, you're actually innovating and adding value into the tech stack of the company, that keeps your— I would say it's not just the product knowledge depth, but it also talks about your tech depth. Right? So I have encouraged that even pre-AI days. That's why I would always veer away from test automation tools and try to encourage teams to write your own ingrown framework, right? You know, and a lot of it is my own upbringing in the industry. My first manager would make me write my own network monitoring tools. We didn't have money for SiteScope, so we were like, hey, guess what? You are gonna write your own ISM tool, or you're gonna write your own ping monitor, right? Starting from that. So, and there is a level of freedom to think and a level of innovation that you start thinking in terms of, okay, what are the other ways in which I can make my life easier? So, that is the culture that you have to encourage your team. But that doesn't just arrive at, When it starts at the hiring level, right? You hire the individuals who have that thirst for knowledge and who want— it's not about knowing a particular language or knowing a particular scripting. It's never about that. It's about how you can try different approaches and never give up, but constantly have that thirst to learn more. So my team, which I have grown, is each and every one of them thinks that way. When we did the pivot to automation, our pivot was no different than adoption to AI. We were like, okay, go into Test Automation Open University, which is all open source, and go pick your own learning tracks. Go learn what you want to, whichever it is. And I always like to do a little gamification of it in terms of, okay, if you learned a brand new tech stack, come and give me the demo. If you do that, whatever money you spent towards it, not only will I reimburse it, I may give you a spot bonus as well. And we have our innovation corner catalog. So every engineer gets their name across the tool they wrote or a new framework they built or an efficiency. Even it could be a simple curl command generator for all I care., but you will have your own named artifact alongside. So it kind of makes every other engineer think, I want my name out there as well, right? So it's that culture that you have to encourage and say, this is how, and a lot of folks typecast quality as, hey, you just write tests, you're just gonna be validating it, and you come in at the end of the chain, no matter even if, with Agile or no Agile or their own Agile, whatever it is, right? But I have never believed in that. I always tell them, don't stay silent, swim upstream, ask the right questions. Not just like, hey, what is changing? How are you going to implement it? Which repos are changing? What is the architecture behind it? Don't you think this would cause performance degradation? But in order for you to ask those questions, you have to have the depth. In order for you to have the depth, you have to have that know-how and understanding. So it's kind of like a vicious cycle, so to speak.

Yeah, totally. How do you keep your management informed on the value you're actually adding? Because obviously, you know, your team knows, but how do you— because you sound like an awesome director, uh, manager to have— how do you let your higher-ups then know what value you're adding? Rather, because a lot of times they see QA as like a as a cost center, not necessarily revenue generating.

Right. You know, see, the whole part is you— if you walk into a room full of execs and tell them, hey, my tests are 78% covered, that's not what they're listening, right? What they want to know is what is the— how fast can you ship? What they want to know is what is the time to market? What is the feature velocity? What is the burn rate? You don't go and say, "I have blah downtime." Go and say, "This is my resource burn rate." Talk to them about revenue protection. Talk to them about risk mitigation, right? Take the tech numbers that you're receiving, But at the same time, speak it in a language that your executives can understand. And by the way, there are a lot of executives who are very tech savvy. They completely get it, right? You can talk about my root cause analysis time has come down 10x, right? So you can say, hey, what was taking me 30 to 40 minutes now takes 3 to 4 minutes. You're going to immediately have folks give you the attention. Immediately folks are going to say, hmm, so how did you get to do that?, right? And, or you could, you can actually say, hey, I have taken in these workflows, and while this is all automated, so that means my time between two releases is now down to X minutes. It used to be blah hours before. And actually, Joe, I was in listening to one of your webinars, I think, and I hope it's with Leapwork, where they spoke about Netflix releasing upwards of 400 times a day. And see, that's the age we live in, right? While Netflix, I'm quoting, is just an example over here, but every ecosystem, every company is in that chase. And you want to deploy faster, you want to deploy multiple times a day. So while the validation team is absolutely paramount and important. You should not come across like how you mentioned as an overhead. You need to come across as the risk scaffolder. You need to come across as the one that pivots. So I always present my metrics in 4 pivots, right? It's quality, scalability, performance, and availability. So I always pivoted around these. Because you wanna make sure, and I explain it to them, that hey, if your functionality does what it's supposed to do, then that's just single-threaded quality. But if it continues to do it over and over again under load, that's reliable. If it is always available no matter what time of day, that's availability, right? If you can move from $100 million ARR to $500 million ARR, that's scaling, right? And your design is ready for scale. So that's the— you— the onus is just not on my team. The onus is on me as well to understand the business model, to understand the ask, and then veer my team towards it.

Love it. So just a random thought came to my mind when you said, no, we reminded me about Netflix, 400 times they release a day. How do you not compare yourself against another company though? Like, how do you have like a manager come to you and say, look, Netflix can deliver 400 releases a day, how come we can't? Like, do you have a, is that like dangerous, you think? Yeah, yeah.

All the time, right? And by the way, Joe, I am an ambitious, competitive person. So I do, in all honesty, I do compare. I always tell my team, That is why, remember I've told you, I never say no to speaking to a vendor or a third party, 'cause I'm like, hmm, what are you guys doing that I'm not doing, right? And yes, there are a lot of folks who come to me and my execs come to me, and I'm not just here, this in past companies, hey, so-and-so is releasing multiple times a day, or so-and-so is doing that, why can't we do it? But you have to explain and understand, This is their architecture. This is our architecture. This is the device. But having said that, you want to release multiple times a day? Sure, we can do it. But are we putting the cart before the horse? Do we have our system designed and ready? Is it something that we can do? While we all heard about Netflix releasing 400 times a day, Liebherr also mentioned that they have thousands and thousands of tests embedded beautifully into their CI so they can actually just plug and play whatever PR comes in and do that deploy because the associated test runs with it. And that is the future, is, you know, and that too with AI, take all this laborious task of automating and making sure that these tests are self-healing, With every UI update, your automation is not breaking, how we're actually healing. Give that task off to AI, make that run with it, okay? But at the same time, and show it to your management and executives that this is a force multiplier. Yes, you can do it. You wanna release 1,200 times a day, we're ready for it, right? But take in your platform partners, make sure that the CI tests are embedded, Make sure it's lightweight. Do your homework before that.

Superwoman, of course you're able to get this done. A lot of times we learn more from our failures though. So do you have any failure stories that maybe during AI, your AI, uh, journey or your transformation that maybe people can say, okay, all right, I'm not alone here, uh, type deal?

I have tons of failure stories. I'll tell you why. Because I don't view failure as to me, success and failure is very iterative, right? We have an industry that's changing by, I won't even say by the week, now it's changing by the day, right? So you have to quickly adapt. More than calling it a failure, I would tell my peers to think of it as the gap between vision and execution. Okay, so you may visualize that I want to do X, Y, and Z, but when the rubber meets the road, you're going to run into infrastructure issues, you're going to run into cost issues, you're going to run into implementation adoption issues, right? Even within our own QA team, our engineers are very, we're all creatures of habit. So you're very used to doing your— you're using your own set of tools or your own set of implementation mechanisms that you're used to. When you throw in something new, you're going to think twice before using it because you're like, hey, I'm already on this racetrack and you're asking me to change the wheels as I'm moving fast. I don't know if I can do that with the same efficiency. So that's kind of like the holdback that typically folks have. But I don't think that that's— it's not like a huge setback. It's more like you have to work and get their muscle memory to be rewired in a way that they adapt, you know, continuously as the change happens. And I also would like to say that it When you're trying to bring a change, don't expect it to happen overnight. It is perfectly OK that it is going to take time. But that is why, like anything else, break it down into smaller phases. Set your expectations also with your stakeholders, with your leaders. Bite-size your expectation and bite-size your work. Sometimes you may be able to get like, one or two phases in, but then it's going to look like a victory. But at the same time, if you get a setback, it's easier for you to get over what was that setback, what should we do it. If it's infrastructure cost, if it is actually personal issues, all of that, and come with a mitigation plan for it. So, I mean, nobody has a recipe for success because we keep changing things. You are going to have like a setback, but I don't view it as, oh my God, this has failed, so scratch it out. I view it as a learning curve.

Love it. So I want to dive into a few things there. It doesn't happen overnight. So you have incremental improvements. Earlier you mentioned 4 areas. I think it was quality, scalability, performance, and availability.

Yes.

So when you have a user story, do you then try to say, how can we apply for this user story, these 4 key areas, and how can we improve during this sprint so every sprint we're getting better and better? Like, how does that work?

You know, when you have a user story, um, you have to give it a full due diligence, right? And I'm not— it doesn't really matter whether you are having an agent look into it or your real self look into it, but you have to understand Which repos are these user stories going to be touching? How many configurations go behind it? Are these going to be between, behind feature flags or not? Now, and then what is the depth of invasion? Those are the reasons why an engineering implementation or a tech diagram is equally important to review and ask those questions. Because if it is, for example, let's say, say it's a database change and you are gonna have, you have to find out, is it gonna cause a performance degradation? Hey, is the scale absolutely required right now? Is it a full rewrite? And you have to keep your performance scripts and tests and synthesizing of that kind of data, whatever obfuscation from, prod data to staging data, whatever is required, you have to get that going and place that into the time it's going to require to analyze and run these through, right? But you have to think in those terms. Failure to doing that is only going to give you a very small increment, like, okay, yeah, it looks good. But once you have a large customer come in and do a massive download of data or pull a 3-month, 6-month report, That's when your system should not come grinding to its knees. It's very important to ask for and institute these practices. That is why it's important to partake in architecture reviews, design reviews, and ask for performance modeling, not just like, "Hey, I'm bringing in this change. Have you run a model against it? How do you think?" What are you trying to bring it from a P95 to a P99, and what is your 99th percentile? So these are the questions that you have to ask.

How do you not get lulled into just trusting the AI, saying, "I don't need to ask it, AI did it"? 'Cause it seems like we can almost be almost too AI positive. Like you just mentioned, if an agent does it or you do it, I don't even know how many people have an agent to do testing, so. How do you know when to question it or when not? You know, when I first started browser automation, I'd get hypnotized by the browser, you know, just watching it. AI is like, you know, 100 times that. So how do you avoid that?

Yes. But, you know, it's in our DNA, Joe, trust yet verify, right? I say that even to my children, right? Hey, I'm sorry, I'm a quality person. I'm not gonna, I'm not gonna take your word for it, but when I see it, I'll believe it, right? It's actually the same at the ethos of it. It's the same concept. You let the AI generate, but you as a human being, you are the judge, right? You have to start with the economics. You take AI, use it for the ones that AI has made it cheap, right? That automation aspect of it. You go ahead and you can let your AI agent do it. But the judgment, you really have to make sure that if it is a PR risk review or if it is a change impact analysis, then you are the one who are going to judge as to what is at most risk. And then you have the one who has to strategize that. AI is not going to be able to do that because AI does not have the product depth as a quality engineer will do, right? And not just go by dev notes because the AI-generated dev code cannot be the same one that does the tests, right? That is like doing your homework and grading it too. So you have to make sure that you rack what's most at risk,, and then you orchestrate a whole workflow based on this risk scaffolding that you did as a judgment tier, right? And this, I would say that the QA engineer holds that judgment because they are the ones who understand the customer's intent and not just what, and what they expect to, or what the customer wants to have. And not just what the code was written for, right? And so that gap is only possible by a QA architect or a QA engineer. So I would say that, do I trust the AI for the mundane stuff? Sure. Like how you said browser automation, no problem, go ahead. Browserless automation, just go ahead, kickstart CLI, run it, no problem. But at the end of the day, even back in those times, we were still doing an end-to-end validation to make sure that, you know, a user acceptance phase is ready and ready to roll out. It's the same concept.

Gotcha. So, I don't know where I got the stat, maybe I'm wrong, Um, I, I think you're leading a global organization over 150 people across US, Canada, I think you said India, Pakistan. Uh, a lot of people listening to this, like, I have a team of 8 and I can't put it together. How are you doing that over that many people, that many time zones? How are you able to not only implement your vision for the AI transformation, but make sure that the whole team's on board and doing it correctly? Keeping everything in a row? How do you manage that complexity?

And it's not just right here in my current organization. I have worked with teams in Ukraine, Vietnam. Cross-geography is not something that is new, and that has now become the norm, right? For, again, while revenue drives it, cost drives it, manufacturing drives it, for a whole bunch of reasons, every large-scale enterprise is split across geographies. But you cannot fundamentally forget that at the ethos of everything, we are all humans, right? And it is humans getting work done out of humans for usage by a human, right? Your end customer. Fundamentally, how I hire is the same whether it is my team in the US or whether it's my team in India or in Ukraine. It doesn't matter. You are supposed to have the same tech jobs. Everybody goes through the same interviewing process. And speaking of vision, you do have local leaders, team leaders in every site. And these, you know, it's a trickle— while it's a trickle-down, I invest a lot in my teams. I try to bring in, I try to always tell them, you are the last stop of the product before the customer gets their hands on it. And that kind of pride you have to instill in your team. At the same time, you have to say, you guys are product quality experts. You are the ones who understand the product the best. So, These large visions of, or rather philosophical principles have to be split and sent down your directs and then their directs. Everybody has to be aligned on it. And when you give them the technical innovation freedom, everyone starts viewing that lens very differently because let's face it, All of us love grabbing the latest tool, playing around with it, trying to bring in the latest code, making sure your name goes up on that catalog which says, "So-and-so developed this particular tool, go ahead and use it," right? And then we also measure our metrics in terms of who is adopting, how have we brought this efficiency down, How have we been able to release better? What are the largest problems that we have solved? And even smaller things where they bring in the value is, and with AI, we're solving for real-world problems that our engineers are facing every day. Let's assume your root cause analysis is taking you a long period of time. You go ahead, you write an agent that takes a look at a customer issue, generates a root cause analysis, takes a look at previous logs, and is actually spitting a report in minutes rather than taking forever. And keep those office hours. And Joe, above all, I keep these skip-level one-on-ones all the time, and they're not formal. I could just ping someone on Slack and say, "Hey, how are you doing? What's going on?" How can I help you work better? And believe it or not, I still keep in touch with those that I led back in 2008, 2007, as much as I keep in touch with them today, those I'm leading today.

Love it, love it. So as you know, a lot of testers I speak with are really antsy. They're really scared now. We're at the halfway point to the new year. You know, you have a lot of people on your team. How do you see organizations hiring going forward? Like, what skills are they looking for? Do you think we still will have a QA tester role that need to be filled, or is it going to be a different role? And what kind of skills do you think people listening need as we move forward?

You need to— I would say you need to be The skills should still be the same, but the passion to lean into technology is a must-have. It's earlier on, maybe, okay, you could be a manual tester. I would say that's still there in certain setups. Let's say you have a lot of hardware built in and that kind of cabling is required and every cable decides a different type of a feature. That all there, I'm not discounting that, but that aside, you still need to have that thirst for knowledge. That's going to be your primer skill. The second is be aware of all the technology and do not be shy to use it. Uh, organizations are going to need human beings regardless of how fast AI agents come together, right? Again, for the same reasons that I'm talking about is that kind of product quality depth. I mean, AI-forward leaders who understand the product, who understand the architecture, who understand the customer usage are the ones who are gonna have their seat at the table, not just those who are spending their time in bringing in and honing in and streamlining the automation because that is the cheaper part that is already done. Use your AI agents for it. And I would also say that for every leader listening, don't be shy to use your AI agents as a resource multiplier, right? Think of the person as having a virtualized self who is able to help a coworker. I used to be a robotics coach, and I used to call these as cobots, collaborative robots, right? So I would say, think of, find a cobot that can help flip a burger, right? So the same way, you are going to still need human employees who are going to be that judgment layer, who are going to be the ones that understand and who are going to be the ones that triage your Sev 0 or Sev 1 at 2:00 AM when the production or when a large customer issue happens because you understand your product, you understand the customer usage patterns.

Love it. And you may have already answered this, Nadine, but before we go, is there one piece of actual advice you'd give to someone, especially a QA leader just starting the journey? And what's the best place to find or contact you?

Okay, so two parts, right? So, the best piece of advice that I can give is don't view quality in silo. View quality as understanding as to you are the voice of the customer. Keep that in mind first, and you want to, Your voice of the customer will help improve customer adoption, and you are the customer success in the engineering org. That's first. Second, view quality as a person who understands the product. So that's going to come in here. But at the same time, second, focused on how was it developed, understand the tech stack, understand the container, understand the architectural depth,. That is going to help you with scale and performance. Third, understand what does it take to be a highly available product, because nobody wants downtime coming in over and over again. Um, how can you reach me? You can reach me on LinkedIn. Um, and, uh, I have decided to write a 5-part series on the voice of QA in the AI era. I've started the first part. I'm going to be publishing my second part shortly. So I would love for folks to chime in, interact with me, reach out to me. And like I said, even vendors who reach out to me, I always tell them, "Hey, thank you for reaching out, but we already automate this layer. We already do this. Is there anything else that you can offer?" I have no problem saying that because, hey, everybody puts in the time and effort, so you should respect that.

Love it. And you can find links to all this awesomeness down below. Thanks again for your Automation Awesomeness. The links to everything of value we covered in this episode, head on over to testguild.com/a597. And if the show has helped you in any way, why not rate it and review it in iTunes? Reviews really help in the rankings of the show and I read each and every one of them. So that's it for this episode of the TestGuild Automation Podcast. I'm Joe and my mission is to help you succeed with creating end-to-end full-stack automation awesomeness. As always, test everything and keep the good. Cheers. Hey, thank you for tuning in. It's incredible to connect with close to 400,000 followers across all our platforms and over 40,000 email subscribers who are at the forefront of automation, testing, and DevOps. If you haven't yet, join our vibrant community at testguild.com where you become part of our elite circle driving innovation in software testing and automation. And if you're a tool provider or have a service looking to empower our guild with solutions that elevate skills and tackle real-world challenges, we're excited to collaborate. Visit testguild.info to explore how we can create transformative experiences Together, let's push the boundaries of what we can achieve. Oh, the TestGuild Automation Testing Podcast. With lutes and lyres, the bards began their song. A tune of knowledge, a melody of code. Through the air it spread like wildfire, through the land, guiding testers, showing Secrets to behold.