Conversation between Andy and Kevin Surace about AI's impact on productivity, headcount, and cost savings. Kevin argues AI will automate manual QA tasks (using tools like GENIE), greatly reduce need for manual testers and QA scripting, and elevate remaining QA roles toward ensuring coverage, data quality, and visibility. They compare similar effects in marketing and customer support.
I remember a few years ago being with a CIO in London. Big, big cio, probably one of the top five in London across Europe. And I was testing a presentation I'd built from on a product and I got halfway through and he said, look, just, just let me stop you here. All day long, people come in and out of my office and they're pitching me productivity advantages and I'm just not interested. Because if I added up the grand sum total of all the presentations in any given day, I'd actually be a revenue generator for the business. So I'm only interested in one thing, really. It has something, it has to actually stop something in my business. So in other words, I was doing a. I no longer do A at all. And I can go. All those people that were doing A. We don't require them any further, sadly. Or that's a good thing because we can redeploy them to a more important task. And I think that's the courage that you're kind of expressing here, which is, no, the real sum game here is you are not going, you're going to have a fraction of the number of software quality assurance people you have today. What becomes then of software qa? What is the new world of software QA going to look like? Because this actually, as you pointed out, this is an important function in any business. Do they become managers, arbiters, overseers, overlords of AI. What's the role for the QA professional?
Well, we can already look at this in some other industries. So what? Marketing teams that have taken on AI across the board actually didn't lose any people, but they become the robot overlords of the AI. There's no question. In fact, if you're in marketing today and you're not leveraging AI to help write your content and create your content, you shouldn't be in marketing. That's just what's happened in two years. Right? But then we can also look at customer support and call centers. Call centers start to be augmented by AI, and eventually a lot of those heads are getting replaced by AI, at least at tier one. So then you have to look at software QA and say, look, there are different tiers of capability here. There are manual testers. AI is going to do that job. Let's just be clear about that, right? AI can take those test cases with GENIE and automate them. And anytime I've automated a manual test case, I didn't need the manual tester. It's just, that's worked that way for 30 years. That isn't a new phenomena. Now I've got AI that does it 100 times faster, probably. I can get rid of almost all manual testers. That's interesting. Then you say, well, what about QA automation engineers? Well, here's AI that automates without necessarily the need for a whole team of QA automation engineers. No one's going to be writing Selenium when we look five years from now. I think that's a ridiculous thing to say. Well, people will still be writing. No, they're not going to be. It doesn't make any sense. And they probably don't need to use a recorder because this thing does the job of a recorder. Just does it with no humans around. Okay, smart. So QA automation engineers are going to be lifted up. I think first of all, there's a shortage of high quality QA QA engineers, right?
Absolutely, yeah.
And so then you say, what do you want to do as a QA engineer? I want to get back to real quality assurance. Over 30 years, we took our QA engineers and made them scripters. And most of the time, a lot of people are spent scripting and then maintaining those scripts. Well, now I can script and maintain the scripts. Your job is to say, how do we get the coverage that we want? How do we produce more data input? How do we get the right data outcomes? Right. How do we assure that. That we know all of the quality issues before a user tells us? That is a whole different level. And that's actually what QA is supposed to be about. We turned it into a monotonous task of writing scripts and maintaining scripts, and it should have never been that. And that's because the tools aren't that good. AI relieves us of that and really changes what we do in the industry. So if I'm a QA QA engineer today, I really want to understand how we drive the highest quality and most visibility. Something we don't talk about in QA a lot, the most visibility to our managers or CIOs. Right. In it, we talk about visibility. We want visibility into issues in the network. Right. So we know about what that is, but we just don't talk about in qa because we're so far behind in just executing our test cases, we've forgotten what it is to try to achieve 100% code coverage, 100% application user flow coverage.
We're just 100%. I mean, talk about that for a second, because I think that is so misunderstood. The idea that we actually test the whole app or the whole Environment. And actually we only test a fraction of it in real terms. Right?
That's exactly right. So we have actually a number of terms in the industry. One is test coverage. What is test coverage? Test coverage is a totally made up thing to make people feel good. And here's what I mean by that. There's a business analyst that has created a set of test cases, and then there are QA people that either manually or with selenium or with a recorder, try to implement all of the test cases. And if they can hit something close to 100% test coverage, what they say is everything the business analyst told us to do, we did. Now here's the flaw in that. That has nothing to do with what your users are doing. Nothing. It just says here are the most important user flows and outcomes, as we think are the most important. And we need to test each one, one time. And if we do that, we know that our overall application coverage or user flow coverage is probably terrible. But I did my job. And because we've had limited budgets in QA and limited headcount and limited time, all we can try to do is just get the regression testing to the test coverage number of 100%. Now, what we have found over the last seven years is that if you achieve 100% test coverage, on average, you probably are only covering about 10% of what users actually do. And when I tell people that, they may acknowledge it, but then they'll say, yeah, but I'm not measured to that. I am measured against test coverage and I'm just trying to do that. So I hope that the whole industry changes the way we measure success to did we get the visibility of everything a user can do now that we have AI working for us? Right. Did we get the visibility of everything a user can do? So I've got the visibility of what the business analysts that are the most important cases, and I've expanded that more than 10 times to get very high application user flow code coverage. Right? That's what I want. I want those numbers. Nobody's ever had the time for that. But we have customers using AI script generation, even before genie, right, that have achieved numbers like that. And they will rave about it on how it changed the way they look at quality assurance. So I want all these people to go and be risen up, right, to a whole different level of quality rather than I'm behind on my scripts, that is the wrong measure.
