Test Automation Won't Save Your QA Career, but These Skills Will

Hey, what if the AI coming from your automation job doesn't even have to be good? What if it only has to be slightly worse than what you're building right now and cheap enough that your company buys it anyway? Here's an uncomfortable question. When was the last time your C-level executive even made a single decision based on your test cases' pass/fail ratio? Ever? My guest today has had thousands of tests work for him, and he says If your organization doesn't value testing, you're doing it wrong, and that's on you, not them. And that's a hard truth. But hey, it's Joe, and welcome to another episode of the Test Automation Podcast. Today I'm joined by Keith Klein, Director of Quality Engineering, a guy who has been leading testing for global banks and has run teams of thousands for many, many years. He's been on my wishlist for a while to get on this show to interview. Keith has actually made waves recently with the talk called Test automation is dead and wants to know exactly what he meant. Because if he's right, a lot of us have been building our identity on the wrong thing. And in this episode, Keith explains why the mechanical part of testing is about to be cost near zero, why agentic test platforms are mostly boys with toys until somebody shows him a high-risk case study, why he thinks confidence engineering is a trap that leads you straight into confirmation bias, and the one question he asked testers at Barclays that instantly reframes what your job actually is. And he also gets into the Agile Manifesto as the biggest unintended consequences story in software history, why he never tells a client whether to release, and the reading list he hands people who wants to think clearly about risk. And you want to stick around all the way to the end because Keith actually drops a line about overconfident experts I've not been able to stop thinking about since. You don't miss it. Check it out. Hey Keith, welcome to The Guild.

Thank you very much. Happy to be here.

So you've been on my bucket list for a while. I'm almost up to episode 600. I can't believe I waited 14 years until I got you on the show. So thank you for joining.

Hopefully it'll be worth the wait.

So Keith, I've been bingeing on your content on your website and on YouTube. Um, like I said, it's really unstructured. The first thing that really caught my attention was you did something with BrowserStack on— it's like the death of test engineering. It was really— I guess it was kind of like a, a, a bonbon type of, uh, of, uh, of talk. So I thought maybe we'd stop there because I think it's top of mind for everyone with AI. So yep, what were you trying to convey in that? I'll have a link for that down below for why you think the test engineering in that, in that, that particular session you had is dying.

Yeah, I, the primary, you know, idea I was trying to convey was that if you are, if your identity is tied up in creating automated checks or producing artifacts, that world is changing dramatically around you. And we've had for a long time in this business, people and vendors and companies that try to equate tests with testing. And AI right now, I was saying the other day, my friend, it doesn't even need to be better than what a lot of the automation I've seen right now. It just needs to be slightly worse and people will be biting their hands off to buy that. Because that kind of shallow automated checking is the most mechanical part of testing, right? And that cost is going to near zero. And just like everything else that happens in software delivery, testing is usually the first line of attack for trying to commoditize it. Get rid of people and equate it to some false analogy to manufacturing in that we're just artifact producers. So, if that's your whole approach, and frankly, the last, I'd say, 20 years of everybody needs to code, everybody needs to automate, the kind of cynical methodology that's been pushed through a lot of the agile trends. Has really driven a wave of people in this business that don't know how to test. And that you see it. I mean, I talk to these vendors all the time and they are actively working on that problem on how to— they haven't worked it out yet, but they're getting there. They don't even need to be better. They just need to be a little bit worse. And they're getting close to be able to wipe out a lot of that work.

No, I 100% agree. I must be naive. I always, I, I, I always love automation. I talk about automation, but I always come in, I, I don't—

I feel a little bit in the dragon's den here.

I always assume like, you know, people know testing, and for the things that can be automated, I'm that little sliver of that thing that I can help them with, right? Not that you can automate everything, but what I'm disturbed by is that that's not true because I've been doing in-person events around the country and what I'm shocked by is, and like you said with AI, that no one really knows what testing is anymore. They don't— like when I started, you, we were testers. You happen to learn maybe some tools to help you with testing, but the automation wasn't the testing. I don't know if that makes sense.

So yeah, it's a subtask.

Yes. Yeah, exactly. Um, so where did it get all mixed up and why? Like how do you, How do you change it now? Because to me, like you said, it's been 20 years now and there's people that have entered the field of testing that don't know how to test. So what do you do?

I mean, I think to, to answer, try and answer both those things in turn. So I don't have an exact date when it started to go, you know, kind of pear-shaped, but I think with the kind of explosion of software, maybe mid-'90s, When every— you could do things with software that you couldn't do before. And so, you know, there's this, there's a powerful, it's like the anthropomorphism of AI, right? The wish thinking that goes around automation that like, you know, testing is producing a widget and we just need to spray a system with all these widgets and run through. That's why I called it the To Infinity and Beyond, right? 'Cause a lot of the automation approaches I've seen with organizations is really trying to count to infinity and not looking at risk or, you know, where areas of the system we haven't tested. And a lot of these organizations that I consult with, some really, really big ones, their test approach actually injects risk into their business. You know what I mean? The way they do things actually makes it more risky for them to release software because they have this overreliance of what I call these short-term dopamine hits of green checks. And that gives you this false sense of confidence that things are okay. And so the speed has— the Agile Manifesto is probably the biggest case study of unintended consequences in the history of software development where people were trying to do things practically and be pragmatic and focus on people in the business. And it gave birth to this gigantic methodology-driven standup ceremonies, artifacts, you know, that just made it seem like, oh, in order to, you know, the kind of productivity paradox of, well, the more we do this stuff to try and speed up, the more it actually slows us down. They had to clear space out automation, I think, you know, all those things converging together really created this environment where, you know, you need to know how to code, everything should be automated, automate first, XP, which is really a design technique and not an automation technique. But like, there's— people are looking at the enterprise for ways to save money, and it gives the illusion of efficiency. And that's where it It's to me, it started to really kind of take a steep turn.

No, I, once again, I definitely agree. I know when testing interviews started to become more like developer interviews, uh, where they, where it's all about, like you said, coding. And I know you need to understand coding, but it had nothing to do with testing. Um, so yeah.

And to, to, to your point, Joe, sorry to interrupt, but like people, you know, you, and I know you've interviewed Michael and, and, and, uh, you know, I am not anti-tool or anti-automation. You know, I mean, it's supposed to extend your reach.

Yes.

You know, it's a tool. And that's why I say, like, you know, if you look at AI as a competitor, yeah, you're screwed. If you look at it as just another tool we need to learn about and use and figure out what works with it and doesn't, it's not a threat. Yep.

Yep. That's a good point because once again, I talk to people that don't know Michael Bolton or James Bach, and they're like, oh, they're anti-tool, no automation. I'm like, I just think online it's portrayed that way, but they're really not that. It's exactly like you said, where you're not against tools when it makes sense. Right. Is what it sounds like.

Yeah, it's— that's 100%. And I've known those guys for a long time. Right. And I've worked with them. I've done training classes with them. I count Michael as a personal friend. They are not anti-automation. You know, the problem is they're kind of anti-bullshit and a lot of what you hear from people who are selling a tool, right? It's in their interest to create— this is, you know, I was talking about this the other day as well, is that like that I wrote a piece called "The Great Capitulation," right? Where like they have to drive the value of testing down to make it into a check-driven exercise in order to sell their tool, right? Because if— and also a lot of people I've met in the automation business, no offense to your listeners, but you know what I mean? When you hear the talk, you're going to hear how I feel anyway. But like, they're not— they don't think deeply about the testing part. And part of it is the strong attractor of the difficulty of coding. And trying to figure out the technical exercise of creating meaningful automation is difficult and probably takes up a lot of the oxygen in trying to do that. But that's— it's not the point. And I think that they, Michael and James, don't have a lot of energy for those conversations. So they probably come off a bit more anti than they actually are. But nobody builds more testing tools than James Bach. I'll tell you that right now. That guy is a coder and develops tons of stuff. Has done for years.

Absolutely. But, you know, you all have been beating this drum for a while and it still feels like it's a— like, you work with— I mean, I know you probably deal with a lot of different teams, a lot of different companies that are huge, and like, do they even care about— does that mean like, like, how do you make them get past the BS and actually care? Like, hey, you're going to lose money. Like, is that the only way to make them?

Yeah, absolutely. You know, I don't You know, and they do care. And when you talk about it in the right way, right? And so, I think too many, and I've been banging on about this for, you know, one, I'm in the unique position that I've had thousands of testers work for me. So, I can control the narrative a bit. And I also get a lot of exposure to management and C-level folks. And, you know, very plain speaking about it and don't get into, I've always tried to focus my energy and language on business risk. And so when you talk about that, and then you've got some other testers are just talking about pass/fail ratios and test case counts and, you know, all these weird metrics that we produce, which are proxies for something else, and they can't explain what the proxy is, you know, I glaze over at those, you know what I mean? You can't translate it into something that's tangible for somebody who's responsible for the business cares about. So when you have those conversations, yeah, they actually work. And believe it or not, you know, we'll just go there, that whole testing and checking, you know, debate, man, that really changed my approach on how to describe what the value in test automation is, what it can and can't do. And it frames it in a way that you can understand the limitations so that when you see a sea of green checks, we want to understand Well, what didn't it cover? Where should we be uncertain? What are the risks of that going wrong? As opposed to just looking at it and saying, yeah, we ran, you know, 10,000 tests on this thing. They're all green. Well, who cares? You know, like, why does that matter? And I think when, yeah, when you actually frame the debate that way, they do listen.

So how do you explain what testing is? Is it simply testing is finding risk?

It's risk management. We're trying to find serious problems fast, right? I mean, that's, that's what we're trying to do. So I never tell a client you should release or you shouldn't release. I say like, you know, and people say, oh yeah, you have to make a decision at some point. Yeah, we have to. But my job as a tester is, and the person who's responsible for testing, is to look at that information as objectively as possible and say, here's what we know, here's what we don't know, you know, and here's where there might be some consequences. So what do you think we should do? You know, that you have to present the information so people can make decisions, you know. And so, yeah, testing really is a risk management exercise, you know, and part of that's finding out, you know, who will be impacted by it, what matters to those people. Like, there's lots of ways to do that, but there The key thing is to focus on risk because that's— I always talk to testers about this. Who has ever been in a management meeting or a meeting with a C-level person or sponsor and they made a single decision based off of their test case pass/fail ratio? Nobody ever has. You know what I mean?

Right, right.

So, if you're not making decisions off of this stuff, what are you gathering it for? You know, like, what's the point? What story is it trying to support and tell? So, yeah, so I think if you position it the right way and are consistent. And the other thing I would say is that, like, I tell testers that, and have for a while, if an organization doesn't value testing, you're doing it wrong. So I own my own value proposition in an organization. So testing uncovers all the important stuff that a business should need to know to help manage its technology, system, or business risk. If they're ignoring that, then you either, one, don't speak the language of those people so that they understand what the risks are, or two, you're not focused on the right stuff, meaning you're not getting, you're not getting risk information for them, right? So if they just look at you as a manufacturing, a widget manufacturing, you know, person, then yeah, that's going to be real easy to say, let's commoditize that. What's— we're going to get the cheapest rate. Where you wouldn't— you would never do that, uh, if it was— if you were thinking about it in terms of managing business risk.

Oh, that's interesting because a lot of testers will say it's a culture issue, but basically it's a— you own it. You're probably not giving them the right, uh, the right data that they need to make these decisions on for, for them.

Absolutely. Interesting.

Yeah.

And we get our house in order first, understand our business, understand like a lot of people feel like, and look, I'm not going to excuse every organization, right? There are some culturally bad places that just don't care, you know, but like there's ways to manage that. You know, there's lots of bad actors and not everybody can just quit their job and go somewhere else. I don't believe in that. Philosophy either. I mean, people get trapped inside of systems and we all got mortgages to pay and everything else. So, I don't think every— I don't encourage testers to die on quality hills. If you make a recommendation or you find something and the organization decides not to do something about it, job done. Your job was to raise that risk and move on. People get to— where I see people having a bad time with that is they too closely associate themselves personally to the work that they're doing. So if you don't like my test approach or the way I test, then you don't like me. And I, I've seen that result in a lot of, you know, hurt feelings.

So how do they get over that then? Like, uh, how do you not put so much faith in yourself then when I don't know if that makes sense.

Like, yeah, well, part of it is, you know, I get what you're saying. Part of it is, you know, don't look for that out of your role, right? And I was just talking to a friend the other day about this, honestly, was that like, you know, your job is to be objective. You're not highlighting risks or problems, right? It's not stopping anything from happening, right? People don't want to be looked at, oh, well, I just saw this the other day, you know, in fact, I might have been listening to one of your podcasts. It was somebody was saying, somebody was saying testing is the bottleneck again in AI, you know, and testing.

Oh, yes, yes. I remember that one. Yeah, yeah.

Yeah, testing is not the bottleneck. Like, if you look at risk management as a bottleneck, I'm not sure you should be in this business. You know what I mean? You'd never say it that way. You'd never, oh man, all this risk management, all this protecting our revenue, man, that is really a bottleneck right now. You know, so like, look, looking at it the right way and what your job is and understanding that that slight, you know, the objective view is your job, right? And that's the uncertainty, right? As I've said before, is, you know, a feature, not a bug in testing, right? We want to be finding places where we're uncertain. So yeah, it might not be for you, right? If you want everybody to like you, Or, and they can still like you as a tester, but if you want your job to be generating, you know, people feel, oh yeah, you know, you helped get the project up by not telling us something, or that's not the right way to look at it. You know, I think your job is to provide that objectivity. And when you do do that, that's doing your job right, you know.

Do you see that testers sometimes put themselves like in a lane where, you know, there may be security issues that may be exposed or maybe performance issues, but they're just focused in on one type of testing? Do you see that as a way of limiting yourself where you're not giving your organization all the information for the type of risks that they could be encountering?

No, I don't view that as a problem. And I think the industry has supported that for quite some time. And I think there's like, Like, you know, I consider, like, I don't know if you know Nate Custer, but like, you heard that name before, but he's like an automation architect, right? That guy's a specialist at designing automation. That's a specialist role. And I, when I want a tricky problem, I'll call him in, right? So that's, that's, that's the special role. Non— I hate saying non-functional testing, but performance engineering. I've been in the enterprise too long, you know, like, you're cranium-depleted skull can't take new acronyms. But performance engineering, you know, those are specialist skills. And I'm not of the mind that like, well, you know, everybody should be concerned about quality. Some people are better suited than others in an organization to do testing. I'm a big fan of independent testing. And, you know, I can already hear, you know, out in the social sphere, oh, well, he's saying that because he works in a testing organization. I'm not that cynical, you know what I mean? Like, I do think there's a role for independent testing. And a lot of the, you know, really, really great developers that I've worked with or engineering organizations appreciate that as well because there's different mindsets, there's different skill sets. So, I don't mind specialization. I think that it's good.

Now, obviously AI is muddying the waters. I've seen companies that are just I guess vibe coding is derogatory now, agentic development where their product managers are developing the software, not even thinking about testing, not even thinking about development best practices. It's like we're getting outta whack here. Do you see that and how do we fight it? Is the genie already outta the bottle? Like where we all drank the Kool-Aid now with AI, no one else is gonna listen to us no matter how loud we get. Yeah.

I mean, it's hard to answer that question, not like, 'cause like AI is really, well, they all are, all technology systems are, but they're kind of, socio-techno systems, right? And so it's hard to look at, and I would call vibe coding, I say slop. I mean, I don't have any of those terms. It's hard to look at that outside of the broader context of some of the economic realities of subsidized tokens, how these organizations work. We were just reading today about OpenAI and Hugging Face, the jailbreak they did. So it's like there's security issues around all that. You know, I was working with an organization that was a couple of months ago and, you know, they put these really powerful tools in the hands of a whole bunch of people who didn't really know what they're doing. And just because it passed their technical, you know, review board, they didn't realize that some of these folks were actually building the Death Star and existential risk to the firm. So, we had to like, you know, start to deconstruct that thing. You know, you had to get the plans from, Princess Leia there, you know. So, but it's, you know, it's— I, I think the genie's out of the bottle to a certain extent. I think there's going to be— I know we're always starting to see it, people rethinking some of this. And, and I, I said late last year, um, I can't remember what talk I was gonna— like, people are gonna— you're gonna see the ones who are too far over their skis on this and start to pull back a little bit because The risks are real, are real. The also the productivity gains and ROI has hasn't even remotely come close to any realization yet. I mean, tokens are so what we even can see from these organizations because they don't even show their books, right? Like how heavily subsidized they are. Did we really get anything out of this? You know, you look at the App Store, how much stuff slop gets put out there, slop apps, and like nobody's using them.

Yep.

You know, so there's a productivity paradox around, you know, how much effort and money it took to build this stuff and did we really get anything out of it? Did the effort to make it eat up all the potential productivity gains? So, you know, I think there's going to be a place for it. I don't think, me personally, this is me speaking up, LLMs are the way to AGI or any of that. I think it's going to be a very powerful tool that's going to get rolled back to us. It's going to settle to a certain level.— that people use it as another tool because the economics of it. And there's a really great piece in The Atlantic recently about how inefficient training is getting and more expensive and everything else. I just don't see how the economics work for like really, really broad applications. And my final thought on that is I see people building a lot of stuff. I don't really see anything new, you know? So like, particularly if they bring it back to testing a little bit, You know, I've reviewed and had demos and talked to just about every major tool vendor's Agentic, whatever platform, Testinator, and, you know, they're not doing anything creative or new with it. It's just, you know, trying to put, you know, the current automation and quickness on steroids. So, which I wasn't barely interested in that before, so I don't know why I want to, you know, strap on a whole bunch of risk on top of it.

All right, so like you said, you're not against AI or automation. I'm just curious to know, like, let's go with each one. Where, where is automation helpful?

So again, I think it's in extending the reach of testers. So like doing things you couldn't do with, with people. I think there's some economy of scales on that. I, I, I work in enterprise as well, so there's lots of stuff around, um, you know, in trading systems that you can automate, you know, simulations of, of environments of, you know, that stuff's really good. Spinning up things down, up and down quickly, you know, looking at, I like, I can't stand shift left, but I like automation being run, you know, as I always talk to clients about, like, let's get the, let's try and compress the error injection and error detection lines as close as possible. So if you can run functional checks at the unit level that detect coding errors or things like that really quickly, I'm not a big believer in the cost of defect curve going up. You know, I think that's kind of junk science behind that, but I would rather see that done as soon as possible. So that's in a good discipline as well. So anything we can test and check earlier than we get into the more complicated and expensive tests, maybe to set up, I'm a big fan of all that.

Love it. How about AI? Same answer or is it a little different?

Uh, yeah, so AI, I mean, I don't know really what it's good at yet in terms of, um, testing. So I've seen a lot of stuff. I think I tend to look at it in two ways. One is almost like a research assistant, um, for ideas, um, for things that like, you know, what, what, you asking it questions. And again, only if you're on really good models, you know, there's definitely to a certain extent, you get what you pay for with some of that. So, I think if you have good models and deep reasoning, you know, you can ask good questions and they can give you good answers for test ideas. And particularly if you can, you know, I've got some risk models that I use and ask it questions, what's the probability of this? Or some Monte Carlo evaluations of, variations of tests, that's really powerful and easy to do. So I want to put that in the hands of a tester to model things quickly, get some ideas, and again, focused on risk. So kind of like as a research assistant for test ideas, the idea of like— and I don't really think they believe it either. I think there's a little bit of cynicism in the kind of agentic test platform, you know, we've got a bunch of agents. Because when you get into the details of that, 2 things is that one, the— when you're adding on layers of obfuscation and opaqueness and, you know, uncertainty of a system on top of another system on top of another system, at what point do you go, okay, what— how do we know what good looks like? And so that's a problem because, you know, they do make mistakes. I don't like to use the word hallucinate, but, you know, 'cause that, you know—

Makes it think it has agency or—

Yeah, or it has an understanding of reality.

Exactly, right.

I think hallucinate from, you know, but like, and honestly, tangentially, that's probably one of the biggest risks with using AI in general is that anthropomorphism. Yep. It's such a powerful thing. You think it's thinking and it is not. It is a machine that, you know, I think it's Emily Bender calls it a text extrusion machine. And that's what it is. And so I use it, you know, all the time. And I'm very, very careful with not anthropomorphizing it. So there's that piece around the, you know, the agentic testing, which I'm not really sure the value of that yet. Because I don't trust it.

Right, right. So that's a good point. Then how do you fight back? Because like you said, automation is almost like hypnotizing. You see these browsers moving, you're like, oh wow. You see these eject demos. I've seen a bunch where give it a website, it goes in and finds all the paths. It shows areas that there's no code coverage for. And you're like, how do you fight back to say, okay, all right, this is not thinking. I need to be objective here. Like, I don't know if you've seen people just lulled into kind of being trapped by it.

Well, I'll say there's two things. One, I don't see that being used in— like, I was at a demo the other day, a public demo, and I kind of got to a point where I couldn't take it much longer. And I said, when questions came up, I said, okay, well, show me where The model that you described, which was an agentic testing system that was generating tests at the same pace in which they were, you know, generating code, right? So they had, the thing was that they had like 10x or something, the amount of PRs in a week or something. And so they said they had to like 20x their amount of tests they were spraying at this system, you know? And then it was all human in the loop and all this other, you know. Nonsense. And so, I said, "Okay, well, look, show me where you've used this. Describe to me the case study where you've used this on a serious client on a system that mattered for a project that was extremely high risk, right? So, like, meet those three." And of course, they didn't because I'm also, you know, the projects that I'm used to working on are like in banking, KPMG does like national payments infrastructure stuff. We do like government work. We do like large banking transformations. You're not gonna spray your, if your website goes down, somebody might lose some money somewhere. If the national payments infrastructure goes down, mom and pop don't get their pension checks, you're in the news, right? You'll lose government contracts, you'll like that type of stuff. So I don't see this stuff being used in, what I would consider to be real serious ways yet. So that's one of the ways I push back on it. It's like, yes, it's like, I call it boys with toys, right? It's like, it's primarily what you see. So I never really went in for kind of vendor demos anyway. So asking, like knowing what questions to ask and pushing back and having some practical experience is always a good ward against, you know, claims of magic.

All right, so this question might get you in trouble, it might get me in trouble. Um, all right, so clients in UK versus USA, do clients in UK care more about quality and testing? Do they get it better?

Um, I, yeah, they're, they're, it's very different cultural environments. I think I've seen good and bad in both. Um, I, I think there's probably a somewhat more risk-taking cultural attitude in the US than there is in the UK. But that's also, I would say, probably more stable and trusted technology environment. So, you know, it's different culturally, but the risks are somewhat the same.

Perfect, perfect. Not as controversial as I thought it would be.

Did you see the way I tiptoed right through?

Love it, love it. So I'm going to start wrapping things up. I want to go back to that keynote. Uh, it wasn't a keynote, it was something you did at BrowserStack with that talk, Test Automation Is Dead.

I'm catching loads of flak for that, but I still—

Yeah. Yeah. So what is the antidote then? Like, I mean, cause like obviously what you said that it, maybe I'm wrong. So if you equate testing with checking, is your words, then you are, your career is dead. But if you believe testing is testing the way testing has always been testing, then it's not dead. Is that the main gist of the—

Yeah, I think if I try to be, you know, be specific about it, is that like if you, if your identity is wrapped up around competing with a machine to create automated artifacts, right? You're in trouble because as I've talked to these vendors, they are actively trying. What do you think the Agentic platform is there to do? Do you think that's there to create 1,000 more testing jobs? No, because the people who are buying it aren't thinking about that. They're looking at that 10x means 10x people, right? That's not 10x productivity. And I know in the enterprise, we always say the only currency we trade in is headcount, right? So, you can— that's the ROI of test automation has been snake oil forever since we started automating that flight registration system with WinRunner. How many years ago that was, you know, because nobody ever— I've never automated a job away, right? It doesn't— it's always grown. And the testing industry grows year after year after year. So if— but if right now in this moment, the cynicism in the— in tool vendors, the cost pressures in the enterprise and organizations, you know, software companies, and we're all over the place, to either move to AI or just in general in the economic war, there is going to be significant downwards pressure to adopt these tools. Like I said, even if they're not better, even if they're just slightly worse, to 10x your job away. So you need to know more than just writing an automation framework because as I said in the talk, pretty soon they're going to be able to regenerate that automation framework in an afternoon. And guess what they're going to get with it as well? All the instrumentation, all the things that you didn't do, your whole spaghetti code of stuff you've never maintained because 3 guys before you or gals before them didn't maintain it. They're going to get all that in an afternoon and not going to need you for that. So, you know, that's really my core point. Testing is going to evolve a little bit. I think in some ways go back to what it, you know, used to be and not so heavily tool and automation focused. And so that's where I think people should be focusing their energy. And there's a whole other piece of this, which we didn't even talk about, which is around, you know, model evaluation and performance and, you know, how to figure out if the actual— we're talking about using AI to test something. There's a whole other piece of this, which again is reusing stuff from, you know, the '60s and '70s around machine learning algorithms and how to test that stuff, which has got different names now around how to tell if the AI is actually working, testing for drift, for bias, for all that stuff. And as the regulatory environment changes, particularly in the EU, right, and, and, and other places outside of the U.S. that aren't so highly politically motivated and actually have some teeth, that's going to change rather dramatically really quickly.

So you think those are skills people need to learn more about than model evaluation and performance of the AI?

Absolutely. Look, you know, to be perfectly clear, you should— the testing is about learning about a system. You should learn about every bit of it. You know what I mean? It's like you should learn the language of, you know, AI and how that industry talks about what it does and all these things, you know, you should absolutely be learning about all that stuff. You can't ignore it. And again, if you're not threatened by it, it's just another tool to learn. You know, I mean, like Selenium, you know, when that came out, you know, you learned how to use it. What is it? How do I incorporate this in? Yes, it's going to have an impact in the industry, but at the end of the day, it's another tool in the box. You know, but your core skills around critical thinking, risk management, you know, translating that into business speak and helping to direct that and direct teams and, you know, that stuff doesn't change.

Once again, this might be a naive, a contrarian view then. Don't you think it's a good time to be a tester then? Especially with all this AI slop, like, is a world of hurt coming? Especially as you mentioned, we didn't cover with tokens in the wild, like how applications just not really being tested are going to drive up costs and things like that. Yeah.

Yes, 100%. You know, I mean, absolutely. And it's, again, I look at it as a kind of the, the reason why I think there's going to be a compression on that is you also look at like, look at the stuff Anthropic puts out on Claude code, right? Again, it doesn't do things dramatically differently, but it's getting better every, every release, right? And so what do I mean by that? Is that like, it's got all that stuff built in. So unit tests, all the instrument, they're compressing that, you know, air injection, air detection, you know, even as they go, right? So like, that's what's gonna take, I think that work is disappearing. I don't think that it's like they're gonna, it's all gonna get fired because they have AI do it. I think it's the actual need for it, a good chunk of it is going to disappear. Because of the economics, because of the technology, and the cynicism of a lot of our, you know, dear friends in the test tool business, um, and the economic pressure to do that. Look what's happened over the last 20 years with everybody having to learn how to do automation, you know, like good quality testing work has already been devalued by organizations. So it's like we have a pretty long case study to draw from on what the reaction is going to be.

Absolutely. All right, Keith, before we go, two quick things. First one, one piece of actionable advice you can give to someone to help them with their testing they could do right now.

They can do right now. So, do you want some stuff to read?

Absolutely. I love reading. I actually recommend this book if you haven't read it yet, Prophecy: Blow Your Mind. Anyway.

Yeah, yeah. I actually, if I was listening to your— was it your last one you did? I think it was about AI that it was a monologue. I think maybe you were talking about it. I actually picked up the book. So it's on my desk. Look, if you know anything about me or like, you follow what I talk about, some core things that I always tell people to read, right? One, Rethinking Expertise by Harry Collins about how to translate risk to how to talk to people, different people about different things. It's a great, great book on that. How Complex Systems Fail by Richard Cook. It is the number one source I go to for trying to find organizational risk and ideas for how to then look for them in an organization. Organization. And an older book that I like is called Measuring and Managing Performance in Organizations by Robert Austin. It's about how goal displacement— we didn't even get into the whole confidence engineering stuff, but like that, why I dislike that. And you don't test to be confident about something. And how you— I've written some stuff about it as well as like how to help organizations focus on the things that matter because as soon as you start measuring something, organizations and people act differently. So how to recognize that? From the AI world, my go-to resources are Emily Bender and Alex Hanna, Timnit Gebru, she's involved, they're all involved in the Distributed AI Research Institute. Dagmar Monnet, fantastic work on how do we even know what we're testing? She's a professor in computer science in Berlin. Olivia Gamblin talks about ethical AI and how to navigate that. And then if you just want some real raw red meat, Ed Zitron, if you haven't listened to him, he's really into the economics and the bubble and lots of information on that. And as well, I've started an AI testing and assurance group on LinkedIn. If you find me on LinkedIn, we have a group there and it's no slop. There's no vendors. It's just people posting research papers, testing techniques, things like that. But I guess the number one thing I would tell testers to do is start to look— and I used to do this at Barclays all the time— is talk to me about the last critical defect you found and what the business impact would have been if it went live, right? That helps you reframe what you're doing, right? So talk— Look at the way you're testing and try to understand what risks to the business would've happened if that testing didn't happen or if that defect you found went live. It's a really, really good thought experiment to start to get you focused on risk, business impact, and why you're doing what you're doing, which is not to just create tests.

I can't let you go, Keith, 'cause you did mention something about confidence engineering. I don't wanna gloss over it 'cause I've actually been going around the country Talking about confidence engineering, why is that dumb? Why is that a bad idea? I need to know.

Well, it's, you know, I get it. You know, it falls into the— and I was— it's whenever you start seeing confidence pop up in the testing world, and it's not— it comes up when people don't know what to do, right? And so they start— it's goal displacement, right? Which is going— like hitting the bullseye on the wrong target. So, the reason— once you start, you know, when you make your exercise to get confident about something, you're instantly, if you understand how emotions and, you know, confirmation bias works, you're going to start looking for things that make you confident, which you can't do. I don't want to understand. That's what I'm saying. Uncertainty in testing is a feature. It's not a bug, right?

Right, right, right.

We want to understand what wasn't tested, what don't we know. That's way more important than what we've already seen.

The—

and then the idea of engineering that, that is a slippery slope towards all sorts of goal displacement and confirmation bias. So I, I, I— that's not our role, right?

You don't—

you test for information. You don't test to get a type of information or one that makes you feel good or one that, you know, is going to make everybody happy. That's, that's not the point. And also the, the thing that bothers me the most about the current, you know, incarnation of it, and there's been various— I was going on about this in the early 2000s about people talking about they could— they actually had a mathematical, you know, algorithm for how you could measure and track confidence, you know what I mean? Which is nonsensical as well. But right now, getting confidence in AI, the pill you have to swallow is that the less I know about something, the more I'm expected to feel confident about it. You know, and there are very senior people in our business that talk on all sorts of shows and podcasts who say things like, you just got to trust the AI, man. You just got to trust it. You just got to go with it. Once, you know, my AI friends, all this kind of stuff and talking about just trusting things that, you know, my bullshit detector starts blinking red really quickly. And I also feel like we're in danger, right? Like, I don't want to do that. That's— I want to know more about a system in order to trust it. I don't want to know less because there's already so many things about AI that we just don't know, right? They are truly black boxes, not as a test technique, but because we're not allowed to see the inner workings of it. So, I don't like any of that.

So, more unconfidence engineering or risk engineering. I think we coined a new term.

It's a sign of maturity, actually. Being comfortable with uncertainty is, to me, you know, that's, you know, like, read Feynman's stuff on this, you know what I mean? He's like, you know, you don't have to know everything. He's like, he's afraid of people who say they know everything. The people who are like, you can be, and also you can be incredibly confident about something and incredibly wrong. You know what I mean? It's like Daniel Kahneman wrote all about that as well. He is like, you know, you can overconfident individuals sound like an expert. They talk like an expert. They write like an expert, but more than likely they are in the grip of an illusion.

Right?

It's like, I, you could be very confidently incorrect.

Right. Well done. All right, Keith, best way to contact you.

Uh, my blog, www.qualityremarks.com. I'm on LinkedIn. I'm, you know, I post and write and do all sorts of stuff all the time, whether anybody's listening or not. So, uh, pretty, pretty regular on these things.

And we'll have links to all this awesomeness down below. Thanks again for your automation awesomeness. The links to everything of value we covered in this episode, head on over to testguild.com/a5. 9, 8. And if the show has helped you in any way, why not rate it and review it in iTunes? Reviews really help in the rankings of the show, and I read each and every one of them. So that's it for this episode of the TestGuild Automation Podcast. I'm Joe, and my mission is to help you succeed with creating end-to-end full-stack automation awesomeness. As always, test everything and keep the good. Cheers. Hey, thank you for tuning in. It's incredible to connect with close to 400,000 followers across all our platforms and over 40,000 email subscribers who are at the forefront of automation, testing, and DevOps. If you haven't yet, join our vibrant community at TestGuild.com where you become part of our elite circle driving innovation in software testing and automation, and If you're a tool provider or have a service looking to empower our guild with solutions that elevate skills and tackle real-world challenges, we're excited to collaborate. Visit testguild.info to explore how we can create transformative experiences together. Let's push the boundaries of what we can achieve. Oh, the Test Guild Automation Testing Podcast with Lutz With their lutes and lyres, the bards began their song. A tune of knowledge, a melody of code. Through the air it spread like wildfire, through the land. Guiding testers, showing us secrets to behold.