The AI Illusion: Why Testing Still Needs Humans

Hey, I've been going around the country for my TestGuild IRL tours, meetups basically around the different cities. I noticed a disturbing trend. There seems to be a massive decrease in the amount of folks that actually understand the fundamentals of software testing. And I also noticed a large part of my audience and myself, who am I kidding, are aging out of the industry. In theory, I could actually retire in 4 and a half years. Will I? Probably not. But What happens to all this foundational testing knowledge when people do start aging out and retiring? You might think, won't AI solve this by consolidating all the testing practices into a global resource? You would think so, but in fact, I would say AI is actually making things worse, much worse. We're being lulled into a false sense of security thinking that AI can actually think and therefore replace us on our work. And to put bluntly, this is what some philosophers call, I usually don't swear, but this is what they say, and there's books on this bullshit. So in today's episode, I wanna go over a few books and resources I think you absolutely must read to help turn this around. And by the way, these are not affiliate links and this episode is not sponsored in any way. Let's get into it. All right, the first book is called Prophecy: Predictions, Power, and Fight for the Future, From Ancient Oracles to AI by Carissa Vélez, Vélez. Look at this. I'm only halfway through, but you could tell it's really blowing my mind. It answers many of the questions I've been asked actually at my TestGuild meetups and IRLs. And the biggest question is usually, why is big tech pushing the narrative of these all-knowing AI large language models? Why are leaders and politicians backing them? And should we be worried about our jobs? This book actually argues that predictive algorithms and AI are ultimately All About Power, Not Accuracy. She dives into how big tech pushes the narrative to secure massive government contracts, to build infrastructure and make politicians and institutions completely dependent on them. You know, now for example, they spend billions of dollars on lobbying to cement this control and shield themselves from liability. But this book is not about politics. It covers philosophy, history, politics is part of it, and just the way we think in the narrative that's being taught and told to us about AI. And you might be asking, but what about the AI intelligence we keep hearing about? Well, the prophecy book brings up the philosopher Harry Frankfurt, who defines bullshit as speech that is persuasive but completely detached from any concern from the truth. Now follow me, I'll eventually connect this to testing. The author actually argues that large language models are ultimate bullshitters because they're designed to generate answers that sound plausible with absolutely no regard for what is actually true. It's just a pattern recognition machine. It's trying to predict the next question or the next word in a sentence that you're prompting it for an answer for. Also, there's a great resource that Michael Bolton pointed out to me in episode 544 last year, which is a free online course called The Bullshit Machines that I highly recommend you take. It's really easy, really fun to follow and it gives you a sense for what LLMs really are and you can experience it for yourself. But back to the book, another point Carissa makes is that machine learning is just tracking correlations. It understands nothing about causes. And for example, it only knows historical trends. It doesn't know anything about the future. And sometimes we think it's predicting the future when it can't. It just goes by maybe what happened in the past. And as we know, what happened in the past doesn't always happen in the future. Also, it's full of examples that give you insight into how LLMs really work. And for example, the book mentions an AI that was trained to distinguish between huskies and wolves, and it got really good at it. But what researchers noticed is that the AI was just focusing on the background. Photos of wolves simply tended to be or have snow in them. It wasn't actually recognizing the animal. It was just tracking a shallow correlation. So even though the animal might have technically been right, it was still what we would call bullshit. So I'm not done with the book yet. I'm halfway through, but a huge point so far is the author of Prophecy explicitly states that systems that are not sentient beings cannot care and therefore cannot value. Seems obvious, but I think we're actually adding more and more anthropomorphic attributes to AI when we're dealing with it. We think it's thinking, it's really not thinking. We think it's reasoning, it's really not reasoning. Humans understand the world including cause and effect, comes from our physical interactions, like feeling the warmth of the sun, the pain of a burn, or the outrage of injustice, or the frustration of our end users. And she goes on to say, because an AI is an unfeeling entity with no skin in the game, a reward for an AI is nothing more than a mathematical weight used to reinforce a connection. It experiences no pleasure, no pain, so it literally cannot comprehend human values and More importantly, it never will. It also touches on something I think we need to be more aware of as testers as well, especially with our end users, that is empathy. When it comes to empathy, it gives an example of chatbots, another great example, and it calls it impersonators, and that they are actually created to be manipulative by design. So the tech companies really don't care if what they're saying is true. What they care about is sucking you into the conversation, using it more and more, eating up tokens. Obviously there's certain things it's good at, but it has some great examples of how, uh, these chats can go right off the rails. It has some really good examples. I highly recommend once again you read the book yourself. And it also goes over an example of how when you mimic other humans, when you're talking to them, it really mirrors your emotions and you can start connecting with them. It builds empathy and trust. It signals true empathy and connection. But when an AI mirrors you, when you're dealing with an AI, it's always going, that's great, JoeBot, and I, I actually agree. It's usually language as if it's, it understands what you're, what you're, what you mean. And it really doesn't. It simply means we're being bullshitted, right? And AI is designed to sound human and pass as an empathic being without actually possessing any empathy. And because of this complete lack of feeling and lived experience, AI cannot understand truth. The truth matters to humans in a way it can never matter to an unfeeling entity. And as testers, I always say we're truth seekers. And also this reminds me, a complete lack of actual understanding and empathy in AI brings me to another brilliant mind in the testing space, Tyreke King. So at our TestGuild IRL in Miami earlier this year, Tyreke did this awesome session around like human experience testing, and his insights tied all this together, I think, really perfectly. So for example, Tyreke acknowledged that AI and agents are absolutely going to automate more and more of the software lifecycle, but he asked a really critical question. What happens what happens if AI takes over and fully automates all of the testing? Well, as he puts it, at the end of the day, you would have a situation where everything is tested, but nothing is experienced. And because the software we create, well, most of the software is for our end users, this simply is not gonna work because AI isn't sentient. It doesn't feel, even though it makes you think it's feeling. It fundamentally lacks empathy. It's never gonna have empathy no matter how much these tech companies are gonna try to tell us it does. Software doesn't live on an isolated digital island. It lives in our real physical world that only we can perceive and understand fully. And also Tyreek pointed out, because we are emotional beings, testing has to encompass more than just clicks on a screen, and it has to evaluate the entire end-to-end customer journey across 3 different levels that he touched on. I have a link for this as a resource down below. He did a whole session on this as well at other conferences that's on YouTube. So he goes over digital touchpoints, physical touchpoints, and experiential touchpoints. AI can't do any of these, and AI can't walk into a room, can't drink your coffee, and it certainly can't feel the stress and frustration of a delayed flight or a clunky physical-to-digital handoff. He also gave a really cool example of the journey of a customer at IKEA and all the decisions and experience that goes into testing that flow that an AI could not replicate ever. And this is actually where our empathy as human testers becomes one of our greatest assets. So Tyreek argues that while AI handles basic functional checks, human testers will need to test the one thing machines fundamentally cannot, the human experience. So we need real people to evaluate how software actually feels and ensure that AI systems we are building are aligned with actual human values. And once again, back to our book Prophecy, it really hammers home the point that LLMs, large language models, are built to be fortune tellers, not truth tellers. She also goes over the history of fortune tellers and our obsession with numbers and economics, the whole thing. So definitely it's a must-read. But anyway, uh, it's really just generating answers that are statistically plausible, but entirely detached from our actual concern for what is true. And as testers, truth is our number one concern. And on top of that, machine learning models do not understand why things happen. They only track shallow correlations without understanding the underlying causes. And like I said, this book is not specifically about software testing, but all of it applies. We are increasingly responsible for dealing with AI-based systems, and management keeps telling us we need to use AI more and more, but I don't think they really understand, uh, the ramifications of what we're gonna go through because of that. All right, so what does any of this have to do with actually the practice of testing? Well, this brings us to our next batch of books by Wayne Roseberry, "Writing Test Plans Made Easy," "Drawn to Testing," and "Drawn to Testing Again." First off, when it comes to, I think, the knowledge of testers in general, I think it's being lost. "Writing Test Plans Made Easy," definitely should check out. Some great points here. Really great for beginners and people that have been doing this for a while that want a refresher. He goes over a bunch of things of why it's important. One thing is the opinions of people, why it's important and why AI, you need real people, not AI. And also it goes over things like areas of the product, like risk that I think a lot of people overlook, but it's a really great breakdown of test plans, why they're important and why they're still important and what kind of value you still get from them. It's knowledge that I think a lot of people are missing. So this is fundamental. I think everyone should read this. What's even more important is these two books. They may seem like, because he has a lot of drawings in it, it really breaks things down really simply. You see it has a lot of not AI-generated images either, it's images he's created. It makes it really fun and it makes it really practical of examples of how he breaks down his thinking of the fundamentals of testing. But he has a bunch of areas on AI as well, how he sees it. And I think it's really helpful for you as a kind of a counterpoint to this prophecy, how this applies to testing. So for example, Wayne really echoes the lack of comprehension that we just talked about. So for example, in the book, "Dried to Test," he has this great example on words are not enough. And he argues that language itself does not contain meaning. Rather, words are just keys used to unlock meaning. And the actual meaning is individually stored in human minds, not AI, built upon our biological brains, our bodies, and lifelong history of physical and emotional experiences. And obviously, because AI lacks a body, a human mind, and lived experiences, it can generate the words, the keys, right? And he goes over this, but fundamentally cannot understand the meaning of truth behind them. So funny how, like I said, it seems like— that's why I like to do— that's why I always recommend going across discipline when you're reading, cuz you never know how one applies to another. And this dovetails beautifully with this book, even though they're not related pretty much at all, but they're saying basically the same thing. This book came out after his book as well, so I had no idea about it. So I really think these books by Wayne really bridge the gap between the philosophical warnings of the book Prophecy and our day-to-day work as testers. Tackles a bunch of things as well, not just AI, but it goes over the fundamentals of testing. He has a really good chapter here on flakiness, how it's not really flaky, how we're missing out on a lot of things. He worked at Microsoft. I think he might still be at Microsoft, I'm not sure, but he does a lot of breakdowns of how they found actual issues rather than just ignoring the flakiness, how they turned out to be really big issues. So a lot of things that fundamentals I think we need to know also be tackled here as well. And he also shares a bunch of real-life stories. One is about vibe coding. His team used an AI tool called Cursor to build an application based on a requirement document at first, It looked amazing. Usually if you've done anything with Vibe Coding, it usually does. Krista built a clean visual app in seconds, but then they noticed a massive bug. The app wasn't saving any of their changes. Why? Because the requirement document didn't explicitly say it needed to save. And as Wayne points out, the AI doesn't magically know your intent. Without explicit human direction, the AI couldn't build a functional application. And this is exactly why Human testers, I think, are irreplaceable. We understand context, intent, and missing requirements, things an AI literally cannot comprehend and will never comprehend. I don't even care if you put AI in a robotic body, still not a human, still not gonna understand. And I can't find it now, but he has a really good roadmap, a visual roadmap showing that in AI-based systems are more than just AI. He points out that there's plenty of places to look for bugs outside of a model itself. Such as the data preparation, the post-processing, and the system boundaries interacting with external systems. And he also says how AI is a high-risk component and we have to treat it with the sober testing strategy, not a drunken AI hype strategy. And once again, ultimately this all comes down back to testing fundamentals. We can't rely on machines to do our critical thinking. And Wayne highlights the psychological trap he calls "I'm doing something" bias. And I don't know if you could see this, but he illustrates this with the cartoon of two people, one sitting at a computer saying, not now, I'm getting the automated scripts to pass, while the literal alligator right here is creeping up behind them. And the point of this is trying to do something engaging, like messing around with your AI scripts, can lead us to ignore the massive dangerous problems right in front of us. So we have to be ready to capture information, be careful of distractions, and tackle the alligators in our code. All right. Hopefully you found that helpful. I haven't done a breakdown of books in a while. As you can tell, I'm a bibliomaniac. So here's your challenge for you today. Don't eejoe and let this knowledge die with us. Don't assume AI is going to consolidate our craft into a magical prompt. It's not. Make sure to read Prophecy. It's a tough read for some people, but I would highly recommend getting through it to really understand the hype and the power dynamics at play. And also make sure to read all of Wayne's awesome work. To ground yourself in what it actually means to think like a tester and take the free course at the bullshitmachines.com URL. All these resources you can find down below. And speaking of fighting for the fundamentals, the best conversations I've had on this topic haven't happened online. They've actually happened in the room at our TestGuild meetup IRL stops. So we're going to more cities coming the rest of the year, and the next two are happening in Chicago and Minneapolis. This July. Chicago is July 16th, and we'll have Jason Huggins back with us, the creator of Selenium, along with Paul Smith, talking about what they're building with Vibium, uh, hands-on. So you're going to get a really cool hands-on experience. Also, we're going to have an awesome panel of practitioners actually doing the stuff in the real world. And Minneapolis is coming July 14th. We have a great group of speakers coming your way there as well. And a huge thank you once again to TestMew, our exclusive tour sponsor, for making these free for the community. And once again, you can find a way to register for all these events down below. Hope to see you there. All right, so that's it. And as always, test everything and keep the good. Cheers.