JoeBot and Adam Carmi expose the gap between AI hype and reality in testing. LLMs are good for brainstorming but fail in CI/CD pipelines, causing false positives and false negatives. Real production data shows only 15% productivity gains from AI-generated tests, far below vendor promises.
Here's the uncomfortable truth about AI and testing that most vendors won't tell you. 99% accuracy sounds great until you realize that just 1% error rate means your entire automation pipeline is untrustworthy. How do I know this? Well, I just spent an hour with Adam Carmi from Applitools at Automation Guild last month, and what he said stopped me dead in my tracks. What did he say? Well, let's break it down. So if you don't know, Adam's been in the AI for testing game way before ChatGPT even existed. Way before half these AI-powered tools were even founded. Applitools has been running AI in production test pipelines for almost a decade now. I think the first time I talked to Adam was way back in 2015 on my TestGuild Test Talks podcast. But here's what he told me. As a tool vendor, speak with all these companies, teams are being sold a story that doesn't match reality. You know the hype. It all says LLMs will replace testing. But here's what Adam's actually seeing happening at thousands of companies. LLM-based test generation, developers get maybe 15% production boosts, not the 70% vendors promise. AI agents writing and executing tests, more like non-deterministic chaos that breaks pipelines. And the promise of good enough accuracy, organizations have zero tolerance for false positives in their regression suites. And this isn't just a theory, this is data from thousands of teams running tests in production. So here's the breakthrough insight Adam shared that actually makes sense. There are two completely different types of AI in testing. You have LLM generative AI, and then you have deterministic AI. So LLMs, you know them, you love them, you use them. Your ChatGPTs, your Geminis. These are truly amazing for brainstorming test ideas, generating drafts, helping you think, but they're probabilistic. Every time you run them, you get a different result. That's an awesome feature for creative work, but it's a deal breaker for automated test execution and quality engineering. Then you have deterministic AI. This is purpose-built AI trained for one specific job. For example, visual validation, element detection. It does the same thing every single time. No surprises, no close enough. And here's where teams are really getting this wrong. It drives me crazy. They're trying to use LLMs in places where they need deterministic AI. And I think my man Adam said it perfectly. Teams expect deterministic, consistent, reliable test execution. The notion of regression testing is something they count on. If a test passed, they know everything is okay. They don't compromise on it. Translation, your automation pipeline isn't a place to experiment with mostly accurate AI. And look, you know me, I am not anti-LLM. I use ChatGPT and Claude constantly. For test brainstorming, writing docs, amazing. They're helping create little proof of concepts, awesome. But for actual test execution CI/CD pipelines, I need to know that if my tests pass, they actually passed, not probably passed or 98% sure they passed. 'Cause here's what happens when you trust probabilistic AI in your pipeline. False negatives waste hours debugging phantom bugs. False positives let real bugs ship into production. Your whole team stops trusting your test results. And just like always, once trust is gone, your automation is worthless regardless of how much AI you have baked in. You're back to manual testing. And Adam's doing something I think is really valuable. He's running a webinar on March 25th where he's breaking down exactly where LLMs belong in your testing workflow and where they don't. Even if you miss this or you see this video later on, you could still register down below and get the instant replay. And this is what he told me he's going to show you: a framework for deciding where to use which type of AI, how to avoid common traps teams are falling into right now, and what AI you can actually trust looks like in a production pipeline. And if you join Automation Guild, you know this isn't vendor pitching, or you heard any of my interviews with Adam. He's always been refreshingly honest about what works and and what doesn't. So if you're a QA lead or engineering manager trying to figure out where AI actually fits in your software development lifecycle, this is definitely worth your time. So make sure to check it out using that special link down below. Even if you can't make the live event, you can still get the replay by registering down below. And if you want to see Adam's full Automation Guild session where he goes deep on LLM versus deterministic AI distinctions, there's also a link below for that as well. Bottom line, stop chasing AI trends and start asking, what problems am I actually solving? And is this the right type of AI for it? And that's how you build testing systems people can trust. And as always, test everything and keep the good. Catch you here next time. Cheers.
