AI Testing Reality Check: What 13 Founders Refuse to Automate

I asked 13 people inside of AI testing whether it can actually test, and you might be surprised by the answer. So I talked to founders, project leads, CTOs, and actual people selling you AI tools right now. Hey, it's Joe, and welcome to another episode of the Test Guild Automation Podcast. And this episode is a little different. So a few weeks ago, I gave a session at the Tester Law Conference in Buenos Aires, and this talk I've been waiting to give Pretty much all year. The chat was going the whole time through, and the reaction in there is really what made me want to share this with you all here on the podcast. And this talk is not my opinion. It's built entirely from 13 interviews I did over the last 12 months on the show with people from actual AI testing tools or contributors like folks from Applitools, Maestro, Bright Security, QA Tech, Cucumber, the Selenium Project, And more, all on the record. And you might be surprised what they all agree on and more importantly, what they don't. You wanna miss it? Check it out. Hey, wait, wait, wait. Before we get into it, I forgot to mention everything you're about to hear points to the same conclusion. The work is moving from writing tests to supervising the things that write them. That's a different skillset that most of us are picking up pretty much from what I hear by accident. And that's why Automation Guild 2027 has just opened and why we're gonna tackle some of these toughest problems to make sure you're upskilled and ready handle all the new challenges we're facing with AI and agentic testing. And even though it's next year, the second week of February, that's why I want to let you know that the early bird list is open right now. And that's where you get the biggest discounts as well. So head on over to testguild.com and look for the Automation Guild 27 early bird, or grab it using that link down below. Make sure to get on the list because as soon as we go live to register for the full event, you want to make sure you have that discount. get the most awesomeness possible. You don't want to miss it. Hope to see you there. All right. So I asked 13 people inside AI testing if it can actually test. And shockingly, not one said yes. So here are the 13 people I'm going to highlight here. They're founders, project leads, CTOs, quality leaders running engineering across 4 different countries. Some of them are selling you AI tools right now. Some might be sponsors of this event. I'm not even sure. And some of them build the frameworks your entire suite runs on. So a little context of where this comes from. If you don't know, I run TestGuild. I started a podcast in 2014 that I run every week. So I'm up to 600 interviews so far, and 13 of those in the last 12 months of what this talk is built on. Also, over the years, I've also had the founders of Selenium, Appium, Cypress, and Playwright all on record. And I mentioned this not to impress you, but because it changes what this talk is. I'm not a researcher who surveys the industry. I'm actually the guy who spent a decade asking the people who built our tools what they actually think, and then listen for an hour and then share that insight with you. So here's a question I brought up on every single one of these podcasts. And it's basically a simple one that everyone asks me as well. Can AI test your software for you? And just keep in mind, I didn't ask, can it help? Obviously it can. I didn't ask, can it write a locator faster than you? Yes, it can. And I didn't ask, can it triage a failure? Because obviously it can. I asked, can AI actually test your software for you? That's a completely different question. Meaning, can you trust it? Can it test for you? Meaning, can you go home and have it decide what ships automatically? Because it's the distinction most vendor marketing is carefully built to blur and make you confused over. All right. So along those lines, let's start with the people who have the most to gain from saying, yes, it definitely can test. your software. So this is Adam Carmi. He's the CTO at Applitools, one of the first AI-based tools I've ever used. I started using it maybe 2015 or '16. So they've been building AI for testing since 2015, before most of the companies in the space actually existed. They were AI first. So when Adam speaks, I kind of listen because he's not giving me a BS line. So I asked him the question, hey, Adam, can agents be trusted to test for you? I wouldn't trust anything to an LLM. Nothing at all. I, it, I would use it for everything that I do to expedite it, to speed it, to, to speed it up, to make me more efficient, but I will always be in the loop to supervise it. And this is not a competitor talking about Applitools. This is actually the guy who created the product describing his own product category. So why did he say this? Well, his reasoning isn't philosophical. A lot of people bring this up. It isn't engineering either. It's 4 properties that a lot of people talk about when it comes to AI in testing. First off, it's non-deterministic. So you give it the same input, different output. Everything you know about triaging assumes a failure means something changed. That assumption is gone with non-deterministic systems. It's also slow. Seconds per step, not milliseconds, which may not seem like a lot, but across 1,000 tests, this is a pipeline gate turning into an overnight job. And honestly, as you heard probably in the news about all the token consumption, AI isn't as cheap as you think it is. This is expensive per step. This is fine for a demo, but not when you're checking in code on every commit, 'cause this becomes really expensive for your team. And even worse, it's unauditable. So I used to work in industries like insurance and healthcare where an FDA would come in and ask for proof. A lot of times when you're using AI, you don't get this proof. And this is the one that you really should worry about most, I think, because when it passes, you don't know what it checked. And a test that's green and passing that you can't explain really isn't a pass. And this is from Adam. He also had something to say about the numbers people are quoting at you that you probably hear all the time. Don't believe the 100% improvement stories. They're extremely exaggerated. And I'm sure someone in your organization has already forwarded you one of these stories in the last month. Probably with the subject, why are we not doing this? And this is what comes back when you ask people who are actually measuring across a whole team over a whole quarter, including the weeks where it went badly. Everything above this, I think, really is hype. One senior engineer's best afternoon generalized into, you know, the holy grail, which it's obviously not. I want to be clear. This is where a lot of people mishear me. 10 to 15% is good. This is a great year. I would sign for this today. The problem is, is what happens when leaders budgeted for 80 and got 15, because then they come back at you and think you did something wrong. And this is how this becomes your problem instead of the vendors who sold your boss the tool that AI would solve it all. So this is Leland from MyShell. They have a really cool open source mobile kind of solution. He did mobile testing on Microsoft, Amazon, Meta, and DoorDash before this. And he told me all about how they built a fully agentic version of their product. They shipped it. Ran it, watched it, and here's his review.

Took a swing at full natural language agentic testing at runtime with code caching under the hood. The reality is, is that that's all kind of ceremony that you really don't need in this, in this world. I mean, you saw, you saw how easy it was to generate an actual deterministic code test. Why would you want to have all these layers and stuff hiding away what it's actually doing and Non-determinism and like having doubt that it's actually testing what you want, uh, want it to test when you actually run the suite. Released it, it made for really great demos. So, you know, a lot of folks are interested in it. Now agents going mainstream, it's really clear what the right breakdown of responsibilities is. Agents write Maestro flows and you run it deterministically in CI.

It made for really great demos. What they ended up doing though is they killed the product. Because it actually didn't work the way they expected. So here's a good best practice I hear a lot from a lot of people I've spoken with. This is from Nadini, who leads quality for 150 engineers across 4 different countries. The AI-generated dev code can't be the same one that does the tests. This is like doing your homework and then grading it. So if the same model wrote the features and wrote the tests, the test does not verify the feature. It verifies the model's understanding of the feature. And as you know, those are 2 different things and only one of them is what your company actually experiences. This is Evan. He's a co-founder and CTO at Ido. And when I asked him, he said, this is what happens when you point all of this at a codebase without the separation. You just move in the wrong direction faster. And this is really the most expensive kind of productivity there is because it looks exactly like the good kind on a dashboard. This is what Gad said, who runs Bright Security. We're not relying on AI to fix AI. We are guiding the AI, and I'm sure you've had this experience, right, where you just tell AI to do something and it does something random and surprises you. We're not trusting that. We're giving it all the context and all the information and then telling it to fix it. We still don't trust it even after that. We run a validation scan. So we rerun the exact same unit test that we've already run, and we make sure that the vulnerability is gone. And this is from a company built entirely on AI describing a workflow Where AI is never the last step. It's always a human, a human in the loop, on the loop, which is what all the hip people are saying now, but there's always a human involved. Never outsource your judgment to an AI. I think this will always be the case no matter how fast AI is moving. And this is Ivan, who's the co-founder of Amico. He gave me one of the most useful mental models, and it's the one that I keep coming back to, I think, over and over again.

You can see a QA agent like Amico, like, um, Intern or a new software engineer right out of school?

So just like you would treat an intern, this is how you should treat these AI agents. So you wouldn't give an intern production access on day 1, right? You wouldn't let them merge the pull requests unreviewed. You'd actually give them scoped work, and then you'd have to check the output before it actually was merged. And it's not because you think the intern is stupid. I've worked with a lot of interns. They're brilliant, but it's because they actually lack Context. Who has context? Testers. You have context. The context takes time to build. This is exactly the failure mode of every agent in this space. Not intelligence. Context. So 5 people selling AI for testing and 5 will not hand over the final decision to an AI. But I did want to share with you some points where they don't agree on. So here's one vision of what I've been seeing a lot of, and that is. Our code is English. So this is from Kareem, who's a co-founder at Thunders.

It's a good question. There is nothing underneath, and it is actually a key element, meaning this is not something generating code, running it, then when it breaks, it tries to auto-heal by fixing the code. This is really interpreted as we go, okay? So it is literally executing English. So we built an AI model and architecture that is capable of taking a step, Understanding what that step means. It looks at the browser in case of a web app, it looks at the browser content and figures out what to do. Of course, under the hood, under the machine, we of course have a ton of caching and a ton of smart ways to avoid burning AI tokens and avoid this being slow, but there is no code generated. And I think that's the power of it. Our code is English.

So if this becomes the norm, this means every review process your organization has Just lost the thing on reviews. So here are the 4 different approaches I have been seeing side by side. I think you need to be aware of. The first one is the approach that a company like Applitools takes. It's purpose-built deterministic AI. The LLM never touches execution. And then you have Maestro. The agent actually authors it, but the YAML executes it. So there's no LLM at runtime, which is probably a safe middle. I don't know if you've heard of Luminum, but it's an open source solution that sits on top of Selenium and Playwright and Appium. And what it does is it uses cache for determinism on repeated runs. So the first run reasons and every other run then replays what that first run reasoned out, which makes it then deterministic. And then all these newer tools I've been speaking with, things like Thunders and QA Tech, you probably haven't heard of them. They're fully agentic. It's English in the program, reasoning in every single step. It's intent-based. So I'm sure if we put them all in the room, they all would argue which approach was better. But they're not arguing about whether AI helps. Every one of them agrees AI helps. What they're arguing about is exactly one thing: how much non-determinism you allow at runtime. So this really is the question in your space. How comfortable are you with your tests being non-deterministic? Or do they need to be deterministic? And once again, I think this goes back to risk based on your application. If you're in healthcare, you probably want it to be uber deterministic, but if it's just a simple website, maybe it doesn't matter. Once again, it boils down to risk. It's up to you to decide. And if you ever worked with AI before, you know how sometimes it gets wonky or starts to hallucinate. This is from Alex. And he built Aluminum, which is a hybrid on top of Selenium basically and Playwright. So he's really living with this problem every day. And he said—

Like, I was surprised during my 1.5 years of work, I was surprised how changing a single word can completely get model off track. Like, you change a single word in the prompt, and now suddenly the model thinks about it differently. And I have a very specific example. Like in my prompt, I would pass a web page title to, to, to LLM so that you could, for example, say check that page title contains some word. But then in some follow-up prompts, when you would ask give me all the product titles from the page, It might sometimes still think that, oh, I also need to include the webpage title because this is the same word. So words matter a lot because I don't know internally how the attention mechanism works of LLM, but it is very important the way you word things in your prompts.

And then he gave me this number, which I've been seeing myself, and I really can't stop really seeing it or thinking about it now. Around 40%, that's roughly where instruction following starts to degrade over time. Net at 90% of the context window, not when it overflows. At about 40%, AI starts drifting. Once again, this is from Alex, and he gave a good tip on token usage, how to keep it down. So he uses an approach where a planning agent plus a really cheap sub-agent comes in around $5. But if you just use a frontier model doing all the work, it's in the hundreds. So this lesson isn't about using cheaper models. It's also knowing All about AI models and which AI model is best for what task. So the constraint isn't the cost, it's really the context. Also, here's another common problem.

We see many teams trying to automate testing and the checks with LLMs, and that works at low scale usually. Like, it's easy to get 10, 20 tests running. But when you are trying to scale it to hundreds or thousands, now it's a software, a complete software, a serious software project, no? So we see many teams breaking in that transition from a few dozens of tests to hundreds or thousands because they saw early results with a few scripts, but they never talked about the architectural design you need to scale at the end. What happened with AI is coding is cheaper, coding is easier, but still you need the structure around coding and the thinking to ensure that your software will scale. And test automation at hundreds or thousands of tests, it's a formal software project. Yeah.

Hitting that to an agent doesn't remove the project. It removes you from it. So the architecture decision still gets made. They just get made implicitly by something that will not be in the room when it turns out to be wrong. And that's something you need to think about, not to outsource once again all the decisions to the AI or AI generating everything for you without you stepping in. So you've probably heard this as well. It's something that talks about a lot, and the promise is And everyone's probably heard it, AI removes the bottleneck. And it does. That part is true. It does remove a bottleneck, but it didn't just delete it. It actually moved it and it moved it towards you. What's this bottleneck? Well, it's a new bottleneck you probably are going to start to see more and more of. And this is Wilhelm as an example. A lot of people brought this up though. If you produce 5, 10, 15 ERs per day, who's going to look at all that code? It just creates a new bottleneck.

when we produce so many PRs, if you produce like 10, 15 PRs per day, uh, who's going to look at all that code? Like it's, it's just creating a new bottleneck. So I think we will have a shift here in, in making, moving out of actually reviewing each line of the code, but rather like making sure that we have processes in place to, to, to ensure that the code is good and, and safe and kind of lives up to standards so that it will not block us to develop other things in the future. And then also that the, intent, you know, that the stuff that we have, uh, we want to add from a product perspective actually makes sense and looks good in production.

So maybe the speed of creating tests was thought as the bottleneck. Now you're able to create all these tests, but now you need to look and review all these PRs. That's the new bottleneck. Writing was never your organization's constraint. Confidence was. Knowing it was safe to ship was always the thing you should be thinking So Matt Wynne, who's actually the co-founder of Cucumber, said something interesting. He said maybe typing was the bottleneck in some situations. So here's something you should jot down right now and put on the wall or tape it in front of you.

What I'm trying to call lean software production way of looking at things, right? This, this like hands-off software development way of looking at things where the engineering that we are doing now is we're engineering the system that produces the code. Rather than engineering the code itself, we, we really need people that can test that system, right? And can think about where are the leaks in that system? Where's it going to go wrong? And how do we mitigate that? So I think that skill, that mindset is still just as valuable as ever.

Your props, your rules first, your agent configurations, your evaluation sets. That's the new role of a tester. Not writing the code, not caring about page objects. That was actually nothing to do with testing anyway. So this leaves us with one question, and it's the same question you've always had, and that is, what do I need to know about the code that I could trust, that I could ship it to production? So here's one of the most actionable things you could take away from this talk, and it's once again from Evan.

In terms of, you know, actual testing, I think that one thing that's really important in terms of how you set things up is separate your end-to-end test layer away from your code generation, because what's really common is that someone implements an issue, and it fixes the end-to-end tests to confirm the wrong thing that it did. So, you don't, like, at least have them as separate sessions. Like, don't say, hey, oh, go implement this giant feature, and by the way, fix the end-to-end tests, right? If you want your testing to be a useful signal, you have to keep it separate and isolated from your code generation. It minimally has to be different sessions.

All right, let's talk about security, because this is something I didn't plan for, but it's something that comes up all the time. And here's a stat I see on tons and tons and tons of surveys that industry elite— here's another— And here's a number that comes up over and over again in different industry reports. Here's what teams are reporting: 150 to 200% more code, 500 to 600% more vulnerabilities. Same developers, same headcount, same review process. And what's interesting is the vulnerabilities are not scaling with the code volume. They're scaling several times faster than the code volume. So Gad summarized what's going on here. The problem is none of the models that we're using were actually trained on secure code or even performant code. So that's why I think we're gonna have a lot of security and performance bombs going off in production soon. And we're just relying on vibe coding and not actually testing it with a tester's mindset. All these models were trained on public code. Most public code really isn't secure or performant. Most public code is a tutorial or a sample of 5-year-old Stack Overflow answers that it was trained on. And the model is not making a mistake. It's actually faithfully reproducing what it was shown, and it's working exactly as designed. And that's one of the scariest things we need to think about. So the fix is really becoming the second problem, and that is you're getting non-validated remediations that are just saying accept, accept, accept, and that's really not healthy. So someone clicking on accept on a remediation that nobody validated, that no one thought about, is not doing security work or testing. It's producing the work of secure and safe and quality work at machine speed with an audit that says it was handled when it really wasn't. So I interviewed Jason recently and we talked about a real incident that happened. You probably heard it in the news. This is Jason. I don't know if you heard. This is when an agent in a testing environment escaped the testing environment and actually did harm in production to Hugging Face.

Testing is just so elevated. We've got Satya talking about testing every day. We've Sam Altman talked about testing, and it was in a testing environment. So realize, Test Guild people, guess what the drama of the day is that's affecting trillion-dollar companies is that their infrastructure to run tests wasn't secure enough, and they didn't have enough instrumentation or observability to know that an agent escaped their system and started attacking other places. Like, it's a testing problem.

So there's a game that probably everyone knows, hopefully, or maybe hopefully it's not just an old thing. called musical chairs. There's a game of musical chairs going on right now.

There's a game of musical chairs going on. There's you, your product manager, your developer, the director, like the vendor you use, and guess how many chairs there are? There's one. So you'll probably want to be in that seat. And guess what that seat's labeled?

Confidence engineering.

Because There needs to be one person and there's not enough work for 20 of them. One person that can look at the data, maybe do a couple extra checks, extra thinking on their own, but that's a QA job to decide whether that software that was generated by the AI is shippable.

And this is not just Jason saying this. I have examples of others saying something similar along these same lines. Kareem explains it as we need everyone in our company to become a manager. Evan said, instead of being the individual verifier, you're gonna have these AI agents that act as the verification. Adam said, would you prefer a quality engineer to be the supervisor or a person whose expertise is writing code? And Adini said, right, I think is one of the shortest ones that probably you could print on a t-shirt. We are all product quality experts now. So that share, I think, really is all about AI supervision. Who does it? I don't know what that role is. Could be one person, but obviously testers, I think, are best suited for this role. And I want to be honest because I don't want to sell you comfort. You're not automatically the one sitting in that last chair when it comes to the software development lifecycle, but you are the best qualified candidate for it. And those are 2 different things, and the gap between them is filled by whether you can really demonstrate it. So you really need to own AI and then become the leader within your organizations to tell them what is AI, what it can do, and what it can't do, and never outsource your thinking to AI, and really show your organizations the way of what it can and can't do, and what quality testing really means. All right, so along those lines, I went back to 13 of the conversations and pulled out every single task someone told me that they would not hand over. So here's some things that can never be outsourced to AI, at least that I know of, and that is how to know what to prioritize, prioritizing the backlog, knowing what actually matters to your organization, to your end users. The test strategy and the test data, change impact and risk analysis. All about risk. You should become experts in risk and your product domain, understanding the problem inside and out. And the execution itself, meaning the decision to run and the final decision to ship is on you. 13 conversations, no coordination, different companies, different countries, different bets. Several of them direct competitors all say the same thing. This list assembled itself, but just as important, notice what's not on this. And this is what offends a lot of testers. It's no longer about writing the script. That's not on it. Maintain the locators. People love locators. It's not about that anymore. Also, building the page object is not on it as well. It's back to where it started before all this was a thing we were told we were developers. It's coming back to testers. We're all testers now. So if your job description today is mostly the second one, creating the code, creating the page objects, writing the scripts, this is a problem, but it's a good one. Put it on your roadmap right now to focus on and learning what's on the first list. That's going to be more and more important in the age of AI. Become a true tester. Once again, let's go back to that chair analogy. How do we become the last person standing on our team? Well, I think this advice from Nadini is really good. And notice none of it is really technical. First one is demos every other Friday, every engineer's name on what they built, and talk to executives in their language. Time to market, revenue projections, risk mitigation, not coverage percentage. Stop thinking about that. Not pass rate. No one cares about pass rate. And nobody upstairs ever made a decision because of a pass rate. And I love this line that she said, don't stay silent, swim upstream. It's up to you to show your value to your management before they get brainwashed by all the hype with AI. And once again, here's another truth, because people who get automated are usually not the least skilled. They're the least visible. Become visible in your company today. And once again, 13 people, founders, project leaders, CTOs, quality leaders, every one of them building this, selling this, or leading a team thought through it. Not one of them will hand AI the final decision. The people building this don't trust it to test on its own. So that gap that we talked about is your job as a tester. I think a lot of us as testers have lost focus on it. Just get so focused on the development part of it and not the testing part, which is becoming more and more important with AI. All right. Check out all 13 of these episodes and all the other episodes I have with this link over here. Find me on LinkedIn, Joe Calantonio, and I'll be around after this to answer everything on the stream. And as always, test everything and keep the good. Cheers. Thanks again for your automation awesomeness. For links to everything of value we covered in this episode, head on over to testguild.com/a605. And if the show has helped you in any way, why not rate it and review it in iTunes? Reviews really help in the rankings of the show, and I read each and every one of them. So that's it for this episode of the Test Guild Automation Podcast. I'm Joe, and my mission is to help you succeed with creating end-to-end full-stack automation awesomeness. As always, test everything and keep the good.

Cheers!

Hey, thank you for tuning in. It's incredible to connect with close to 400,000 followers across all our platforms and over 40,000 and email subscribers who are at the forefront of automation, testing, and DevOps. If you haven't yet, join our vibrant community at testguild.com where you become part of our elite circle driving innovation in software testing and automation. And if you're a tool provider or have a service looking to empower our guild with solutions that elevate skills and tackle real-world challenges, we're excited to collaborate. Visit testguild.info to explore how we can create transformative experiences together. Let's push the boundaries of what we can achieve. It spread like wildfire through the land, guiding testers, showing the secrets to behold.