Is AI Coming for Testers, or Are You About to Win Big?

Are you falling into the selector's trap that's probably burning like 80% of your team's time? Is Apple's new self-testing agent in Xcode 27 a genuine leap or just more vibe coding to clean up later? And are testers really first in line to lose work at AI or first in line to win the next big role? Find out on this episode of the TestGuild News Show for the week of June 15th. So grab your favorite cup of coffee or tea and let's do this. So I discovered this tool late last year, Quality, and they like the show so much that they wanted to sponsor this episode. And they told me they have some new features they want you to know more about. And basically what they do is really fundamental. Instead of you juggling tons of spreadsheets and 3 or 4 disconnected tools, you write, plan, execute, and track your testing, plus defect tracking and reporting all from one place. And this is gonna save you a lot of time. And the part testers probably wanna take a look at first is Hooty. They have a built-in AI assistant. You hand Hooty plaintext requirements, a UI screenshot, or full BRD, and it generates the test case in plaintext or Gherkin, then executes them automatically and logs defects with full context. This looks pretty affordable since the plan starts around $12 per month, and there's also a free 14-day free trial with full functionality, no credit card required. So check them out using that link down below and tell them Joe sent you. Next up is a video I think is really gonna catch your attention, and it's from my friend Manoj Kumar Kumar on his MK Labs channel on YouTube. And he takes a deep dive in this video on why testers are unusually well-positioned for AI. His argument in the video flips the usual line. Everyone says testers need AI, but he flips it and says AI needs testers, and that QA folks are better set up for AI engineering than most backend hires walking into the same room. And in the video, he walks through 4 different phases that are already built in QA that map straight into AI directly. First, you think probabilistically, the way you handle a flaky test that passes 40 times, fails on the 41st time. Second is you think in edge cases, which is just adversarial testing and prompt injection defense wearing new names. Third is you think in systems, tracing a failure across retrieval model and orchestration, the way you trace an integration test. And fourth is, You know how to classify failure modes. So a hallucination is really a functional bug and drift in a regression. Amanoj also makes the case that the instincts testers spent years building map directly into AI engineering and evaluation. And the only real gap is to close the vocabulary that we're using. He points to 3 roles hiring right now, AI engineering, AI evaluator, and agentic workflow engineers. And I agree, the window is open for roughly 24 months before the title becomes What is the new norm? Highly recommend you check it out for yourself using that link down below. All right. So I know a lot of Apple nerds follow this and also mobile testers. So here's an article for you that I think you're really going to enjoy. And this is how Apple just rolled out Xcode 27 at WWDC 2026. And the headline for testers is what its coding agents can now do on their own. So Xcode 27 gives agents the tools to validate their own work so they can run longer without a human babysitting them. So this includes writing and running tests, trying ideas in isolation with playgrounds, checking visual changes with previews, and driving the simulator through a new feature called Device Hub. And Device Hub is really worth looking at on its own because it pulls physical device management, simulators, resizing, and interactive app testing into one workspace. So the agenting side runs on models from Anthropic, Google, and OpenAI. And Xcode now supports the ModelContext protocol an agent-client protocol so you can wire it inside or outside tools and agents. So obviously this is cool because Xcode 27 coding agent can write and run their own tests, check previews, and drive the simulator through the new device hub. And so if you're an Apple platform QA person, some of your test running is now moving into the agent loop. All right, next up, are you tired of spending time on brittle locators? If so, this next one is for you. This is by Gurvinder, who released OpenQA, an open-source AI agent harness for browser test automation. And on LinkedIn, you can see his pitch starts from the frustration of a lot of us. Recognize QA engineers spending most of our time babysitting brittle selectors and step definitions. So his answer is a new open-source solution he created called OpenQA, which allows you to write plain BDD feature files, a scenario that says navigate to a page and a to-do item, and then confirms the item shows up and the AI agent does the rest. So it navigates by intent, not by CSS locators. So when the interface gets refactored, the agent adopts on its own. Teams manage the feature files and nothing else. Every run still produces CI-grade evidence, HTML reports, trace viewer, and screenshots. And it works with Playwright BDD, Cucumber.js, and any major LLM including Claude, OpenAI, and Gemini. He calls it the quality engineering response to the vibe coding era. And another open source solution that caught my attention I found on LinkedIn. It's by Karishma. Who created a new solution called Astra built as a QA automation ecosystem? So this looks like it comes in two parts. The first is a Playwright and Python test framework built on PyTest, the Page Object Model, and a lot of reporting. The second part I think is what testers want to look at, and that is an AI agent that reads test cases from Jira, Azure DevOps, Excel, or plain text and generates PyTest scripts. What sets it apart is how it fights hallucination. So instead of a one-shot generation that invents method names, the agent uses RAG to index your real codebase with local embeddings, then runs a GPT-4 tool loop that has a verify actual page object methods and fixtures before it writes a line of code. So the generated code gets syntax checked, and if two pieces of retrieval context conflict, generation aborts and flags a human. So once tests are written, it updates the Excel tracker to mark them automated, but definitely something worth checking out if you wanna learn more about how to use RAG and AI agents in testing. Next up is this week's webinar of the week. This week's webinar is all about using AI to close the gap between AI-written code and human-tested code, which is happening this Wednesday. Even if you can't make it, make sure to register 'cause we send out the recording and you can also watch the replay. And they told me here's the problem they're gonna dig into. Most testers know what AI testing should deliver. Well, only 16% of QA teams have actually adopted it. Even though 3 out of 4 call it strategic. The argument is that bolting a general purpose AI assistant onto your workflow gives you brittle throwaway results you end up rebuilding every quarter. But Ole, CTO of TestGuild and the creator of SoapUI that you probably have all used back in the day, walks you through TestGuild AI, the native AI layer of the platform with live demos. So you'll see autonomous agents that pick what to run before execution, then triage failures, and then separates flaky tests from real bugs after it. He also covers generating runnable tests from plain language intent in the framework your team already uses and how to expose test context to any AI tool through the TestGuild MCP server. So I highly recommend you register now using that link down below. Hope to see you there. All right, we talked a lot about AI agents, but how do you evaluate them? Well, you might wanna check this out. It's Dynatrace has just released DT-Evals, which is an open source command line tool for evaluating the quality of LLMs and AI agents using real production traces. This is by Christoph, a principal product manager at Dynatrace, which frames the problem this way. AI apps fail differently than regular software. They can respond fast with no errors and still hand back an answer that's inaccurate, ungrounded, or even worse, unsafe. But DTEval scores these responses with an LLM judge across dimensions like relevance, faithfulness, hallucination, toxicity, bias, P2 leaks, and prompt ejection. You can run it in CI mode where it emits machine-readable outputs and fails the pipeline when a quality threshold is breached. So your AI checks sit in the same release process as your latency and error gates. And this last article really brought me back, back in the day when I used to have to do performance testing with mainframes and people are still doing this now. And here's a case study in how it was done recently. All right, this is by Adam, a software test engineer on z/OS at IBM, and he's gotten k6, the open source load testing tool, to run natively on IBM z/ mainframes. This is important because did you know mainframes still handle about 90% of credit card transactions worldwide, and the world's top 100 banks still run on IBM z/OS? And this talks about how IBM's existing load tools, TPNS and WSIM, live deep in the platform while k6 brings test as code, JavaScript, CI triggers, and Grafana dashboards modern teams already know. So the catch was that k6 didn't run on IBM z/OS architecture, but that didn't stop Adam because he goes over how he forked k6, hit two build errors, and fixed both with one-line code changes that were merged upstream, making k6 now installable through kopen, the package manager from z/OS. All right, and for links of everything value we cover in this news episode, head over to all those links down below. So that's it for this episode of the TestGuild News Show. Once again, I'm Joe, and my mission is to help you succeed in creating end-to-end full-stack pipeline automation awesomeness. And as always, test everything and keep the good. Cheers.