WebdriverIO, AI Agents, and 7 Questions to Ask Before Trusting AI Testing

Faster AI test runs or cheaper ones? Which one should you actually pick? Is your WebDriverIO suite ready for the biggest change the project has made in 15 years? And are you still copying CI jobs every time you add a test environment? Find out more in this episode of the Test Guild News Show for the week of October 5th. So grab your favorite cup of coffee or tea and let's do this. Have you ever wondered how to grade your AI? If so, I have an awesome resource for you. We're running an awesome webinar next week on Who Grades the AI? 7 Questions to Ask Before You Trust AI Testing. So if you're evaluating AI testing tools, you probably run into this. Terms like agentic, autonomous, self-healing can describe different capabilities, yet they all look similar to a polished demo. That's the focus of our upcoming webinar with Dinesh, who's the co-founder of Autotester. He's gonna walk through 7 questions testers, QA leaders, and developers should ask, plus the evidence a credible answer should produce beyond a vendor simply saying yes. And he told me he's gonna break down the difference between AI assistants, test generation, test platforms, and quality systems, and where each one stops. And he also covers 2 questions many evaluations miss. Does the system actually learn? And who checks the quality of its work? And he's gonna actually demonstrate Autotester live, showing things like coverage generated from a requirements document, the resulting test code, a healing record, and auditor scores from 0 to 100 backed by evidence. Hope to see you there. It's happening next week. You can register down below. And speaking of checking AI's work, I just saw this announcement from WebDriverIO all about a major new release. So Christian just announced that WebDriverIO version 10 is out, and he calls it one of the biggest changes in direction the project has made in 15 years. So this is all around how he says more code is now written by coding agents, Nobody can review every line of it, so trust has to come from verification. How's that done? Well, let's dive into the latest release. Well, the biggest feature seems to be WDO Session, a CLI that lets an agent drive a browser, phone, or desktop app step by step, then export what worked as a regular test. There's also browser.act. This allows you to write a step as intent. A model works it out once, and after that, it replays as plain code without spending any more tokens. This is a big feature, I think. And if the UI changes, a healing step only counts when it produces the same network requests, navigation, and page change as before. Also has some new dev tool traces to show you and your agent why a run failed. And as always, it runs on real Chrome, Firefox, Edge, and Safari, plus iOS, Android, Electron, Tauri, and native desktop apps. Sounds awesome, right? But a quick heads up before you upgrade. Version 10 requires Node.js 22.19 and Appium 3. This sounds like an awesome addition. I think every tester should check out using that link down below. Let me know how it works for you. And speaking of keeping suites manageable, here's how one team runs a single Playwright suite across every environment without duplicating a single test. This is by Jay. He talks about running the same Playwright regression suite across multiple environments usually means duplicate tests or a pile of copied CI jobs. But, but Jay lays out how his team avoids both. while testing the SalaX secure messaging web client with Playwright and GitLab CI. And in his words, his goal is one regression suite in multiple environments. So application URLs live in environment variables and a single variable tells Playwright which one to use. So switching environments locally means changing one line with no test code changes. But differences— the team follows a single rule: if one test differs, tag it. If a whole folder belongs to an environment, exclude it with test ignore settings. And all that filtering sits in the Playwright config, which keeps environments checks out of the test. On the CI side, GitLab's parallel matrix turns one job definition into parallel runs for each environment, all executing the same command. A reporting job then combines the results to one daily summary showing pass, failed, and excluded tests per environment. Adding a new environment is just one matrix entry. So if you're stuck maintaining separate suites or CI jobs per environment, This is a really good approach you should definitely check out and learn more as well. All right, so as you know, I love performance testing. So this next article caught my attention on LinkedIn. It's by Petros talking all about a new release of an open source npm package. If you don't know, Petros is a test automation leader and Cypress ambassador. He's released Cypress Performance UI, a free open source npm package that brings performance data into your Cypress runs. He says he built it because he wanted something lightweight that ties functional end-to-end test browser performance data and visual reporting into one workflow. And what's really cool, as you can see here, is the package collects native browser metrics and generates visual HTML reports. And according to the README, no Lighthouse is required. You install it as a dev dependency, then register it with a single import in your support file or with an explicit registration call. It adds 3 commands as well. One collects page load metrics from windows.performance, another wraps the W3C Performance API timing values, and the third writes an HTML page load report. The timing is focused on including response start, DOM content loaded, load event end, and first contentful paint. And staying with Cypress, as I was creating my new show, I just saw this. It was just released like an hour ago. It's by David Ingram, a senior SDET and Cypress ambassador. He makes a simple argument: if your test suite has users, you should treat it like your product. So those users include developers opening a PR, the QA engineer triaging a nightly failure, And more and more, the AI agent who actually relies on your test to tell it whether it broke something. He points to published numbers to show how low the bar often is. Atlassian reported flakiness caused up to 21% of master build failures in its Jira frontend repo, with reruns costing more than 150,000 developer hours a year. And Dave also believes as well, the test suite should produce standards like reliability, usability, performance, and maintainability. And reliability comes first. Tests should wait on real signals instead of hardcoded sleeps, and a flaky test is a bug that gets an owner and a ticket. And his own results back it up. As the only QA engineer at his company, he runs about 650 end-to-end tests on every PR in roughly 12 minutes with a flaky rate under 1%. Also, another must-know open source tool made an announcement on LinkedIn, and this is from Alex, who just announced how Lumium 0.23 has a new feature you need to know more about. So this version 0.23 updates tackles a trade-off a lot of teams pairing AI with coding assistants will recognize: speed versus cost. The headline is a new direct MCP mode. By default, Illuminium works as an agent using its own LLM requested to translate natural language instructions into tool calls, which keeps costs down. Direct mode skips these requests and hands Illuminium's toolset straight to your coding assistant. And according to the release notes, test runs faster but burn more tokens and cost more, roughly 2x difference across the board. Alex's name for it obviously is token maxing. Synchronization also got faster. Illumium waits for the app to settle before capturing a state, and the team cut the starting headroom from 500 milliseconds to 50, reducing overall sync times by 10x. And rounding it out is there's a new Maestro driver for iOS and Android. If you haven't checked out this open source project yet, I highly recommend you do. And once again, you can find it down below. How does chaos engineering fit in with AI? By Arun, talks about chaos engineering in the AI era. If you don't know, Arun's a senior performance and resilience testing architect, and he says chaos engineering has moved from niche experimentation to maintained practice, and AI is a big reason why. He writes how AI can now generate chaos scenarios across application, data, and platform resilience, work that once took deep expertise and manual effort. But Arun says the hottest pot is still validation. In a distributed microservices architecture, AI can suggest killing pods for one service or several at once. Whether that scenario actually applies depends on details like multi-AZ setup, pod affinity, and he says this still requires and takes human judgment. And finally, a startup less than 6 months old says its open-source eugenic testing framework hit number 1 on GitHub. What is it? Let's check it out. And this post by Simon, the co-founder and CEO of TestArmy, he says the team open sourced an end-to-end testing framework called E2E 5 days ago, and the reaction really caught them off guard. And according to Simon, the project hit number 1 on GitHub trending. npm is already showing 40,000 weekly downloads, and several projects have moved their full test suites over. So what is it? Well, let's check it out. So E2E covers web and mobile apps. You describe a goal in natural language and an agent works through the app to complete it. For testers, the useful part is that you can mix these agent steps with regular locators and assertions in the same test so you can check exact results. When a later assertion verifies an agent step, its actions are recorded and replayed on the next run with no model calls until the app changes. Tests with no agent steps don't need a model at all, so you can bring your own subscription API key or local model. And the docs include migration guides from Playwright, Cypress, Selenium Detox, and Maestro. Whoops, I almost forgot. Also, the call for speakers for our 11th Annual Automation Guild is going to close this week. So if you haven't already, here's your last chance to submit your idea and send it our way so we can start choosing sessions for our upcoming Automation Guild in February. All right, so that's it for this episode of the Test Guild News Show. I'm Joe. My mission is to help you succeed in creating end-to-end full-stack pipeline automation AI awesomeness. And as always, test everything and keep the good. Cheers.