Test Automation Crisis,AI Takes Over Performance Testing and more TGNS175

Is your automation actually making things worse? Could AI agents replace your entire performance testing workflow and the first AI orchestrated cyber attack just got blocked. What does that mean for AI safety? Find out on this episode of the Test Skill News show for the week of November 16th. So grab your favorite cup of coffee and tea and let's do this. Hey, before we get into the news, I want to thank this week's sponsor, Zap Test AI, an AI driven platform that can help you supercharge your automation efforts. It's really cool because their intelligent copilot generates optimized code snippets while their plan studio can help you effortlessly streamline your test case management. And what's even better is you can experience the power of AI in action with their risk free 6 month proof of concept featuring a dedicated ZAP expert at no upfront costs unlock unparalleled efficiency and ROI in your testing process. Don't wait. Schedule a demo now and see how it can help you improve your test automation efforts using the link download down below. All right, next article is by Titus, who published a new article that argues that test automation is failing across the industry not because teams can't write or execute tests, but because test results don't provide clear actual information. And titus points to 2019 data from Source Labs showing that three out of four organizations pass rates below 90%. That suggests that most teams lack the signal quality needed to understand their test failures. When test fails, teams face an impossible choice delay releases for timely investigations or ignore failures in ship anyway. And the core problem he identified is that the industry obsesses over test authoring and execution speed, while the real bottleneck is evaluating and resolution. He goes on to say that test automation has six stages and there's no value until all six are complete. A test suite hasn't finished until every failure is understood. Yet most teams are drowning in failures they can't diagnose. Practices like automatic retries and self healing may appear to improve pass rates, but they actually hide critical information rather than strengthen the signal. And it also challenges common solutions that many organizations use for automation disappointments like switching from one tool to another or using AI powered tools, which won't fix the underlining issues if teams still can't determine whether the failure indicates real bugs or false positives. Also, shifting testing entirely to developers hasn't improved signal quality, and BDD tools have added overhead without making test maintenance easier. All right, now, speaking of making testing easier instead of harder, here's a tool that I just learned about. Even though this was published maybe two weeks ago, and this by Hersheet, who announced that he built a lightweight visual K6 script builder designed to help engineers and QA teams get started with performance testing quickly. And he said he created the tool after observing teams struggling with the initial learning curve of K6 code and we're missing boilerplate, incorrect headers or confused pacing can sometimes consume hours of setup time for newbies. This tool provides a user interface for adding API requests with method path headers and body parameters, and allows users to chain them together. It supports functional checks per request, including status and JSON validations, manages environment variables, staging and production environments, and configures think time with either fix or random settings to simulate real user behavior. It includes a live code preview in the editor with export code and download functionality. It it describes this tool as teaching first, explaining that teams can prototype flows such as product browse or checkout without writing even a line of code. The load generates a working script that users could then iterate on as they dive deeper into K6 features. Another tool that's been around for a while just got a critical update. Geico Adsic just announced a few days ago that Bug Magnet is now available as a command for AI coded assistance, including claude code and cursor. According to the announcement, Bug Magnet can assist with writing characterization tests, which are tests that document existing behavior to make future changes easier. It supports test coverage and gap analysis, helping identify areas that lack adequate testing. The tool can also analyze undocumented or unclear behavior and state transitions within systems. It provides support for exploratory testing and advanced coverage analysis. And finally, it applies common testing heuristics to detect bug and feature gaps. And the tool is also available on GitHub that you can find in the link down below. So, as you could probably tell, I'm a bibliomaniac, so this next LinkedIn post caught my attention, and this by Sai, who just announced he wrote a new book called Appium Insights Strategies for Successful Platform Agnostic Test Automation, and it's now available. If you don't know, Sai is the director at Lambda Test and Appium member and contributor. He describes this book as drawing from nearly a decade of working on mobile automation and open source projects, and by looking at the table of contents you see the book goes beyond basic Appium and goes into deep dives into Appium things like architecture, internals, troubleshooting and practical scenarios drawn from real project experience. In his LinkedIn post, he calls it really a guide for readers who want to understand how Appium works under the hood and how to build more robust platform agnostic automation strategies. Definitely a must read. Definitely check it out for sure. Next up is our webinar of the week. So at the Test Guild we're going to be running a webinar that takes place this Thursday that examines how agentic AI is being being applied to functional testing. We'll be joined by Don Jackson, who's a perforce technical evangelist who's going to focus on addressing issues with traditional functional testing methods that struggle to keep pace with modern application development, including slow test creation, resource intensive maintenance and challenges achieving cross platform coverage. And he also said attendees are going to learn how agentic AI creates, executes and maintains tests strategies for building cross platform tests using natural language methods to reduce script maintenance and testing time, and how Perfecto AI addresses complex testing challenges at enterprise scale. Definitely something you don't want to miss. You can register for it down below. Hope to see you there. So we talk about AI a lot on the new show, but how about tools that actually test AI itself? Well, this next article by Crystal talks about how Tesio is making a major bet in this space. Now it's an expanded focus on AI safety and reliability through comprehensive testing services designed to address three critical risk areas, hallucination bias and privacy threats. And according to early adapters data cited by the company, 82% of AI bugs originated from misinformation and high severity accuracy failures. So what's cool about Test IO is they are a managed outsourced testing platform that helps organizations identify and mitigate these risks before AI systems reach production. And they target specific vulnerabilities that have emerged as companies increasingly deploy AI powered features and products. And I think this really reflects a growing industry concern over AI systems reliability. As more software products incorporate AI capabilities, testing teams face new challenges that differ significantly from traditional software quality assurance. Hallucination problem where AI systems generate faults or even fabricated information has become really problematic for customer facing applications. All right, this next one might be controversial. It's all about AI driven performance testing. So Akesh talks about how AI and large language models are pushing performance testing from a reactive process to a predictive one. Instead of writing scripts, configuring load tests and finding bottlenecks late in development, he argues this whole cycle is becoming outdated. He describes how AI trained on years of defect data and logs can spot performance issues as developers now write code things like nested loops that cause exponential slowdowns, inefficient database queries, or even early signs of race conditions. The focus shifts from finding problems to preventing them. Andy raises a big problem. If AI can actually predict where systems will fail under load with high confidence, how much traditional load testing is still needed. He suggests a new workflow where every code commit triggers an AI performance review that analyzes the code dependencies, patterns from thousands of past issues. Manual testing then narrows to small sets of edge cases AI can't predict. And he looks further ahead to distributed AI agents embedded across systems, agents that simulate users, learn from real behavior, navigate UIs and API autonomously, and detect performance degradation before users notice. So I think it's really controversial because he's talking about how this moves from hands on testing to guiding these AI systems, train them to recognize bottlenecks, decide what's worth testing, and to find patterns of good performance. And according to him, he sees the role leaning towards more machine learning, data analysis and system thinking in the future. You probably heard by now, but if not, have you heard about Anthropic's AI Cyber Attack? This article goes into detail how Anthropic has announced and successfully disrupted what it calls the first reported AI orchestrated cyber campaign. The sophisticated operation used Anthropic's clawed AI to automate attacks against approximately 30 high profile organizations, including tech companies, financial institutions and government agencies. It says the attackers bypass Claude safety guardrails through clever prompting techniques. They broke attacks into small, seemingly benign technical tasks and assigned Claude a person as an employee of a legitimate cybersecurity firm conducting defensive testing. And the AI powered framework automated 80 to 90% of the attack process, identifying vulnerabilities, writing exploitation code and extracting sensitive data. Human operators only intervene at critical decision points, allowing the campaign to operate at speeds impossible for human teams to match. So really mind opening if you want to read more about it and full links of everything of value we covered in this news episode. Head on over to the links in the comment down below. So that's it for this episode of the Test Skill new show. I'm Joe. My mission is help you succeed in creating end to end full stack pipeline automation awesomeness and as always, test everything and keep the good. Cheers.