Automation Guild Kickoff, AWS Nova, Karate v2, and AI Testing Reality and more TGNS177

In a land of testers far and widely journey seeking answers, seeking skills, seeking a better way o the Test illinovation Testing Podcast Guiding testers with our nation awesomeness. From ancient realms to modern days, they lead the way. Oh, the Testicle Ocean Testing Podcast.

Vibe of Ice why teams are getting burned by something called Vibe Syndrome even when the AI generated tests look solid on Day one what long running Test event just announced registration for its 10th annual edition? Find out in this episode of the Test Guild News show for the week of December 7th. So grab your favorite cup of coffee or tea and let's do this. First up is the biggest news item of the week, and that is registration is officially open For Automation Guild 2026, our 10th annual live online event. If you care about leveling up your automation skills in the age of AI, this is the one week you don't want to miss. We're running February 9th to the 13th and this year's lineup is one of the strongest we've ever had. And you have two great options. One is choosing a three day Functional Automation Pass or go all in with the five day complete Event Pass, which adds two extra days of non functional automation training. Either way, you get lifetime access to every session recording live Q and A replays, transcripts, chat threads, and all the resources shared by our speakers. And this isn't a one and done watch it and forget it event. It becomes a permanent learning library you could come back to all year long. Every attendee also gets to join our private Automation Guild community, and you're automatically enrolled in our Test Automation Accelerator, which gives you a brand new live training session with the Community Expert every month from March through November. So what are you going to learn this year? Well, we already have 29 high impact sessions confirmed covering the skills testers tell us they need most right now, like Resilient Test Design, AI Driven Automation with Playwright Cursor Maturity Models, Performance Testing with AI Testing, LLMs debugging, CI failures, API testing, Best practices, Quality strategy, agentic AI workflows, practical chatbot testing, and how to navigate the new responsibilities emerging in quality engineering. And right now is the best time to register. Especially if you're looking for that special something for that automation geek in your life. Definitely register now using that link down below. Hope to see you there. So if you're into playwright, you probably heard of this framework. If not, Yarn has just announced the Release of Serenity JS version 3.37, which introduces a more flexible approach to composing playwright fixtures, and this release addresses fixture composition challenges and playwright tests by allowing teams to combine multiple base tests including component test runners, add custom abilities to actors without boilerplate code, and build multi layer test suites that blend UI and API, visual and component testing. And according to this announcement, the new API provides cleaner test setup, clearer intent and more scalable structure as test suites grow. And the full announcement with examples is also available@SerenityJS.org or using that special link down below. All right, next up is another article that caught my attention on LinkedIn posted by Scott at Agile Labs and he posts how AWS has just released Amazon Nova act for general availability. What is it? Let's check it out. So this is a new service designed to help testers and developers build and manage AI agents that automate browser based workflows. And according to the authors of this post, Nova act delivers over 90% task reliability at scale while offering what they describe as the fastest time to value compared to other AI frameworks. Novaact addresses what AWS identified as a core challenge in building reliable browser automation at enterprise scale. The service is powered by a custom Amazon Nova 2 Lite model and includes capabilities for web quality assurance, testing, data entry, data extraction and checkout workflows. AWS states that most models today are trained in isolation, separate from the orchestrators and agents that actually execute tasks which reduce reliability. The difference here is Nova act uses reinforcement learning while agents run inside custom synthetic environments called web gyms that simulate real world user interactions. This vertical integration across the model orchestrates creative tools and SDK all trained together is what AWS claims unlock higher completion rates. Really cool development you definitely should check out. Let me know your thoughts in the comments down below. All right, another article that caught My attention on LinkedIn is by Peter Thomas, the creator of Karate Labs. They just announced Karate version two. So this is a complete ground up rewrite of the framework after eight years and millions of tests run around the world. In their December 2 announcement, they made it clear this isn't because the original Karate was broken. It's because they learned a ton of how teams actually use the tool and they want to prepare it for a future where LLM coding agents play a major role in automation. The rewrite focuses on solving three big issues. First, teams that weren't using Java still had to install it just to run Karate standalone version, which created friction, especially inside enterprise wide lockdown desktops. Second, the team says LLM thrives with clean discoverable command line interfaces and scriptable APIs, so developer tools need to be agent friendly going forward. And third, after eight years of feature edge cases and architectural workarounds. They hit limits that simply couldn't be patched over. So Karate 2 aims to aim to remove all this friction I've always been impressed with Peter has accomplished with karate and now it looks it's taking it to the next level. If you haven't tried karate yet, I highly recommend you do Once again using the links down below. Once again, I'm always scrolling through LinkedIn so here's another article, this one by Oliver, who's published a practical guide on testing AI models through the software development lifecycle. And in this post he talks about when code or user experiences are generated by a machine, the risk and cost of failure increases significantly, shifting the problem from code quality to trust. So testing AI models differs from traditional deterministic software because models change their behavior relying on shifting data and sometimes cannot explain their own decisions. Oliver frames the approach around the Machine Learning Operation Lifecycle or mlops, addressing each stage with specific testing strategies. So for example, for data validation and preparation, he recommends treating data pipelines and schemas as code with data unit tests and historical comparison. For data drift, he also introduces the concept of test slicing to audit data for bias by creating slices based on sensitive attributes like age, gender or location to review composition before training begins. And for integration and deployment stage, he establishes reproducibility as a gold standard and and qas must verify that accuracy and recall metrics remain within the range of the original model. And he also covers post development, identifying model drifts detection as the biggest immediate threat to AI quality. All right, speaking of AI, OWASP has just released a version of their AI testing guide. Once again, I found this on LinkedIn. This was posted by Jeff Williams and this is actually being called one of the first open community driven standards for trustworthiness testing of AI systems. And it also describes and explains that AI testing differs fundamentally from traditional software testing because AI models learn, adapt, generalize and fail in non deterministic ways. The guide addresses risks that extend beyond conventional security testing. And obviously it's owasp, so it's open source and they always welcome contributions from the community. And you don't want to miss our Webinar of the Week, our last webinar of the year at the Automation Guild. We're going to go over how to transform qe with the 3R AI framework, reduce, realize and reinvent. And we're going to be joined by the folks at T Sigma AI. I'm going to go over and show how to leverage AI Rag to intelligently validate and enhance requirements and also going to go over their platform and how they're actually able to do all this as well. If you haven't registered yet, I highly recommend you do using that link down below. Hope to see you there. Next up is all about the Vibe syndrome. What is it? Well, Raj has asked the question in this article, are you creating Vibe syndrome in your test automation? This actually stands for what he calls very intelligent but ephemeral output that works today but fails tomorrow. And the problem Raj identifies is straightforward. Junior testers can now use AI tools like ChatGPT and Copilot, generate test cases and build automation frameworks in days. But three months later, when these frameworks start collapsing, nobody can explain why the code looks professional. Tests pass initially, but the foundation is missing. Raj breaks down what happens when teams rely on AI without understanding core principles? Teams use AI to scaffold frameworks without knowing why they choose one design pattern over another. When the UI architecture changes or scaling becomes necessary, the entire structure falls apart. And he goes over patterns. He's seen across three examples where code has looked good but lacked architectural vision, design pattern mastery and clean code principles. And he says the solution isn't avoiding AI. He argues that testers need to build their foundation first in areas like software architecture, design patterns, clean code, domain knowledge and tools, agnostic design. And he also introduces an alternative to Vibe, which he calls vice, which stands for Visionary Intelligence and contextual expertise. As I mentioned at the beginning, if you want to learn more about automation in the age of AI like we covered in some of these news articles, you don't want to miss this year's Automation Guild that you definitely should register for right now. All right, for links of Everything value we covered in this news episode, headed over to the links in the comment down below. So that's it for this episode of the Test Skill New show. I'm Joe. Thank you. My mission is help you succeed in creating end to end full stack pipeline automation awesomeness and as always, test everything and keep the good. Cheers.