5 AI Projects Every SDET Should Be Building, Jev , AI Agent Hacks and more

What are 5 projects every SDA should build right now? Google just confirmed its AI model hacked into 3 real companies during a routine test. And what new AI model claims it can make decisions up to 200 times faster than existing LLMs? Find out on this episode of the Test Guild News Show for the week of September 20th. So grab your favorite cup of coffee or tea and let's do this. All right. Quick Test Guild update before we get into it. We just opened registration for 2 more Test Guild IRL events. And we're heading to Austin and Dallas the first week of November. If you want to connect with other testers in person, grab your spot now at testguild.com/irl. You don't want to miss it. Hope to see you there. So if you're like me, you might be tired of fighting with legacy Windows apps that no automation tool really can touch or work correctly. If that's you as well, you don't want to miss this next story. So this comes your way via Andy Knight, who commented on Paul's post all about how Paul has just released Automancer. So Automancer is a C# engine and command line tool for automating Windows desktop applications. Paul built it for years of stitching together tool frameworks and PowerShell scripts for legacy window apps, knowing that the industry largely moved automation efforts to the web with Selenium and Playwright, while Microsoft's own WinApp driver has gone largely unmaintained. So Automancer aims to fill that gap. It resolves locators through a fallback chain covering modern UI automations, older interop layers and raw Win32 enumerations so testers could focus on picking locators rather than debugging why one fails. And it also uses the pollUntilTrue async pattern popularized by Playwright, bringing that approach to Windows applications for the first time, according to Paul. Also, every CLI command is structured as a simple text-in, text-out operation, a design he says makes it compatible with how large language model tools calls already work, positioning it for AI-driven RPA-style automation. And Paul is asking testers who maintain Windows desktop app tests to try it against their toughest application and report where it breaks. Also, the project is completely open source and available on GitHub, and you can try it for yourself using that link down below. So speaking about building your own tool like Paul did, here's a roadmap for exactly what to build next if you want to grow your AI skills as a tester. And this is from Siddharth, who's a friend of the Automation Guild. He's spoken many times there. He's an SDAT, He published this on LinkedIn outlining 5 AI projects he recommends test automation engineers build to help grow their skills. The first is a test case generator that takes a user story or API spec and produces functional, negative, and edge case tests with a review queue instead of a one-shot output. The second is an AI-native Playwright framework that moves from requirements to test scenarios to locators to execution with automatic locator validation and failure-aware retries. The third he recommends is an API testing agent that reads an OpenAPI spec, generates and executes tests, and flags contract violations. Fourth is a test failure root cause analysis agent that traces a failure through stack traces, logs, and recent code changes and can connect to things like Jira, GitHub, and Slack. And fifth is an autonomous system that continuously analyzes the test suite for flaky tests, duplicate coverage, and low-value tests, tracking metrics like flaky and me time to repair. So from beginner to advanced, I think these are 5 areas you should definitely focus in on to help hone in your skills. You can find out more about it using that link once again down below. All right. Probably the biggest story last week is all about Jev. So what is it? Well, TypeSafe AI, founded by former OpenAI researcher Diego Almeida, has launched what it calls the first System 1 model, a new class of AI built for automation rather than conversation. The model named Jev skips text generation entirely and instead outputs structured type decisions a software can use directly, paired with calibrated confidence scores on every answer. And according to the company, Jev matches existing large language models on decision-making task while running 40 to 200 times faster with end-to-end responses between 700 and 500 milliseconds. And pricing is also sharply lower at roughly 4 cents per million input tokens, with output-free. Typesafe says the architecture makes type errors mathematically impossible and eliminates hallucination on structured outputs. And Jev is available now. I was waiting on a waitlist, but they just made it publicly available to everyone. So I highly recommend you check it yourself. Leave a comment, let me know what you've used it for already. So I knew it wouldn't take long, but dang, it was quick. Someone put it into play already using Playwright test using Jev. Who was it? Let's check it out. Once again, Jason Auburn's on the forefront. He's already put this to the test. So Jason connected Typesafe's AI Jev model to a Playwright browser session to see how it performs inside an actual testing loop. So he built a small storefront with intentional defects and let Jev choose from a set of predefined probes, like applying a coupon twice or entering a negative quantity. While Playwright executed the actions, after each probe, a second Jev call judged whether the results violated a stated requirement. The system made 2 calls per probe, one to pick the next action and one to judge the outcome, each returning really quick. He said within 120 to 175 milliseconds. Across 63 probes and 126 API calls, Jev flagged 21 cases tied to 5 known defect types, including compounding discounts and accepting negative quantities. But Jason obviously found real limits. One valid checkout scored nearly even between a bug and no bug, a distinction the interface didn't surface. A separate run had earlier produced a false flag on a normal action. And if you want to learn how to do things like Jason has been doing, he's actually released an awesome resource for this as well. It's a new book. All about how AI tests software. It also carries a foreword from James Whitaker, who's a former engineer director at Google, who calls it among the most important books for testers working in an era of non-deterministic AI-built software. Jason structured the book in 9 parts, moving from AI checks, which he frames as the highest return, through interactive AI tests, contextual test thinking, and what he calls the AI test harness. And later sections cover human expertise inside the AI testing loop. And if you'd like a free copy, all you need to do is click my link down below and then comment in that post that I have linked there. All right. I'm always looking for real test cases, and this one is from Anthropic's CI redesign. So Anthropic engineer Sachin details how agentic coding pushed his company's CI system past its limits. As Claude began authoring the bulk of code and taking on more and more PR reviews, CI job volume rose 25 times over 6 months, driven partially by a tenfold increase in the number of tests across the codebase. And because of this, the team's test impact analysis service, which decided which tests run on each change, started falling behind under the load. They applied 3 successive patches, a big machine, shared processing, and daily restarts that bought 70 days, then 29 days, then less than a day before failing again. While the listener component fell too far behind, it missed recording test results, which meant the selection service worked from stale data and sometimes ran tests that were already flaky rather than the ones that mattered. The team eventually redesigned the service around a stateless, horizontal, scalable architecture using an in-memory data store, a build that he said took 3 weeks with Claude's help versus what he estimates would've taken a quarter of a year earlier. And the key takeaway here is that he advises engineering teams to plan for 10 to 10 times their expected scale from the start rather than patching incrementally. All right, AI is back in the news for hacking. Google confirmed its Gemini AI model gained unauthorized access to 3 outside companies during a cybersecurity capabilities test in May. The incident occurred during an evaluation run by a third-party AI security firm, Irregular, when Gemini was assigned a fictional company to attack that happened to share a name with a real one. And according to Heather Atkins, Google's Vice President of Security Engineering, the model found public information online and guessed credentials to access systems it believed were part of the test. In each of the 3 cases, Gemini stopped once it realized it had accessed a real company rather than a simulated one. The company obviously said the event didn't meet its bar of AI misalignment, attributing them instead to the model mistakenly believing it was operating inside a closed test environment. And once again, Jason can be an awesome resource for a lot of these new AI escapes we've been hearing about with this awesome breakdown in this video from a Microsoft engineer. All right, for links to everything of value we covered in this news episode, head on over to those links in the comment Down below. So that's it for this episode of the Test Guild News Show. I'm Joe. My mission is to help you succeed in creating end-to-end full-stack AI pipeline automation awesomeness. And as always, test everything and keep the good. Cheers.