Sauce Labs AURA, Playwright Tracing, Loop Engineering and More

Did you know that more than half of enterprises admit they ship code with known critical defects? Would you let an AI agent block your release build? And what happens when the AI passes a test by breaking into the place where the answers are kept? Find out more in this episode of the TestGuild News Show for the week of July 27th. So grab your favorite cup of coffee or tea and let's do this. First up is all about one of the first sponsors I ever had from the TestGuild., and that is Sauce Labs. So Sauce Labs has launched Aura and it's calling it an AI unified release assurance platform. Their CEO Prince frames the problem as a verification bottleneck no team can staff its way out of. And the research behind it is the real story. Sauce Labs cites data showing developers now produce 741% more code while release velocity has risen less than 20%. So what is it actually? Well, Aura is Sauce Labs' new AI unified release assurance platform And the keyword here really is unified. This is not a new product you have to buy. If you're already a Sauce Labs customer, Aura gathers what you've been using, device testing, visual checks, accessibility, app distribution, analytics, crash and error reporting. It connects them all into one continuous learning system. It also pulls intent out of requirements and pull requests, generates and runs the tests, correlates failures against nearly two decades of execution data, then feeds production errors back into your coverage. And alongside the launch, the company published a Wakefield Research study of over 400 US executives and engineering leaders. 53% admit they routinely ship code with known critical issues, and 80% have already traced a production incident to AI-generated code. And 65% say their worst quality incidents last year cost half a million dollars or more. So Source Labs says enterprises running Aura see 90% fewer production instances, 40% faster releases, and 38% of engineering capacity reclaimed. So I think this is exactly the type of platform we need in the age of AI. And I know your leadership is going to start asking you how you verify AI-generated code. You want that answer to be ready. So I highly recommend you check out Sauce Labs' new platform using that link down below. All right. Next up is about a tool I've been hearing more and more about, and that is Bruno. They just shipped a new version 4.0 on July 23rd. It's flagged as a major release with breaking changes, so make sure you read that page down below before you upgrade. And the headline is AI you control. Bruno added a new bring your own key assistant that works with OpenAI compatibility providers, a sidebar that reads your active tab, and AI help for docs, scripts, and tests. There's also a security tab showing what context gets shared. So what happens when you combine OpenTelemetry with Playwright? Well, let's check it out. So Oliver announced the release of Playwright OpenTelemetry, which is an open source Apache 2.0 project from NForm. And the problem he opens with on his LinkedIn post probably sounds familiar. A test clicks, great project, the spinner hangs and the test times out. But you open the Playwright trace and see the click, the spinner and the network requests and a final screenshot. All accurate, but none of it tells you why it failed. The answer sits past the network boundary and finding it means lining up timestamps across a few tools and guessing which log line belongs to your test. This makes the test itself the root of a distributed trace. A reporter turns test steps, hooks, and actions into spans. A fixture injects a W3C transparent header into browser requests. So any service already running OpenTelemetry attaches its own spans underneath. There's also a trace view that syncs Playwright screenshots, filmstrips against the timeline. And to use this, all you need to do is make two changes: add the reporter to your Playwright config and import the instrumented fixture in place of the standard one. So production is way too late to discover bad telemetry. So that's why you should start using a tool like this right now. You can check out more about it using those links down below. All right, so the original creator of Puppeteer and also one of the main contributors to Playwright at Microsoft just released something new I think you need to know more about. So Andre just announced on LinkedIn that Flakiness.io, a test analytics service for GitHub, is now generally available after 2.5 years of development. And the problem it targets is one you know. A pull request shows some checks were not successful and now you're opening workflows, scrolling terminal logs to find the failing tests and checking recent commits on main just to get a baseline. But Flakiness.io annotates every test results with its commit and cross-references against a repository history so you can flag a new failure as a regression and report that back to GitHub as a check. Clicking through, you get the actual failure plus screenshot videos and Playwright traces. Andrei notes most services charge per upload results around $40 per 10,000. He points out that the Playwright team was generating 280,000 results per commit back in 2023, and would have run over $1,000 per commit. So this is a really cool solution because per-result pricing penalizes testing because adding one or more configuration raises your bill. But tools like flakeness.io are starting to charge for storage instead. And this is another tool I think is definitely worth checking out as well. And yet another tool came on my radar from LinkedIn that I think is going to help you answer the question of whether you let an AI agent block a release or not. So the co-founder and CEO of Agenti QA says the company's GitHub Action is now live on the GitHub Marketplace. And one of the main pain points he's trying to address, say you point a non-deterministic agent at a CI pipeline and you're one transient timeout away from a red build. Once that happens, developers disable the gate and never trust it again. So the action is built around a fixed exit code contract. 1 is a planned failure, 2 is a usage or config error, 3 is a retryable infra error. Under the default policy, plan failures and config errors fail the job while a transient disconnect warns a team without blocking the release. And two of the details really stand out here is a run that executes zero plans is bucketed as a config error, never a pass. An exit zero only counts if a valid JSON envelope backs it, so polluted output cannot fake a green build. And yet another new tool came my way via LinkedIn, but this one is more for mobile folks. And for folks that are really worried about security findings. So this is from Vishal, a security engineer at PhonePe announced that the company has open sourced Dorfin, an automated dynamic application security testing framework for Android. And this is fairly new, it already has 132 stars on GitHub. So how does it work is you give it an APK in a container device or emulator, it decompiles the app, traces taint flows from attacker-controlled input to the vulnerability sink, and then tries to prove the findings is real. The check covers intent redirection, WebView issues, context provider path traversal, hard-coded secrets, unprotected exported components, and more. The part testers really should pay attention to is the cool verification step. For findings an LLM triages as true positives, Dolphin generates a proof of concept ADB command and runs it on your device. Default mode is interactive. So each command appears in a review box and nothing executes until you approve it. There's also an auto-approve and skip-verify flags. So this is cool because I think it really helps close the gap between a static finding and a demonstrated exploit. So you definitely check it out once again using those links down below. All right, let's talk about loops. This is from Raj. I haven't heard from him in a while, so really was excited to see this new post he created called Prompt Agent Loop. In this post, he separates 3 ways of working with AI by the unit of work. Prompt engineering, where the unit is a single turn. Agentic work, where the unit is really a task in the agent that reads the repo, runs your suite, and picks its own next step. And loop engineering, where the unit is the system and you design what finds the work, what independently checks it, and what conditions mean stop. His one-line version: prompting is asking, agentic is delegating, and loop engineering is designing the thing that delegates. And as a tester, what really popped out to me is that inside a loop, your suite stops being a report a human reads and instead becomes a controlled input a machine steers by. Nobody compensates for a bad signal. The agent sees red, believes it, and changes working code to make the red go away. And he also mentions that a 60% suite that is never wrong beats an 85% suite that lies twice a week. And this is just another, I think, example of how signaling reliability now outranks coverage percentage that a lot of people have been focusing on. And assertion outranks execution because agents are already good at driving a browser and really bad at knowing what should have happened. So next up is what I think is one of the most disturbing news items we've had on the show in a while. And this came on my radar from, once again, LinkedIn from Adam. And so what happened is OpenAI was benchmarking how good its models are at hacking with the safety blocks turned off on purpose. The sandbox had no intent, just a package proxy. The models found a zero-day in that proxy, escaped, moved laterally until they hit a machine with internet access, then figured out Hugging Face probably hosted the answer to the test. They broke into Hugging Face production and stole the test solution out of the database. And so just think about that. The system under test optimized for passing the test, not for doing the actual work. And that's something we need to be more and more aware of as we dive into AI in our testing processes. All right, for links to everything we covered in this news episode, head on over to those links in the comments down below. So that's it for this episode of the TestGuild News Show. I'm JoeBot. My mission is to help you succeed in creating end-to-end full-stack pipeline automation awesomeness. And as always, test everything and keep the good. Cheers.