Playwright MCP vs CLI, Mutation Testing and AI Agents Flying Blind

Playwright MCP or CLI? Which one is really eating your AI budget? If you had 12,000 green unit tests, why would bugs still get through? And AI agents can ship code faster, but can you see what breaks in production? Find out in this episode of the Test Guild News Show for the week of September 28th. So grab your favorite cup of coffee or tea and let's do this. So is Playwright CLI a better choice than Playwright MCP server? This is a conversation going on on LinkedIn all over the place, but here's an article that caught my attention. It's by Irvin Dimitri. who put both to the test and shared the results on LinkedIn. He says the key difference is how page state gets delivered. Playwright MCP places a full accessibility snapshot directly in the tool result, and on complex pages, those snapshots pile up in the model's context. That drives up token use and leaves less room for reasoning and code generation. The CLI, on the other hand, returns a link or path to a snapshot file, so the agent only looks at the page data when it needs to. His benchmarks were split. For running existing tests, the CLI used 1.2 credits compared to 1.5 for MCP. But for exploratory testing to identify a critical flaw, MCP used just 0.6 credits while the CLI used 5.3. So his verdict is it depends. Well, the CLI fits perfectly for browser work inside a larger coding task involving secure files, test runs, and logs. But MCP works best for exploratory testing when the agent needs to interact with the browser. So once your test runs, where do the results go? If your answer is a CI log that nobody reads and you're a Cypress user, this next one's for you. This is Greg, who reports on InfoQ that Grafana Labs has published an approach for monitoring Cypress test suites. It converts test results into Prometheus metrics, sending them to Grafana Cloud so your results stick around after a CI job ends. It works through Cypress lifecycle hooks, so the beforeRun hook sets an identifier for the whole run and the afterSpec captures each spec's pass and fails count And duration. Cypress runs are short-lived, so the results go in the Prometheus push gateway where Grafana Alloy scrapes them and forwards them on. If you attach a GitHub Action run ID, any metric change can be traced back to the CI run that produced it. The payoff obviously is awesome trend data, that dashboard that could show a spec getting slower or a test that has started failing intermittently. Greg also does note that this does not replace Cypress Cloud or Allure. It just puts your test results next to the telemetry your engineers already use just to give you a bigger picture. So definitely check it out using that link down below. So last week I covered what I think is going to be a big, big story going forward this year, and that is Jev. And here's a write-up from QA Wolf talking more about how they see Jev being used. This is my Goran. And he points to an article how they're actually implementing Jev in a really cool use case. So one example they give is how do you test an AI assistant that never says the same thing twice, which makes it non-deterministic? Well, They explain how you can use Jev, which takes some state and a set of typed questions and returns decisions with probabilities to build 2 AI primitives for Playwright. The first is to satisfy checks meaning, so the parcel is on its way can pass a requirement that the reply confirms shipment while the we're preparing your order should fail. The second act handles navigation. If a support link moves into a menu during an A/B test, act can still reach the support composer within step and time limits you set. They also go into detail on how Playwright still verifies structured state and side effects like 201 responses and an open ticket in the right queue. If no ticket exists, the test fails no matter how convincing the assistant sounds. Uncertain judgments and evaluator errors fail the assertions too. So I just think this is one of many use cases we're gonna see coming up, popping up for Jev, especially with QA. And I'm curious to see what others are gonna have in the future. If you're a tester, I still feel like fuzzing gets no love. So this next article caught my attention. It's all about how GitHub's AI agent takes over fuzzing Grunt work. And talks about how fuzzing still needs a human in the loop. And Antonio Morales wanted to show how much of that work an AI agent could take over. So writing on the GitHub blog, he introduces the fuzzing Taskflow, an open source pipeline for C and C++ projects built on the GitHub Security Lab Taskflow agent framework. He points it to a GitHub repository and the agent finds entry points, writes harnesses, runs AFL++, reads the coverage reports, and improves the harness from there. Each round doubles its time budget. It stops once 2 iterations in a row gains less than 1% line coverage. Every crash gets its own markdown report with a verdict, such as a real vulnerability or a bug in the harness itself, along with the root cause analysis and suggested fix. He also says the fixes are marked review required because the agent does get things wrong. All right. So another big question that comes up on the show all the time is if an agent writes your code and your tests, can you trust a green build? Well, this next article tackles that. And Paul talks about how 12,000 unit tests, all green, and the bugs still got through. Paul says every line of code at Swamp is written by an agent. So are the unit tests. But then they say the user acceptance testing suite has run about 2,000 times, and 80 of those runs caught real defects in code that passed every single unit test. He says that the agent writes the function and the test from the same understanding. If it missed the requirement, the test encodes that mistake. And when a test fails during the agent loop, the fastest path to green is often rewriting the tests. He also cites a study of over 86,000 agent-authored test patches where 8.2% had weak or no explicit oracle signals. He calls agent-written unit test specifications, not verification. They're useful map for the next agent, but a green build isn't proof the code works. What actually verifies? Tests the agents can't rewrite. So Paul points to the architectural fitness tests, property tests, contract tests, and black box tests as the release boundary with the constraints owned outside the mutation loop. So how can you prove though your test can actually catch a bug? Well, one team's answer is to inject bugs deliberately and see if the tests go red. This is by Rohan, a research fellow at Antithesis, writes that the company has built a new agent skill for mutation testing that injects artificial bugs to confirm a test actually fails when a property is violated. He also says simple code changes like swapping a plus sign for a minus mostly causes crashes in distributed systems. So the skill teaches agents to inject subtle race conditions where most of the time everything works fine, but edge cases go wrong. And they talk about a use case of how they use this on an open source database. They started with 13 safety properties. The agent ran 19 mutation injections across 46 test runs, about 24 hours of fuzzing. 11 properties were successfully falsified. The other 2 couldn't be, which means either the test setup needs reinforcement or experts should revisit those properties. And along the way, the test found 3 new bugs in the database RQLite. All addressed by the Arculights creator, Philip O'Toole. Just another cool example of another technique you should definitely check out as well. Well, as you know, I'm always excited about new books, and here's an announcement from the one and only Elizabeth Hendrickson. So Elizabeth and Joel have a new book out, Signals and Levers: System Thinking Tools for Unlocking Software Delivery. If you don't know, Elizabeth is a former VP in R&D at Pivotal and the author of Explore It, which is an awesome book for testers as well. And Joel, who's a co-author, has spent more than 25 years delivering software, plus a decade helping teams stopping change theater. The book uses tools for statistic process control, systems thinking, and economic theory to help you identify real problems and find the levers that create lasting change. And based on an Amazon preview that I can see, each chapter covers a lot of practical techniques like building causal diagrams, separating signals from noise, framing problems as testable hypothesis, and visualizing trade-offs. It also has a Better in 30 Minutes exercises that lets you practice these techniques Right away. And it's also currently a number one bestseller on Amazon's computer performance optimization category. If you know Elizabeth, you know this is a must-read, so you can check it out using that link once again down below. Next up is a Follow the Money segment. So Raindrop just announced its Series A, bringing its total funding to $5 million. Along with the money, Raindrop launches a new product called Raindrop Simulations. Up to now, Raindrop has focused on catching problems in AI agents that are already in production, but Simulations moves that check earlier. It's cool because it runs on every pull request, replays real production traffic on your existing test cases against a proposed agent change, then applies Raindrop's anomaly detection to the results. And this is one big reason why I think it really matters for testers, because traditional evals mostly measure the failures your team's thought to write a test for, but simulations also looks for behavioral changes nobody anticipated before they ship. All right, so I found this next article on LinkedIn from the CTO of Dynatrace, all about enterprise AI is moving faster and flying blind. And it goes over how AI agents can write and deploy code fast, but according to the author, they have no answer to what happens once that code is running in production. And in this post, he cites a January 2026 Neuron's IT Agent Report that found fewer than 1 in 10 enterprise applications is fully observable. He also points to March 2026 SolarWinds survey where 77% of IT teams lacked full visibility across hybrid environments. And one thing that really popped to me is his math on chained agents. So one agent that's 95% accurate sounds good, but put 10 of them in sequence and the cumulative accuracy, according to him, drops to roughly 60%. Each wrong action could trigger the next before any human knows something went wrong. For links of everything of value we covered in this news episode, head over to those links down below. So that's it for this episode of the Test Guild News Show. I'm Joe. My mission is to help you succeed in creating end-to-end full-stack pipeline AI automation awesomeness. And as always, test everything and keep the good. Cheers.