Playwright MCP Cost, OWASP's Warning, and the Question Nobody's Asking

Is Microsoft quietly steering you off Playwright MCP? OWASP tested 12 AI defenses. Most of them break. Does yours? And 46 responses in and one number already looks wrong. Find out more on this episode of the Test Guild News Show for the week of August 10th. So grab your favorite cup of coffee or tea and let's do this. All right, quick one for me before we get into it. The Test Guild State of Automation 2027 survey is live and the first 46 responses are already telling a really strange story. 96% of you are already using AI in some form, but only 2 people said they weren't touching it at all. But when I looked at the budgets, just 4 out of 46 said their tooling budget is going up. It's flat everywhere else. So AI is everywhere, but the money isn't. What are people actually using for testing? Well, nearly a third named a general purpose coding agent as their primary automation tool. Like Claude, Copilot, Cursor, not a testing tool at all. Now I know 46 responses is nothing. I've run this survey over 10 years, so I'm not going to trust a trend on a sample that small, which is why I need you. Just 5 minutes. It's completely anonymous. It really helps me know what content to create in the new year as well as topics you want in Automation Guild. So help me out using that survey link down below. Look forward to seeing your answers. All right, next up is all about token usage. And this came my way via LinkedIn from Quality Engineer Lead Bradley, who's flagged something worth checking out for anyone running AI agents against a browser. Playwright's own MCP server documentation now tells you in its own README to think twice before defaulting to MCP. So quick background for anyone that doesn't know, Playwright MCP lets an agent like Claude or Copilot drive a real browser using accessibility snapshots, the same structured data screen readers use instead of screenshots or CSS selectors. Restyle a button, swap a div for a section. The automation will still hold as long as the accessibility semantics didn't change. But here's the update. The MCP repo README now has a dedicated Playwright MCP versus Playwright CLI. This section that steers coding agents towards the CLI plus skills approach instead of MCP for most tasks. Why? Well, using CLI calls skip loading the large tool schemas and full accessibility trees into the model's context. MCP remains the pick specifically for exploratory automation, self-healing test loops, or long-running work where you need continuous browser state. But you'll save a ton of money using the CLI approach whenever possible. Something definitely worth checking out using those links down below. All right. Rajesh Deb just released a really long post on LinkedIn, and this is all about what are we as testers standing for now? And he covers in this how testers are talking endlessly about automation, AI-generated tests, genetic testing, and which frameworks to learn while the systems being built decide who gets shortlisted for a job. Which transactions look suspicious, and what healthcare information somebody receives. And he says, we're asking a very small question at a very big moment. And that question is, how do we test this fast? And he goes into job anxiety, protecting your livelihood, learning the tools, understanding AI, building agents. But he warns against confusing learning the next tool with becoming more valuable. And says the thing that survives every framework change is judgment. Believe me, over 30 years almost in this industry, I definitely agree. The shoppest line is aimed really at practitioners. When a vendor claims to like 70% faster, he says some of us suddenly lose the urge to ask how that number was produced. We all fall into this trap. So definitely a quick reminder of ways we should be thinking in the age of AI and how not to outsource our thinking to AI and all the hype around it. Definitely worth checking out once again using those links down below. Next up is how Microsoft has detailed a round of test reporting updates in its Microsoft Testing Platform. Let's check it out. So if you don't know, MTP powers .NET tests, the Test Explorer in Visual Studio and VS Code, and test runs in CI And it works with MSTest, NUnit, xUnit, TUnit, and Expecto. And to get the reporting feature, all you need to do is turn on report.gh in a GitHub Action job, and a failing test becomes an annotation on the line of code that failed. Skipping tests show as warnings, and the run writes a summary to the workflow page. And Azure DevOps also has a flag that allows you to have live publishing that streams results into the test tab while the run is still going. All right, Jason Auburn is at it again. He's always doing something new and innovative. So if you're doing anything with AI and testing, you definitely need to give him a follow. But this new one is he just announced Carbon, which he describes as the first test framework built to live inside AI coding agents. This is definitely a trend we've been seeing here as well. So definitely on point. So why do this approach? Well, his argument starts with what traditional frameworks do not have. They sit outside development, Receive scripts, execute instructions, and reporting results. They know which tests ran, but they do not understand the product, the code, the change, the developer's intent, or the risk the implementation created. Coppen, however, runs alongside the coding agent so it can reason over the codebase architecture dependencies, test requirements, current changes, and the conversation that produced them. And the shift Jason calls important is that QA agents can now talk directly to coding agents. Coding agent explains what's changed and why, and the QA agent challenges those assumptions, finds untested behavior, and hands back those failures, the evidence, and the likely cause. Coding agent repairs it. The QA agent verifies, checking for regression, and reassesses confidence. So he says the purpose is not more green checks. It's whether the evidence is strong enough to trust a change. Speaking of trusting changes, especially when it comes to security, this next post comes your way. By Arseniy Estat Ashesh on LinkedIn, who has open sourced the pseudocode for a security harness he built around AI coding agents, which he says he now runs it daily within Claude Code. Once again, on point what Jason was saying earlier. So he styled this as a layered supply chain defense blueprint, and what pushed him further to develop it was the TanStack supply chain compromise report earlier this year. And he had the package on his machine, though not the affected version. His argument is that agents now write code and install packages on their own, so security rules that actually to get enforced are no longer optional. So this harness puts a gate in front of every npm and pip install that scans for malware, refuses anything published too recently, and rehearses the install before running it for real. All right, are you struggling with getting test coverage gaps on pull requests? Well, this next one's for you. So as I was writing this article about Amico, which previously been on my podcast, they told me they have a new feature they think a lot of people should know about. So on LinkedIn, Amico announced they're promoting a background agent they call a worker aimed at filling test coverage gaps on pull requests. And according to the company, you configure it once, tell it what to watch and what to do when it sees something. And the example they publish, as you can see on LinkedIn, watches a repository, reads the diff on every new pull request, identifies behavior nobody covered, writes Playwright tests, and commits them to the branch along with a report listing every file it read and every case it chose. And the co-founder and the company as well have told me they really are focused on the human. The human still reviews and decides, and it's available free during their early access. And OWASP just released the top 10 for LLM applications for 2026. And the one thing that really popped out to me is a theme that seems to run through this whole document, and that is a passing test result is not evidence of a safe model. And the document you can download is huge. It's over 122 pages. And really it goes over a few big things. OWASP is telling teams to test against adaptive attackers. The research it cites found that static attack success near zero, while adaptive success topped 90% across most of the 12 recent defenses. And as you go down this, the pattern really repeats itself. So under supply chain, a tampered model spread misinformation while evading standards benchmark evaluations. Under data and model poisoning, an attack that erodes refusal behavior while preserving general accuracy is described as undetectable through standard evaluation. So like I said, a really thorough breakdown of how LLM and how you need to be testing it now. It's definitely once again worthwhile for your time. And last up also I found on LinkedIn is from Andrew Frey, who has published a review of an Amazon white paper on building an AI gaming testing agent. And he offers a 4-part scorecard other teams can reuse. And it covers ability, accountability, affordability, and alternatives. So the setup runs plain language test cases against a mobile RPG. Andrew says the most interesting capability is letting non-automation engineers write tests in plain English, though he notes the white paper gives no detail on what the test actually tests, how they affect the team, accountability, Talent ability is when he pushes the hardest. The example output is only a summary of the game state each turn and the tool chosen next. So if the model's wrong about either, you really can't tell. He calls the video output a minimal requirement at this stage. And on cost, Amazon claims $1,660 a month for 200 nightly tests, which Andrew equates to over 100 hours of human QA. At scale, AWS mentions cost approaches $3 per test per month. And once again, for everything we covered in this news episode, head on over to the links down below. So that's it for this episode of the Test Guild News Show. I'm Joe. My mission is help you succeed at creating end-to-end full-stack pipeline AI automation awesomeness. And as always, test everything and keep the good. Cheers.