Is Performance Testing Dead,LLM Testing, AI Reliability, TestMu and More TGNS166

What global online conference is just around the corner? Are you testing LLMs the right way or setting yourself up for hidden failures? And is performance testing really dead in the age of AI? The answer might surprise you find out on this episode of the Test Skill News show for the week of August 17th. So grab your favorite cup of coffee and tea and let's do this. All right, if you're looking for fresh insights and community networking, I got the event just for you. I'll be kicking off testament tomorrow, August 19th, but make sure to register if you missed day one because the event is going on to the 21st and I believe even if you can't make it, everyone that registers will get a recording. They also have an excellent lineup including speakers like Andrew Jones, Andy Knight, Michael Bolton, Debbie o', Brien, Simon Stewart, and a bunch more. They also have over 60 sessions covering topics like AI accessibility, CICD and how to future proof your career and more. So don't miss it. Register now, use that link down below and hope to see you there. So are you wondering how to test LLMs? If so, here's a great article on LLM testing by Monica and it goes over from correct correctness to confidence understanding LLM testing and it goes over challenges on how to validate large language models and unlike traditional software testing where you know the outcomes, with LLMs the same input doesn't always mean the same output. This makes old pass fail QA approaches insufficient in this case. Instead, Monica recommends testers should focus on confidence, not just correctness. She outlines five key dimensions like roundness, safety, robustness, consistency and costs. Also, classic testing methods also need to adopt and she goes over functional testing means check in ranges of valid outputs, regression testing means comparing behavior before and after updates and safety testing involves prompts designed to reveal bias or toxicity. In our conclusion, LLM testing is about building trust in evolving unpredictable systems and testers must measure acceptable ranges, not perfect answers, while exploring tools like prompt, fu, deep evalve and LangChain to monitor and govern real world use. Definitely another must read article that you can find more about using the link down below. So talking of LLMs, there's an interesting real world implications of how AI models behave in testing that you need to know about as well. A new study from Sonar has revealed significant security and code quality challenges across leading coding models, with important implications for software testers working with AI generated code. And so the research analyzed over 4,000 Java program assignments across five major models and while all models demonstrated strong capabilities in general rating satanically correct code and handling common algorithms. The security findings present serious concerns for testing teams and according to Sonar CEO Tariq, understanding each model's unique strength and failure patterns is crucial for safe implementations. The study found that all evaluated models reduce critical security vulnerabilities and including hard coded credentials and path traversal injections. And for Claude Sonnet 4, nearly 60% of vulnerabilities reach blocker level security, while Lama 3.2 version hit over 70%, with GPT4 reaching 62.5%. And perhaps most concerning for testers, the research reveals a troubling pattern. Better functional performance often correlates with higher security risk. The study also identified what researchers call distinct coding personalities, and also all models showed a consistent bias towards messy code, with over 90% of identified issues being code smells which indicate poor structure and future technical debt. And with gartner predicting that 90% of enterprise software engineers will use AI generated code assistance by 2028, Sonar emphasized the need for a trust and verify approach using tools like sonarqube and other platforms to analyze AI generated cod for security flaws. And definitely something testers need to be involved with as well. So speaking of having testers in the loop and AI and LLMs in this whole thing and came across another approach that might help you with this as well. And this was published by Peter Souza, who's introduced both the PDF in a video talking about a QA framework he's created called Human, Created to solve what he calls the AI production paradox. And he goes over while AI can generate code and plan quickly, humans still slow things down with approvals, which could create a bottleneck. So we created Human, which stands for Human Centric, User Focused, Multidisciplinary and nuanced. This method shifts QA from being a gate at the end to more of a continuous partner from the very start of a project. And he breaks down that the human centric means trusting instincts through exploratory gut checks before code is written, user focus anchors testing on real user experience, not just specs. Also talks about how to integrate QA across design, product engineering, legal and compliance, and also focuses on system level workflows and readiness reports beyond ticket checks. This framework also includes seven practical templates for bugs, automation, pull requests, epics and more, and ready to plug into tools like jira or Monday.com and Peter also contrasts the old QA as a gatekeeper model with the new QA as a partner model, arguing teams to adopt continuous risk assessments from day one. So next up is the webinar of the week and this is all about how to prevent AI driven failures before they cost you Performance Testing in the Age of AI so I'll be joined by two experts, Stephen Filoni and Don Jackson, focusing on how AI enhances rather than replaces performance testing and the presentation will examine four specific areas where performance testing was incorrectly predicted to become obsolete. First one is microservices architecture, the second one is cloud testing environments, the third is test automation frameworks and the fourth is is AI implementation and also going to emphasize that while AI creates operational efficiencies, performance testers remain necessary to guide these implementations effectively. And this session is going to cover historical context, current state and the future direction for performance testing in AI enhanced environments. So make sure to register now using that link down below. Even if you can't make the live session, I'll be sending out a link to the replay in a few days after the event so do you want to help shape the future of software testing tools? Well, I just found a leading software testing platform that they're conducting a market research understand how development teams discover and evaluate testing tools. The survey aims to help testing platforms better understand the decision making process behind testing tool adoption within engineering teams and participants who actually complete this brief survey will be entered into a selection process for gift cards from major retailers including Amazon and Starbucks. I love Amazon cards. I always do surveys like this because you never know. I just came across a new announcement about Chaos Engineering pioneer Gremlin launched an AI driven reliability intelligence tool for SRE. DevOps has launched reliability Intelligence, which is an AI driven solution to help teams detect and fix reliability issues in complex systems with AI coding assistance, speeding up production deployments by 70%. As we've seen in previous news items today, Gremlin argues that outages and failures are becoming a bigger risk. The platform combines automated fault injection, continuous resilience analysts and a MCP server for LLM integration. And some key features include experimental analysis to compare test results against baselines to spot anomalies and explain failures Recommended remediation turning millions of test patterns into specific accessible fixes An LLM integration letting testers query data and build custom dashboards with natural language and Colton, the CEO of Gremlin, says this tool provides actual recommendations based on deep understanding of your system and dependencies. I also found this next announcement on my LinkedIn feed. It was posted by Scott Moore to an article on how Traceloop has released Open LLM Metry, an open source observability framework built on top of opentelemetry, and this was designed specifically for monitoring large language model applications. OpenLM Metry extends OpenTelemetry, meaning it could plug directly into existing observability tools like Datadog, a Honeycomb, a New Relic or Splunk, and it supports instrumentation for nearly every major LLM provider. Also, the setup is lightweight with just two lines of code to start capturing telemetry. It also ships with a traceloop SDK while still outputting standard opentelemetry data, making it easy to integrate without losing compatibility. And this is big. So instead of black box LLM behavior, you can now get actual telemetry connected to your existing monitoring pipelines. Cool stuff. I want to thank this week's sponsor Zaptest AI, an AI driven platform that can help you supercharge your automation efforts. It's really cool because their Intelligent Copilot generates optimized code snippets while their plan studio can help you effortlessly streamline your test case management. And what's even better is you can experience the power of AI in action with their risk free 6 month proof of concept featuring a dedicated ZAP expert at no upfront costs. Unlock unparalleled efficiency and ROI in a testing process. Don't wait. Schedule your demo now and see how it can help you improve your test automation efforts using the link down below. All right, and for links of everything value we covered in this news episode, head on over to the links down below. So that's it for this episode of the Test Guild News Show. I'm Joe. My missions help you succeed in creating end to end full stack pipeline automation awesomeness. As always, test everything and keep the good Cheers.