AI Testing Is Bigger Than You Think, 5 Areas Testers Must Own

1 Sept 2026 · 40 min
40:29

In this TestGuild podcast episode, host JoeBot interviews Swathi Seela, a principal software quality engineer with two decades of experience. Swathi presents her "AI Testing Landscape," a practical framework that divides AI‑related testing into five overlapping domains: (1) AI‑assisted testing, where testers use LLMs and copilots to augment test design and execution; (2) AI evaluation and trust, focusing on verifying LLM outputs, spotting hallucinations, and distinguishing agreement from evidence; (3) AI development quality/governance, covering prompt versioning, RACI responsibilities, and observability of AI pipelines; (4) testing products that embed AI features, which requires probabilistic validation, repeated queries, and new assertion strategies; and (5) testing the AI models themselves, including model‑level observability and RAG debugging. Swathi discusses common failure modes—over‑confidence, hallucination, infinite‑loop token consumption, and diminishing returns when AI is over‑delegated. She shares a concrete two‑step delegation workflow using a custom MCP tool that generates test cases via GitHub Copilot, outputs CSV files, and then requires human vetting before uploading to Azure DevOps, illustrating how partial automation balances speed and reliability. The conversation also explores the evolving role of testers, the need for both traditional testing fundamentals and AI literacy, and practical advice for newcomers: start using AI tools, break tasks into small, verifiable chunks, and develop a personal evaluation rubric. Swathi emphasizes that AI is a tool, not a replacement, and that testers must retain judgment, maintain domain expertise, and continuously upskill. The episode closes with actionable take‑aways, contact information, and links to resources.

Chapters

More from JoeBot

1:00
youtube.com

Loust or JMeter?

2d ago
1:11
youtube.com

Who Grades the AI? 7 Questions to Ask Before You Trust AI Testing

3d ago
1:00
youtube.com

Your Users Don't Use Emulators

1w ago
2:55
youtube.com

What Actually Happens When AI Self Heals a Test🤔

1w ago
36:02
youtube.com

Locust and Azure Load Testing: Create and Debug Load Tests with AI

1w ago
10:00
youtube.com

WebdriverIO, AI Agents, and 7 Questions to Ask Before Trusting AI Testing

1w ago
0:45
youtube.com

Should AI Check Its Own Homework

1w ago
0:42
youtube.com

The $700K/Month Device Testing Mistake

2w ago
40:57
youtube.com

AI Agents Passed the Tests. Real Devices Didn’t.

2w ago
9:57
youtube.com

Playwright MCP vs CLI, Mutation Testing and AI Agents Flying Blind

2w ago
29:33
youtube.com

AI Testing Reality Check: What 13 Founders Refuse to Automate

3w ago
See all content