Cut Weeks of Manual Test Design into Minutes Using GenAI

In a land of testers far and widely journey, seeking answers, seeking skills, seeking a better way. O the Test Illumination Testing Podcast guiding testers with our nation awesomeness. From ancient realms to modern days, they lead the way. Oh, the Testicle Automation Testing Podcast hey, let me ask you this. When was the last time your test suite kept up perfectly with changing requirements? Probably never, right? And if you like most teams, you probably have felt the pain of brittle test, fuzzy requirements and spending weeks on manual scripting, only to end up with gaps in coverage. And that's exactly why I'm so excited about today's episode. We're going to be pulling back the curtain on a brand new course the Test Guild has just released in partnership with keysights called Turning Requirements into Reliable Tests Using AI Augmented Design. And don't worry, this episode is just a highlight reel. You're going to hear from industry experts like Noel Russell, Jonathan Wright and Ian Schopp. They'll share why manual testing is overdue for a rethink how context makes or breaks test design and how gen AI can help teams slash weeks of work down to minutes, all with real world examples and real world insight. And if you like what you hear today, you'll definitely want to check out the full course by registering down below. But before we get into it, I just want to let you know that Test Guild has officially opened our call for speakers for our 10th annual Automation Guild conference that's scheduled to run February 9th to the 13th of 2026. This online event focuses on end to end automation testing and seeks presentations that address real world challenges facing the testing community. So I survey our community every year and ask them what's the number one thing that they're struggling with. And then based on their input, I compile a list of topics they asked for. We'd love to hear your idea for a session at the next Automation Guild. Once again, you can register as a speaker using the link down below. Let me know your thoughts and love to see you there. All right, so let's kick things off as we hear from Noelle Russell. You've probably heard of her. She's one of the leading voices in AI ethics and strategy. She's worked at companies like Amazon, Microsoft, and now runs the AI Leadership Institute. So in this course, Noel explains why AI models aren't just tools. They're like living systems that need careful handling from day one. And with all the code vibing going on, this is definitely something you need to keep an eye on. She uses the metaphor that really stuck with me that AI is like a baby tiger.

Now, I've come to think about this in a very specific way. As a matter of fact, my journey went to, you know, as I went to Microsoft, my job was to really look at a bunch of research models and help those research models get to production. I used to use a term like herding cats, right? Like, I'm trying to get everyone, like doing the right thing and managing engineering work. But when I think about AI models, cats wasn't a really good analogy because it wasn't really like hurting cats. It was more like herding tigers. Baby tigers, as a matter of fact. That's why I have them all over the place, right? So baby tigers. And the reason I kind of think about AI models like baby tigers is because they're very cute at the beginning. Fluffy, adorable. People want to work with it. People want to, you know, like, it's very novel and exciting. People are enamored with these models in the beginning. I mean, just remember the first time you used ChatGPT or a large language model, right? And you're like, you entered in a question, it gave you an answer and you're kind of like, oh my gosh, that's so cute. That's pretty good. That's not terrible. We get excited about that because it's doing something we never thought it could do. However, in this early stage of development, when a model is in baby tiger mode, we really need to ask some very important questions that are hard to ask when you're excited. They need someone in the room, hopefully you, that will raise their hand and go, well, have we thought about like, hey, baby tiger, look at those paws. How big are you going to be? Or razor sharp teeth at birth, like, what do you eat? How much do you eat? Where are you going to live? What happens if I don't want you anymore? These are critical questions to ask as you are building your AI systems. And the earlier you ask them, the better.

All right, so I love that image. It's so easy to get swept up in the excitement of Gen AI, especially when it gives us something clever in seconds. At least it looks clever until you start diving into what it actually generated for you. But as Noel said, if you don't ask the right questions early about things like size, scale, security, governance, performance, you're going to have a tiger on your hands. And I've been obsessed with vibe coding for the past three months. And I can tell you this is truly a real danger. It generates something that looks amazing until you start messing around with it and go oh, my gosh, the security leaks. This is a performance. It's not scalable. There's all these bugs that are built in because it doesn't really know a tester's mindset as it's creating it. Noel also drops another stat in this course, that for every $1 invested in responsible AI, companies are seeing $3.70 in return. And that's proof this isn't just about ethics, it's about business value. Now, in this course, Noelle also shares an awesome framework you can use called poet, which stands for Precision, Optimize, Ethics and Trust. Now, I won't spoil all the details here, but it's a simple way to make sure your AI strategy scales safely while still delivering your company roi. All right, so what about the hidden costs of manual testing? Well, next we're going to shift from AI strategy to a challenge testers know all too well, and that is manual test design. So Jonathan Wright, the keysight's chief AI officer and automation cyborg, has been in the trenches of enterprise testing for decades. He has a bunch of knowledge. If you ever see him at a conference, definitely pick his brain because he knows everything about everything about AI and automation. And in this course, he shares why relying on happy path test cases leaves teams exposed to a bunch of risk. So here's a clip where Jonathan uses the humble login requirement to show just how many edge cases can slip through the cracks.

I think if we're all quite honest with ourselves, we talk about happy paths with happy data. So let's take the login example. Valid username, valid password, and then you press login, log on, maybe sign in, and then you successfully sign into the application. Now if I was writing that as a requirement or even a model, I would have, you know, unsigned up, user, forgot my password. I'd have 5 or 6 different happy path and maybe negative paths through the system. Now, using model based testing as a technique, how many permutations are there through there? You could say, oh, 20, 30, 40 different values, different valid passwords, expired passwords, first time they're logged in, with some requirements for a yule of an agreement for privacy policy or something else. Right. Or terms and conditions. I don't know how that system works and I don't know how that system works for you. You might have some business rules that say, actually if you're in a different location, like I'm in Seattle at the moment, it's going to send me a two factor authentication message to my mobile device and then I'm going to have to Pass that into my application so that I can securely sign in. Fantastic. So how do I know that I've got to go through those steps or I have those business rules? Well, I kind of teased a little bit before it around the sources of truth. And when we talk about sources of truth or sources of information for a second, we're talking about context. And context is absolute king, because that's how your system works differently to any other system in the world.

All right, so isn't that eye opening? Something as simple as login to the system can balloon into dozens of scenarios. Password resets, geolocation rules, two factor codes, business specific workflows, and a bunch more. But here's the reality, especially from what I've experienced on multiple different Sprint teams. Most teams don't have the time or bandwidth to design all those tests manually. So they just test the happy path, maybe a couple of edge cases, and hope it's enough. And as we know, it never is. But Jonathan also makes a powerful point here, that context is king. Without embedding your domain knowledge, you as the tester in the loop, the regulations, the policies, the business rules, your test suite doesn't reflect real world behavior. So that's why you're needed so much more than now as a tester with domain knowledge with the assistance of AI to make you even more awesome than you already are. And that's why manual test design is overdue for a rethink. And in this course, Jonathan shows how modern teams he's interacted with all over the world are combining small pieces of messy requirements with AI and context documents to produce highly accurate test suites that evolve with the system. Look, as I mentioned, Jonathan knows software testing. He knows automation. He's not just drinking the AI Kool Aid. He's a real world practitioner that sees this working across multiple enterprises with multiple teams. So when he talks, I listen. And you should too. So how do we use automated test design with gen AI? Okay, so we've seen the risk of minimal testing, but what's the alternative? So here's where Ian Shop comes in. Ian is a product manager for keysight Generator, which is their gen AI powered test automation engine. And in the course he walks through how generator works and why it's changing the way testers across multiple organizations and teams and companies they work with are doing testing. So here's Ian given a high level introduction.

So keysight Generator is an AI powered test generation engine deployed securely and on prem that embeds the power of generative AI into your testing process. It's designed to streamline test asset creation directly from your natural language requirements. It combines modern AI technologies with a simple intuitive UI to put this generative capability in everyone's hands. And it ingests your real inputs so you upload your real system requirements as well as the domain specific information that Jonathan was just discussing before, things like policy documents or glossaries which might provide that necessary additional context. And then Generator will create test assets from your requirements using all this information on the fly. And then it outputs ready to run artifacts. It generates gherkin scenarios as Jonathan was talking about in feature files as well as conventional manual test cases with execution steps and expected outcomes that then slot straight into your QA workflow. Now, because Generator is fully deployed on your own hardware and runs entirely behind your firewall, no IP or customer data ever leaves your network. So it's ideal for teams in tightly regulated or compliance heavy industries like healthcare, aerospace or defense. But at the other end of the scale, an agile team who operate in retail for example, that might find they struggle to keep keep up with test authoring when faced with fast changing priorities or scope change will also get huge value from it as it can generate tests really quickly from those fast changing requirements, allowing the team to be more adaptable to change.

All right, so this is where it gets really exciting for me. Generate isn't just cranking out random test cases, it's building traceable rate or run assets based on your actual requirements and context files. If you work at an enterprise with really complex things like healthcare or insurance or finance, this is awesome. And as you can see, the speed is jaw dropping, especially if you had to do this manually over the many years that I have. And According to them, one customer cut manual scripting time from three weeks down to just 30 minutes using this exact approach. So let's hear another short clip where Ian explains how Generator removes ambiguity and even catches scenarios human testers might miss.

So to begin with it removes ambiguity, natural language processing within the embedded model, interprets fuzzy requirements and produces clear unambiguous scenarios. So there's no more guessing and you get consistently written tests and scenarios every time. As we mentioned, it protects your IP and ensures data compliance. Its secure on prem deployment, keeps all source requirements, context documents and any other data inside your network satisfying gdpr, HIPAA and audit mandates. And it can take your test authoring timescales from weeks to minutes. It generates traceable test suites really fast, eliminating the slow error prone scripting grind. And we've seen this in real world examples. One of our customers who usually expected manual scripting of test cases to take three weeks, they managed to get the same number of test assets out of generator in just 30 minutes. That's such a valuable time saving. And then testers can spend that time executing tests, analyzing results, or doing exploratory tests to find those intriguing edge cases. Instead, Generator's Context store adds domain depth so you import your documents that hold the industry specific terminology and edge cases, and these are used by Generator to build rich context relevant tests. So much that we've heard from customers that it has generated scenarios their human testers would otherwise have overlooked. And then we know systems don't stay still. So we ensure it's easy to keep tests Evergreen a simple one click generation updates assets whenever a requirement or your context changes so you can be sure nothing goes stale.

All right, so think about that. Instead of weeks of back and forth clarifying requirements, Generator turns fuzzy statements into clear, consistent tests that a context aware and even services edge cases testers might not even think of. And that frees up you and your team to spend more time executing tests, analyzing results and doing exploratory work. The stuff humans are usually best at. All right, let's wrap this all up. From Noel Russell we learned that responsible AI is like raising a baby tiger. It's exciting, but it requires foresight and discipline and it can deliver serious ROI when done right, which is definitely something you'll learn in this course. And from Jonathan Wright we saw the hidden cost of manual testing and why happy path testing just doesn't cut it anymore. And for me, we saw a real world example how a keysight generator is transforming test design, cutting weeks of work down to minutes while improving coverage and more importantly, compliance. And remember, this is just a highlight reel. The full keysight course goes way deeper with real demos, frameworks and step by step guidance you could take back to your teams right now, so don't miss out. Make sure to register now returning requirements into reliable tests with AI augmented design using that special link down below. All right, so thank you Noel, Jonathan and Ian for your automation awesomeness. For links of Everything value we covered in this episode, head on over to test guild.com a561 and if you haven't already, we'd love to have you speak at the next Automation Guild. Make sure to submit your idea now before the call for speaker ends in a few weeks. All right, so that's it for this episode of the Test Guild Automation Podcast. I'm Joe. My mission is to help you succeed in creating end to end automation awesomeness and as always, test everything and keep the good. Cheers. Hey, thank you for tuning in. It's incredible to connect with close to 400,000 followers across all our platforms and over 40,000 email subscribers who are at the forefront of automation, testing and DevOps. If you haven't yet, join our vibrant community@test guild.com where you become part of our elite circle driving innovation, software testing and automation. And if you're a tool provider or have a service looking to empower our guild with solutions that elevate skills and tackle real world challenges, we're excited to collaborate. Visit testguild.info to explore how we can create transformative experiences together. Let's push the boundaries of what we can achieve with lutes and lyres. The bards began their song A tune of knowledge, a melody of code through the air it spread like wildfire through the land Guiding tester showing the secrets to behold.