AI Testing LLMs & RAG: What Testers Must Validate with Imran Ali

In a land of testers far and widely journey, seeking answers, seeking skills, seeking a better way. O the test Illumination Testing podcast, guiding testers with automation awesomeness. From ancient realms to modern days, they lead the way. Oh, the test is on a major testing podcast.

AI is writing code faster than most QA teams contest. So how do you validate software when the output isn't even deterministic? I get asked this all the time. So I decided to create an episode that breaks down how real teams are actually testing LLMs, RAG systems and AI agents without losing control of quality. And joining us to break it all down is AI expert Imran Ali, founder of AI Testing Group. So he really knows his stuff. You don't want to miss it.

Check it out.

Hey, quick reminder, if you're thinking about leveling up your AI testing skills in 2026, Automation Guild 2026 is right around the corner. It's also the perfect last minute gift for the tester in your life or for yourself if you want to future proof your career. In the new year, Imran will actually be leading a 90 minute hands on workshop on LLM and rag testing where you'll learn how to evaluate retrieval quality and embedded accuracy, test chunking strategies and rag pipelines, detect hallucinations, validate consistency, reasoning and safety and agentic workflows, and understanding where traditional testing works and where it fails for AI systems. If AI powered applications are anywhere on your roadmap in the new year, this is the kind of skill set that separates future ready testers from everyone else. So check out Automation Guild 2026. Grab your spot now while you still can, using that special podcast code down below. Hope to see you there. Hey Imran, welcome to the Guild. Hiya. Great to have you. Really excited. Reached out on LinkedIn and I looked at your background. I'm like, wow, this guy has a lot of background around all the things that are hot right now. AI agents, autonomous qa, LLM rags, enterprise QA transformations, all the things. So I thought it'd be great to have you on the show. So thank you for joining us. I guess before we get into it though, maybe a little bit more about yourself.

Yeah, absolute pleasure to be on. Joe, thanks so much for inviting. So look, I've been in the industry now for, you know, over two decades, been involved in software testing, yeah, probably since 2003. Started off as a manual QA tester, moved into automation around 2010, 11 from there moved into leadership roles. And yeah, over the last few years I guess my focus has really shifted fully in terms of how AI can try to remove some of those bottlenecks that teams struggle with. And we've created a startup AI test group that tries to help teams turn requirements into testing, really using AI powered solutions, AI powered workflows, going from requirements straight through the whole kind of QA process and workflow.

Nice. So you start off as manual testing, manual qa, and then obviously you got into AI. What convinced you about AI? Because a lot of people that have your experience sometimes like AI is not real or it's overhyped, like how do you know what's real and what's not when it comes to AI, what got you into it?

I think, to be honest, when GPT came out back end of 2022, I knew from that very moment when that was released and all the hype was kind of, you know, surrounding that, that this was going to have a massive impact on QA. So literally that very month, I think it was December 2022, I purchased the domain aitestgroup.com not knowing at all what the hell I was going to do, how it was going to come into play. But I knew that was kind of a real, kind of massive shift in terms of how things were going to work in the industry and just generally everywhere. So I think AI really has the ability to make things so much faster and easier in the industry, but everything needs to be checked, everything needs to be validated, and there definitely needs to be that human intervention and interaction involved. So I think efficiency, saving time and also money for companies as well in terms of being able to do things more quicker.

Great. So I assume you speak with a lot of enterprise teams. So I'm just curious to know, when you speak with teams or people that are interested in what you provide, is there like any big gaps you see that you think you know, AI testing, what AI testing really is and what it means in practice to what they may think it is and what you have to like, say, okay, it can do this, it can't do that. Do you have any of those conflicts ever?

I think to be honest, if we're really being transparent, there are definitely conflicts and gaps. So I think one of the most common things that kind of get asked is how data is stored and what confidence enterprise clients have in terms of that data being held on their side and not being leaked into those LLMs. So that's a big concern. The other concern or kind of conversations that are commonly had around how really accurate AI is in really complex scenarios where you need real time decisions, because of course AI is fantastic when it's looking at historic information, but then you've got to size something real time, which requires really, really kind of complex understanding and thinking, which sometimes the model itself can't give you. So that's where that kind of, you know, cross section of a human interaction and validation coming into the AI is so critical because ultimately you can't leave it to the workflow because start to finish you might get something very, very different, quite wrong to what you first wanted. So I think those two kind of key conversations are commonly had and experienced.

So do you see people trying to think or say that QA is no longer needed, testers are no longer needed, we'll just create these agents and we're good to go all the time?

Yeah, absolutely. So I think, you know, I think testers generally are very worried about that scenario. I don't think that's happening anytime soon at all. Just because the models and the agents, the workflows are all fantastic. But there is so much handholding to be done, right, so much investment in terms of getting the whole infrastructure set up to do this properly that it's going to require a lot of, you know, QA resource and skilled QA resource to get this set up over, I would say a number of years to be honest, before we can really say, yeah, end to end, we can kind of pass it through different kind of LLMs check one by one to make sure the output is correct. So I think we're kind of quite far from that happening, to be honest.

And this comes from someone that's actually building end to end automation QA pipelines from requirements, I think, all the way to reporting. So I guess when we talk then, what does fully autonomous QA workflow actually look like then in your world?

Yeah, so we don't have the words fully autonomous in our dictionary because not yet anyway, in terms of AI, because like I said, each step is checked. So how our workflow works, you pass in requirements, whether they're behavior driven development, gherkin scenarios, user stories, whether they're old fashioned requirements passed in and they get converted in test scenarios, positive, negative, all the edge cases. So that's done. But at that point we check if there's been any hallucination. So they'll be checking in terms of, you know, are those tests, broadly speaking correct? So skim through of those tests and the coverage and there's a point there where you can edit, add, delete on those scenarios. So that's one check and then from there the automation starts. So we Have Playwright automation with MCB server as quite commonly kind of used across different agents. So the automation run starts and the user can see that automation running that whole journey. So again, that can be looked at, viewed just double checked as cursory kind of, you know, kind of glance to make sure that that's probably speaking correct. And then you have results in terms of a report which gives you not only tests passed, you know, tests run passed and fail, but also, you know, key insights in terms of where the failures may lie in terms of the code. So we get kind of risk analysis coming back as well. And again, in the tester pool those checks are happening. So I guess the key takeaway here is every juncture there is that human validation checking happening and when the bugs get raised into jira, those bugs kind of, you know, check. So there is definitely huge amount of time saving, massive amount of work being saved by this approach. But there is, let's say, 10% of the work being done in terms of the checking. Yeah. And again, also the prompting is really key. So I think we're kind of moving where the QA functions and works into different areas. The prompting, the validation, the checking, understanding some of the pipelines or the LLMs and the rags and those kind of things, it's just kind of, you know, it's not replacing. But you're shifting, shifting that work and understanding how that comes into play.

Absolutely. And you said something about keep up, I would think with the way AI can generate code right now and the type of velocity developers have at producing, is it inevitable that we have to use AI at some point as testers just to keep up? Not that we're being replaced, but that we're going to get so much thrown at us now that just to keep up with it, we're going to have to utilize it, I assume. Am I wrong?

I think so. Because look, we know that probably half the world's code is being written by, you know, copilot, cursor, Claude.

Right.

So we were talking about massive companies, you know, huge, huge kind of corporations investing, you know, kind of so much in terms of AI generated code. So that is being done, has been done. The same level of AI nowhere near is being used for qa. So I think QA is way behind in terms of the usage of AI. So yes, automation engineers might be using those tools to write some of the automation code, but not test. So I think that's where there's a massive gap in terms of where we're behind in development, in terms of usage and we're trying to address that gap and make sure that we are, we have the ability to keep up with development because I think we're finding with the speed of development, we're getting faster and faster releases. Code's been churned out, deployments happening at ever fast speed, but issues are arising. Right, because we know that, you know, the AI generated code is causing issues in production. And when that's happening, not only are we seeing more issues come into production, but also they don't know how to fix those because of course that's AI generated. So when they can't fix it, they don't know the code as well. So they might get an issue which might take them 10 times as long to fix. So I think all of those feed into the narrative of us trying to use AI to get early testing, to speed up testing and keep up with development.

Absolutely. All right, so right now where do you see the most realistic opportunities for autonomy? Is it test creation, execution, analysis, maintenance? Is there one area where you're like this is where if you're not doing it yet, start here?

I think the obvious one is test case generation. It's a quick, quick win. I mean it's a big bang for buck. I mean, I think if you're not doing that, I think start there because it's an absolute no brainer. You can just put into any model your requirements to kind of get the output in terms of those test cases. So a decent model, nice up to date version will give you a good set of coverage, positive, negative and edge cases as well. So I think that's a really, really quick win. But I think beyond that as well, we can certainly use these tools. I mean we have got a, a good, robust tool out there to be able to create automation. So I think those days of writing these really long, convoluted frameworks, I've been there, spent a decade and a half writing those. I think that can really be massively, you know, increased in terms of efficiency with these tools. So I think we're going to be moving to codeless tools and also these kind of tools that create those automation for you. Real time, you're looking at the code, looking at those, the automation run in front of you. So I think the test case scenario is definitely automation, I think is fast coming up. I think it's fast growing. I think we're going to see in a year's time companies worldwide just, you know, kind of just using these tools to create the automation. So you know, you're creating tests, you know in real time. I think what's happening is, I mean, historically we might have spent as testers, week one of a sprint, right, writing test scenarios for the stories that we've been given in Agile Sprint. So, you know, at least minimum half of a sprint, if that can be done in minutes, potentially at a higher coverage, we're talking about huge, huge wins in the industry in terms of, you know, like time savings and keeping up with the development pace. So I think the companies that don't use those at a massive disadvantage.

100% agree. So how do you get it though that so the AI agent really understands the application well enough to make reliable testing decisions or saying this is a good test case or this is not a good test case.

Yeah, that's good answers. So think the key thing is here is prompt engineering. So even in any software engineering team, the output's only going to be as good as the input, right? So if you've got poor requirements, if you've got poorly understood, poorly written user stories, you're going to get poor development and poor code and poor testing, right? So the testers are not going to understand those requirements properly and therefore testing is not going to be at the right, you know, the right level. In the same way, if we don't provide those requirements in these platforms, we're not going to get the optimal results. Right. So again, so it's just be able to make sure, again, you know, shifting left, making sure those requirements we're passing in are really, really well written, are well understood, giving us the best chance for test scenarios to be accurate. And then that obviously then kind of feeds downstream into all the other things, the automation, the bugs, the reports and all of that. So it's making sure that, you know, the input's correct and most accurate and not ambiguous and those kind of things.

So that's interesting. If the requirements are bad, like anything, you'll get bad. If you do manual tests and bad tests, you'll get bad results. Same thing with AI. So if a company's just thinking, hey, they have an existing, you know, legacy applications and the requirements are bad and they've always had bad testing, AI isn't necessarily going to solve that right off the bat, Right?

Exactly. Yeah. So you know, input output, right. So I mean, you know, you need, again, it's not going to solve that issue. So that's not something that we, we try to solve. I don't think it's our job to solve. We certainly obviously kind of, you know, review the requirements with the bas or whoever's you know, rifle knows for us, but ultimately it's all based on the requirements and of course the models as well. So, you know, if you have a good model and you're combining it with rag, that gives you a good chance of getting the right output as well. So I think, you know, we leverage RAG massively and that's a way to be able to kind of get bespoke domain information that's up to date and accurate.

All right, so speaking of rag, you become known for your LLM testing and RAG evaluations. Why do you think testing AI model is fundamentally different from maybe testing software? Or are they?

They are. Massively. Massively, yeah. So I think the traditional approach of black box testing, for example, is so different when testing LLMs. Right. So you put in the same question in GPT five times, you get five for answers. Right. So how do you test that? You haven't got an exact output to test against. So the whole mindset, the whole kind of the way of testing is different. So you're testing, you know, on the way testing on the correctness, the relevance of the answer, and you're testing on things like consistency, hallucination, traceability, safety. So you've got those aspects that are so much different to before. Yeah. And then the way that you test is again, you're using techniques and tools that you would not ordinarily use. So using things like, for example, you know, fuzzing edge cases, so you have an answer which is, broadly speaking, meeting the output and you have got, you have some tolerance around that answer. So those kind of things, adversarial prompts and kind of, you know, you might have custom evaluation sets. So all those things kind of come into play which are different to the old kind of way of testing.

So this is where I think a lot of testers kind of resist. At least when I do a webinar, I'll see in the comments you can't test something that's undeterministic. But with AI, RAG and lms, you have to be able to test something that's undeterministic. And it is possible, it sounds like with some certain guardrails in place.

Exactly, yeah. So using certain techniques, you know, and benchmarks and tools, it is possible. And we're seeing, you know, in the industry more and more jobs come up which ask for these tools and these skill sets and they're highly in demand. So, you know, absolutely, you can do, but it requires different mindset and definitely upskilling in those techniques.

So what's the mindset? And upskilling, that's needed.

Yeah, so like I said, I mean, so if you try to learn some of these, so blue, for example, rouge, rag, S, dp, val, some of those tools out there and understanding. So I think couple of a good course, the istqb course is a good one to give you some good foundational knowledge in terms of how you go about testing AI models. I think if you can look at those, you'll get some good guidance. Like I said, you know, it's just different stuff. Like, you know, you might test, for example, traceability, you know, can you cite where the information came from? You might test, for example, the hallucination rate, you know, the answers, you know, how they're supported by context, for example, consistency of the inputs and the outputs. And like I said before, you know, kind of the fuzzing and the edge cases, you might use LLM as a judge. So you might use the output from the LLM and you might use another LLM to give you some guidance around, you know, the answer of that. All these things, you know, you can learn creating custom evaluation sets tailored to the domain, understanding rag, understanding vector databases, embeddings, all of this is really, really important. Information that needs to be understood, skills that need to be learned if you want to go and embark on this journey.

So you just mentioned RAG again. I think a lot of people know what LLMs are now, but I'm not so sure about RAG. And if so, like, why should they care about RAG? Are more of their applications that they're testing, implementing rags, and that's why they need to know how to. To test them.

Exactly. Yeah, yeah. I mean, so look, LLMs are incredibly expensive to, you know, design, implement. So if you want a massive corporation, you're not going to be, you know, creating an LLM anytime soon. Right. But a rag, on the other hand, is much easier to kind of, you know, get set up. So I think testers are going to be involved in testing rags more frequently and more sooner than LLMs. So basic understanding of a RAG system, how it's used and how to test it, I think is going to be critical in the next year.

So a lot of people, when we talk about AI, always talk about hallucinations. I don't know if it's, if that's a major issue that you see, and if so, is there a way to measure hallucinations to see?

Yeah. So hallucination definitely is an issue and it's a massive worry. Right. Because obviously if you're relying on the model to Give you the answer and you rely on that to kind of, you know, give you the test report hand to your manager or your stakeholders and it's giving you, you know, hallucination results. It's obviously kind of scary thought. Again, I think the way to do it is just kind of to have a skim through of your results at each point and kind of use your experience, using knowledge, your domain understanding and to be able to kind of, you know, get around that also with the rag. So when you're putting in information into your RAG system, so your bespoke company information, all your data, your manuals, all the information very bespoke to your particular work and domain, that again helps you with the hallucination. So again those chances of hallucination are less when you're combining the RAG with the LLM models.

Nice. So it sounds like people definitely need to learn more about RAG, how to test RAG as a skill for 2026. Are there any other skills you think they need to know? I know sometimes people ask me, should they still be learning languages like Python or machine learning or prompt engineering or tool literacy enough to start? Like where would you say people should focus their energy on, especially as we go into the new year, what they should learn for career growth?

Yeah, so I think Pythons are kind of a no brainer if someone hasn't already got that in their belt. You know, Python obviously is kind of the most widely used language worldwide. It's going to be used ever more so going forward. So I think that's an absolutely, you know, a really critical language. If you haven't got that, I think APIs are not going to go away. So you know, knowing JSON, understanding that, you know, being able to, you know, use API testing, again, core fundamentals of that are important. DevOps, you know, again, you know, CICD, you know, fast cycles, shifting left is going to be happening more and more. So I think you know, kind of those things, you know, playwright as the kind of the key web automation tool out there. So I think those kind of core skills, you know, very basic fundamental automation technologies that have been out there for a while. I think couple that with your AI learning, learn how to test LLMs, learn some of the techniques, some of the tools around that, learn how to test rags, if you can, try to understand a bit about LangChain, a lang graph, how agents orchestrate, that's again useful going forward. And be able to use these platforms and understand where they go wrong and be able to troubleshoot when they go wrong. I think are the key skills going forward.

Nice. So I know you're not a fortune teller, but in three to five years, do you see QA being like an AI first discipline? I assume everything's going to be QA first, but maybe I'm wrong. Overhyped or underhyped that you think QA in three to five years will be an AI first discipline.

Do you know what, Joe? I think next year is going to be the asset test. I think if we're sitting here next year, this time next year, and everything's progressed from now until then in terms of the usage, the reliability, the accuracy, the model is getting better, the platform is getting better, and AI adoption increased. I think yes, the answer to your question is we're going to be kind of AI first. I think there is a worry that if this is not adopted properly and the right infrastructure isn't there, this could be overhyped as well because ultimately the AI buzzword has to be adopted. So, you know, we've got to adopt this in QA and really give the right output that's meaningful. And I think companies at the moment are trying to embrace, but we haven't seen that full embracement of it yet. So I think there is still that kind of question mark. Is the output going to be good enough for us to use 90% autonomous and 10% manual, in which case this is going to be AI first, or is it a case of we are going to have to rewrite 80% of these test cases, the automation, automation scripts? And I think that question hasn't been fully answered. And I'm speaking as someone who's in this game, you know, who's kind of developing, you know, platforms. I want to see things play out over the next 12 months.

I'm always curious to know when I speak to entrepreneurs, why focusing on something. So why did you focus on Azure Venture to focus or what inspired you to build an AI first type of tool or system or platform?

Yeah, sure. So having been in the industry for over two decades, I saw teams drowning in manual effort and recently with AI capabilities exploding, testing hasn't kept up. So I wanted to build a company that adopted AI and helped testing really kind of close that gap. So combining deep QA experience with modern AI workflows with the aim to try and reduce those bottlenecks and improve quality without the hype, but actual real business value. So wanted to kind of help organization just have that capability to turn requirements into testing using AI powered software and workflows.

Absolutely. Okay, before we go, is there one piece of actual advice you can give to someone to help them with their AI testing efforts and what's the best way to find or contact you?

Yeah, so again, I think there's so much free material out there at the moment. Go on Udemy, YouTube, Coursera. Anyone who wants to learn can do. So I've just put a post out literally this morning just around kind of, you know, key things to really focus on the year ahead in terms of trying to get ahead in the market. I think really only 1 out of 20 testers are probably really upskilling at the moment. So if you're willing to put in half an hour, 40 minutes a day, try and learn those skills that I mentioned, those tools and technologies, you're probably ahead of 95% of people out there. So I think it's just kind of, yeah, just get on those courses. Lot of them are free, A lot of them are very, very cheap. Knowledge is so cheap out there. It's just in case of enrolling those courses, getting yourself a decent machine. So I think if you're going to do this, you need a RAM of at least 32, ideally 48 or 64. So I think your equipment's important and then it's just really kind of finding the answers to your question. So each time you're, you're learning about the latest tool out there, it just ask on communities, get out there, ask questions and practice your kind of your learning. So I think all of those things kind of put together give you a really good chance over the next 12 months. You know my on there LinkedIn, we've got our site, aitestgroup.com we've got our platform, our studio that gives you the requirements created into test scenarios, automation, automation scripts, test reports, bugs. We try to do the whole lifecycle with kind of version control history. All of that is on our website as well. And yeah, I'm pretty active in terms of trying to offer advice and help and just generally kind of, you know, trying to, you know, get people to adopt AI more for qa, but also generally the IT industry at large.

And you can find links to all this awesomeness using the link down below. Thanks again for your automation awesomeness. For links to everything of value we covered in this episode, head on over to testguild.com a572 and if the show has helped you in any way, why not rate it and review it in itunes? Reviews really help in the rankings of the show and I read each and every one of them. So that's it for this episode. Of the Test Guild Automation Podcast. I'm Joe and my mission is to help you succeed with creating end to end full stack automation awesomeness. As always, test everything and keep the good. Cheers. Hey, thank you for tuning in. It's incredible to connect with close to 400,000 followers across all our platforms and over 40,000 email subscribers who are at the forefront of automation testing and DevOps. If you haven't yet, join our vibrant community@testguild.com where you become part of our elite circle driving innovation in software testing and automation. And if you're a tool provider or have a service looking to empower our guild with solutions that elevate skills and tackle real world challenges, we're excited to collaborate. Visit test guild.info to explore how we can create transformative experiences together. Let's push the boundaries of what we can achieve.

With lutes and lyres the bards began their song A tune of knowledge, a melody of code through the air it spread like wildfire through the land Guiding tester showing the secrets to behold.