AI Testing: How to Ensure Quality in Non-Deterministic Systems

Hey, welcome to the Test Talks, TestGuild Automation Podcast. I'm your host, Joe Colantonio, and today I'm talking with Adam Sandman from Inflectra, someone who's been building testing tools long before AI became the thing everyone had to have an opinion about. And in this episode, we get into one of the most pressing challenges in quality engineering right now. How do you actually ensure quality when you're testing AI systems that don't give the same output twice? Well, Adam breaks down what non-deterministic testing really means, why the old rules no longer apply, and what testers and QA managers need to start doing differently today. You're not gonna wanna miss this one, but first, a quick word from our sponsor. Before we get into it, this episode is brought to you by Inflectra, makers of Spira Test, Repease, and the full Spira platform for test management and software development. Hey, real talk, if you're still piecing together your test management with spreadsheets and hope, It's time to check these guys out. They've been at this since 2006 and they got over 80,000 daily users across 82 countries and they back every subscription with free support. Go to Influxor.com and start your free 30-day trial right now. No credit card needed, no contract required. So definitely check them out using that link down below. Hey Adam, welcome to the Guild.

Oh, thanks for having me here today, Joe. It's a pleasure.

Awesome, awesome. As I was saying in the, in the green room before we started, and I've been following you for a long time, and you've been doing this a long time, way before AI came into play. So I'm just curious from your perspective, what do you think has changed about software quality once AI has entered, you know, the software development lifecycle?

I mean, there's two big things. I mean, really fundamentally, one is that people are now using AI more and more to do the development and using coding assistants called Claude Code. So it's lowered the barrier of entry for code. I mean, people who maybe didn't even think they could build an application can now put something together. Teams are being asked and tasked by their managers and by private equity and anyone who really has a stake in the outcome of building something is you need to use AI, you need to accelerate what you're doing, or your competitors are going to be doing that. So the imperative is you have to use AI for the development side of things, which is either allowing the same people to do more with effectively a digital human team, you know, 2 developers who are experienced can now develop 6x the functionality that they could have done before, or a less experienced team that maybe would've had to work with someone else can now do it in-house. I mean, the idea is a product owner could potentially build an MVP themselves without a traditional software development full-stack team. That means The imperative on the quality side is going to be, we are dealing with significantly more amounts of code, potentially human-machine co-written. How do we test that at scale? When 3 years earlier, if I had come on this podcast and talked about testing and quality, it would've been the bottleneck then. I mean, whether it was test automation, manual testing, exploratory testing, there's never been enough time to ensure the quality of applications we built before. And now we've raised the stakes with, with AI-driven development. In many ways, our backs are against the wall, right? As, as quality professionals, the stakes got higher, the volume got bigger, and potentially the, the, the quality has gotten lower. That's controversial maybe, but we think, we think we look at it that there's a greater chance of, of potentially lower quality code going into the development lifecycle maybe. That raises the bar on the quality side of things. That's A. Now B is, We're also building AI systems. So whereas A was about using AI to build anything, whether it's AI or traditional system, we're also building AI workflows, agentic AI tools, generative AI functionality into our apps. I mean, the UI of tomorrow or today is maybe not a web page. It might be a chatbot. I mean, even in our toolset, we're looking at, do we need a traditional user interface? Do we just need a chatbot where you can ask it, what's the quality of my project? Give me the answer versus navigating through 6 layers of dashboard and clicking through 4 pages of grids. So in those situations, we've got deterministic systems are being built faster with AI and non-deterministic systems need to be tested. These are both challenges that are hard and they're both happening at the same time. So I feel like from a software quality professional audience, the stakes are getting higher. The risks of these systems causing negative outcomes is greater. And yet, if you look at where software is playing a part, software is eating the industry, eating manufacturing, software-defined manufacturing, real-time systems. Software is more and more prevalent in our lives. We're replacing discrete hardware systems with software instead. So that means the risks, in a classic risk matrix, that the impact of a mistake is more severe and the probability is getting more likely because of AI. You put these two Trends together, and now you're looking at, you know, large-scale outages affecting humanity. And the only thing that stands between chaos and Armageddon are the testers and the quality professionals. So I feel like we're in a very privileged spot here, Joe. I mean, we are the bastion that's gonna keep the world safe and vibrant for everyone else.

Yeah. You know, I don't know. It's just cuz I, we, we're testers by heart, by nature that, uh, we think, you know, our role is almost being elevated with AI, especially with all this AI slop. Being created. But in my experience, testing has always been seen, and QA has been seen as like a roadblock or a cost center. And you just said, you know, we could never keep up before, and now with all this code, it's almost impossible to keep up. So how do we keep up then?

Right. Well, I mean, obviously, fighting for the budget is one thing. And you're absolutely right. A lot of companies see quality as a hygiene factor. They see it as, well, you know, I have to do some testing. I have to I don't want my customers testing my product, or I don't want to be seen publicly advocating for that anyway. But you're right, testing is a cost center. It's always the first to get cut. You look at any project, what's the first thing they look for savings in is always the QA or the quality side of things. So you're absolutely right, that is a big problem. So if you're a QA manager or, you know, a VP of development or app dev, whatever title you happen to have, you aren't going to get a significantly larger budget for more people., but you may have to therefore do more with less. So the same question is being asked of the development team, which is basically instead of having as many developers, can you use fewer developers using AI or can you use the same, you know, same developers and do more with AI? I think it is ultimately gonna be the testing professionals are gonna have to get comfortable with using AI to try and do more with the same team. Now that doesn't mean we're gonna have less people, we may even have more people. So you may, you know, if, if, but you're gonna have to do 10x the level of quality assessment. And we've seen some amazing things with AI. Like, we, we've been going through some accessibility testing of our own applications to meet some of the, you know, the European compliance laws and some clients. And using AI development tools and AI testing tools and things like Axe, we've done, we've done some amazing things where we did in 2 weeks what we thought would take months. So I do think if testers can upskill, AI is a great friend. The risk, of course, of always is People get scared by AI, but it's also, it's a tool we'll need to use. We'll have to get comfortable with it. It's also understanding the risks of using it too. It's not like it's a, it's not a, a Hail Mary. It won't fix everything. It's not going to be a salve for all the wounds, but it's a great tool in our arsenal.

Absolutely. And like you said, it is just a tool, but for some reason, a lot of testers I speak with, still uneasy with it, especially when they hear nondeterministic systems, because we've been so used to deterministic, you know, bare metal, we had control of everything. Now it's out in the wild, who knows what the heck's going on. And a lot of testers I know too feel uneasy. So what's maybe the biggest misconception testers have about testing AI today or using AI for testing?

Well, I think that— I wonder if it's a misconception. I think it's just a different approach. So the big— and I think it's also educating managers. Because as you say, with a deterministic system, in theory, if you've got a system that you don't change and you test it with the requirements and you run it through a bunch of tests, for every given test data input and a stable system, you do that same test repeatable, you should get the same answer every time. Assuming that it's the same environment, the same test data. And you know, one classic question a tester would always say is, well, how come it's broken today? What did you change? And that's always one of the ways you can try and get to the root cause of why something happened. Well, with a non-deterministic system, you can have the exact same system, the exact same test data, and that's what, that's what's different, the exact same environment, and the answer is different every time. And that means that to get to 100% testability is probably impossible. So it's educating our managers, our product owners, executives, sponsors, and saying, it isn't a question of do we test this or do we test it completely? It's what is the acceptable level of risk we're willing to tolerate in this AI system for this use case. And that will be different for every use case. 99.999%, 5 nines, may be fine for an e-commerce chatbot, probably is not fine for an air traffic control system using AI, some kind of route planning or deconflicting system, right? Maybe there's no level of allowable failure. That's a good question. So I think it's maturity around understanding that. And then the second part would be, Just like if you try and do performance testing with a functional testing tool, you're going to fail. Like imagine you open up— I open up 20 browsers and I ran a functional test 10 times. I've done load testing. That's no load testing. Use a load testing tool that simulates traffic. It's just different. In the same way, there are tools that we're working on, other people are working on to help you test non-deterministic systems. And it actually is more like load testing. It's like I'm going to write agents that are going to deploy a million input agents on your chatbot at scale, and they're going to simulate 10 million different input permutations. You're going to have 10 million output agents reviewing all the results. You put them together, and then you're going to correlate 10 million attempts on one chatbot covering things like safety, racism, hate speech, escalation when clients are frustrated, whatever the parameters are that you're trying to build this chatbot to support. And tools will do that. And then now, and they'll analyze it across all those factors and tell you for each of those different factors, like hate speech, 99.999% of the time it didn't do that. Did it give helpful answers? 89% of the time, you know, et cetera. And you go through all these factors and then you can make an intelligent assessment and say, is it fit for purpose for this use case? Yes or no? And that is a business decision, as we all know. But the tester's job is to use the right tools to ask the right questions to give the metrics that will give you intelligent data. And these are going to be different metrics than before. They're different questions than before. And I think once you start looking at it that way, it is ultimately no different than a performance engineer who may have come from a traditional functional testing route having to relearn their craft for a different use case. The other thing I think is really important, Joe, when you talk about the non-deterministic systems is the non-deterministic system is a bit of the system. It still might run in a web interface. It might still run on a mobile app. You still need to test that the UI works, that data layer works, that's deterministic. So you need to piece apart your application into what's deterministic, what's not, and then what are you going to test appropriately? Same way when you're doing load testing, you might load test the REST API and the UI you know is going to scale fine, you don't need to load test that. So it's decomposing the app into the right bits and testing in the right way. And we've been doing this for 20 years. It's a new medium, but it's the same approach that's worked before, breaking it down into testable units and applying the right testing methodology for the right part of the system.

I love it. I love what you said about, you know, what does even pass/fail mean anymore for some of these systems? It's more about risk. And I think that's a— that was a great point you brought up. So, and I guess another thing that in your answer that you kind of alluded to is, uh, testing's not going away, but our role of what we do might change, and we just need to embrace it just like we embrace anything. It's not being replaced. We just, like you said, may focus on other things now, or have a change our approach to how we do testing.

I mean, I think that's very true in, in any discipline, whether it's, I mean, software developers right now are going through massive change management in their role. QA is the same, business analysts, project managers, scrum masters. I think we all have to risk, risk management people, you know, risk managers are a professional that have been kind of locked in their room with their spreadsheets. We need to bring 'em outta that room and integrate them into the scrum teams, into the development teams. I think there's a lot of different disciplines. That we'll need to bring into the, maybe it's the testing or the wider development, software development folds. People who are used to, data scientists, people who are used to looking at data and looking for bias, looking for causality versus correlation. There's a lot of those things that are very useful when you're talking about testing AI that are not traditionally included in the room. So I think the room needs to get bigger, the skillset needs to get better, We need to look at, you know, different, different skills that we'll need to have that maybe we didn't need to have 2 or 3 years ago when we were basically testing web apps, mobile apps, and maybe some APIs in between.

Yeah, 100%. I guess another thing you brought up too is, uh, the tooling. A lot of pe— AI's a tool, but yet the, the, the tools we use for testing have to change as well. Can you use a non— a tool that was made for AI or pre-AI, uh, today effectively? I dunno if that makes sense, but Don't you have to change your toolset sometimes as well to fit the tooling?

Oh, for sure. Yeah, absolutely. Yeah, I think that's true. And of the two aspects, there's the testing a non-deterministic system. There are going to need to be, and we're working on some new stuff. We're going to have a video, I think, in this podcast. It's called Surewire. It's a new tool for testing AI systems. That's specifically for testing the non-deterministic part. It's going to allow you to deploy agents at scale that will test your AI system and give you that statistical output of, for each of these different criteria you've given it, how did it match? How did it perform? And that's a specific use case around AI. The second thing I think is, everyone's looking at, even for the non-deterministic parts, as we said earlier, if there's gonna be so much app developed by a team that's semi-human, semi-machine, sort of a hybrid team, the people testing even the web component or the mobile Testing the deterministic parts, we'll need to do that more quickly. We need to cover more ground. We're not gonna have the time to do it. So we'll need to embrace, you know, AI-enabled testing. So even the parts that are maybe traditional apps, we're gonna have to look at the time that a human's gonna spend writing a Playwright script or a Selenium script or a Cypress script to use if you're open source. That's gonna go away. You're gonna need to use tools that effectively do this for you, whether that's open source or a commercial tool like one of ours. Ultimately, It's gonna be AI-assisted testing in the same way that developers right now are getting used to the idea of AI-assisted development, which is just as alien. I mean, the idea that you're gonna basically write in English or another natural language and have the AI build the app and have a markdown file. We're moving to the— we're moving potentially to an age where software is written in spec, in markdown files, requirements document basically. And the AI takes that together with the context window of your entire code base The codebase describes your kind of system as is, your spec describes your system as desired state, and the AI has to basically interpolate these two and say, well, this is what you want the system to do, this is what it does today, let's enhance it, let's refine it, let's refactor it. I mean, that's a huge sea change in development. And if you think about the risks that's going to bring, it's incredible. Um, and that means now the other benefit though is institutional memory. One of the biggest challenges we've had for the longest time is if you've got an application that's more than 4 or 5 years old, who knows how the system works? Who knows how it really works? You know, that every developer who was on the original team would say, oh, I know that that one logging module, it's really dodgy. Don't you, don't, if you touch that thing, it's gonna be tested all day. And no one knows how it works anymore. And the requirements user stories were out of date. So in theory, AI becomes this living memory because it ingests your entire code base. It can tokenize it. It looks at all the tests and all the requirements. And you've got your user manuals over time, in theory can build a model that's the application warts and all. And then a new team rolls on or new people join, there's institutional memory that's codified. So that could be really incredible because that's something we've never had before. We've always had this institutional knowledge that gets lost when the original team rolls off once, and no one wants to be building the same app or testing the same app for the rest of their life. That's just boring. And, you know, and the teams go bigger and smaller as the project moves, you know, in scale. So having institutional memory is great for a tester because now you've got effectively the entire history of 20 years of development team that you can ask questions. So a tester could even say, hey, you know, developer AI agent, why did we code it this way? It seems like this is a really strange way we did this. And they might come back and say, yes, but we had to support IE 11. And, you know, in those days we did it this way, but we can refactor it now to get rid of that because that's not a requirement. So Those are the conversations that you'll have with the live development team, but when that team is gone and there's a new team, they don't know that. So suddenly the AI can make the application less brittle because potentially you could say refactor this application to support modern web tech. We removed the requirement for IE 11, thank God. Um, and this is real, this is real stuff we deal with. We've got some parts of our test management tool that we wrote in 2006 that were written for IE 11 and, you know, early Firefox and, you know, that stuff's still probably in there somewhere and With AI now, we can start to remove some of that and we can have it test it. We can make sure that we understand what are the requirements. So it's that folklore and myth is now institutionalized in the, in the, the knowledge of the project. Now, imagine it is a tester. You have access to that. Doesn't that, that's a huge advantage of what we had before.

All right. That's a great point. I think a lot of people just think of AI forward, like, uh, on greenfield apps, but not legacy apps. And I used to work for a healthcare company and I swear that the code base was 30 years old. And one of the original people that developed it was just about to retire, and they were all panicked. This is way before AI. So that's a good point. AI could have been applied to understand it and actually help the team not only understand it, but also not be afraid to refactor, because a lot of times you get all the crud because people are adding new functions because they don't want to test existing or touch existing functions or features.

Yeah. And we've tried this ourselves. I mean, an interesting one we've had is we're moving Spira, our test management tool, from Entity Framework 4 to Entity Framework 6. You know, I started the company and I wrote some of this code 20 years ago, and I'm not a developer anymore. I'm the CEO, so I'm not supposed to be touching the code. They've been using tools like Kuro to actually ask it to do the migration, the modernization. And it's, and this is a brownfield. It's very old code. We've got a lot of custom extensions we wrote. It is, we have, we've been, we've been afraid to do this migration for the last 5 years. We are almost done with it, and it's taking us probably 6 months, maybe 4 months. And the quality is high. It rewrote all the unit tests for us. It's written more unit tests. So using AI as your developer, development pair programming partner to write your unit tests, to understand the code, to go out to Microsoft's website and try and tell us, you know, why, why did it used to work this way? It's been invaluable for our development team and our testing team. Both teams have benefited by having access to this living knowledge and this basically an, a dispassionate judge of the code base. Like, you know, I, I, you know, I may say, why did I do it this way? Because I'm proud of doing it this way. AI has none of that bias. It's gonna say, right, Adam did it this way because he didn't know what he was doing. He was, you know, he was a developer of a small startup and now it's a big company. The CEO, he's too busy. But this is how we should have done it in the future. So I think it's, I think people look at AI as just this vibe coding, forward-looking, but I think as the context windows get bigger for these AI models, they're starting to ingest the entire code base. Like you can give it your entire Git repo. When you start to be able to have that level of context and the version history and the blame history in Git, suddenly it's able to really help brownfield testing teams and brownfield development teams improve the quality and avoid that whole, as you said, the crud, the, I'm afraid to touch this piece because if I touch it, it's, everything's gonna break. And we all know which system, bits of a system have that.

Love it. Love it. And I guess this has always been true and always will be true. I guess if you have bad requirements, regardless of what you created, it's never gonna be good. You never can get quality 'cause the requirements were never good. So can you use AI then to, on a brownfield, requirement set up to say, okay, you have these requirements, but you don't have code coverage for this, or, you know, it's not— you missed this other context or this other piece of document that isn't even required, isn't even included in the requirements.

I don't know if that makes sense. Yeah, there's— yeah, absolutely. There's many ways of doing that. I mean, one thing you can do is you can ask AI. We have functionality already that will, you know, just— is the requirement— well, first of all, is the requirements— are the requirements well written? Are they consistent with each other? Are they Do they have specific examples? Are they, you know, are they precise in what they're asking for? That's like the first level, like, because you can certainly use AI to first, first layer is, are the requirements well written, consistent, consistent with each other and make sense? And the second thing would be, well, let's look at the code and let's look at the tests. Are they consistent? Like the test is testing for A, but the requirement says B. AI can help you find those mismatches and patterns. And then you can even look at the, if you've got a large context window, you can open up the whole code base and say, The requirement says A, the test says B, the code does C. What's the right answer? That's a whole other question. And then you can ingest things like the user manual, and that maybe gives— tell you that the end user thinks it does D. So it's written to do A, it's tested to do B, the code does C, but the users are thinking it does D. This is all real. And AI can help you start to bridge some of those gaps. And then obviously a human would come in and say, listen, If the users expect D, maybe D is the answer, or maybe the users don't know what they're doing and they've been exploiting this feature and who knows. But the answer is you can now for the first time start to look for those patterns and find those mismatches. And then there's other things where you can do where you can also start to think about if I work, you mentioned healthcare, if I'm working in healthcare, I have to not just test that this application does what it's supposed to, but is it breaking the law? Is it breaking 21 Part 11? Is it breaking HIPAA? And you can use AI to start to ingest regulations, policy documents, Medicare, Medicaid rules. We've got clients testing in these environments and the system requirements say A, but the law says B, those requirements better quickly say B and the system better say B or you're gonna get fined or you're breaking the law. So again, AI can do some of these policy level compliance testing, which we don't think about as testing, but it's testing. It's verification that the system is fit for purpose. And as I mentioned, if the stakes get higher, if you're a financial trading platform and you conduct an illegal trade, billions of dollars of fines could happen because you did that. AI could help try and prevent some of that, look for those cases, look for where we may have a mismatch of requirements. And I think the other thing about AI, which is also a thing to forget about, is if more code is being written, it's going to be more in production, and we know the testers can't keep up. What is the biggest Achilles heel of any kind of automated testing has always been maintenance. AI can also help with maintenance. A lot of tools, ours included now, do things like self-healing. And self-healing is one of these things that, as I said, the holy grail you've heard about for 10 years, 20 years. And they go to any website, probably including ours, and you read some mumbo jumbo and you watch a video that doesn't answer the question. And basically it's coming soon. It's a bit like quantum computing or cold fusion. It's coming soon forever. I think AI and the vision technologies and the recognition is starting to make things like self-healing maintenance is becoming a feasible solution. That may be expensive depending on the AI cost, but it's gone from being a pipe dream to maybe an expensive solution. And as we know, the models get better, the context get larger, like anything, it's going to get cheaper to the point where it will become an affordable, realistic solution. And I think this is probably the year we're going to see that as well. So I think automated test maintenance and, you know, really making the job of the tester easier from a UI standpoint and an API standpoint is going to be better. We've got the new non-deterministic systems coming out which are going to make that side, I think, like, you know, testing at scale feasible. And then you're going to have the ability to have this living knowledge base of about the application, the requirements, the documents, the end user. Now you've got tests having access to not just the system as aspect, but the system as is, and even the user feedback, you start to ingest user feedback, maybe support tickets, you can start to get a lot of really useful context. Yeah, people forget this. Your support base, your support knowledge base, your tickets. If you ingest like ServiceNow or Salesforce or any, and pre-sales even, you know, customers have all these questions about your product. You know, the job of a tester is to deliver an amazing user experience to users. Maybe an amazing non-user experience to agent, AI agents. Then we have to, you know, who are our users? But we're going to have to test this entire experience realm. And there are now tools and data to start to do some of that, which is pretty exciting, right? I mean, for the longest time, people would say, I'm testing blind. You know how many times testers in our team would even say, you know, I wonder what our users think of this. You know, I have my opinions, the developers have their opinions, the product owner has an opinion, the scrum master, the different stakeholders. But What do our users really think? Well, if you mine your entire support, support ticketing system and knowledge base and forums, you might start to get some of those answers and tease those out and certainly get insights, which would be interesting.

Yeah, 100%. Uh, how do people know how to do this though? Like, uh, like when you go into an organization, like, like what do you say? Do you say like, you, you're missing on all these opportunities? Or, you know, if you had limited resources, you probably wanna shift left as soon as possible and it's easy to do now with AI. Like, how do you get a team on board with the new guidance or guardrails for this rapid development? Like, where's the best place to start?

So, yeah, I mean, that, yeah, and that's a really good point. And it's hard to, I don't know, the answer is maybe, you know, you know, like it varies, it depends, is probably the real answer. But, uh, to some degree it's also look at, look at where the pain is. Also start with people were more likely to adopt anything new or deal with change when it's gonna make their life better. So in general, if you can help someone get value out of it themselves, not just the organization, it's going to be much easier change. So meaning that if the first thing you want to do with AI is something that takes away a repetitive, boring, horrible task that I had to do, I am more likely to embrace it than if you tell me, this is extra work for you, but for the good of the organization, I'll do it if I'm asked to, but I'm definitely going to be less proactive with it, less willing. So I think it's fine to oftentimes find the use cases to start with where AI is going to really The testing team is X people, or the, you know, the development team is X people. How can I make those people more productive, less drudgery, excited about this? Let's take that use case first. Now they've got some more time. Now they can maybe look at another use case that maybe isn't— it's adding work. So for example, testing a non-deterministic system is going to be new work to test something new. Now that's more on my plate. But if you can reduce the time, number of times I'm fixing my Selenium scripts for the web bit, Do that first. Oh, I've got 3 hours back in my day. Now I can start to go deeper into the, into the non-deterministic side. That's one approach. The other approach might be, you might say, you know what, if we, people are attached to what they're doing today, maybe if I show them the non-deterministic stuff, it's more sexy. Maybe that will be more enticing than refactoring what they've done before, which they like doing. So a lot of it depends on, I think, the team you're working with and what's the motivation for the people you work with. If they really are jazzed to learn new things, it might be giving them an AI that's going to be ingesting all the support tickets and using that to do something. But if, on the other hand, everyone is so time constrained that they haven't got time to think about that, even if it's the coolest thing on earth, they just haven't got time to even look at it, then you've got to start looking at how can I save time on their current tasks. I think some of it depends where your team is today, and then aligning the priorities of the organization with how you can align it with the team again, because if the team is stressed and dealing with too much stuff already, adding one more thing is not the right solution. The key is to take something off. So I think that to me would be, I mean, I'm not a professional change management, but certainly when we've talked to clients, whenever you talk to them about tools and products, the first thing is, if the product's going to add work to someone's plate, they generally don't want that tool, even if it's the best thing ever. If that tool is going to save them time, then they're going to like it. And that's the, so you will often say to people, Actually, it's a trivial example. It's nothing to do with AI, but we went to an organization where the team was not writing requirements. There were no requirements. They just didn't have them. But they were doing testing with spreadsheets. We tried to go into the organization and talk about requirements. They're like, I don't have time for requirements. I'm sorry. We don't have them. We've been good enough without them for the last— people know what they're testing. Go away. And there was no adoption. But then we came back and said, what if we use AI and our test management tool, replace all the spreadsheets, and now instead of spending 4 hours each day collating the results of the spreadsheets and putting them into a report, That's all free. And they're like, oh, well, that would save me 4 hours a day. So we did that first. Then they said, well, now let's talk about requirements because I've got some time now to do that. So even though doing requirements was probably the most important thing, there wasn't time to do it. So I think it's really important to look at the team and see what's the hook that's going to make them want to change their role. And that way it becomes something that's going to incent them to do the other thing and the other thing after that. So I think, I don't know, at least that's my bias is work with the people you have. I love it. To make their lives better. I mean, that's a real story, by the way, from a customer in Australia.

Yeah. Great approach. And that's why I love speaking with vendors is you see a lot of things and you like, a lot of times people have blinders on. They're just in their little role in their little company. They don't realize everything else and all the things that work for other companies. That's why I love vendors cuz you're solving real world problems. And you're proving it because people are buying the products to do it anyway. So, as a vendor, I'm just curious to know then, are there any patterns you're seeing with teams using a platform like Inflectra when it comes to managing AI or helping scale quality better or easier?

That's an interesting one. Yeah, I mean, we're seeing a lot of teams interested in, first of all, very interested in AI. Now, there's always the compliance and the rules. Can they use AI? I would say that's been a big change, particularly in the US. We've seen companies go from last year where we're talking to banks, manufacturing companies, insurance, healthcare. AI would've been, maybe we'll look at it in a demo, but we're not gonna use it. This year in the US, we're seeing, yes, yes, we want AI. So definitely they recognize they need AI to speed up what they're doing. And I think right now the use case is still simple. I'll be honest, it's things like reviewing my requirements and giving me feedback, writing test cases, some degree of this whole self-healing visual testing, if it can be made to work. People want to take their current process to some degree and make it less work, make it easier, make it faster. We haven't seen yet the non-deterministic stuff hit mainstream. I think that's still a little bit out there. I think people are grasping with the fact that we're going to start building these agents and we don't know how to test them, but the agents are not in production yet. So I think that's still, I think, later this year, maybe next year. The biggest use cases today have still been around my current software development lifecycle, doing things the human way with AI assisting. Now, people are starting to talk to us about what I think is next generation, which will be spec-driven development, taking vibe coding mainstream. I think that's the next level, which will be, if we're going to start doing that, then that's going to change testing in a more fundamental way, because now we're dealing with this markdown file that is the application. That markdown file could become the source of the tests. Retooling what an agile team is, when it's going to be a human-digital team. I feel like that's what people are starting to, you know, the conferences were talking about that. I haven't seen that in any of our clients yet. Not yet. It's really still AI assistance of the largely human-driven technological lifecycle. I mean, what's in production in the real world live, but everyone is certainly playing with these new things and everyone is planning for for it. So I would say we're at this inflection point where we know AI is making regular development faster. We know we can help speed up regular testing, but in terms of how it's going to completely change the lifecycle from an organizational people roles, that's happening, but not, it's only just starting to happen. So I feel like we've got a little bit of way for that to go. At least the customers in our space, which tends to be large organizations, banks, insurance, healthcare, manufacturing, They're not the digital natives. Now, if I went to a startup in San Francisco, I think I'd have a different, which is not our core ICP, our core audience. I think you'll see a different profile. So I'm just saying amongst the enterprises that we deal with versus the startups and digital native companies, I do think it's coming, but it's not there yet. Gotcha. If that makes sense.

Yeah, absolutely. 100%. I think, you know, we almost 30 minutes into it and we haven't even mentioned like someone's listening to go, this guy really knows his stuff. Maybe a little bit more about Inflectra. Like, what is Inflectra? Why did you start it? What are you doing now? What kind of things can you do that you're creating that can help with some of these issues we talked about?

That's a great question. I mean, well, thanks. Started the company 20 years ago, so we'll be doing this for 20 years. It's our 20th anniversary this year. So 2026 is 20 years.

That's awesome.

I love you. Well, thank you. I launched the company in '06, and there was no AI. There was— cloud wasn't yet around. Amazon hadn't released AWS yet. Jira was just launched. I mean, we're talking a long time ago, back to 2006 when we started the company. What was everyone using? You know, ClearCase, CVS, on-prem, HP, you know, Test Director, Quality Center, as it was Mercury Interactive, you know, automation was WinRunner, QTP, Astra Quick Test. So that's when we started the company. My, the mission of when I launched the company was You know, there are some really good tools out there, but they're very expensive, they're hard to use, and most companies don't use them because of the two reasons I just mentioned. Wouldn't it be great to have easy-to-use, agile-forward test management tools, things that would really help, and then ultimately test automation? And that's why I launched the company. It was democratizing quality to anyone of any budget. Now, that was where we started. And then as we kind of, you know, roll forward into the future, what we found over time is that For a company like us, anyone could buy our products, you know, on the line, on the website, but the clients that we have the greatest success with are the clients that care about quality because quality really matters. And so we've started to self-select. Clients that care about quality and where quality is a life or death situation for them and their clients are the ones that we focus on. Anyone could buy our software, obviously, again, but we really want to focus on supporting the clients where it's a mission-critical system or it's life-threatening, or it's a, it could be a banking system. And people don't think of finance as mission-critical, but if you can't get your money tomorrow and you can't make payroll, that is, you know, we're business owners, that's mission-critical. So I think when you look at quality, quality isn't just like literal life or death, but it's where software matters, where if something doesn't work, impacts real people's lives in their daily life. That's where we serve. So over time, we started to focus on these kind of customers. And that's when you start to get into those use cases, you start to realize it isn't just about does the system work, is what are the risks of the system? What are the requirements? You know, how are the teams working together? So we started to realize that for us as an organization, it wasn't just the testing teams, it had to be the management, the CIO, the CTO. And so we started, when we grew the company, we added things like Spira Plan to bring governance and portfolio management into the organization, tying together the development tools so the development team and the testing team and the business analysts and the execs and the product owner all see the same thing. And when you start to connect these things together and give visibility, now you can start to give people an insight into what is the cost of quality, what is the cost of lack of quality, so that when the executive sees, oh, by not having quality in this one release, our support tickets went up 400%. And our customer satisfaction NPS score went down 300%. That impacts my bottom line. My customers are no longer happy to use my product and they're using my competitor's product. So that starts to make things real. Or if it's a government client, we had clients where a good example was the healthcare system wouldn't talk to the organ donation system. And so you couldn't track when people, if people passed away, people couldn't donate their organs. And so suddenly there aren't going to be organs available for people who need them. So these real situations meant tying that data together across these systems, uh, and having that visibility. So as a company, we've gone from just being testing tools for testers, which I think is where we started 20 years ago, to be quality for everyone, compliance for everyone, safety, and then risk. How do we bring all those factors together so that we're basically making the management see there is a cost to quality, there is a value to quality. And as you said at the beginning, it's Joe, it's not just a cost center, because once you start to show people it's a revenue center, it's a customer satisfaction play here. Now the tester goes from being a disposable cost asset, which is unfortunately, you know, we always face that. Oh, the first line item that's cut on the budget is, is, is quality. We start to show this is actually value. This is your bottom line. Your business depends on quality. It should matter. So I think as a company, we've had to build functionality that brings us other audiences into the interplay. So it's not just the testers themselves in their own echo chamber with their own metrics and their own KPIs. It's showing it to the product management, the CIO, the, the risk management office, whatever, all the— showing the organization's leaders that the testing team is key to your business success and growth. Um, I think that's, that's what we do as a company at Deflecto. We're, we're showing you the value of, of this, not just, is there other many bugs? Are there any, you know, test run, that, that, those, that's just data. The key is showing them the value. And then you look at some of the other things we're doing around deterministic, non-deterministic testing and, you know, using AI to, to do self-healing. A lot of that is just making things easier, faster, and cheaper because we know there's always gonna be a cost pressure. So I would say we aspire to show value, but at the same time, we wanna use AI to show that we can, with the same team, do more because that's ultimately gonna, gonna end up demonstrating the value and the adaptability of the QA team. Any organization right now is looking at where can I use AI on my marketing team, on my go-to-markets, on my, you know, sales, marketing, RFP writing. People using AI in operations, in finance, they're using AI everywhere. And if the QA team is using AI to improve quality 10x, that's gonna be recognized in the organization. And now they're saying the quality team are leaders. They're showing us how this business is using AI to improve our customer satisfaction versus the AI, the QA team, they don't like AI, they're laggards. We have to show leadership and like, no, we understand AI, we understand the risks of it, we understand the benefits of it. We're using it intelligently to increase quality and improve the bottom line. What is the sales team doing? You know, like, put on someone else.

Now, I love it. And I went through that transition from vendor to open source and going through open source made it difficult to me. When we started, find that visibility and traceability, 'cause you had all these other tools you need to integrate with. So this almost sounds like an end-to-end visibility risk software development lifecycle platform you've built.

Yep. My understanding. Traceability, visibility. Yep. You are 100%. We've been doing some brainstorming on, on taglines and I'm, I'm gonna capture that one. That's a great one.

All right.

Okay. Awesome. See, you're in the marketing team already now. Thank you. Yeah.

All right. All right, cool. All right. I guess before we go, is there one piece of actual advice you can give to someone to help them with their, uh, implementing right now with AI, and, uh, maybe the best way to find and learn more about Inflectra.

Okay, I'll answer the second part first. So to find more about Inflectra, go to inflectra.com. Very simple. Uh, go on LinkedIn, look for Adam Sandman or Inflectra. I travel a lot, go to conferences. I'm gonna do more videos on this stuff this year, so hopefully I have some fun and interesting content. Now to learn more and play around with AI, uh, depends on your role. I, I mean, I'm, I'm actually from a developer background, so I love tinkering. I would say I'm an action-oriented learner. Tinker, you know, go install Visual Studio Code or Kiro or Cursor or, you know, try something out. Go get, you know, if you're open source, if you haven't got budget, you know, use open source tools like Playwright, MCP, you know, play around with those things like MCP servers, you know, just try it out, see what works, look at what's interesting and don't believe what us vendors say, try it out yourself, you know, but also Don't be afraid to tinker. I think a lot of people are afraid right now that, oh, I don't know what I'm doing and oh, it's wrong. And no, no, well, I think everyone's learning. Like a simple example, we wrote some, a prompt in our system to generate test cases. We go to a demo and the test, and at 99% of the time we've done this, it's always been perfect. Right in front of the demo in, in, it was at Automation Star in Europe, all the test steps came out in Spanish. The requirements are in English, the test steps are in Spanish. We're like, What on earth's going on? It was something to do with the prompt and the way we'd written it, that, that, then the way we'd encoded it. We have no idea. So, you know, that's, is that a failure? Well, that was a failure, right? Our system didn't work. It failed. But instead of being embarrassed by failure, we said, this is non-deterministic. We use a non-deterministic testing tool. And that was the inspiration for our non-deterministic new testing tool, Surewire. So what I say to you is, if you're going to tinker and play with stuff, when you hit failure, which you will, that's going to inspire you to the next thing. That's, I think, what's really important here. So my advice would be, be courageous, don't be afraid, play with stuff, break stuff. If it doesn't work, try something else. And know that's all part of the journey. We're all figuring this out, which is exciting, right? It's like when the internet came out in '98, '99. No one knew about web applications. No one knew about J2EE. I remember writing Perl scripts. We were all learning. And it was a pretty miraculous time. And I think we're at this time again. So I think Yeah, that would be my advice. Just be curious, be courageous, and, you know, and share with other people on LinkedIn, other places. Share what you're doing, learn from each other. We're a community, and we're all here to learn.

Love it. And to actually try it for yourself and Tinker or try AI with Inflectra, use the link down below now and try it for yourself. Thanks again for your automation awesomeness. The links to everything of value we covered in this episode head on over to testguild.com/a580. And if the show has helped you in any way, why not rate it and review it in iTunes? Reviews really help in the rankings of the show, and I read each and every one of them. So that's it for this episode of the TestGuild Automation Podcast. I'm JoeBot, and my mission is to help you succeed with creating end-to-end full-stack automation awesomeness. As always, test everything and keep the good. Cheers. Hey, thank you for tuning in. It's incredible to connect with close to 400,000 followers across all our platforms and over 40,000 email subscribers who are at the forefront of automation, testing, and DevOps. If you haven't yet, join our vibrant community at TestGuild.com where you become part of our elite circle driving innovation and software testing and automation. And if you're a tool provider or have a service looking to empower our guild with solutions that elevate skills and tackle real-world challenges, we're excited to collaborate. Visit testguild.info to explore how we can create transformative experiences together. Let's push the boundaries of what we can achieve. Oh, the TestGuild Automation Testing With lutes and lyres, the bards began their song. A tune of knowledge, a melody of code. Through the air it spread like wildfire, through the land, guiding testers, showing the secrets to behold.