#332 How to Build AI Your Users Can Trust with David Colwell, VP of AI & ML at Tricentis

Generative AI is transforming industries at an unprecedented pace. But as AI changes how you or your team work, one thing is clear. Your skills also need to evolve. At datacamp, we offer everything you or your team need to adapt and thrive with AI. Whether it's business users looking to get the Most out of ChatGPT and Copilot, or developers and data scientists looking to fine tune models, you can learn the entire AI skills spectrum on Datacamp. Power your AI transformation today. Start learning@data camp.com.

One of the things that AI has done is it magnifies all of the efficiencies and inefficiencies of your organization. We've been helping our customers shift their, what we call it shift testing left. It means move it earlier into the cycle. AI really just accelerates that. They can shift testing left quicker, they can do more of their testing earlier, they can automate certain sooner, they can get their requirement validations, they can contribute in peer reviews. All of this is done earlier, but it doesn't change the fundamental equation. If you're rolling out a ton of AI code and you have bad security practices now, you've got worse security practices because you're doing more of it in an ungoverned way. Every person in your organization is probably telling you, or at least a lot of the executive leaders are saying, use AI. Be more efficient, be more productive. But it's a dangerous trap to fall into because we mistake the volume of output for completed work. And that's really where as organizational leaders, we need to take a step back and say, that person that's generating 500 confluence pages, talking about how they're going to be doing their work and great plans and everything like that, is probably not generating 500 pages worth of value. They're generating 500 pages worth of stuff. You should start to measure what is actually important to you.

Welcome to Data Framed. This is Richie. We've reached the point where anyone can build things with AI. It's a different matter to be able to build things with AI that are really good. To build something your users trust, you need to thoroughly test your work. And with AI assisted software development, that whole process has changed. Beyond testing, you need a great AI product team.

I want to know who to hire.

And how do you help them succeed. I our guest is David Colwell. He's the VP of AI and Machine Learning at the AI software quality company Tricentis. David's got 15 years of experience across data science, AI, software testing and automation. Basically all of our favorite things here on the show he works on Vision AI, a computer vision based AI automation capability, as well as AI agents and AI quality assurance. Let's find out how to make AI software higher quality. Hi, David, welcome to the show.

Hey Richie, pleasure to be here. How are you today?

Life is good. I'm trying to keep things jolly just to kick things off. I like a good disaster story. So can you talk me through what's the biggest AI disaster you've seen?

I have been in the field for a little while, so I've got stories on both fronts. When it comes to biggest AI disaster story that I've seen, I think my favorite is every time lawyers try to use artificial intelligence for their cases because AI is deceptive in that how good it is at seeming like it knows what it's talking about. And we ascribe a lot of intelligence to somebody that is able to speak with our lingo, that someone that knows the words that we use and that uses them correctly. So I've seen maybe four or five now different examples of when legal teams get their hands on AI and they give it a simple task like summarize this brief for me and you know, generate some arguments that I might want. And because it uses all the right legal terminology, because it's going, yep, you can see in this section, you know these reasons, and here's the rationale and that conflicts with this law or something like that, they start to overly trust it at the beginning. And there was an absolutely comedic brief. I think it came out of the New York Circuit court, which lawyer had created their entire arguments on a very serious case completely using ChatGPT. And it read very naturally. It was a very good, well constructed argument that was full of completely hallucinated citations. And the resultant, I don't know what you call a legal slap back, but whatever the, whatever the opposing council wrote back in their motion was more or less them scratching their head and saying, well, your honor, we don't really know how to respond to this, but every citation in this appears to be completely made up. And the judge going through reading it issued this scathing opinion saying, I have no idea how you have been this stupid, but I'm trying to read all of these citations. I cannot find a single one of them. I mean, the lawyer was lucky to keep their license. They got sanctioned, they got publicly slapped down about it. And the defense they came back with was, I thought that this was like a search engine that I could just kind of put things in and I only got facts back out. And so I started to trust it. I think that's probably the only reason that lawyer still has their license to be honest was because the judge put them into the ignorant and not malicious category. But I think that all of the major AI disasters that I've seen that are at least things that I could have a bit of fun with, not the ones that have truly gone off the off the track, fit into that category of people having decided to use AI for something that they should not have been using AI for, and putting a bit too much confidence into AI's ability to sound like it knows what it's talking about and speak with great confidence and a vast amount of ignorance.

Absolutely. I mean, it's a brilliant story and it's very easy to enjoy. Like a bit of Sheldon Freud there. It's like, okay, lawyers doing stupid stuff. Brilliant. I think a lot of people are being pressured to use AI more and more for this kind of content generation. So I think we're going to see a lot more examples of this where we get just mistakes happening because AI did something that looked plausible or generated something that looks plausible, but it's just not. So how do you protect against this? I mean, where do you start with stopping the tide of these kinds of errors?

It's difficult because on one side you've got. Every person in your organization is probably telling you, or at least a lot of the executive leaders are saying, use AI. Be more efficient, be more productive. And speaking as a sort of people leader myself, sometimes I do I fall into this trap as well, where I'll be looking at my team and I'll be like, come on, you've got access to a tooling. You could have done this much quicker. You know, I believe you could be doing twice as much if you just used all the tools that we had at hand. It's a good thing because in many cases it is true. But it's a dangerous trap to fall into because we mistake the volume of output for completed work. And that's really where as organizational leaders, we need to take a step back and say, that person that's generating 500 confluence pages, talking about how they're going to be doing their work and great plans and everything like that, is probably not generating 500 pages worth of value. They're generating 500 pages worth of stuff. And this person over here who's trundling along getting their work done, don't discount that. You should start to measure what is actually important to you. Measure things like your velocity, how fast you're getting work done. Measure the things that are Meaningful and not the outputs. And that's often what we got used to doing, because we're used to a world where the only way you get to an output is a human having gone through the steps to create that output. Now that that cycle is broken, we now are focusing a lot more on how do you determine that the output. Is this the right output? It's the thing that you wanted. And there's different ways that we go about this in the industry. There's things like benchmarks are great ways of analyzing how good is the output generally for the task at hand, if you've got a volume of data. But there's also enterprise testing practices which customers are implementing. And this is a lot of the work that we do in Tricentis where in order to validate that the AI has done the right thing, you need to put these kind of validation points in place so that we can confirm that, yes, the output. For example, let's say you've got an AI generating code, it wrote a lot of code, but that code is causing the test to fail. So that's actually a net negative. You are worse now than you were before. So having those tests there allows us as people to gain kind of a 30,000 foot view of what the AI is doing and say, yep, it looks like you're on the right track. And it also allows really smart engineers or smart AI practitioners to go one layer deeper and say, well, I'm going to integrate that into my work cycle so that I can not just wait for someone else to tell me that the AI got it wrong. I can feed that information directly to the AI and allow it to kind of self correct along the way. That's the game that we're playing at the moment. Us and these artificial minions that we've generated is our role shifted from typing a lot of things out to peer reviewing and setting up frameworks so that the artificial minion doesn't get too far off track. I think that's where most organizations are trying to crack the nut of AI Productivity is less about generation and more about verifiability.

Okay, yeah, I mean, certainly he's a content creator. Oh, yeah, I've got to create all this content. Like, you know, we've got to put out a podcast every week. And yeah, not going to have, not going to replace myself with AI completely, at least on the show. But yeah, having different ways of measuring what success looks like is more about the quantity of high quality output rather than the quantity output itself. Okay, So, I mean, since you mentioned the Software testing example. This is like one of the big use cases for AI. Talk me through. How does your workflow change then? So suppose you've got all these agents helping you out. How does your workflow change to make sure that you're still creating high quality output?

Yeah, it's a good question. So the first thing that you have to start with is real facts and data about what it is we're trying to achieve. And we're a very data focused organization. In fact, we recently did this 2025 Tricentis Quality Transformation report. You can find it on our website. It's a good read. It surveys a large number of organizations across the globe through different countries, different regions, different size organizations. And there's a lot of interesting statistics in there. But the main goal for us out of that report was to say, are we actually solving the right problem here? Are we dealing with the challenges that our customers are facing? And some of the statistics that came out of it were. It's interesting that the quality engineering teams had the same challenge that the development teams had. They were being told, you need to be moving faster, you need to focus on speed. I think it was, I mean, it's gonna be hard off the top of my head, but I think it was like 43, 45% of organizations that their primary focus for their quality organization was on speed. Now, as quality engineers, that's a good goal to have, but it puts you in that productivity bind where you really are the last line of defense in this sort of AI waterfall that's coming down because upstream. And this will be telling you a real story from my, my personal history in developing AI organizations. You enable your product teams with AI and they start using AI to generate requirements and generate what needs to be done. They may even be using AI to come up with ideas of what we could do and how we can help our customers. So they've got now more ideas and more things being generated. And that traditional kind of bottleneck that would happen when you would tell the product team, okay, great, can you go write all this down, put some thought into it, come up with what your priority lists are and then start siphoning those to the team. That natural bottleneck of work that they had to do before they could communicate, the vast brain dump of ideas they had to the engineers disappeared overnight. So now they've got like a funnel where they can just throw huge volumes of things down to the engineers and the engineers are picking them up and going, wow, okay, I guess all these things are coming my way. I better Use AI to start creating the code. And so now you've got fast ideas, fast code, and you get to the last point where you're supposed to be going, all right, not just is this idea done correctly, that's an important part of testing. But is this idea benefiting the business? Is it actually meeting the original goals that were defined? This is something that you'll often call like static testing. Just validating that the requirement that you got is actually a good requirement to solve the use case that it elicits. And then once you've got those, are we using the right tools and products to be able to deliver it? So is it, you know, the correct outcome now when you take that mission of making sure that it's the right thing and that it's built right, and you say, now your main goal is speed, that starts to come with sacrifices. And the quality teams are saying, well, I'm, I'm snowed under. Like, I, I can't handle this massive pipeline of work that's descending on me without some help from artificial intelligence. And so you end up with artificial intelligence being both the cause and the solution to the problem, where people say, all right, because there's this huge volume of work, I need to be moving faster. The only way I can move faster is to use artificial intelligence. This is where it becomes critically important that these organizations have access to good, reliable AI tooling to use for this. And it really does split into two parts. There's a whole section about how do you test an artificial intelligence itself if you're building a product based on that? But the more common problem is just how do you test artificially generated content? So artificially generated code, artificially generated requirements, et cetera. And this is where if your testers are kind of trying to also upskill into becoming artificial intelligence engineers, at the same time, they're going to be completely node under. We recommend that they look for companies, and this is a selfish plug because I work for one of them that produce this type of tooling. My day job, day after day is just to find ways to use AI to make testers lives easier. We have to accept into ourselves the burden of figuring out how to not just generate more what we affectionately call internally AI slope, which is just AI generated content that isn't adding any value. We have to figure out how do we make it relevant, how do we make it accurate, how do we communicate degrees of confidence to the customer when the AI may or may not know what it's talking about so that we can unlock the speed that these quality engineering teams are seeking in a way that doesn't sacrifice the original mission statement of protecting the production environment, protecting the end consumer from more AI generated slop. I'll give you one example of where AI has proved critical to solving the problems that AI creates. When AI writes a requirement, or realistically, when AI writes any kind of narrative document, it is extremely prone to filling in the gaps of what I'd say it thinks you're thinking, but I'm not sure it thinks. I think it more or less does mathematical calculations and we ascribe to it the anthropomorphic thinking label. But when the AI gets a requirement from you, you can almost imagine that in its head it's reading all the other requirements it's ever seen. Like, okay, let me think about this requirement that I saw online or these JIRA tickets, or these issues in GitHub or wherever it's seen. This nature of things. And what it comes back with is, well, all of those talked about security and scalability as well. So I'm just going to add some security and scalability things into this requirement. And that one also talked about, you know, policy making decisions. I'm going to add some policy making things because your requirement was just close enough to those ones for it to trigger those ideas. Here's where. The first of the major AI syndromes, the diseases that infect humanity when it's exposed to artificial intelligence, if you'll excuse the somewhat dramatic phrasing, the first of these diseases is that we feel like we can't slow the machine down. We don't want to be the bottleneck. And so when you get these huge narratives generated by AI, the temptation is that we skim rate them. We go, oh yeah, that looks about right. That looks about right. Yep, that's roughly what I'm after. We don't subject them to like detailed review where we're going through and going, do I need that? Why are there security requirements in this AI? Like, I know security requirements in general are a good thing to have, but are those the right ones? Are they relevant to my problem? It is unnatural for us to do that because we feel like the work is complete and with the final step before we hit the go button, dust it off and tick. Our KPI is us reviewing this thing. And so we naturally want to get through that step as quickly as possible. And that is where most AI errors creep in, is the AI adding things that don't belong there because it hallucinated up an idea of what it thought you wanted. The Cure for this is, yes, we need to retrain humans because ideally that thing never ends up there in the first place. We need to teach people that it's okay to be the bottleneck. But we also have got specialized AI agents that are part of our quality agent componentry in Tricentis that is looking for that. Where it's been tasked with saying, hey, read this requirement. Do you think that any parts of it are superfluous? Is there anything here that really isn't relevant to the business, is not addressing the core concern and if there are, raise those to the human. Because, yeah, AI is very malleable. It'll take on whatever role you assign it. And so when it's the, when it's the requirement creator, it's taking on the role of. Let me think of all the things. When you give it a role of, okay, you're a critic, you're trying to find missing components. It does a pretty good job of that. So AI is both a massive productivity benefit, but without a good testing practice that's got good AI enabled tools, you run the risk of throwing a ton of AI slop into your customer's lap, at which point they're likely to pick it up and say, I didn't actually want this. What is this thing that you've done? A lot of stuff, but it's not very useful stuff. So yeah, that's how we think about testing. It's kind of like that final bastion of defense against huge volumes of AI rubbish being thrown into production.

I have to say, a lot taking on there, but you have me worried sort of in the middle where you're saying, okay, actually one of the really useful skills now is just reviewing AI slop. That sounds like a terrible job. So I'm glad that, yeah, do that all day would be horrible. So I'm glad that there are agents to help you out. That. It seems like that's quite a common architecture pattern is you have like one agent creating stuff and then you have a second agent that's going to be reviewing or acting as a, as a critic there. Does it change your approach to testing as well? Because you mentioned that quite often quality people come at the end of the process and they're trying to decide, okay, is this any good or not? Seems can you move quality earlier into the process to I guess eliminate that final bottleneck?

You can and you should. So one of the things that AI has done is it magnifies all of the efficiencies and inefficiencies of your organization. So we've been helping our customers shift their, what we call it shift testing left. It means move it earlier into the cycle. We've been helping our customers do that for quite a while. And so AI really just accelerates that. They can shift testing left quicker, they can do more of their testing earlier, they can automate sooner, they can get their requirement validations, they can contribute in peer reviews. All of this is done earlier, more so with the help of AI. But it doesn't change the fundamental equation. And that's something that's important to remember, is just like if you're rolling out a ton of AI code and you have bad security practices now, you've got worse security practices because you're doing more of it in an ungoverned way. And if you've got good security practices, you're moving faster in a safer way. So you get, you multiply the benefit. You can almost imagine AI is like hiring a bunch of extra people that aren't particularly well trained. If your process can handle that and integrate those people into your environment and get good value out of them, and then you are going to get even more value out of artificial intelligence. But when it comes to testing something that our internal QAs do a lot of now, we've spent some time training them in this regard is we've let them know that you are now sort of the custodians of AI risk at Tricentis. And part of your job is to analyze the requirements and to say there will be a desire to use AI to solve some of these. You have to determine if AI is the right solution in those cases. Because there's a bit of a saying that we've ingrained into everyone. It says, if your use case cannot tolerate consistent, persistent, incurable failure, then you can't use AI. Because we train AI to make mistakes. Sometimes that's part of the. It's baked into the cake. AI training involves a degree of accuracy, which is another way of saying a degree of error. If you invert that, if we ever had an AI that was 100% accurate, we would have reinvented a database. So we know that AI can make errors. So when our testers are looking at use cases, they will ask questions right up front in the beginning and say, what will happen when. When this fails, what will happen when the error happens? And if your use case doesn't have an answer for that, either by saying, well, when an error happens, there will be a person to review it, or when an error happens, there is a way of mitigating that via recovery mechanisms. Later on, if the business owner or the product owner doesn't have a good answer to that question, then you can't use AI for that. It would be a good example, perhaps is back to the original one that we came to like. The AI in legal is actually a good use case, but the lawyer should be reviewing what the AI puts out and saying, yes, is this right or is it wrong? This is something we call human in the loop, a bad case for AI. This actually has happened with a couple of companies is, well now your AI with no lawyer review is giving legal advice to laymen, people that do not understand the law. Now when it makes a mistake, who's going to review that? You expect an engineer to pick up a legal document and say, oh, you misquoted the law there. Or actually that's not what that case that was in 1986 said about how we apply this. No, there is no potential for human review in that use case. So we would call that a use case defect where you have a critical flaw in your use case. And we expect testers to shift all the way back there to consider themselves not just as guardians of quality in organizations, but to be guardians of use cases and determining whether that risk is acceptable to the group. So longer answer to your question, which was about can we shift testing left? Can we make it faster? Absolutely we can. We can be moving all the way to the left than we should be, but we should also be moving quality back into the definition phase to determine if this is the right thing to be doing in the first place.

Okay, yeah, certainly you can have something that's very high quality but completely pointless as well. Yeah, you can invent stupid things that are, that are great quality. So I do like the idea of thinking about what you should build as well as having all those sort of different steps in order to ensure high quality throughout the process. Process. It does sound like a lot of it is about getting the fundamentals right. So you mentioned the idea of you've got to have like good security policies throughout. I guess the role of like, of things like data governance in there as well. So can you talk me through that? Like what do you need to put in place before you can actually create high quality AI products?

So this is going to get a little bit into designing an AI product organization because there's been a shift in the last three years when it comes to how much data you need to get productive AI. If I cast my mind back six, seven years ago when I started rolling out AI product to Our customers. You needed to lead any of your rollout times with six months odd of data gathering, data labeling, data cleaning, data science exercises. So you could not even get to the first results of are we getting a good enough accurate model developed to deliver this outcome? Until you already had a data set and the data set sizes were quite large, like you'd be thinking tiny data sets would contain thousands of records that had been labeled and realistic size data sets you were into the millions of records that you would require. So it was a very big exercise and there was a huge focus on data lakes and data pipelines and everything to make sure that that worked. What changed was transformers, which is kind of the underlying architecture of large language models currently are sort of a bit more general purpose machines. And we learned that there is a lot that a model that has been trained on this vast volume of data that they have access to, like the generally the whole Internet, more or less that those models can actually solve a large number of problems with very, very limited data. I think the original paper that hypothesized this was from OpenAI. It said large language models are few shot learners. What that means is with a task the language model is unfamiliar with, you can give it a few examples of how to do that task. And its inherent ability to understand natural language and sort of translate between those tasks yielded a pretty good outcome with no training at all. Now what this meant is that an entirely new category of data volume emerged and that was you have benchmark ready data. That means that you are not training the model, you're not fine tuning the model. In fact, you are not touching the model at all. All you are doing is tuning everything else, tuning the context, tuning the prompt, tuning the pipelines and the tooling and everything else. And that set of data that you need is only enough data to be able to tell you if you are accurately solving the problem. So you could be 10, 20, 100 that volume of data. Now what that changed about the way that we as product organizations considered use cases is instead of having a narrow set of use cases where we said we can only really operate in a world where we've got a huge amount of data to be able to validate whether it's true or false, our use case horizons broadened massively. We could now start to look at in the world of testing, for example, we went away from oh, we can just do stability on whether a test is broken or not, or should a defect be raised or not, areas where there was a lot of information density. We then were able to say, oh no, we can generate test cases from requirements, because even though we don't have a huge amount of data on that, the language model is doing a fairly good job. And so we can just expand that out or we can have it build automation, we can have it build the test case data sets, we can have it deal with the queries to the database. So the world became our proverbial oyster in the space where we didn't have a lot of labeled data. Now the challenge that this starts to create is for a data organization. If you don't have good data hygiene, you'll start to suffer from one of two problems. You'll start using data illegally or you will start to inadvertently expose data leaks. So imagine that you're operating in this new world where you say, all I need to do is give a few examples to the machine and then the machine will be able to figure it out from there. Well, what's in those two examples? Do those examples contain customer confidential information? Do those examples contain sensitive things like API keys, for example? Do you know what's in those examples? Because we learned pretty early on that anything you tell the AI, sufficiently motivated user can get back out of the AI. This is something called context extraction. Or the AI could inadvertently say like, oh, we gave it an example of how to set up a project in GitHub that contains the actual API key that you use to connect. And then a customer says, hey, how do I connect to a project in GitHub? And it says, oh, you should use this API key because I forwarded my input. So data hygiene becomes critical in environments where you are no longer going through that rigorous process of take the data, label the data, review. The day like that was a six month exercise. Now product teams left and right just picking data up from here and there and putting it into the large language model. So you start to need a really strong data governance organization that is really, really confirming what's going in, what's going out, what is being used to train the model, what is being presented as context to the model. Do we have a good handle on who owns that data? Do we have the rights to use that data? Does that data contain sensitive information? And you need all of this not just because it's the right thing to do, which it is. You need all of this because your customers will demand it of you. We got certified with ISO 42001, I think about five, six months back, or might have been sooner than that. It was a big exercise in going through and saying, yes, we have confirmed that we know where like the data lineage of each record that gets presented to the AI, that we can confirm when AIs are trained and when they're not, who owns that data, that we have rights to access that data, that it doesn't have any sensitive information in it, even down to the point where we are presenting the correct data to different prompts that go to different customers so that we never end up in cross data pollution environments. So when we say that this new world of AI, we tend to use AI and generative AI synonymously. So I'll split them out. In the world of generative AI, if you're running a product organization, you need to get a very good handle on where your data is, who owns your data, what type of data it is, what sensitive information it contains. And you need to be able to have that process automated to the point where your product orgs can confidently use the data they have access to and not have to ask all these questions individually because if they're asking it individually, then you're answering individually to your customers. So it becomes even more important to have strict data governance policies just the same as it is to have strict security policies. Because this will help you move fast in a world where AI is doing a lot of the lifting for you.

Absolutely. That sounds like great advice. I have great data governance policies. But at the start of this conversation we were talking about how when you're with quality control engineers, the end of the process, they become the human bottleneck for this and then you need AI to help them out. There is the same true if you're working in a data governance position, it can be.

So we have a lot of our data governance processes automated so you can say things like we know that you can only present data to the AI that the person using that product has access to because our authorization controls within the, within the company have already automated that where it starts to get into the human, necessary human oversight. Now I really do see this as the evolution of the data scientist is there is an entire new currency being built up around how much gen AI ready data do you have? And this is more of the data that you have the right to access that has been properly cleaned and confirmed can be used for things like fine tuning. How much of that do you have, where is it, who owns it and how has it been processed? So as an example, if we want to generate test cases, one of the goals that we have is to improve the accuracy of the test case generation. And over the long and medium term we know That a solution to improve that accuracy is fine tuning. How do you get better results out of an AI? You can add better context, you can add better prompt, or you can improve the base model. That's what we call fine tuning. Now, if we rely on all the product orgs trying to figure that out, how to fine tune what the right data is, et cetera, then we're going to end up with a governance nightmare with everyone having their own view as to what's correct and what's incorrect. And so we have our internal group. Sorry, banged my microphone there. Our internal group focusing on how to get that data, Genai. And that means that it needs to have an automated process set up for each use case, for ingesting the appropriate data, for turning that data into anonymized data that can be used safely in the algorithm, for setting up spot checking policies so that people can go in and review subsets of that data, make sure that it is in fact anonymized, and then implementing training practices and policies that will then actually do the part that data scientists and machine learning engineers are most used to, which is put the data through the model and do the benchmarking. So that thing that used to be the hardest part of everyone's job of yep, get the model, train the model and do it, is now somewhat commoditized. And the part that used to be someone else's problem, which is how do I get the data and move it into this state where it's ready to be, is now very much the challenge of data scientists. So that's the new bottleneck that emerges. The good news is here that's kind of a trailing concern. So the pattern that we have in our organization is that you have to go to market with tens of data. So where you know how to validate that, that's in the benchmarking territory. You know that the process is good with a model that has not been fine tuned, with examples that are not created from real data. So there's synthetic examples that have been introduced, ingested into the model. And the reason you want to do that is because there is a huge usability question with every AI product, which is we think we're solving the right problem, but we don't know if we're solving the right problem in the right way. And the only way to determine that is to get it into the hands of the people that are doing it and to observe them, to say, all right, how do you interact with this? Like the best example here would be there's a benchmark out there called Swebench, which is the software engineering benchmark. And if you look at the deviation between what the questions are in Swabench and what real engineers ask the AI to do, it's a massive gap and it's filled with usability challenges. You need to get there as quickly as possible. Once you've identified that, yes, this is the right problem to solve and that we've kind of nailed down those usability challenges, that's when you move into the world of collect the data, clean the data, get the fine tuning set up. Because now you're trying to optimize, you're trying to go from it's the right problem we're solving into we're solving that problem quickly, cheaply and accurately. And that's the realm of where data scientists play. So they do become a bottleneck over the long term because as you start to put more and more AI capabilities into production, you start to have more and more of these cases where you're like, no, we need this optimized and that optimized. But the lucky thing is that this is almost always trailing deployment. Deployment. So you're getting value to your customers quickly. It really becomes the internal fuel for your AI organization is to build that data up over time.

Okay, so that's interesting. You mentioned the changing role of the data scientists there. So I think, yeah, there's been this sort of long standing thing where you have to spend way too much time cleaning data and then you get to the fun part of modeling. Actually the modeling part's been sort of taken away. There's more data claim to be done, more data governance to be done. Actually, since, since you've gone from being a manager of a data science team and then machine learning teams and now you're working, you're running an AI department. Talk me through, like, what's the difference? Like how is your, the profile for who you're hiring changed?

Yeah, so I think the big shift went from AI being a thing that was done in a, a piece of the organization. Like you would have your data science team, you'd have your machine learning team and they would operate sort of like a services team where there would be a request coming in from business somewhere saying, hey, we need to improve our fraud detection. We've tried all these algorithms and we can't quite get it. Can you guys do one of those magic model things that you do and then see if we can get better results out of that? Or sometimes it would be a reach out. The first examples we had In Tricentis were reach outs where we said, hey, we know you've got this problem. Do you know we can solve that problem with a computer vision object detection model model, for example, or improving the accuracy of self healing with regression models. So it used to be that AI was kind of locked away in a corner to the degree of go do your magic stuff and then come back to us and we'll try to figure out how to deploy that giant ball of math in a scalable way. The big unlock here was not actually large language models. It was commoditized access to large, large language models. Because now you've got every product organization under the sun can just connect to an OpenAI endpoint or an Azure endpoint or a bedrock endpoint. And now they have AI. They've got what they wanted out of the box. And so the biggest challenge at the beginning was making that decision. Do you just let everyone loose? Because you will gain a lot of cool use cases and you'll get fast delivery by not putting any gatekeeping in place. You can say, go, go nuts. And this almost made it feel like the entire discipline of being AI people would be relegated to niche companies that deliver, like AI models or do what we will call the AI labs. And as a, you know, as a machine learning engineering person, I found that confronting and the natural feeling was to say, oh, no, you can't possibly do that. You don't know all of the activation functions that we know. The challenge was realizing that that didn't matter anymore for a large number of these organizations and a large number of product teams within those organizations. And so we had a very challenging discussion internally about like, okay, well, what is it that we do now that every place in the organization can roll out AI? And we started to see that every product team was able to take AI to production, but they very rapidly started running into problems that we'd solved decades ago. So they would say, hey, the AI is not doing what I want it to do sometimes. And my tests are failing because I've written a specific test saying if I do X and I expect yes, because they didn't understand that they were dealing with a probabilistic machine, that it would sometimes answer the question like this and sometimes answer the question like that. And they found it frustrating and irritating. And they're like, we don't know how to handle this. And we're like, what do you mean that's. You should benchmark it. Doesn't everyone know that? And that was when we realized that, no, everyone doesn't know that most product orgs are used to operating in that world of I input X and I get x plus one out, because this is an adding function, it's reliable, and it's unit testable and everything else. And so AI engineers evolve to take on that role of sort of being like AI architects or AI advisors across these organizations, teaching them, no, that's not how AI works. You should treat it like this. Or, okay, prompt engineering is just another way of saying, we've got a magic book of prompt spells that we roll out, and for some reason they make the AI behave differently. The AI engineers were the ones saying, no, no, this is why it makes it behave differently. So it was unlocking that connection between outcome and cause. And unlocking that connection allows you to improve. Because up until then, we found tons of product with tons of AI rolling out where they didn't have any way of improving it because they didn't know what was actually making it tick. And so the best AI engineers got into that role very quickly and said, all right, I am no longer the gatekeeper that says you can't use AI unless you come through me. I'm the accelerator that says you can use AI however you want, but if you want to get good outcomes out of it, I'm going to come and help you get there. And so they started doing much more communication, much more holding town halls and saying, hey, here's what's working, here's what's not. Let's communicate the why behind it to people. Not because we think everyone will pick up the back propagation algorithm overnight, but because everyone that we can educate to use AI better will unlock the whole organization to move faster. So that was the journey for the AI engineers. The journey for data scientists, though, is was much harder because at the beginning it very much felt like, why do we even need data scientists? Because we've got this machine that can just operate off very little data, that can do its job realistically without any custom data. You could just invent the data on the spot and say, hey, here's what this use case might look like. You know how to do this task, Go do it. And so at the beginning we were like, oh, what really is the purpose of having a data science team? And then we discovered that just like with AI engineering, when you propagate out the anyone can do it attitude to the group organization, you fail to internalize all of the things that that team already knew how to do, all of their expertise that they had built up. You're now just kind of Asking everyone to absorb that expertise overnight. And so when it came to, like, how do you design a good benchmark? How do you come up with the categories that you need to measure across? How do you govern against bias in your benchmark? How do you ensure appropriate distributions? What does your confusion matrix look like? Like, the only thing I talked to one team and I said, okay, for this, you're going to need a confusion matrix. Because I was still in the mode where I was like, of course you do. This is the way we operate. And I got confusion back from the team, but I didn't get a confusion matrix back because it's a term that they were completely unaware of. They had to Google it and they're like, how do you produce this? What's the point of this? So that's when I realized you actually do need to start propagating out data science as a exercise, as an expert team, almost a guild, if you will, that are able to accelerate everyone else. And that was how we unlocked AI productivity as an AI product team. Now, now, how does this change your hiring? Well, you still do need very good machine learning engineers and you need very good data scientists. But we started to see that to roll AI product out, you also needed engineers that can fill the gap between those engineers that can say, okay, that's what the data looks like and that's what the machine learning solution is going to accomplish, and this is what the customer needs. So I need to sew all of these together in a rapid way to develop a prototype to take it to production so that we can see if we're solving the right problem. And so we were hiring more Python engineers, we hired more machine learning engineers, we hired a few more data scientists because we saw that AI was becoming so pervasive within the organization that we needed to get very early on capturing all of these skills so that we would have a sufficiently sized enabling team to now go start working with the product teams. At this point in our journey, every single one of our product teams has AI people working in that team. They've got machine learning engineers, they've got data scientists, they've got these Python prototyping engineers that are living and breathing their business use cases, but they are implementing the machine learning and data science practices that we built up over time. So I guess a summary of that journey is every organization is going to face this journey and you will face the same degree of confronting feeling, there's no better word for it, that we did. When you look at where the organization is heading and you get this sense that that's not my role. Like, I don't see that anymore. Like, you're not doing, you know, giant data analyses, building and training models. Like, there's no home for me there. And the encouragement that I would give is. There certainly is. You have now become an expert enabler, not a gatekeeper. And that's actually a very satisfying role because what gave us satisfaction in our jobs was not I trained a model. That was. That was the work. What gave us satisfaction was I solved that use case. I gave value back to the customer. We aren't a. We're not a university. We don't do things just so we can write a paper based on them. We do things so that our customers can get some value out of it. Right now, one machine learning engineer can roll out five times more delivery by enabling other teams than they could have in the sort of centralized gatekept model. So it is a very satisfying outcome. But there's a lot of. There's some pride, there's some kind of historical baggage that you need to give up in order to get to that location.

No, I do love the idea of existing data scientists, existing machine learning engineers being enablers for the rest of the organization. I'd love it more about, like, how do you go about, like, teaching all these sort of, maybe less technical people or just people with a different background, like those data skills, those machine learning skills. Yeah. What does that rollout look like?

That rollout is challenging. If you, if you do it smoothly, good executive support is necessary, and executive support generally comes from. It is very hard to go through and confirm that you're doing AI safely, that you're doing it well, that you're doing it securely. Like that burden is high and necessarily so because AI is very powerful and everyone is both enthusiastic and concerned about it. And so having a sort of centralized function of we know how to do the benchmarking, we know how to do this, and then distributed execution, where you send them out, is a model that is, at least from an executive point of view, makes a lot of sense because they go great. I can centralize my community of practice. I can make sure that there is a way that we do this within our organization. So getting that buy in early on is critical. After you've got that buy in, the challenge that you're going to face almost immediately following that is people don't feel the pain of doing an AI rollout wrong until they put it into production, usually. And that's when they'll have said, oh, we've Got this great Q use case. We rolled it out to customers and then like support ticket after support ticket are coming in or they're being told, all right, it needs to do better at this task. And the engineers start to feel that frustration of what do you mean by better? How do I metricate better? How do I go from here to there? Like, if I change the prompt now, it breaks all these other tests that I'd written and these ones might be like, how do I handle this? And so if your organization is a low level of maturity, then you'll find a lot of pushback from the engineering teams being like, no, no, you guys are, you're the lab people. We move fast, we break things. You know, we've got this, this speed to market that we can. And that will disappear over time as they start to feel that pain. It's important to engage early on in the education and say, we've got a lot of good examples that we run internally of saying, hey, we rolled out this thing. Check this out. It does this, this and this better. Now we upgraded the model. It does these things better and these things worse. Is it better or worse? And just like letting the engineers feel that pain of going, I know, like, how did you, why did it get worse over here? I thought models only got better over time. No, models get different over time. And they. It's like doing a heart transplant of your product every time you change a model out. So you don't want to end up in this locked place where you can't change anything for fear of breaking something. That's actually relates very closely to engineers. Because that's what it's like doing a code refactor when you have no regression tests. So that rollout starts with, yes, getting executive support, but it also starts with education and showing people, hey, this is what we do. This is why we do it. Not just hiding in the research environment and saying, you all don't get it. You don't understand our statistical brilliance. You need to get much, much better at communicating the why behind the what you do. Because that's the secret to unlocking people's minds and going, yeah, I actually do want that thing that you do. That sounds very useful. Please come over here and build a benchmark for our team. Give us some class based balancing to tell us whether or not we're doing the right thing or whether we've got bias in our model. Help us understand what level of hallucination do we have? How do we measure the distance between these phrases? That's something that doesn't make sense in a lot of engineers minds because it's not the world they've lived in for a long time. So if you can communicate that out, it makes that journey a lot smoother. It makes it easier to go from step to step. If people are convinced that they want the thing that your team offers and not just a being told yes, the go, consult the grand intelligent people in the lab and they'll emit a Excel sheet that will tell you whether you did good or bad.

Okay, yeah, I can certainly see how it's both good advice that you want to just, just send all the sort of knowledge you build upon the job around the rest of the organization, but no one's actually going to listen to it until they've built something that doesn't work. And then, and then, then they need your help. One thing I've realized, we talked a lot about benchmarks throughout this, but we've not actually explained what a benchmark is. You just want to give a quick description of like what one of these benchmarks for your modern performance is going to involve.

So you've already picked up on me using those, you see those terms, right? So old habits die hard. So the easiest way to understand a benchmark. I grew up from a very young age with PC games, so I was used to this. It's like a benchmark is just where you run a bunch of different scenarios and you measure how good or bad it is and you say, all right, overall this one has this much in the good column and this much in the bad column. The benchmarking is not that much different in the world of large language models. So you'll have a bunch of use cases that you'll have to think through. So let me give you an example that we've got internally. You've got a task which is you want to take in a JIRA ticket and you want to generate a set of test descriptions out of it. Like, what should I be validating to make sure this JIRA ticket is correct? Without a benchmark you've just got JIRA tickets and then output it's a set of test cases and like, are these the right ones? Are they the wrong ones? Like, I don't know who reviews it. All right, give it to a test. And we were doing this for a while, to be perfectly frank. So cool. Give it to some testers and tell. They can tell you what they think. And they'd be going through and be like, oh, I like this one, that one's a bit rough. We're getting this, like, really fuzzy feedback from people about, like, was it good or bad? And so what we did was we went through and gathered a set of these geotickets and we curated what we thought were the ideal sets of test scenarios to be generated for different categories. So what are the functional tests, what are the security tests, what are the performance tests? And then whenever we change the AI, which is kind of that combination of context, prompt, model, training data, all of that, it would kind of loosely call that vai because people think, oh, you only benchmark new models. No, you benchmark any change to that stack. So when we change that stack, we would go in and say, go back through, feed it in a set of JIRA tickets and look at the outputs and tell us, did it get the test cases that we created and how many of them did it get? How many of them did it miss, and so on and so forth. And that's a benchmark because we get out of that a set of statistics saying, this is how good it was, this is how bad it was, this is where it got better, this is where it got worse. Now, we view that as a testing activity because those are the test cases of an AI product where you go in and you say, how. How right is it and how wrong is it? And if you do that well, you unlock engineering teams to iterate much faster. Because when engineering teams are trying to build their AI product that Stacker talked about earlier, they will often be operating, and this sounds stupid, but it is very true. They will be operating off feeling when they are seeing the results, because they've got no. They've got no data to use, they've got no benchmark. And so they'll be going, that looks about right. Oh, I think that got better at doing this. And it's literally them iterating over and over again like that. And it feels very productive because you're like, yeah, I'm getting like, the results are better and the results are better and better until you throw someone else on it and they're like, oh, it keeps changing. It feels wrong to me somehow. Now you've got two people arguing about their feelings when it comes to how good your AI model is. And the thing that we tell our engineering teams is, yeah, that happens in the team. Now imagine you've got 20,000 customers arguing with you about how they feel. The model is like, how are you going to. That's like a whole football stadium full of people doing a group therapy session with your results. Do you want to engineer based on that. No, you want to engineer based on some outcome. You want to know if what you did improved, degraded or stayed the same. So build a benchmark, spend that time. And based on our internal statistics, you will get to accuracy, which is the currency of progress in AI is accuracy. You will get to accuracy four to eight times faster if you have a good benchmark than if you have a no benchmark. And when we say a good benchmark, a good benchmark should not just say 20%, 22%, 28%. A good benchmark should also break down for you along different business value, axes, categories of things. So for us it was like, how many functional tests did it build? How many non functional, how many of these were edge case, how many of them were high value or, or low value or et cetera, et cetera. So when we're giving the results back to the engineers, we're saying that change you made improved the amount of functional tests that was generating because, well, you put some stuff in the prompt saying focus on functional tests and etc. Or maybe you gave better examples or better context with regard to the functionality, but it degraded the security tests. Now, is that good, is that bad? That's back to the point product manager and the engineering team to figure out, but at least now they know what to look for. Is it bad at generating security tests? All right, well, maybe we need to add some changes to improve security context. Is it generating a lot of edge cases but missing the straight through paths? Maybe I need to rearchitect how the algorithm works and maybe split that up into different agents and get them to tackle these problems independently. So creating a good benchmark, which we view as the tester's role is, or it's actually the tester combined with the data scientist to figure out what the benchmark is. The tester brings the categories and the risks and the things that he wants, he or she wants to see. So that unlocks productivity for the engineering teams as well as gives you something to go back to your customers with and say, all right, this new version improves in these areas. With these regards, the last point on benchmarking is how do you keep your benchmarks alive? And you need to be gathering results from production. So you often can't gather all the data from production because customers will not want to give it to you. They'll say it's sensitive. But customer tolerance for you using their data for benchmarking is much higher than you using their data for training, because training ends up in the model and can be emitted all over the place, benchmarking is a bit safer. So we have customers that opt into having their data be part of the benchmarking model because that gives them some comfort that hey, when you're seeing if the model is good or bad, when you're improving it, you're actually trying to improve it for me, because my data is part of that. So there's some reward for them in doing so. But you need to make sure you're continually kind of collecting that data and putting it into the benchmark so that you're keeping it current and you're dealing with those shifting user behaviors over time so that you're not just, you know, hallucinating about what you think the inputs and outputs are going to be, but you're gathering real inputs and outputs from production. It's very much something that the data scientists lean into in our organization.

Yeah, certainly it's a fascinating topic. And I guess I suppose an example of the benchmarking is then if you've got like a customer service chatbot, like anything the customers ask the chatbot that would then be used for benchmarking in the future. Is that the idea?

Yes, if you, if your customers allow you to.

Yeah, of course, of course, yeah. Always ask for customers consent before you capture in the data. Okay, brilliant. So just to wrap up, what are you most excited about in the world of data and AI?

So when it comes to what I'm most excited about, it's also what I'm most nervous about. I think it's, it's going to sound super nerdy. It's everything to do with knowledge management. Because we've gotten kind of to the point where if you give AI the right information, it generally gets a pretty good answer out of that. So the whole game we're fighting at the moment is how do you take this vast sea of unstructured information and turn it into something that the AI can very easily get accurate, up to date information out of. And I'm seeing a lot of movement in agentic search. I'm seeing a lot of movement, improvement in knowledge graphing, for example, and knowledge searching. And I think that those spaces, those are the key unlocks for the future of artificial intelligence. In the more near term view, the less nerdy view of what I'm excited about. It's a lot to do with agents. I'm very excited about agents becoming our. Everyone says like an agent's going to be your peer. I kind of view it as an agent's going to be your employee. Your employee as, as an employee of an organization, you're going to have your own sets of agents that you build and manage and that do work for you. And I think that that excites me because for our customers, that feeling of being snowed under and being drowning in the volume of work that they have, I see agents as a cure to that. And we at Tricentis, we are rolling out these agents for our customers. We've already launched one, there's two more in the pipeline to help our customers, you know, get this value sooner. That excites me because I think the world is a lot more fun when you can delegate some of that dull work to agents to get done. As long as we don't lose our, our intelligence and our ability to review and understand when the AI is right and wrong. I see that as a very exciting future. So the nerdy side, knowledge management gets me because I'm still an AI nerd at heart. But on the actually coming soon and productive and out there in the world, agents are something I find incredibly exciting.

Oh yeah, right there with you. Yeah. Knowledge graphs in like the graph Transformers. Very, very exciting on a, on a technical front. But yeah, the big stories of this year for sure is AI agents automating dull work. Fingers crossed, all the dull work goes away. Yeah. And we don't just end up as AI slot reviewers. Finally, I was want more people to follow. So whose work are you most excited about at the moment?

I follow a lot of what the AI Frontier Labs are doing and for me, the most exciting company out there still is Anthropic. I love the work that they're doing in being responsible with artificial intelligence. They're very closely aligned to our domain. So they just recently released like the 4.5 model of Claude. That model is already a very good coder and it's just starting to get better and better at these technical tasks because we're in their sort of enterprise space. I follow a lot of what they do with. They put MCP into the world. They've released the CLAUDE agents. I think that that channel has a lot of potential still to milk out of it. When it comes to the knowledge graphing work. It's kind of a toss up for me. There's so many companies that I follow in this space that I couldn't really pick a winner out of all of that. And when it comes to research, this, this sounds kind of boring, but I'll tell you why I find it so exciting that Stanford were doing a lot of research on understanding the true productivity of artificial intelligence. And that research is incredibly valuable to me because it's in the same way that you want to benchmark your use case to figure out if you're progressing or not. If you can get to a structured and rigorous way of determining whether the AI result is not just an accurate result, but whether the net productivity of the system involving people and AI is increasing or decreasing, then that gives us the real information that we need to determine the path forward for our entire organizations. So I really enjoyed the research they did there on determining the productivity of coding artificial intelligences. They recently presented that at the, I think it was the AI Engineering World Summit. But they're doing a lot more research on governance of those agents and improvements of productivity in that space.

Absolutely. I mean there's so many people will claim, okay, yeah, you're going to get a 10 times productivity improvement from using AI. Not really going to happen very often. I do, like, that's like real academic research going on just to measure like, how much are you actually going to get by adopting a technology. So, yeah, the workout, Stanford's really great stuff. Yeah, good choices. Thank you so much. And yeah, thank you so much for your time, David.

Thank you, Richie. It's been a pleasure. Sam.