#329 Building Trust in AI Agents with Shane Murray, Senior Vice President of Digital Platform Analytics at Versant Media

Generative AI is transforming industries at an unprecedented pace.

But as AI changes how you or your team work, one thing is clear. Your skills also need to evolve. At datacamp, we offer everything you or.

Your team need to adapt and thrive.

With AI, whether it's business users looking.

To get the Most out of ChatGPT and Copilot, or developers and data scientists looking to fine tune models.

You can learn the entire AI skills spectrum on Datacamp. Power your AI transformation today. Start learning@datacamp.com. I kind of have an underlying philosophy that you can't truly establish readiness until you start building. And that is the best way to be ready. Actually from working with teams who are building AI, the ones that are actually going through the prototype to production to scale up phase are the ones that are finding out what readiness really means for their organization. Human in the loop I think is essential. I actually probably wouldn't be building a solution without having factored in some human in the loop to it. And I think both for as like a quality gate, but also to build that trust with users, the ultimate consumers of it, who have a say in it being reliable.

Welcome to Data Framed. This is Richie. For all the promise of AI agents, a major barrier to them being useful is that they don't always work, that is they are untrustworthy. Today we're going to look at ways to improve trust in your AI agents with a big focus on data quality. I also want to discuss AI readiness. Every CEO claims they want their company to be AI ready, but I'm not really sure what that means. In this episode, we have a repeat guest, Shane Murray. At the time of recording, he was field Chief Technology Officer at the data and AI observability platform Monte Carlo. He's just switched jobs to be Senior Vice President of data digital Platform analytics at Versant Media. In fact, he's returning to his media roots since he was previously Senior Vice President of Digital analytics at the New York Times. So let's find out how to build trust in AI agents. Hi Shane, welcome to the show.

Hi Richie, nice to see you. Wonderful.

Yeah, great to have you here. Now, I love a good disaster story. So just to begin with, can you tell me what's the biggest AI disaster you've seen?

From my previous days as a data leader at the New York Times, we talked about disasters as those that might appear on the front page of the New York Times. As many teams I think do, I would say from the people I work with, people are generally fairly conservative. So I haven't been firsthand to disasters. But the ones that kind of come to mind are, I don't know if you saw there was a McDonald's chatbot that leaked, I think it was like, 64 million job applicant records. And the security researchers found that the default password was 1-23456. Comically. So, I mean, I. I think these reputational ones are the ones you. You really worry about. And similarly, the one that comes to mind just from having been in the field of media, but it was the Chicago Sun Times who had a summer reading list that was essentially all fictional books. Right. You know, not fiction books, but actually fictional books. And so. So these ones that kind of get to the heart of reputation and trust, I think are the. Are the ones you. You watch out for.

Yeah, absolutely. It's all very well having, like, a forecast that's a little bit off, but when you're causing, like, problems for 64 million customers of yours, that's a really kind of. It's a bad situation. Something you definitely want to av. Maybe that's too negative. So we need the flip side to this. Talk me through some success stories you've seen.

Yeah, I'll maybe reference one I've. I've read about, and then one and one I've seen firsthand, just working with customers here at Monte Carlo. But the one that struck me just like reading what data teams are doing is the Stripe team. And I haven't actually seen, like, a deeply published post on this, but one of their team members was talking about how they've essentially built a foundational model for payments using billions of transactions, which is essentially replacing, you know, feature engineering, you know, what we call now traditional ML models with embeddings capturing the relationship between payments. And I just thought that was super impressive. And, you know, most people tend to talk about traditional ML and then generative AI and the fact that they're using these LLM approaches to rethink these sort of foundational business problems around fraud detection and things like that I just found super interesting. Yeah, have you seen that one?

No, I hadn't heard about this. That's actually kind of wild because. Yeah, you think about machine learning and generative AI as being very separate things, but actually, yeah, an embedding model to. It's normally used for, like, detecting how close different words are in meaning to each other. So I guess detecting how close payments are, that seems like a good way to detect fraud. So. Yeah, very innovative there.

Yeah, yeah, definitely that one. And then I just say, like, from. From customers I work with, where I'VE seen the most value, even though it's not maybe the, like the autonomous agent use case is these teams that are basically taking what, you know, what were kind of robotic process automation steps and, you know, unstructured to structured data. And so an example, a customer I work with pilot travel centers who have, you know, 750 travel centers in the U.S. they're the biggest diesel fuel provider, they're the biggest subway franchisee in the US and they actually took these bills of lading that the truck drivers essentially take photographs of and send in. And then it goes into this, what was a previously human driven processing loop and used LLMs to extract the data from those images and then dramatically speed up their kind of financial data pipelines, which I thought, I think that's become like a prototypical use case, but, you know, super interesting one and really valuable to organizations.

Yeah, certainly it hits on like a lot of the kind of current themes, like the idea that, well, okay, images are now data. You can extract very useful structured information from them. And then also like one of the big goals is like, okay, let's speed up some processes, automate stuff and then we can go faster. All right, nice. Hitting all the key themes in one story. I like that. I want to talk about how we get to these success stories. So I think a lot of organizations, one of the big sort of management buzzwords has been about becoming AI ready. I'm still not quite sure what that means. So talk me through. What does being AI ready mean to you?

Yeah, I think it can be a very ambiguous term that goes without definition often and probably similar to terms of the past, these kind of terms of art in data and business. But you know, I think I kind of have an underlying philosophy that you can't truly establish readiness until you start building. And that is the best way to be ready. I know that's kind of a cheat answer, but like actually from working with teams who are building AI, the ones that are actually going through the prototype to production to scale up phase are the ones that are finding out, you know, what readiness really means for their organization. And that can be organizational. That can be from a systems perspective. You know, are they cloud first? Do they have the right, you know, management around unstructured data, that sort of thing. And then I'd say just like more, more practically where I see teams focused on in this space, one is, do they have reliable AI ready data? I think everyone's kind of accepted this idea that for your AI to be good and ready, your data has to be Ready? So that's kind of one dimension. The other is this thing we talk about with customers, which is once you are building AI, do you have the means to evaluate and monitor the quality of that as you're scaling it up, the quality of the outputs, the quality of the process to build it? And then thirdly is kind of when the agent maybe goes askew, do you have the ability to understand it and to intervene and to prevent bad outcomes?

That's interesting because those ideas are kind of the different ends of the development cycle. I love the idea that in order to figure out like what you don't know, you just try building stuff and wherever you fail, that's going to be useful information. Or like, this is what we need to learn about, this is what we need to do better. But then also, yeah, like the big problems are like, well, okay, how do we actually scale stuff? How do we understand like what's going on with these agents we're putting out into the world? Because if you can't track it and yeah, disasters are going to happen. Let's talk about what skills you need to have in order to like start building with this stuff. You talk me through like, basically like what roles are going to be involved and what skills do they need in order to build stuff.

Yeah, the, this is something I've been thinking about quite a bit and talking to, to data leaders about in the space because it's, I, I think there are a lot of data roles in teams these days. You know, I can, can probably list off at least six or seven different roles that you might have in a large data organization. But I think, I think AI obviously is kind of changing how we need to think about this and the role of machine learning teams, the role of data engineering teams. I think when I, if I break it down into what I see as necessary, one is this idea of context engineering. And I think at the moment I'm seeing the people that do that often have the title of AI engineer, but in many cases it could be a data engineer or a software engineer. But it's really the idea that you have to engineer good context. That could be a rag pipeline, could be a set of prompts, could be hitting different APIs that are bringing in the right information. How do you make sure that you have trusted and reliable context serving this agent? The second. I've heard some debate around whether this is kind of a product role or a more of a data science role, but how do you define and evaluate quality? Right. And often this is a mix of human feedback Right. How do you take in human signals about, you know, what's relevant, what's irrelevant, and then how do you use LLM as a judge techniques to also measure and monitor quality? And I kind of think at the moment this is a data science job and where I see it being done best, it's being done by data scientists because it is a lot of thought about how you measure the quality of an experiment or how you measure these non deterministic outputs in a way that is scientific. Right. And then the third kind of role I see often is more as you start to scale up, which is kind of the platform engineer. Right. How do you instrument your stack to be able to support many agents to build upon? And we're a bit early for that, but I'm certainly seeing teams think about how do you have extensibility and scale, how do you have standard frameworks for experimentation and for observability?

That's interesting stuff. And so on that second point about how it's data science teams that need to be monitoring the quality of the AI work, I think that's fascinating because I get the sense over the last few years there's been a drift from data teams being the people responsible for doing AI to engineering teams being the ones responsible for doing AI. So it's nice to know the data teams still have a purpose here.

Yeah, I'm often seeing that it's, it's the data teams that are potentially staffing that talent within the product or software engineering team. And so it's a data scientist that might be doing some of the context engineering, but might also be responsible for reporting back. Like how well is this thing doing? Right. And I still think there's like a huge role for data people to play in this.

Oh, since you mentioned context engineering, this is Andrej Karpathy making up phrases again. Do you just want to explain what is context? Context engineering?

Yeah, I think this term, I'm not as sure of the derivation of it, but I feel like it is the right term after people have started with prompt engineering and then realized it's more than prompts. Right. In some cases you're building a rag pipeline, but basically all the ways that you're bringing in unstructured and structured data as well as prompting the AI to actually make it work. So I mean, it's not a very scientific definition, but that's kind of how I think of this larger space of context engineering. Yeah.

I have to say the term's been quite divisive at Data gam. So all the people who are involved in engineering, like building AI features like. Yeah, trying to figure out how much context you give to an AI. It's an incredibly important task. Context engineering is a real thing. Incredibly important. But then all like the curriculum team. So people are building courses. Anyone in marketing, like, well, giving context to people when you're explaining stuff, that's just what we do all day. We don't need a new term for it. So yeah, definitely divisive. Depends on your point of view as to whether it's a useful thing or not. Oh, so you also mentioned the idea of platform engineering. Just talk me through. What's that?

Like, my experience with data teams over the past decade is like we really gravitated to this world where we need people focused on building the platform that other data engineering or other data analysts or data science teams can make use of. Right. And so building up both the infrastructure and the sort of golden pathways or the tooling that you expect people to use. And so, you know, as I've been talking to probably 150 data teams over the past six months about how they're approaching this and very often they're seeing fragmented agents or AI being built out on the edge in software engineering teams who are close to the problem statement of the customer, but they're all going and building with different tools and different frameworks. And so the idea of the platform engineering team is really about standardization. And often I think when it's done best, you're. You're waiting till there's enough use cases to require a platform. You shouldn't build ahead of the need too much. And then you're also like providing the things that are naturally adopted. You're not a good platform. You don't have to force adoption because people see the benefits of not having to own that foundational layer.

Okay, this seems incredible. It's like having some sort of standardized infrastructure. And I guess a lot of companies, particularly the large of the company, you end up with silos with different teams using different pieces of tech that do the same thing. So I guess maybe talk me through the start of this conversation about becoming AI ready. So what's the infrastructure you need to be AI ready? Like what does a sentinel tech stack look like?

I mean, at the moment it, it feels fairly minimal. Right. I think people are looking at, at orchestration. So something like a langgraph is very common. Right. And establishing like a standard for a gateway to whatever set of foundational models you want to make available to people within the organization to build upon. I do think like platforms Then extend into how do I reliably run experiments. Right. How do I evaluate that this thing is better than the last version and how do I, how do I have observability? And that observability can be the latency or the cost. Right. How many tokens you, you're using? I think some people have certainly run into cases with, with all this AI being built that somehow they'll have a cost spike where a team hasn't realized like what they've done. But it can also be like the traceability of these agents and ensuring you're knowing how they're behaving. So I think we're at the early stages of the platform discussion. You can also, I would say like, you know, there's discussion about whether you have a dedicated vector database or like what is the underlying management of unstructured and unstructured data that supports the agents. So all these things can be components. But yeah, I would say it, it feels we're pretty early in the platform discussion of AI.

Okay, yeah. So I guess you probably want a plan for swapping out some tools at regular intervals in the near future then. Is there a way to make your infrastructure a bit more fluid then? Like, how do you plan for that?

How do you mean?

If you got some sort of fixed tech stack, then you probably, that's not going to work. You're going to want to change stuff at some point.

I don't think I have a great answer for that, but I feel like, you know, investing minimally at the moment and supporting the top use cases. And then also most teams are still just allowing teams out on the edge to go and use what they want. But it's like, how do you make sure you've supported the top three use cases? This is how I've approached platforms in the past. You don't try to solve for a hundred percent, you don't try to enforce too much, but you have, you know, you're 80% of value covered by the platform. I think modularity is probably a key word for this now. And just making sure what you build is extensible to different use cases.

Okay, yeah, I like the idea of modularity. Like maybe you don't want some monolithic stack that you got to swap everything out all at once. Okay, so you were talking before about observability and like the how you test for quality in, in your AI. I guess this all starts with the data, right? So I guess what sort of data quality controls are you going to want before you start saying, okay, we're going to use this data with our AI products or features.

I think AI readiness also has this category of kind of AI ready data. And I think for some teams, as I've mentioned, like it's about actually building the application and, and seeing what needs to be ready in that process. I think also we have this huge branch of work, many teams now approaching conversational BI or whatever name you want to want to give to it. But basically replacing dashboards with natural language and allowing that to access your data. I think for that teams really need to and have been thinking about kind of data certification, right? Which, which data sets have the right coverage in terms of monitoring, have the right incident response processes, have the right metadata around them to be ready. That with AI tends to include synonyms like, you know, revenue equals sales and all the ways that someone using the agent is going to talk about your data. And then, you know, essentially I'd say that's kind of one side is have you prepared your data estate to have agents put on top of it? And then I've also seen data teams who are kind of tackling this idea where they have downstream teams that are building agents and they need to think about how they approach kind of readiness of structured and unstructured data. And the unstructured data is kind of a new component where, you know, you previously had very little visibility on that data. So I think teams are now grappling with the idea of how do I monitor the unstructured data that may be used by, you know, tens of teams downstream.

Actually that's a really interesting point. So how do you even measure like the quality of like is this image any good or not?

So I feel like it's early on, but there's some things that are parallel from structured data, obviously. So is it fresh? You know, is it in a valid file format? There's some of the things that are kind of fairly standard with that. But then I think you've got an image is a, maybe a harder one. But I think of text like semantic consistency and I've seen a lot of teams try to protect for this by having like one person contribute to the, to the governance of the documents to ensure this semantic consistency. If you've got really high volume, variable and high velocity data, then you need to be able to check that someone's definition of risk is the same definition of risk that's understood across the corpus. Right. And so I think there's a need to sort of understand the semantic consistency of the corpus. And then there's also a need to understand the, the relevance of it. And I've talked to a lot of teams that are dealing with kind of rag based pipelines where they need to be able to know when there's a drift in the relevance of the underlying corpus from the, from the model and the output that they've built.

Okay. All right, so just to make sure everyone said this idea correctly, so semantic consistency is this just like business definitions are consistent. So it's like, okay, this is how we define the sales qualified lead, or this is how we define customer lifecycle value and it's the same everywhere. And you want that across all your documents, Is that, is that correct?

Exactly, yeah.

Okay, so is there like a technological solution to ensuring semantic consistency or is it like case of just read everything and hope that it's right?

Yeah, I mean at Monte Carlo we've done some work into, into building these unstructured data monitors. And so, so, you know, I think we're, we're sort of starting to work with teams to see like what they need to monitor and, and what they need to extract, but it's a different problem and you basically need to structure some information from that unstructured data in order to monitor for that consistency. So, and, and much of this relies on the fact that now we have like underlying LLM functions in our, in our warehouse that we can tap into.

Okay. Yeah, so it seems like maybe it's a sort of ongoing research problem is like how do you actually ensure like that document quality control like everywhere. I'd like to have you find maybe like a concrete example of how the data quality then feeds into AI quality. Like how does that relationship work?

You know, I tend to think of this kind of engineering maxim, like as much reliability as you need and no more. Right. Maybe this stems back to the idea of like you kind of have to build the thing and put it in production and start learning. And so the idea of monitoring these things is often learned through monitoring and iteration. How does it affect performance? I think oftentimes people are really attaching to this idea of hallucination. Right. And it's kind of the most commonly talked about problem with generative AI. What I've found from talking to many data teams is that very often like the underlying problem that's causing hallucination is not the model itself, it's actually the context or the data inputs. Right. And so if you have outdated data, Right. Then you're going to have a model that's giving potentially like old information or if you've just tapped into Your underlying like confluence data that's, that's had maybe hundreds of contributors over the last 10 years. You're going to have a mix of high and low quality information. And so I think the practical, well, the models can hallucinate and it's certainly something you need to build monitors and guardrails on. The practical problem that many teams are finding is actually like, all right, how do I ensure the data is reliable and how do I ensure the model outputs are reliable?

Oh, right. So we're back to document quality control where like you've got five year old, well, I mean maybe customer facing support pages that they ought to be kept up to date, but certainly like intranet pages where it's like, okay, yeah, I wrote this process seven years ago, that page still exists somewhere and suddenly it's being pulled in, into your AI helped system.

Well, yeah, I mean I think some of the early use cases have been can we put a, can we put a conversational agent on top of our internal documentation? I think people very quickly find, finding that that isn't a very up to date document source. I mean obviously there's, there's cases outside of that where you can tap into much more fresh and timely documents.

And are there any of the sort of data quality issues you see then or is that, is that the main one?

Yeah, no, I think, I mean the data qualities, the data quality issues that contribute, many of them are things that we've seen, you know, across traditional ML and across analytical use cases still I find like silent schema changes from an upstream, say software engineering team are one of the most common causes of pain or even externally managed data that, that you're ingesting that actually changes. And so schema changes is still this sort of event that if not controlled can be like disastrous for downstream systems. I think then you have the idea of pipeline delays causing stale or incomplete data. So I was talking to a customer who said like we need to know the delay between when the underlying document set is updated and when our model actually starts to use that because this time in between, this latency can, can cause it to be delivering the wrong answers to consumers. Right. And so there's a, there's still a freshness problem that exists in AI, as with any data product that you're building. I think thirdly, you have this idea of, of data drift which can occur from code changes or instrumentation changes, but the idea that your, your underlying data that you're using in a rag pipeline or in a, in context can actually drift into a place where the model no longer is giving relevant responses. And then I think the new one is actually like measuring the, the actual output of the AI. Right. And it could be measuring that for, for clarity or for a lot of teams I talk to measure it for like, does it carry the kind of brand image that we represent? Like, is it speaking as we would speak to customers? Is it grounded in the, the context that we've provided it? Or is it accurate if you have some ground truth? And so all of these things, I think from the underlying data inputs through to, you know, which can be schema changes in freshness through to the, the actual model output. So the quality issues that are affecting AI in production, so many different things.

That can go wrong there. So yeah, certainly with, with freshness I can see like if you're pulling in like weather data or news data, like even a small delay is like, that's going to render the data completely useless and certainly like changing formats like that. You mentioned the schema changes. Yeah. If your data is suddenly in a different form, then that's going to break a lot of things downstream. Okay. So I guess a lot of things that can go wrong. Suppose like your chief Director officer is like, okay, we need to improve data quality. Do you have like a single recommendation for like, where do you start improving data quality?

This is something we, we tackle at Monte Carlo a lot. And I'd say like the most logical place to start and the place of highest impact. Typically, at least in large organizations and many small organizations, you'll have the kind of foundational data products, right? The core data layer that should be easily joinable, that everyone's tapping into and they're building derived products on top of that. So you might have domain teams that are actually tapping into that data. So, you know, I tend to recommend starting with those foundational data because it's, it's where you might have hundreds of users that you can affect downstream. And if you, if you improve the quality there, you improve it downstream. And so typically there I'd start with, you know, the base kind of schema freshness volume checks that you can scale up very easily. Right. And you don't need to be writing manual assertions. You can really just turn these type of checks on with any observability tool and then going, going deeper on your critical data elements and ensuring you have kind of distribution monitoring around those so that you know that by the time that it gets to a metric that the CEO's reading or a piece of data that's, that's really required in an end user system that it's actually been measured along the way and hasn't had any transformation problems or any underlying data problems. So I think that's the, the detection side. And then we typically encourage teams to really start with, you know, basic operational practices if they don't have those, which might include on call rotations, having a clear and understood severity process for incidents so that you can kind of separate the signal from the noise and really amplify the, the signal of a SEV1 or a SEV2 and then doing retros when you have these incidents and kind of reporting on things like time to detect and time to respond. So these are kind of the basic practices that I think every software engineering team has implemented and every data team needs to have implemented these days.

That's a silt dig at data teams. I think there. Yeah, so the idea of having people on call for when things go wrong and doing retrospective, like a postmortem when there's a disaster so you can fix it. Very common in software engineering, much less so in data teams. So yeah, that definitely seems like a good process to implement. So yeah, I guess if you're a data team leader, you're like, how do I do things better? Just basically copy whatever the software engineering team's doing.

I think that, I mean, we don't need to reinvent the wheel here. Like, I think data teams have been behind software engineering teams in most organizations just in terms of adopting these strong like reliability engineering practices. But like data just keeps becoming a more central part of the products we build. And it's. So these are the steps to take.

Absolutely. Okay. And so you mentioned earlier on about you go beyond just observing problems to try and find out like what the root cause of those problems was. Just talk me through how do you go about that? I think certainly once you got like an AI agent and then there's like lots of layers of tech in between that and oh, there's a data problem, it can go quite deep, I guess. So yeah, talk me through what's the process for finding a root cause of a problem.

So maybe I'll, I'll separate it into two pieces, which is kind of how we observe and how we observe agents. And so we've been doing work at Monte Carlo to basically extend our agent observability so that we're capturing the traces of agent behavior, you know, through all the building blocks, which you can think of as kind of let's take a rag pipeline. It'd be, you know, chunking and embedding and retrieving, and then you've got the decisions that an agent might be making. And so, you know, as these agent architectures get more complexity built into them, we've just seen the need to extend that kind of telemetry instrumentation to be able to go and dig into individual responses. Right, because you've got non determinism built into these, you need to actually be able to break it down and look at individual cases and understand essentially the lineage of those. And so part of it is that instrumentation. And then the other part is actually we've, you know, over the years we've, we've really invested in, in troubleshooting and root cause analysis. And what we've built up in Monte Carlo now is a, is an agent for troubleshooting. And so what that agent does is it takes all of the context of an incident and which, you know, includes the, the anomaly itself, but also an understanding of the data lineage and understanding of the different logs that are coming from contributing tools. And then it spawns a series of sub agents which might Explore, you know, GitHub changes or airflow issues, or go upstream and look at all the potential data failures that could be happening. So each of these specialist agents then go and explore hypotheses and you might have ten or a hundred LLMs running in parallel to explore a hypothesis, come back with a finding, share that with the main agent, who then summarizes it within the span of about 2 minutes and suggests next steps. And so that's really the latest and greatest that we're doing on troubleshooting, which is like, how do you take what we've found? With data teams can take kind of 18 or 20 hours of, for an incident to actually go and troubleshoot. And with any alert, give in 2 minutes a review of what might have.

Caused it, of the idea that using like hundreds of agents to then find out what the problem was with the original agent. Now you've got like a hundred more problems after you debug, debug those agents.

It's, I will say, like we've found it to be such a good use case because there's, there's such a clear process that engineers take to this hypothesis testing. And we've also, over the six years of Monte Carlo, built up an understanding of, you know, all the root causes that we can kind of give hints to this agent about potential. But yeah, it's agents upon agents, as you say.

No, it does seem incredibly useful. I mean, I imagine like spending 20 hours going through, like digging through your entire stack to figure out where the problem is. That's got to be an incredibly tedious and frustrating job for any sort of data scientist or engineer. So, yeah, I like the idea of automating that. So you mentioned you've got sort of six years worth of data on like things that could go wrong. Yeah. What are the most common sort of root causes of problems that you've got found?

Maybe I'll focus on like, as we've, as we've talked to teams building AI, what some of the new problems there are. Obviously I kind of talked through like schema and freshness and distribution problems in data that can occur. But one of the interesting ones I've found is talking to teams building AI is the actual, you know, underlying model upgrades behind the scenes. Right. Or prompt changes. And typically you'd think, like these are occurring within teams that should know that changes that are happening. But a lot of these changes are currently invisible. And I think part of it's because we're early. But a change from GPT to 4.0 to 4.1. Right. Can really have dramatic impacts on an agent that you have in production. Right. And so I've heard teams say, like, they'll start getting feedback from their user base that somehow like the thing feels less useful. And we see this using some of the UIs provided by the agents as well. But, you know, data teams need to be aware of those model changes or prompt changes in the same way that they are a schema change.

Yeah. All these foundation model companies, they keep coming up with new models and they're always like, oh, this is the latest, greatest thing. You need to adopt this. But actually need to be very, very careful in an enterprise setting when you're switching over your models, it seems.

Yeah. Even if it's better. I think we've seen as data teams, like you care about consistency in a way above, like accuracy. Sometimes consistency is such a critical factor for data teams. And so a model changing versions behind the scenes or someone switching out a prompt has huge impacts on the consistency of behavior for these. So that's, that's been one. I think another thing that has come up with a lot of data teams is is this idea of embedding drift. Right. Which you could frame as kind of knowledge drift. But, you know, are my embeddings still relevant to the, to the use case that I'm supporting? And then I just say going back, like the thing that's been reinforced through my conversations is if that underlying knowledge base or document set that you're Feeding in is low quality or is delayed, you know that that's going to, to make or break your AI application.

So embedding drift. This is a whole new problem to me. Talk me through what this is about. Like the, the meanings of words changing. When does this happen?

Yeah, potentially I'll maybe I'll give a sort of example that a customer shared, but basically they were doing some work in the US and had some embeddings that were specific to the US and then starting to expand the business into Canada. Right. And the standards of the language or the images in this case in Canada were very different from the US Their techniques, which might have been like few shot prompting or other techniques to actually make this work, suddenly become less reliable for the scope of the problem they're solving. And so I think historically like these sorts of solutions are being built by sometimes one data scientist. Right. But as you start to fragment ownership of that solution, I think you, you have more requirements to have observability or have monitoring on each piece of it.

Yeah, that's absolutely surprising. It's certainly something I fell over in my own life. Moving from, from England to the U.S. it's like, oh yeah, a lot of these words are wrong.

Me too. Moving from, from Sydney to New York experienced exactly the same thing.

Absolutely. Yeah. So it seems like as you are sort of moving into new markets, you really, really need to be careful about like AI performance and, and is that going to change? Okay, so I guess in general all this speaks to the problems of just how you go about changing your, your processes. And like once you start adopting agents, do you have any advice on, on process management? Like how do you go about changing things?

I, I think maybe the, the first one I just call out, maybe it's obvious, but like human in the loop I think is essential. Like I actually probably wouldn't be building a solution without having factored in some human in the loop to it. And I think both for as like a quality gate, but also to build that trust with users, you know, the ultimate consumers of it who have a say in it being reliable. And I think when teams set the expectations with their end users, they're in a much better place and also bring in experts to ensure the quality of the application, then they're, you know, actually more successful in deploying these applications and driving that change management with their user base. I even saw, you know, the same thing with machine learning is that like the closer you get to that end user and build for them and don't build to replace them, Then, then you're going to get a lot more buy in. I think the second one is like the, the thing I've seen teams do is kind of start very narrowly, you know, prototype narrowly, get the buy in, get the wins, then figure out the next scale. So they're going from ten to a hundred to a thousand users in a way that I don't think we considered as much with. You know, we certainly did sort of smaller A B tests, but most products, you know, get launched to production without going through such rigorous kind of step ups in, in the user base. And I think sometimes that'll mean you, you go up and then you come Back to your 10 users and test some new changes. But it really feels like that sort of phased rollout is part of the, the AI adoption process culturally. What, what we've found at Monte Carlo, because of course, like any other company we're adopting AI is you need to give people kind of space and time to experiment. You're not going to be productive on day one of using AI and you're going to do a lot wrong. And so I've found it really useful. At Monte Carlo we have a culture of like sharing the successes and failures we've had with using AI and that's actually converted a lot of people into users who were, were previously maybe a bit shy to use the technologies.

The idea of the phase rollout, that's actually surprisingly revolutionary or radical I suppose compared to like SaaS software, it's like, okay, we do an A B test, we give the new feature to 10% of our users. If it seems to work okay, we roll it out to everyone. But yeah, doing it like 10 users, then 100, then a thousand and gradually building up, that does seem, it's a very different process and a lot more rigorous.

And I think in part it ties to maybe, maybe the weakness of Human in the Loop is that right now. And, and I've seen some surveys that have shown this to back up the anecdotes I have. But Human in the Loop is the primary way people are ensuring quality. And that's very hard to scale up. So you have to start thinking about how to, you know, keep that, keep that at levels that are manageable. As you scale from 10 to 100 to a thousand, you can't keep scaling up your humans in the loop, so you have to scale up more automated evaluation and monitoring approaches.

Okay, yeah, certainly, yeah. Humans seem to be a bottleneck. Everything, if you want to be like, all right, so I guess all this has been building towards how do we get users of AI, users of AI agents to, to trust the product? So do you have any final advice on like, how do you get that trust?

Yeah, I, I had an data scientist say to me the other day that trust is gained in droplets and lost in buckets. I'm not sure who get, who gets credit for that, but I thought it was a really good phrase. I think what we've found over, over the span of Monte Carlo is that there's both what we can do to, to, you know, use observability, to, to make more reliable data products. But then there's a whole human element to this and to rolling out any software that ensures the trust. And so I've seen three things really drive trust in data and I think it extends to AI. But one is like, you have clear accountability, right? You have clear accountability over the data. You probably want to avoid too much fragmentation across the value chain of the product. So, you know, you know who's responsible for the lifespan of this then you have an expectation of what quality and reliability means. So, you know, actually sharing that expectation, whether it's four nines of reliability or whether it's something else. And as I've spoken to people that are launching AI products, they've found if they don't set that expectation up for front, there's missteps with their users who maybe expect something else. So like, you need to know, you know, how, how trustworthy and high quality this product is that you're adopting. And then the third one actually is like, you know, we see when, when customers do go through data incidents, their ability to transparently communicate that to their end users. Right. And communicate uptime and downtime is something that also builds trust. Right. So, you know, people understand downtime of products, but if you don't communicate it well, it can, it can be a failure in, in that trust building. So those are probably the three things and you know, I think it, it just reinforces that that trust is kind of built in, in production.

I really like the idea that as in real life, a lot of trust is about communicating realistic performance. I guess so, yeah, if you don't have 100% reliability, then, you know, make it clear to users that maybe some things are going to go wrong some of the time. They shouldn't be looking to trust it to be right on every single occasion. All right, super. And just finally, I always want more people to follow. So whose research are you most interested in at the moment?

Yeah, I've been reading a lot of, and I'M just thinking. I'm not sure I know his full name, but Hamill H A M E L and his website is hamel.devdv. and he's an engineer who talks and writes a lot about building reliable AI. And so I've been following along with him. I think he's writing some really interesting stuff around, you know, tackling error, analysis of agents and building evaluations. So I've been enjoying him. And then I'd say the other. The book I'm reading at the moment is Empire of AI. Have you heard of that?

I have read the book. It's a very, very good book. Lots of gossip about what's been going on this last decade. Yeah, it's a good read.

Yeah. So that's Karen Howe Hao, who's a freelance journalist. But, yeah, I think she started by being behind the scenes in 2019 with. With OpenAI. And it. It kind of builds from there. The good and the bad, I'd say.

Yeah, yeah. Lots of very, very juicy gossip in that book. All right, super. Thank you so much for your time, Shane. It's been a pleasure chatting.

Thanks so much, Richie. Sam.